aboutsummaryrefslogtreecommitdiff
path: root/src/blockchain_db/lmdb (follow)
AgeCommit message (Collapse)AuthorFilesLines
2019-03-04Merge pull request #5081Riccardo Spagni1-3/+3
d78addcb db_lmdb: don't unnecessarily cast to double on the way to uint64_t (moneromooo-monero)
2019-03-04Merge pull request #5175Riccardo Spagni1-0/+1
85088d9f db_lmdb: fix missing mdb_dbi_close in migration (moneromooo-monero)
2019-03-04ArticMine's new block weight algorithmmoneromooo-monero2-6/+214
This curbs runaway growth while still allowing substantial spikes in block weight Original specification from ArticMine: here is the scaling proposal Define: LongTermBlockWeight Before fork: LongTermBlockWeight = BlockWeight At or after fork: LongTermBlockWeight = min(BlockWeight, 1.4*LongTermEffectiveMedianBlockWeight) Note: To avoid possible consensus issues over rounding the LongTermBlockWeight for a given block should be calculated to the nearest byte, and stored as a integer in the block itself. The stored LongTermBlockWeight is then used for future calculations of the LongTermEffectiveMedianBlockWeight and not recalculated each time. Define: LongTermEffectiveMedianBlockWeight LongTermEffectiveMedianBlockWeight = max(300000, MedianOverPrevious100000Blocks(LongTermBlockWeight)) Change Definition of EffectiveMedianBlockWeight From (current definition) EffectiveMedianBlockWeight = max(300000, MedianOverPrevious100Blocks(BlockWeight)) To (proposed definition) EffectiveMedianBlockWeight = min(max(300000, MedianOverPrevious100Blocks(BlockWeight)), 50*LongTermEffectiveMedianBlockWeight) Notes: 1) There are no other changes to the existing penalty formula, median calculation, fees etc. 2) There is the requirement to store the LongTermBlockWeight of a block unencrypted in the block itself. This is to avoid possible consensus issues over rounding and also to prevent the calculations from becoming unwieldy as we move away from the fork. 3) When the EffectiveMedianBlockWeight cap is reached it is still possible to mine blocks up to 2x the EffectiveMedianBlockWeight by paying the corresponding penalty. Note: the long term block weight is stored in the database, but not in the actual block itself, since it requires recalculating anyway for verification.
2019-02-21db_lmdb: fix missing mdb_dbi_close in migrationmoneromooo-monero1-0/+1
Fixed by hyc
2019-01-22Pruningmoneromooo-monero2-23/+405
The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2019-01-19db_lmdb: don't unnecessarily cast to double on the way to uint64_tmoneromooo-monero1-3/+3
2019-01-16blockchain_db: allow getting output keys without commitmentmoneromooo-monero2-3/+4
Since the commitment has to be calculated for non rct outputs, it slows down a lot unnecessarily if we don't need it
2019-01-16Merge pull request #4984Riccardo Spagni2-22/+28
008647d7 blockchain_db: speedup tx output gathering (moneromooo-monero)
2019-01-07Make get_output_key method constmoneroexamples2-4/+4
get_output_key method is commonly used when working with txs and their key images. Because the method is not const, passing blockchain object though const& or pointers to const is not possible in this context. This is especially problematic in external projects (e.g., projects in moneroexamples) that use monero C++ api to operate on the blockchain and txs. Thus, having get_output_key method will simplify moving blockchain object around through const references and pointers to const objects.
2018-12-31Merge pull request #4946luigi11112-50/+8
6644b9b blockchain_db: remove a couple unused functions (moneromooo-monero) ce594f5 blockchain_db: allocate known size vector only once (moneromooo-monero) 8332698 db_lmdb: inline check_open, it's trivial and called everywhere (moneromooo-monero) 5511563 db_lmdb: avoid pointless division (moneromooo-monero) d1efe3d cryptonote: set tx hash on newly parsed txes when known (moneromooo-monero) 9cc68a2 tx_pool: add a few std::move where it can make a difference (moneromooo-monero)
2018-12-18blockchain_db: speedup tx output gatheringmoneromooo-monero2-22/+28
We know all the data we'll want for getblocks.bin is contiguous
2018-12-12Merge pull request #4901Riccardo Spagni2-4/+68
a48f2dab blockchain_prune_known_spent_data: blackball file is now optional (moneromooo-monero) 17b45725 Outputs where all amounts are known spent can now be pruned (moneromooo-monero)
2018-12-05db_lmdb: avoid pointless divisionmoneromooo-monero1-1/+1
2018-12-05db_lmdb: inline check_open, it's trivial and called everywheremoneromooo-monero2-8/+7
2018-12-05blockchain_db: remove a couple unused functionsmoneromooo-monero2-41/+0
2018-12-04Merge pull request #4872Riccardo Spagni1-10/+6
fc99f177 lmdb: fix gcc 7.3.0 'implicit-fallthrough' warning (xiphon)
2018-12-04Merge pull request #4854Riccardo Spagni1-2/+0
bd98e99c Removed a lot of unnecessary includes (Martijn Otto)
2018-11-27Outputs where all amounts are known spent can now be prunedmoneromooo-monero2-4/+68
Only for pre rct for obvious reasons. Note: DO NOT use a known spent list which includes outputs which are not known spent. If the list includes any output that's just strongly thought to be spent, but not provably so, you risk finding yourself unable to sync past the point where that output is spent. I estimate only 200 MB saved on current mainnet though, unless the new blackballing rule unearths a good amount of large-amount-set extra spent outs.
2018-11-26Merge pull request #4821Riccardo Spagni2-6/+10
fc98f7a0 rpc: speedup get_outs.bin (moneromooo-monero)
2018-11-26rpc: speedup get_outs.binmoneromooo-monero2-6/+10
2018-11-26Merge pull request #4830Riccardo Spagni1-0/+9
dffec258 db_lmdb: error out if the db needs migration in read only mode (moneromooo-monero)
2018-11-19lmdb: fix gcc 7.3.0 'implicit-fallthrough' warningxiphon1-10/+6
2018-11-15Removed a lot of unnecessary includesMartijn Otto1-2/+0
2018-11-14Merge pull request #4769Riccardo Spagni2-21/+26
5808530f blockchain: remove unused output_scan_worker parameter (moneromooo-monero) 1426209a blockchain: don't run threads if we have just one function to run (moneromooo-monero) 6f7a5fd4 db_lmdb: slight speedup getting array data from the blockchain (moneromooo-monero) 99fbe100 db_lmdb: save some string copies for readonly db keys/values (moneromooo-monero) bf31447e tx_pool: speed up take_tx for transactions from blocks (moneromooo-monero) 4f005a77 tx_pool: remove unnecessary get_transaction_hash (moneromooo-monero) 593ef598 perf_timer: call reserve on new timer array (moneromooo-monero) 6ecc99ad core: avoid unnecessary tx/blob conversions (moneromooo-monero) 00cc1a16 unit_tests: notify test special case for the usual weirdo (moneromooo-monero)
2018-11-09db_lmdb: error out if the db needs migration in read only modemoneromooo-monero1-0/+9
2018-11-04db_lmdb: slight speedup getting array data from the blockchainmoneromooo-monero1-6/+8
2018-11-02db_lmdb: log which output was not found in get_output_keymoneromooo-monero1-1/+2
2018-11-01Merge pull request #4664Riccardo Spagni1-9/+23
61304151 db_lmdb: use MDB_MULTIPLE_NEXT where possible for some speedup (moneromooo-monero)
2018-11-01db_lmdb: save some string copies for readonly db keys/valuesmoneromooo-monero1-11/+16
2018-11-01core: avoid unnecessary tx/blob conversionsmoneromooo-monero2-5/+3
2018-10-19db_lmdb: use MDB_MULTIPLE_NEXT where possible for some speedupmoneromooo-monero1-9/+23
2018-10-17Update db_lmdb.cppCactii11-1/+1
2018-10-02Fix rtxn usage in BlockchainLMDB::get_estimated_batch_sizeHoward Chu1-2/+2
Should only stop the rtxn if we actually started it Fixes Coverity 184960
2018-09-29Merge pull request #4459Riccardo Spagni1-5/+16
bcf3f6af fuzz_tests: catch unhandled exceptions (moneromooo-monero) 3ebd05d4 miner: restore stream flags after changing them (moneromooo-monero) a093092e levin_protocol_handler_async: do not propagate exception through dtor (moneromooo-monero) 1eebb82b net_helper: do not propagate exceptions through dtor (moneromooo-monero) fb6a3630 miner: do not propagate exceptions through dtor (moneromooo-monero) 2e2139ff epee: do not propagate exception through dtor (moneromooo-monero) 0749a8bd db_lmdb: do not propagate exceptions in dtor (moneromooo-monero) 1b0afeeb wallet_rpc_server: exit cleanly on unhandled exceptions (moneromooo-monero) 418a9936 unit_tests: catch unhandled exceptions (moneromooo-monero) ea7f9543 threadpool: do not propagate exceptions through the dtor (moneromooo-monero) 6e855422 gen_multisig: nice exit on unhandled exception (moneromooo-monero) 53df2deb db_lmdb: catch error in mdb_stat calls during migration (moneromooo-monero) e67016dd blockchain_blackball: catch failure to commit db transaction (moneromooo-monero) 661439f4 mlog: don't remove old logs if we failed to rename the current file (moneromooo-monero) 5fdcda50 easylogging++: test for NULL before dereference (moneromooo-monero) 7ece1550 performance_test: fix bad last argument calling add_arg (moneromooo-monero) a085da32 unit_tests: add check for page size > 0 before dividing (moneromooo-monero) d8b1ec8b unit_tests: use std::shared_ptr to shut coverity up about leaks (moneromooo-monero) 02563bf4 simplewallet: top level exception catcher to print nicer messages (moneromooo-monero) c57a65b2 blockchain_blackball: fix shift range for 32 bit archs (moneromooo-monero)
2018-09-29Merge pull request #4445Riccardo Spagni1-2/+12
fe125647 Fixup RENAME_DB() macro (Howard Chu)
2018-09-12db_lmdb: do not use base for cumulative distributionmoneromooo-monero1-0/+2
it's confusing and needlessly complicated
2018-09-11v8: per byte fee, pad bulletproofs, fixed 11 ring sizemoneromooo-monero2-17/+22
2018-09-10Merge pull request #4293luigi11111-2/+6
9d65399 is_hdd update (p8p)
2018-09-10Merge pull request #4256luigi11112-7/+3
d6fc870 db_lmdb: resize blockchain database when 90% filled (moneromooo-monero)
2018-09-04Merge pull request #4270luigi11111-2/+13
29dea03 epee: resize vectors where possible in serialization (moneromooo-monero) 76affd9 epee: some speedup in parsing (moneromooo-monero) dc6c069 db_lmdb: speedup the get_output_distribution common case (moneromooo-monero) 76ac5a8 wallet2: ask for a binary output distribution, for speed (moneromooo-monero)
2018-08-25is_hdd updatep8p1-2/+6
2018-08-22Merge pull request #4200luigi11112-50/+0
43f7110 blockchain_db: remove unused get_output_key variant (moneromooo-monero)
2018-08-19db_lmdb: speedup the get_output_distribution common casemoneromooo-monero1-2/+13
2018-08-15Merge pull request #4161luigi11111-2/+2
be02eb9 db_lmdb: demote the 'batch already enabled' log, it's harmless (moneromooo-monero)
2018-08-14db_lmdb: resize blockchain database when 90% filledmoneromooo-monero2-7/+3
instead of a random ratio from 60% to 90%.
2018-07-30blockchain_db: remove unused get_output_key variantmoneromooo-monero2-50/+0
It was actually incorrect, as it would not return commitment
2018-07-27Merge pull request #4129luigi11111-0/+3
ee31383 db_lmdb: don't sync a read only DB (moneromooo-monero)
2018-07-20db_lmdb: demote the "batch already enabled" log, it's harmlessmoneromooo-monero1-2/+2
2018-07-19Merge pull request #4013luigi11112-0/+13
e5592c4 rpc: add blockchain disk size to getinfo (moneromooo-monero)
2018-07-19Merge pull request #3981luigi11112-6/+211
45e419b db: store cumulative rct output distribution in the db for speed (moneromooo-monero)
2018-07-19Merge pull request #3854luigi11112-1/+18
149da42 db_lmdb: enable batch transactions by default (stoffu) 34cb6b4 add --regtest and --fixed-difficulty for regression testing (vicsn) 9e1403e update get_info RPC and bump RPC version (vicsn) 207b66e first new functional tests (vicsn)
2018-07-13db: store cumulative rct output distribution in the db for speedmoneromooo-monero2-6/+211
This gets rid of the temporary precalc cache. Also make the RPC able to send data back in binary or JSON, since there can be a lot of data This bumps the LMDB database format to v3, with migration.
2018-07-11db_lmdb: don't sync a read only DBmoneromooo-monero1-0/+3
This would only throw
2018-06-29add --regtest and --fixed-difficulty for regression testingvictorsintnicolaas2-0/+17
on_generateblocks RPC call combines functionality from the on_getblocktemplate and on_submitblock RPC calls to allow rapid block creation. Difficulty is set permanently to 1 for regtest. Makes use of FAKECHAIN network type, but takes hard fork heights from mainchain Default reserve_size in generate_blocks RPC call is now 1. If it is 0, the following error occurs 'Failed to calculate offset for'. Queries hard fork heights info of other network types
2018-06-20rpc: add blockchain disk size to getinfomoneromooo-monero2-0/+13
This should help new nodes predict how much disk space will be needed for a full sync
2018-06-14db_lmdb: enable batch transactions by defaultstoffu1-1/+1
2018-06-11db_ldmb: print db version when complaining about incompatibilitymoneromooo-monero1-4/+5
2018-06-08db_lmdb: warn about slowness when running off a spinning diskmoneromooo-monero1-0/+3
2018-05-30Merge pull request #3876luigi11111-0/+1
740da1b core: fix automatic safe db sync mode switching (moneromooo-monero) e942d34 protocol: do not switch to unsafe sync mode for just a few blocks (moneromooo-monero)
2018-05-28core: fix automatic safe db sync mode switchingmoneromooo-monero1-0/+1
2018-05-23db_lmdb: save pruned and prunable tx data separatelymoneromooo-monero2-41/+328
This bumps DB version to 2, migration code will run for v1 DBs
2018-05-21speed up get_output_distribution (and precalc common case)moneromooo-monero2-0/+43
2018-04-23speedup get_output_histogram for all amounts when min_count > 0moneromooo-monero2-5/+9
This skips the vast majority of "dust" output amounts with just one instance on the chain. Clocks in at 0.15% of the original time on testnet.
2018-03-16add RPC to get a histogram of outputs of a given amountmoneromooo-monero2-3/+44
2018-03-05Merge pull request #3307Riccardo Spagni1-1/+2
223d7d0c db_lmdb: fix free space reporting (moneromooo-monero)
2018-03-05Correct spelling mistakes.Edward Betts1-1/+1
2018-02-23db_lmdb: fix free space reportingmoneromooo-monero1-1/+2
reported by Brad Richards
2018-02-19db_lmdb: fix return code mixup in for_all_*moneromooo-monero1-12/+12
2018-02-16Merge pull request #3226Riccardo Spagni1-4/+7
e4646379 keccak: fix mdlen bounds sanity checking (moneromooo-monero) 2e3e90ac pass large parameters by const ref, not value (moneromooo-monero) 61defd89 blockchain: sanity check number of precomputed hash of hash blocks (moneromooo-monero) 9af6b2d1 ringct: fix infinite loop in unused h2b function (moneromooo-monero) 8cea8d0c simplewallet: double check a new multisig wallet is multisig (moneromooo-monero) 9b98a6ac threadpool: catch exceptions in dtor, to avoid terminate (moneromooo-monero) 24803ed9 blockchain_export: fix buffer overflow in exporter (moneromooo-monero) f3f7da62 perf_timer: rewrite to make it clear there is no division by zero (moneromooo-monero) c6ea3df0 performance_tests: remove add_arg call stray extra param (moneromooo-monero) fa6b4566 fuzz_tests: fix an uninitialized var in setup (moneromooo-monero) 03887f11 keccak: fix sanity check bounds test (moneromooo-monero) ad11db91 blockchain_db: initialize m_open in base class ctor (moneromooo-monero) bece67f9 miner: restore std::cout precision after modification (moneromooo-monero) 1aabd14c db_lmdb: check hard fork info drop succeeded (moneromooo-monero)
2018-02-16Merge pull request #3231Riccardo Spagni2-2/+10
84a8b2da Don't create readtxn until after txn_safe gate check (Howard Chu)
2018-02-02blockchain_db: initialize m_open in base class ctormoneromooo-monero1-2/+1
It's cleaner this way, since it's a base class field Coverity 136568
2018-02-02db_lmdb: check hard fork info drop succeededmoneromooo-monero1-2/+6
Coverity 136364
2018-01-31Don't create readtxn until after txn_safe gate checkHoward Chu2-2/+10
2018-01-26Update 2018 copyrightxmr-eric2-2/+2
2018-01-10Merge pull request #2971Riccardo Spagni2-4/+6
ae860230 Fix exceptions not finding txpool txes when relaying (moneromooo-monero)
2017-12-28Additional fix for core_testsHoward Chu1-10/+15
Reset thread-local info if it doesn't match the current env. Only happens when a process opens/closes env multiple times in the same process, doesn't affect monerod.
2017-12-28Fix stale readcursor flagsHoward Chu1-22/+15
Reset thread-specific flags when a write txn is started. Also remove some redundant start-readtxn code.
2017-12-20Fix exceptions not finding txpool txes when relayingmoneromooo-monero2-4/+6
2017-12-18catch const exceptionsmoneromooo-monero1-1/+1
2017-12-16move includes around to lessen overall loadmoneromooo-monero2-1/+2
2017-11-14Merge pull request #2629Riccardo Spagni1-0/+1
383ff4f6 remove "using namespace std" from headers (moneromooo-monero)
2017-11-14remove "using namespace std" from headersmoneromooo-monero1-0/+1
It's nasty, and actually breaks on Solaris, where if.h fails to build due to: struct map *if_memmap;
2017-11-14Merge pull request #2742Riccardo Spagni1-0/+6
7c7d3672 Increase LMDB maxreaders if large number of threads in use (Howard Chu) 6738753b Use max_concurrency as-is (Howard Chu)
2017-11-08Protect node privacy by proper filtering in restricted-mode RPC answersbinaryFate2-9/+40
This patch allows to filter out sensitive information for queries that rely on the pool state, when running in restricted mode. This filtering is only applied to data sent back to RPC queries. Results of inline commands typed locally in the daemon are not affected. In practice, when running with `--restricted-rpc`: * get_transaction_pool will list relayed transactions with the fields "last relayed time" and "received time" set to zero. * get_transaction_pool will not list transaction that have do_not_relay set to true, and will not list key images that are used only for such transactions * get_transaction_pool_hashes.bin will not list such transaction * get_transaction_pool_stats will not count such transactions in any of the aggregated values that are computed The implementation does not make filtering the default, so developers should be mindful of this if they add new RPC functionality. Fixes #2590.
2017-11-02Increase LMDB maxreaders if large number of threads in useHoward Chu1-0/+6
2017-10-26fixed an LMDB issue on OpenBSD (#2699) which leads to a crash in monerodston1th1-0/+6
2017-10-15Merge pull request #2640Riccardo Spagni1-1/+1
22b51e06 db_lmdb: include chain height when failing to find an output key (moneromooo-monero) 5db433b3 blockchain: avoid exceptions in output verification (moneromooo-monero)
2017-10-12db_lmdb: include chain height when failing to find an output keymoneromooo-monero1-1/+1
To help debugging logs.
2017-10-07frob level 1 logs a bit for consistencymoneromooo-monero1-7/+7
Level 1 logs map to INFO, so setting log level to 1 should show these. Demote some stuff to DEBUG to avoid spam, though.
2017-09-25Merge pull request #2457Riccardo Spagni2-8/+14
d2d8a41c Use actual batch size for resize estimates (Howard Chu)
2017-09-19db_lmdb: fix use of uninitialized key in for_blocks_rangemoneromooo-monero1-2/+2
2017-09-17Use actual batch size for resize estimatesHoward Chu2-8/+14
And optimize import startup: Remember start_height position during initial count_blocks pass to avoid having to reread entire file again to arrive at start_height
2017-08-29Fix blockchain_import wedge on exception in cleanup_handle_incoming_blocksmoneromooo-monero2-9/+25
2017-08-20Toggle SAFE syncmode on and off automaticallyHoward Chu2-0/+7
If monerod is started with default sync mode, set it to SAFE after synchronization completes. Set it back to FAST if synchronization restarts (e.g. because another peer has a longer blockchain). If monerod is started with an explicit sync mode, none of this automation takes effect.
2017-08-19DB cleanupHoward Chu1-1/+11
Hide LMDB-specific stuff behind blockchain_db.h. Nobody besides blockchain_db.cpp should ever be including DB-specific headers any more.
2017-08-12core: add --db-salvage command line flagmoneromooo-monero1-1/+1
Use to load the database when the primary meta page is corrupted
2017-07-31some include cleanupmoneromooo-monero1-0/+1
2017-06-11blockchain_db: add a txpool tx getter which returns existencemoneromooo-monero2-3/+13
Avoids exception spam for the "nope, not found" case
2017-06-01Don't copy blockchain for coinbase_tx_sumHoward Chu2-3/+13
Changed Blockchain::for_all_blocks() to for_blocks_range() Operate on blockchain in-place instead of building a copy first.
2017-05-25Move txpool to the databasemoneromooo-monero2-0/+235
Integration could go further (ie, return_tx_to_pool calls should not be needed anymore, possibly other things). poolstate.bin is now obsolete.
2017-03-19Partial fix #1876, check thread in block_txn_start()Howard Chu1-1/+2
Don't allow use of existing batch txn if it's from the wrong thread
2017-03-03Remove redundant num_txs() methodHoward Chu2-18/+4
Cleanup of bf1348b7e2b2c72a6d40b23567afaa46b53e6cb7
2017-02-21update copyright year, fix occasional lack of newline at line endRiccardo Spagni2-2/+2
2017-02-21Merge pull request #1727Riccardo Spagni2-28/+9
0288310e blockchain_db: add "raw" blobdata getters for block and transaction (moneromooo-monero)
2017-02-21Merge pull request #1724Riccardo Spagni2-2/+9
cca95c1c blockchain_db: do not throw on expected partial results getting keys (moneromooo-monero)
2017-02-18Handle map resizes from other processesHoward Chu1-9/+52
2017-02-18Can't cache num_txs or num_outputs eitherHoward Chu2-34/+34
Same reason as 3ff54bdd7a8b5e08e4e8ac17b7fff23ad3a82312
2017-02-18Add a few read txns to streamlineHoward Chu2-3/+7
Slight perf gain, but mainly to reduce spam at loglevel 3
2017-02-13blockchain_db: add "raw" blobdata getters for block and transactionmoneromooo-monero2-28/+9
This speeds up operations such as serving blocks to syncing peers
2017-02-13blockchain_db: do not throw on expected partial results getting keysmoneromooo-monero2-2/+9
When scanning for outputs used in a set of incoming blocks, we expect that some of the inputs in their transactions will not be found in the blockchain, as they could be in previous blocks in that set. Those outputs will be scanned there at a later point. In this case, we add a flag to control wehther an output not being found is expected or not.
2017-02-11db_lmdb: fix bad height saved in tx datamoneromooo-monero1-0/+1
The recent change to not keep separate track of the blockchain height caused the reported height to jump early in the lmdb transaction (when the block data is added to the blocks table), rather than at the end, after everything succeeded. Since the block data is added before the transaction data, this caused the transaction data to be saved with a height one more than its expected value. Fix this by saving the block data last. This should have no side effects.
2017-02-08extract some basic code from libcryptonote_core into libcryptonote_basickenshi841-1/+1
2017-01-28Update and use blockchain data files definesMiguel Herranz1-4/+5
2017-01-16Tweak some msg loglevelsHoward Chu1-11/+13
2017-01-16Change logging to easylogging++moneromooo-monero1-17/+21
This replaces the epee and data_loggers logging systems with a single one, and also adds filename:line and explicit severity levels. Categories may be defined, and logging severity set by category (or set of categories). epee style 0-4 log level maps to a sensible severity configuration. Log files now also rotate when reaching 100 MB. To select which logs to output, use the MONERO_LOGS environment variable, with a comma separated list of categories (globs are supported), with their requested severity level after a colon. If a log matches more than one such setting, the last one in the configuration string applies. A few examples: This one is (mostly) silent, only outputting fatal errors: MONERO_LOGS=*:FATAL This one is very verbose: MONERO_LOGS=*:TRACE This one is totally silent (logwise): MONERO_LOGS="" This one outputs all errors and warnings, except for the "verify" category, which prints just fatal errors (the verify category is used for logs about incoming transactions and blocks, and it is expected that some/many will fail to verify, hence we don't want the spam): MONERO_LOGS=*:WARNING,verify:FATAL Log levels are, in decreasing order of priority: FATAL, ERROR, WARNING, INFO, DEBUG, TRACE Subcategories may be added using prefixes and globs. This example will output net.p2p logs at the TRACE level, but all other net* logs only at INFO: MONERO_LOGS=*:ERROR,net*:INFO,net.p2p:TRACE Logs which are intended for the user (which Monero was using a lot through epee, but really isn't a nice way to go things) should use the "global" category. There are a few helper macros for using this category, eg: MGINFO("this shows up by default") or MGINFO_RED("this is red"), to try to keep a similar look and feel for now. Existing epee log macros still exist, and map to the new log levels, but since they're used as a "user facing" UI element as much as a logging system, they often don't map well to log severities (ie, a log level 0 log may be an error, or may be something we want the user to see, such as an important info). In those cases, I tried to use the new macros. In other cases, I left the existing macros in. When modifying logs, it is probably best to switch to the new macros with explicit levels. The --log-level options and set_log commands now also accept category settings, in addition to the epee style log levels.
2017-01-15Merge pull request #1506Riccardo Spagni2-18/+41
3ff54bdd Check for correct thread before ending batch transaction (Howard Chu) eaf8470b Must wait for previous batch to finish before starting new one (Howard Chu) c903c554 Don't cache block height, always get from DB (Howard Chu) eb1fb601 Tweak default db-sync-mode to fast:async:1 (Howard Chu) 0693cff9 Use batch transactions when syncing (Howard Chu)
2017-01-15Check for correct thread before ending batch transactionHoward Chu1-0/+7
2017-01-14Don't cache block height, always get from DBHoward Chu2-14/+29
2017-01-14Use batch transactions when syncingHoward Chu2-4/+5
Faster throughput while avoiding corruption. I.e., makes running with --db-sync-mode safe more tolerable.
2017-01-07blockchain: allow marking "tx not found" without an exceptionmoneromooo-monero2-3/+13
This is a normal occurence in many cases, and there is no need to spam the log with those when it is.
2016-12-01db_lmdb: add info in an error message when we can't get an outputmoneromooo-monero1-1/+1
Will be useful to debug
2016-12-01db_lmdb: guard against going out of sync on unexpected db resultsmoneromooo-monero1-1/+1
m_num_outputs keeps track of the number of outputs, which should be the same as the size of both the output_txs and output_amounts databases. If one goes out of sync, we need to throw to abort whatever it is we were doing.
2016-12-01db_lmdb: minor pedantic tweaksmoneromooo-monero1-8/+8
Add consts in a few places where it makes sense, avoid unnecessary memory reallocation where we know the full size needed at the outset, simplify and avoid memory copy.
2016-12-01db_lmdb: set same packing format for output_data_t and pre_rct_output_data_tmoneromooo-monero1-0/+3
For safety, though it seems to have been the case already. Also add a comment about the necessary layout identity.
2016-10-15wallet: select part of the fake outs from recent outputsmoneromooo-monero2-11/+28
25% of the outputs are selected from the last 5 days (if possible), in order to avoid the common case of sending recently received outputs again. 25% and 5 days are subject to review later, since it's just a wallet level change.
2016-09-29Merge pull request #1128Riccardo Spagni1-1/+1
d2e11f3 db_lmdb: do not error if dropping a non existent hard fork table (moneromooo-monero)
2016-09-24db_lmdb: do not error if dropping a non existent hard fork tablemoneromooo-monero1-1/+1
2016-09-21Fix minor typo in "need resize" message.Myagui1-2/+2
Message observed while synchronizing a node from scratch. "LMDB memory map needs resized" Proposing a change to: "LMDB memory map needs to be resized"
2016-08-31core: faster find_blockchain_supplementmoneromooo-monero2-2/+9
Since this queries block heights for blocks that may or may not exist, queries for non existing blocks would throw an exception, and that would slow down the loop a lot. 7 seconds to go through a 30 hash list. Fix this by adding an optional return block height to block_exists and using this instead. Actual errors will still throw an exception. This also cuts down on log exception spam.
2016-08-29db_lmdb: fix reorg with "halfway rct" coinbase outputsmoneromooo-monero1-1/+3
2016-08-28move the rct commitments to the output_amounts databasemoneromooo-monero2-128/+53
Since these are needed at the same time as the output pubkeys, this is a whole lot faster, and takes less space. Only outputs of 0 amount store the commitment. When reading other outputs, a fake commitment is regenerated on the fly. This avoids having to rewrite the database to add space for fake commitments for existing outputs. This code relies on two things: - LMDB must support fixed size records per key, rather than per database (ie, all records on key 0 are the same size, all records for non 0 keys are same size, but records from key 0 and non 0 keys do have different sizes). - the commitment must be directly after the rest of the data in outkey and output_data_t.
2016-08-28add rct to the protocolmoneromooo-monero1-0/+2
It is not yet constrained to a fork, so don't use on the real network or you'll be orphaned or rejected.
2016-08-28db_lmdb: update reset for recent db changesmoneromooo-monero1-0/+10
- we need to drop the new m_tx_indices database - we reset the version to current version This fixes the core tests failing to initialize.
2016-08-28blockchain_db: add functions for adding/removing/getting rct commitmentsmoneromooo-monero2-0/+120
2016-08-26db_lmdb: do not try to modify the database in read only modemoneromooo-monero1-4/+12
2016-08-11Merge pull request #948Riccardo Spagni2-6/+24
11dc091 Fake outs set is now decided by the wallet (moneromooo-monero) 1593553 new unlocked parameter to output_histogram (moneromooo-monero)
2016-08-10db_lmdb: errors dropping hf starting height db on reset are not fatalmoneromooo-monero1-2/+1
This db is now dropped unconditionally, so may or may not be there in the first place.
2016-08-01new unlocked parameter to output_histogrammoneromooo-monero2-6/+24
This constrains the number of instances of any amount to the unlocked ones (as defined by the default unlock time setting: outputs with non default unlock time are not considered, so may be counted as unlocked even if they are not actually unlocked).
2016-07-26db_lmdb: add some missing api call checksmoneromooo-monero1-14/+31
2016-07-13remove hf_starting_height dbmoneromooo-monero2-76/+2
It's not really needed, it used to be an optimization for when that code was not using the db and needed to recalculate things fast on startup.
2016-04-24Another take on migrationHoward Chu1-570/+94
Delete old indices and recreate them, rather than updating them Maybe not quite as slow as before.
2016-04-08MigrationHoward Chu2-5/+1049
Migrate from DB version 0 to version 1 on startup
2016-04-05Merge branch 'performance' into masterHoward Chu2-715/+559
2016-04-05CleanupHoward Chu2-21/+65
drop obsolete remove_output() fix get_output_key(global), fix crash in blockchain_dump
2016-04-05Cleanup and clarifyHoward Chu2-115/+92
Try to rationalize the variable names, document usage.
2016-04-05Use cursors in some remove functionsHoward Chu1-7/+33
Helps when they're called repeatedly in one txn
2016-04-05Use DUPFIXED for block_info and output_txsHoward Chu2-56/+46
Saves another ~150MB or so on the full blockchain
2016-04-05More outputs consolidationHoward Chu2-491/+263
Also bumped DB VERSION to 1 Another significant speedup and space savings: Get rid of global_output_indices, remove indirection from output to keys This is the change warptangent described on irc but never got to finish.
2016-04-05Use DUPFIXED for output_keysHoward Chu1-23/+37
Saves another 90MB on 200000 block import. Had to bring back compare_uint64 for this, but it's safe since this table is always 64-bit aligned.
2016-04-05Use DUPFIXED for tx_indicesHoward Chu1-50/+60
Small space savings, no measurable speedup
2016-04-05Use DUPFIXED for block_heightsHoward Chu1-22/+29
Only a small savings...
2016-04-05Use DUPFIXED for spent_keysHoward Chu1-23/+18
2016-04-05Add back changes from revert.warptangent1-19/+2
m_tx_outputs doesn't need to be changed, as it's no longer dup list.
2016-04-05Use MDB_APPEND mode with two tx subdbswarptangent1-2/+2
This is possible on those using a tx index as a key.
2016-04-05Schema update: tx_indices - consolidate the tx subdbs from 5 to 3warptangent2-75/+47
2016-04-05Schema update: tx_indices - yet less indirectionwarptangent2-16/+11
2016-04-05Schema update: tx_indices - improve further with less indirectionwarptangent2-17/+50
2016-04-05Add batch warning for further reviewwarptangent1-0/+4
2016-04-05Schema update: tx_indiceswarptangent2-32/+109
2016-04-05Consolidated block infoHoward Chu2-95/+63
2016-04-05Update schema for "tx_outputs" to use array containing amount output indiceswarptangent2-114/+103
This speeds up wallet refresh by directly retrieving a tx's amount output indices. It removes the indirection and walking the amount output duplicate list for every amount in each requested tx. "tx_outputs" is used by: Amount output indices are needed for wallet refresh. Global output indices are needed for removing a tx. Both amount output indices and global output indices are now stored in an array of 64-bit unsigned ints: tx_outputs[<tx_hash>] -> [ <a1_oi, a1_gi, a2_oi, a2_gi, ...> ] Previously it was: tx_outputs[<tx_hash>] -> duplicate list of <a1_gi, a2_gi, a3_gi, ...> The amount output list had to be walked for every amount in order to find each amount's output index, by comparing the amount's global output index with each one in the duplicate list until a match was found. See also d045dfa7ce0bf131681193c97560da26f9f37900
2016-03-30Detect map resize failuresHoward Chu1-1/+3
2016-03-26New RPC and daemon command to get output histogrammoneromooo-monero2-0/+67
This is a list of existing output amounts along with the number of outputs of that amount in the blockchain. The daemon command takes: - no parameters: all outputs with at least 3 instances - one parameter: all outputs with at least that many instances - two parameters: all outputs within that many instances The default starts at 3 to avoid massive spamming of all dust outputs in the blockchain, and is the current minimum mixin requirement. An optional vector of amounts may be passed, to request histogram only for those outputs.
2016-03-20db_lmdb: include the error codes from lmdb api in error logsmoneromooo-monero1-40/+47
2016-03-19Reduce log noiseHoward Chu1-2/+3
2016-03-16Add destructor for readtxnsHoward Chu2-10/+21
Only if we created the readtxn. Was missing cleanups from exceptions before.
2016-03-15Small cleanupsHoward Chu1-24/+28
Only one return and TXN_POSTFIX_RDONLY() per function Only log rtxn_start if the rtxn wasn't already active
2016-03-15More for 92dd4ec6d6251b15954002e72a7c7faa5059a3abHoward Chu1-22/+22
Make sure we stop the right txn too
2016-03-14Merge pull request #723Riccardo Spagni2-21/+21
2abdb2c avoid some val copies (Howard Chu)
2016-03-14Hack for read/write txn mixupHoward Chu2-42/+31
save the thread ID of the writer thread so we don't try to use the writetxn from reader threads
2016-03-14avoid some val copiesHoward Chu2-21/+21
2016-03-09More for bdec7cbHoward Chu1-14/+14
More uses of db error helper
2016-03-05Merge pull request #700Riccardo Spagni1-33/+34
bdec7cb BlockchainLMDB: Use DB error helper consistently (warptangent) c5932eb BlockchainLMDB: Add DB error to exception (warptangent) a49c355 Blockchain: Omit verbose time stats messages by default (warptangent)
2016-03-05Merge pull request #698Riccardo Spagni1-57/+14
ee7a8b8 Get rid of lmdb_cur (Howard Chu)
2016-03-03BlockchainLMDB: Use DB error helper consistentlywarptangent1-28/+27
2016-03-03BlockchainLMDB: Add DB error to exceptionwarptangent1-5/+7
2016-03-03Get rid of lmdb_curHoward Chu1-57/+14
We don't need it now with per-txn cursors.
2016-03-03More for df239428c0f43e2664f2be273996a734ca12c686Howard Chu1-0/+6
Let ARMv7 work again
2016-03-01Make partial revert consistentwarptangent1-1/+1
If user-defined comparator is used, subdb shouldn't be opened with MDB_INTEGERKEY. TODO: Again, this will be added back with future schema updates.
2016-02-29Revert part of 7db89ed2eee2ce31f039783323d665fe7c24d441warptangent1-2/+14
For now, so existing databases work. TODO: add these back with future schema updates.
2016-02-29BlockchainLMDB: Add height and db error to exceptionwarptangent1-1/+1
2016-02-25ARMv7: fix unaligned accessesHoward Chu1-20/+15
And cleanup some key comparators
2016-02-23read txn/cursor stuffHoward Chu2-156/+317
Could wrap more later.
2016-02-17Use MDB_PREV_MULTIPLEHoward Chu1-12/+41
in get_global_output_indices
2016-02-17Fixup after lmdb master resyncHoward Chu1-6/+6
2016-02-17Shutup about VERSION 0Howard Chu1-1/+3
2016-02-17Use cursor in get_output_keyHoward Chu1-1/+2
2016-02-17Use MDB_APPEND mode where possibleHoward Chu1-5/+9
When keys are contiguous and monotonically increasing, this gets denser page utilization (doesn't leave padding in page splits). Can't be used for keys that are inserted in random order (e.g. hashes) In total this only saves around 1.5% of space compared to original DB code. The previous patch accounted for 0.8% savings on its own; the blocks tables just aren't that big.
2016-02-17Use cursors in write txnsHoward Chu2-22/+100
Saves a bit of seek overhead. LMDB frees them automatically in txn_(commit|abort) so they need no cleanup.
2016-02-17Keep a running blocksize countHoward Chu2-0/+15
Used in batch size estimation, avoids rereading already processed blocks during import
2016-02-17Win32 import batchsize tweaksHoward Chu1-3/+6
Reduce frequency of resizes: bump minimum increase from 128MB to 512MB Use a bigger safety margin at small batch sizes
2016-02-13BlockchainLMDB: Deal with DB exceptions at block level with particularitywarptangent1-2/+13
Add another DB error exception type to distinguish failed txn setup from general use of txn. This keeps the error handling flow the same as before the block-level txn setup changes that moved control up a layer to BlockchainDB.
2016-02-13BlockchainLMDB: Add sanity check for inconsistent statewarptangent1-2/+13
This hasn't been known to occur in block-level txn abort, but throw exception if it does.
2016-02-13BlockchainLMDB: Call destructor on allocated txn if setup failswarptangent1-0/+8
2016-02-13BlockchainLMDB: Replace remaining txn pointer NULLs with nullptrwarptangent1-9/+9
For consistency.
2016-02-08Make HardFork object available to BlockchainDB and derived DB implementationswarptangent1-0/+2
This will later allow the HardFork object's DB update functions to be called when the DB transaction that persists across block add/remove is open.
2016-02-08BlockchainDB/LMDB: Refactor block-scope DB txn handling for add blockwarptangent1-19/+1
Move block-scope txn start and stop from BlockchainLMDB to BlockchainDB.
2016-02-08BlockchainDB/LMDB/BDB: Extract DB txn functions for block add/removewarptangent2-0/+42
2016-02-08BlockchainLMDB: Allow two HardFork functions to update DB during block addwarptangent1-4/+4
Note that this doesn't yet cause them to be called during block add.
2016-02-08BlockchainLMDB: extract txn macros used during block add/removewarptangent1-0/+26