| Age | Commit message (Collapse) | Author | Files | Lines |
|---|
|
PoW is expensive to verify, so be strict
|
|
Revert a96c1a4 and adc16d2 (moneromooo-monero)
|
|
2cd4fd8 Changed the use of boost:value_initialized for C++ list initializer (JesusRami)
4ad191f Removed unused boost/value_init header (whyamiroot)
928f4be Make null hash constants constexpr (whyamiroot)
|
|
As a side effect, colouring on Windows should now work
regardless of version
|
|
This reverts commit a96c1a46d4b3854252de75cbe09458ad5d1aecb0.
|
|
This reverts commit adc16d2504d3e76b0115791caf10446684d45433.
|
|
69465e3 cryptonote_protocol: fix '--no-sync', ignore new blocks and txes (xiphon)
|
|
|
|
|
|
2aa1134 daemon: display peer address type in print_cn (moneromooo-monero)
|
|
f17dcde Remove unused txs member in NOTIFY_RESPONSE_GET_OBJECT (Doy-lee)
|
|
|
|
The check added here (in #5732/#5733) is supposed to disconnect behind
peers when the current node is syncing, but actually disconnects behind
peers always.
We are syncing when `target > our_height`, but the check here triggers
when `target > remote_height`, which is basically always true when the
preceding `m_core.have_block(hshd.top_id)` check is true.
|
|
|
|
Any peer that's behind us while syncing is useless to us (though
not to them). This ensures that we don't get our peer slots filled
with peers that we can't use. Once we've synced, we can connect
to them and they can then sync off us if they want.
|
|
When asking for txes in a fluffy transaction, one might ask
for the same (large) tx many times
|
|
Count transactions as well
|
|
Essentially, one can send such a large amount of IDs that core exhausts
all free memory. This issue can theoretically be exploited using very
large CN blockchains, such as Monero.
This is a partial fix. Thanks and credit given to CryptoNote author
'cryptozoidberg' for collaboration and the fix. Also thanks to
'moneromooo'. Referencing HackerOne report #506595.
|
|
|
|
|
|
|
|
b0c552f5 cryptonote_protocol_handler: add block/tx hashes in notify logs (moneromooo-monero)
|
|
c84ea299 cryptonote_basic: some more minor speedups (moneromooo-monero)
e40eb2ad cryptonote_basic: speedup calculate_block_hash (moneromooo-monero)
547a9708 cryptonote: block parsing + hash calculation speedup (moneromooo-monero)
11604b6d blockchain: avoid unneeded block copy (moneromooo-monero)
8461df04 save some database calls when getting top block hash and height (moneromooo-monero)
3bbc3661 Avoid repeated (de)serialization when syncing (moneromooo-monero)
|
|
|
|
This saves a duplicate serialization step
|
|
Based on Boolberry work by:
jahrsg <jahr@jahr.me>
cr.zoidberg <crypto.zoidberg@gmail.com>
|
|
eef164f7 cryptonote_protocol_handler: search for syncing peers in "cruise mode" (moneromooo-monero)
|
|
a54e81e5 daemon: add '--no-sync' arg to optionally disable blockchain sync (xiphon)
|
|
551104fb daemon: add --public-node mode, RPC port propagation over P2P (xiphon)
|
|
|
|
|
|
|
|
6a3608d3 cryptonote_protocol_handler: pad tx messages when using tor/i2p (moneromooo-monero)
|
|
When all our outgoing peer slots are filled, we cycle one peer at
a time looking for syncing peers until we have at least two such
peers. This brings two advantages:
- Peers without incoming connections will find more syncing peers
that before, thereby strengthening network decentralization
- Peers will have more resistance to isolation attacks, as they
are more likely to find a "good" peer than they were before
|
|
|
|
|
|
RPC connections now have optional tranparent SSL.
An optional private key and certificate file can be passed,
using the --{rpc,daemon}-ssl-private-key and
--{rpc,daemon}-ssl-certificate options. Those have as
argument a path to a PEM format private private key and
certificate, respectively.
If not given, a temporary self signed certificate will be used.
SSL can be enabled or disabled using --{rpc}-ssl, which
accepts autodetect (default), disabled or enabled.
Access can be restricted to particular certificates using the
--rpc-ssl-allowed-certificates, which takes a list of
paths to PEM encoded certificates. This can allow a wallet to
connect to only the daemon they think they're connected to,
by forcing SSL and listing the paths to the known good
certificates.
To generate long term certificates:
openssl genrsa -out /tmp/KEY 4096
openssl req -new -key /tmp/KEY -out /tmp/REQ
openssl x509 -req -days 999999 -sha256 -in /tmp/REQ -signkey /tmp/KEY -out /tmp/CERT
/tmp/KEY is the private key, and /tmp/CERT is the certificate,
both in PEM format. /tmp/REQ can be removed. Adjust the last
command to set expiration date, etc, as needed. It doesn't
make a whole lot of sense for monero anyway, since most servers
will run with one time temporary self signed certificates anyway.
SSL support is transparent, so all communication is done on the
existing ports, with SSL autodetection. This means you can start
using an SSL daemon now, but you should not enforce SSL yet or
nothing will talk to you.
|
|
|
|
- Support for ".onion" in --add-exclusive-node and --add-peer
- Add --anonymizing-proxy for outbound Tor connections
- Add --anonymous-inbounds for inbound Tor connections
- Support for sharing ".onion" addresses over Tor connections
- Support for broadcasting transactions received over RPC exclusively
over Tor (else broadcast over public IP when Tor not enabled).
|
|
The blockchain prunes seven eighths of prunable tx data.
This saves about two thirds of the blockchain size, while
keeping the node useful as a sync source for an eighth
of the blockchain.
No other data is currently pruned.
There are three ways to prune a blockchain:
- run monerod with --prune-blockchain
- run "prune_blockchain" in the monerod console
- run the monero-blockchain-prune utility
The first two will prune in place. Due to how LMDB works, this
will not reduce the blockchain size on disk. Instead, it will
mark parts of the file as free, so that future data will use
that free space, causing the file to not grow until free space
grows scarce.
The third way will create a second database, a pruned copy of
the original one. Since this is a new file, this one will be
smaller than the original one.
Once the database is pruned, it will stay pruned as it syncs.
That is, there is no need to use --prune-blockchain again, etc.
|
|
85665003 epee: better network buffer data structure (moneromooo-monero)
|
|
5464725a protocol: change standby mode to not wait sleeping (moneromooo-monero)
85807dfb add a once_a_time_milliseconds class (moneromooo-monero)
|
|
avoids pointless allocs and memcpy
|
|
3dba7f25 protocol: option to pad transaction relay to the next kB (moneromooo-monero)
|
|
|
|
506472e0 protocol: fix use after free when dropping a connection (moneromooo-monero)
|
|
|
|
|
|
To help protect one's privacy from traffic volume analysis
for people using Tor or I2P. This will really fly once we
relay txes on a timer rather than on demand, though.
Off by default for now since it's wasteful and doesn't bring
anything until I2P's in.
|
|
ce9457a cryptonote_protocol: don't serialize an unneeded new block message (moneromooo-monero)
|
|
c24a0af [monerod] Added blocks remaining count during syncronisation. (Jorropo)
|
|
Non fluffy block nodes should now be very rare
|
|
And percent if usefull (% < 99)
|
|
a2b557f 6795bd0 209ec96 ed2c81e a830db2 57ea902 31a895e ba8331c f7f1917 41be339 f025ae9 ef2cb63 dcfd299 5d3e702 2704624 2771a18 0e4c7d0 (moneromooo-monero)
|
|
also use reserve where appropriate
|
|
|
|
|
|
|
|
3880bf39 cryptonote_protocol_handler.inl: remove span read just now that failed to pass some basic tests (stoffu)
da249fd5 cryptonote_protocol_handler.inl: fix return type mismatches (int vs bool) (stoffu)
|
|
pass some basic tests
|
|
|
|
|
|
|
|
0e7ad2e2 Wallet API: generalize 'bool testnet' to 'NetworkType nettype' (stoffu)
af773211 Stagenet (stoffu)
cc9a0bee command_line: allow args to depend on more than one args (stoffu)
55f8d917 command_line::get_arg: remove 'required' for dependent args as they're always optional (stoffu)
450306a0 command line: allow has_arg to handle arg_descriptor<bool,false,true> #3318 (stoffu)
9f9e095a Use `genesis_tx` parameter in `generate_genesis_block`. #3261 (Jean Pierre Dudey)
|
|
|
|
|
|
928c1825 cryptonote_protocol: guard against all threads in standby (moneromooo-monero)
|
|
e3f0980a daemon: don't drop RPC with busy error when running offline (moneromooo-monero)
|
|
|
|
7ed62e63 cryptonote_protocol: fix std::move usage on different types (moneromooo-monero)
|
|
|
|
|
|
69f9a075 cryptonote_protocol: fix missing space in version mismatch message (moneromooo-monero)
|
|
f06603a4 cryptonote_protocol: update target height when receiving blocks (moneromooo-monero)
|
|
|
|
a2c845c7 cryptonote_protocol: internal error consistency size check (moneromooo-monero)
|
|
|
|
|
|
|
|
|
|
|
|
also avoid integer underflow on zero height
|
|
These even had the epee namespace.
This fixes some ugly circular dependencies.
|
|
As a followon side effect, this makes a lot of inline code
included only in particular cpp files (and instanciated
when necessary.
|
|
|
|
It's sent as JSON, so raw binary is not appropriate
|
|
It's nasty, and actually breaks on Solaris, where if.h fails to
build due to:
struct map *if_memmap;
|
|
|
|
monero/src/cryptonote_protocol/block_queue.cpp:208:44: error:
suggest braces around initialization of subobject [-Werror,-Wmissing-braces]
static const boost::uuids::uuid uuid0 = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0};
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
{ }
monero/src/wallet/wallet_rpc_server.cpp:1895:43: error:
lambda capture 'wal' is not used [-Werror,-Wunused-lambda-capture]
tools::signal_handler::install([&wrpc, &wal](int) {
^
monero/src/cryptonote_protocol/cryptonote_protocol_handler.inl:1616:40: error:
lambda capture 'arg' is not used [-Werror,-Wunused-lambda-capture]
m_p2p->for_each_connection([this, &arg, &fluffy_arg, &exclude_context, &fullConnections...
^
monero/src/cryptonote_protocol/cryptonote_protocol_handler.inl:1616:46: error:
lambda capture 'fluffy_arg' is not used [-Werror,-Wunused-lambda-capture]
m_p2p->for_each_connection([this, &arg, &fluffy_arg, &exclude_context, &fullConnections...
^
monero/src/blockchain_utilities/blockchain_export.cpp:181:3: error:
bool literal returned from 'main' [-Werror,-Wmain]
CHECK_AND_ASSERT_MES(r, false, "Failed to initialize source blockchain storage");
^ ~~~~~
monero/contrib/epee/include/misc_log_ex.h:180:97: note:
expanded from macro 'CHECK_AND_ASSERT_MES'
...fail_ret_val, message) do{if(!(expr)) {LOG_ERROR(message); return fail_ret_val;};}while(0)
^ ~~~~~~~~~~~~
monero/src/blockchain_utilities/blockchain_export.cpp:195:3: error:
bool literal returned from 'main' [-Werror,-Wmain]
CHECK_AND_ASSERT_MES(r, false, "Failed to export blockchain raw data");
^ ~~~~~
monero/contrib/epee/include/misc_log_ex.h:180:97: note:
expanded from macro 'CHECK_AND_ASSERT_MES'
...fail_ret_val, message) do{if(!(expr)) {LOG_ERROR(message); return fail_ret_val;};}while(0)
^ ~~~~~~~~~~~~
|
|
0a872798 protocol: kick idle peers by dropping them (moneromooo-monero)
|
|
7adceee6 precomputed block hashes are now in blocks of N (currently 256) (moneromooo-monero)
|
|
This is safer, as we don't risk break expectations (eg, requesting
block hashes and then receiving a late set of blocks). Dropping a
connection means another will be attempted in a fresh state.
Also bump the kick timeout to 5 minutes, to ensure we only kick
really idle peers.
|
|
|
|
3487d810 cryptonote_protocol: fix needless chain hashes downloads (moneromooo-monero)
|
|
This shaves a lot of space off binaries
|
|
The last known hash was calculated incorrectly, causing
further chain hash downloads to restart from the current
chain. When the block queue has close to 10k blocks waiting,
this causes frequent downloads of 10k more hashes, but
with only the last few hashes actually being useful.
|
|
It is unused, as it was apparently a future optimization,
and it leaks some information (though since pools publish
thei blocks they find, that amount seems small).
|
|
Defaults to off, but fluffy blocks are forced enabled on testnet
|
|
c867357a cryptonote_protocol: error handling on cleanup_handle_incoming_blocks (moneromooo-monero)
ce901fcb Fix blockchain_import wedge on exception in cleanup_handle_incoming_blocks (moneromooo-monero)
84fa015e core: guard against exceptions in handle_incoming_{block,tx} (moneromooo-monero)
|
|
|
|
fixes getting those in a loop
|
|
c22d22e2 Cleanup test impact of adding safesyncmode() method (Howard Chu)
9a859844 Toggle SAFE syncmode on and off automatically (Howard Chu)
|
|
|
|
This allows peers who synced past a fork on the wrong height
to reorg to the right chain after they updated their software
to include the new version.
|
|
|
|
|
|
|
|
|
|
Fix sync wedge corner case:
It could happen if a connection went into standby mode, while
it was the one which had requested the next span, and that span
was still waiting for the data, and that peer is not on the
main chain. Other peers can then start asking for that data
again and again, but never get it as only that forked peer does.
And various other fixes
|
|
If monerod is started with default sync mode, set it to SAFE after
synchronization completes. Set it back to FAST if synchronization
restarts (e.g. because another peer has a longer blockchain).
If monerod is started with an explicit sync mode, none of this
automation takes effect.
|
|
5a283078 cryptonote_protocol: large block sync size before v4 (moneromooo-monero)
7b747607 cryptonote_protocol: kick idle synchronizing peers (moneromooo-monero)
|
|
42b34b35 Consistently print peer id in hex and on 16 chars (moneromooo-monero)
|
|
b59cd074 befor -> before (Nano Akron)
|
|
|
|
In case they dropped off downloading for any reason, they'll get
sent to download again.
|
|
|
|
Really unique yet consistent spelling mistake
|
|
Not used yet.
|
|
a1891ebe tests: fix tests build (moneromooo-monero)
|
|
88e83f94 cryptonote_protocol_handler: log versions as unsigned ints (moneromooo-monero)
|
|
0e8d60c0 cryptonote_protocol: fix recv/send idle time before handshake (moneromooo-monero)
|
|
f90bbe2a cryptonote_protocol: keep target in sync with dropped connections (moneromooo-monero)
|
|
|
|
This was broken by the reorg fix, since we now have to add blocks
regardless of their starting height. We now check whether we know
the parent for the first block in the next span, or whether it was
requested. If neither, it's an orphan. If it is not known, but was
requested, we wait to get that block.
|
|
|
|
They're interpreted as characters otherwise
|
|
|
|
Add get_fork_version and add_ideal_fork_version to core so
cryptonote_protocol does not have to need the Blockchain
class directly, as it's not in its dependencies, and add
those to the fake core classes in tests too.
|
|
|
|
When a node is dropped, we stop considering its claimed blockchain
height as a factor in the target height calculation. This prevents
a runaway chain from being still thought to be the target even if
the nodes carrying it are dropped.
|
|
We won't even talk to a peer which claims a wrong version
for its top block. This will avoid syncing to known bad
peers in the first place.
Also add IP fails when failing to verify a block.
|
|
158c3ecf core: thread most of handle_incoming_tx (moneromooo-monero)
f57ee382 cryptonote_protocol: retry stale spans early (moneromooo-monero)
90df52e1 cryptonote_protocol: light cleanup (moneromooo-monero)
84e23156 cryptonote_protocol: avoid spurious SYNCHRONIZED OK messages (moneromooo-monero)
5be43fcd cryptonote_protocol_handler: sync speedup (moneromooo-monero)
|
|
1c9196b0 cryptonote_protocol: fix days behind calc on testnet (moneromooo-monero)
|
|
214fd81e some include cleanup (moneromooo-monero)
|
|
|
|
Connections can be dropped by the net_node layer,
unbeknownst to cryptonote_protocol, which would then
not flush any spans scheduled to that connection,
which would cause it to be only downloaded again
once it becomes the next span (possibly after a small
delay if it had been requested less than 5 seconds
ago).
|
|
|
|
|
|
A block queue is now placed between block download and
block processing. Blocks are now requested only from one
peer (unless starved).
Includes a new sync_info coommand.
|
|
|
|
|
|
|
|
Since I had to add an ID to the derived classes anyway,
this can be used instead. This removes an apparently
pointless warning from CLANG too.
|
|
6fc2dc39 cryptonote_protocol_handler: fix crash in debug log (moneromooo-monero)
|
|
072102cf abstracted nework addresses (moneromooo-monero)
|
|
|
|
All code which was using ip and port now uses a new IPv4 object,
subclass of a new network_address class. This will allow easy
addition of I2P addresses later (and also IPv6, etc).
Both old style and new style peer lists are now sent in the P2P
protocol, which is inefficient but allows peers using both
codebases to talk to each other. This will be removed in the
future. No other subclasses than IPv4 exist yet.
|
|
Integration could go further (ie, return_tx_to_pool calls should
not be needed anymore, possibly other things).
poolstate.bin is now obsolete.
|
|
Found by smooth
|
|
|
|
|
|
|
|
424b76a4 Fix spurious synchronization status msgs (Howard Chu)
|
|
0288310e blockchain_db: add "raw" blobdata getters for block and transaction (moneromooo-monero)
|
|
Only update target height if it's actually greater than the current target.
Only display "synchronized" when current height equals target.
|
|
This speeds up operations such as serving blocks to syncing peers
|
|
- only pause mining once we've got the lock (in practice, it'll
already be paused by another thread if we can't get the lock
at once though)
- do not call prepare_handle_incoming_blocks when we dismissed
all the blocks, it only causes cleanup_handle_incoming_blocks
to complain afterwards
|
|
8bdc86be protocol: speed up sync by minimizing duplicate work (moneromooo-monero)
61dfa310 epee: fix some log macros not printing context nicely (moneromooo-monero)
|
|
In particular, the prepare_handle_incoming_blocks call
is pretty lengthy, and entirely pointless in the common
case where several different connections will prepare
the exact same blocks.
|
|
- fix wrong block being used when a new block is received between
a node elaying a fluffy block and sending a new fluffy block
with txes a peer did not have
- misc a neverending ping pong requesting the same missing txids
when a new block is received in the meantime, causing the top
block to not be the one we need
- send the original fluffy block message block height when sending
a new fluffy block, not the current top height, which might
have been updated since
- avoid sending back the whole block blob when asking for txes,
send only the hash instead
- plus misc cleanup and additional debugging logs
|
|
|
|
0644eed7 Remove boost/foreach.cpp includes (Miguel Herranz)
36dd3e23 Replace BOOST_REVERSE_FOREACH with ranged for (Miguel Herranz)
629e3101 Replace BOOST_FOREACH with C++11 ranged for (Miguel Herranz)
|
|
database before quitting
|
|
|
|
Makes it easier to log just what's going on on P2P
|
|
|
|
This replaces the epee and data_loggers logging systems with
a single one, and also adds filename:line and explicit severity
levels. Categories may be defined, and logging severity set
by category (or set of categories). epee style 0-4 log level
maps to a sensible severity configuration. Log files now also
rotate when reaching 100 MB.
To select which logs to output, use the MONERO_LOGS environment
variable, with a comma separated list of categories (globs are
supported), with their requested severity level after a colon.
If a log matches more than one such setting, the last one in
the configuration string applies. A few examples:
This one is (mostly) silent, only outputting fatal errors:
MONERO_LOGS=*:FATAL
This one is very verbose:
MONERO_LOGS=*:TRACE
This one is totally silent (logwise):
MONERO_LOGS=""
This one outputs all errors and warnings, except for the
"verify" category, which prints just fatal errors (the verify
category is used for logs about incoming transactions and
blocks, and it is expected that some/many will fail to verify,
hence we don't want the spam):
MONERO_LOGS=*:WARNING,verify:FATAL
Log levels are, in decreasing order of priority:
FATAL, ERROR, WARNING, INFO, DEBUG, TRACE
Subcategories may be added using prefixes and globs. This
example will output net.p2p logs at the TRACE level, but all
other net* logs only at INFO:
MONERO_LOGS=*:ERROR,net*:INFO,net.p2p:TRACE
Logs which are intended for the user (which Monero was using
a lot through epee, but really isn't a nice way to go things)
should use the "global" category. There are a few helper macros
for using this category, eg: MGINFO("this shows up by default")
or MGINFO_RED("this is red"), to try to keep a similar look
and feel for now.
Existing epee log macros still exist, and map to the new log
levels, but since they're used as a "user facing" UI element
as much as a logging system, they often don't map well to log
severities (ie, a log level 0 log may be an error, or may be
something we want the user to see, such as an important info).
In those cases, I tried to use the new macros. In other cases,
I left the existing macros in. When modifying logs, it is
probably best to switch to the new macros with explicit levels.
The --log-level options and set_log commands now also accept
category settings, in addition to the epee style log levels.
|
|
Also print its value when printing pool
|
|
|
|
Per discussion of #1359, the wording has changed to indicate that no assessment of the blocks validity is made.
|
|
3f7d6fb5 Fix delayed exit when syncing (moneromooo-monero)
|
|
|
|
|
|
d61bd81 add lightweight block propagation ("fluffy blocks") (Dion Ahmetaj)
|
|
Added a new command to the P2P protocol definitions to allow querying for support flags.
Implemented handling of new support flags command in net_node. Changed for_each callback template to include support flags. Updated print_connections command to show peer support flags.
Added p2p constant for signaling fluffy block support.
Added get_pool_transaction function to cryptnote_core.
Added new commands to cryptonote protocol for relaying fluffy blocks.
Implemented handling of fluffy block command in cryptonote protocol.
Enabled fluffy block support in node initial configuration.
Implemented get_testnet function in cryptonote_core.
Made it so that fluffy blocks only run on testnet.
|
|
This will be when we can't find common ground between the peer's
short chain history and our blockchain.
This fixes bad peers claiming a higher blockchain height from never
dropped, and keeping the node in synchronizing state forever, since
we will never get blocks from that peer.
|
|
|
|
cec090f Revert "Fix fake 'network synchronized, begin using' messages" (moneromooo-monero)
|
|
|
|
This reverts commit 78035d2b6c9922f4cd730df0766aa74f4854ccb2.
The patch doesn't work, and causes constant SYNCHRONIZED OK spam.
|
|
|
|
|
|
c2941cb Show correct amount of days behind/ahead while syncing (tobiasw2)
|
|
|
|
|
|
It's logging the blockchain height, not the top block height
|
|
Some of it uses hardcoded height, which will need some thinking
for next (voted upon) fork.
|
|
|
|
Remove trailing whitespace in same files.
|
|
|
|
|
|
With minor cleanup and fixes (spelling, indent) by moneromooo
|
|
The last relayed time of a transaction is maintained, and
transactions will be relayed again if they are still in the
pool after a certain amount of time, which increases with
the transaction's age. All such transactions are resent,
whether or not they originated on the local node.
|
|
|
|
Bockchain:
1. Optim: Multi-thread long-hash computation when encountering groups of blocks.
2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible.
3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible.
4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks.
5. Optim: Multi-thread signature computation whenever possible.
6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD)
7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???).
8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads).
Berkeley-DB:
1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc).
2. Fix: Unable to pop blocks on reorganize due to transaction errors.
3. Patch: Large number of transaction aborts when running multi-threaded bulk queries.
4. Patch: Insufficient locks error when running full sync.
5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation.
6. Optim: Add bulk queries to get output global indices.
7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3)
8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key
9. Optim: Added thread-safe buffers used when multi-threading bulk queries.
10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details)
11. Mod: Added checkpoint thread and auto-remove-logs option.
12. *Now usable on 32-bit systems like RPI2.
LMDB:
1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect)
2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3)
3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key
4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details)
5. Mod: Auto resize to +1GB instead of multiplier x1.5
ETC:
1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete.
2. Fix: 32-bit saturation bug when computing next difficulty on large blocks.
[PENDING ISSUES]
1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization.
This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD.
2. Berkeley db, possible bug "unable to allocate memory". TBD.
[NEW OPTIONS] (*Currently all enabled for testing purposes)
1. --fast-block-sync arg=[0:1] (default: 1)
a. 0 = Compute long hash per block (may take a while depending on CPU)
b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence)
2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000)
a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions.
b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache.
Fast - Write meta-data but defer data flush.
Fastest - Defer meta-data and data flush.
Sync - Flush data after nblocks_per_sync and wait.
Async - Flush data after nblocks_per_sync but do not wait for the operation to finish.
3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower)
Max number of threads to use when computing long-hash in groups.
4. --show-time-stats arg=[0:1] (default: 1)
Show benchmark related time stats.
5. --db-auto-remove-logs arg=[0:1] (default: 1)
For berkeley-db only. Auto remove logs if enabled.
**Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version.
At the moment, you need a full resync to use this optimized version.
[PERFORMANCE COMPARISON]
**Some figures are approximations only.
Using a baseline machine of an i7-2600K+SSD+(with full pow computation):
1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain.
2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain.
3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain.
Averate procesing times (with full pow computation):
lmdb-optimized:
1. tx_ave = 2.5 ms / tx
2. block_ave = 5.87 ms / block
memory-official-repo:
1. tx_ave = 8.85 ms / tx
2. block_ave = 19.68 ms / block
lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e)
1. tx_ave = 47.8 ms / tx
2. block_ave = 64.2 ms / block
**Note: The following data denotes processing times only (does not include p2p download time)
lmdb-optimized processing times (with full pow computation):
1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000).
2. Laptop, Dual-core / 4-threads U4200 (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000).
3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000).
lmdb-optimized processing times (with per-block-checkpoint)
1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000).
berkeley-db optimized processing times (with full pow computation)
1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000).
2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
berkeley-db optimized processing times (with per-block-checkpoint)
1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
|
|
|
|
new update of the pr with network limits
more debug options:
discarding downloaded blocks all or after given height.
trying to trigger the locking errors.
debug levels polished/tuned to sane values.
debug/logging improved.
warning: this pr should be correct code, but it could make
an existing (in master version) locking error appear more often.
it's a race on the list (map) of peers, e.g. between closing/deleting
them versus working on them in net-limit sleep in sending chunk.
the bug is not in this code/this pr, but in the master version.
the locking problem of master will be fixed in other pr.
problem is ub, and in practice is seems to usually cause program abort
(tested on debian stable with updated gcc). see --help for option
to add sleep to trigger the error faster.
|
|
Update of the PR with network limits
works very well for all speeds
(but remember that low download speed can stop upload
because we then slow down downloading of blockchain
requests too)
more debug options
fixed pedantic warnings in our code
should work again on Mac OS X and FreeBSD
fixed warning about size_t
tested on Debian, Ubuntu, Windows(testing now)
TCP options and ToS (QoS) flag
FIXED peer number limit
FIXED some spikes in ingress/download
FIXED problems when other up and down limit
|
|
commands and options for network limiting
works very well e.g. for 50 KiB/sec up and down
ToS (QoS) flag
peer number limit
TODO some spikes in ingress/download
TODO problems when other up and down limit
added "otshell utils" - simple logging (with colors, text files channels)
|
|
|
