** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY)

Bockchain:
1. Optim: Multi-thread long-hash computation when encountering groups of blocks.
2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible.
3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible.
4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks.
5. Optim: Multi-thread signature computation whenever possible.
6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD)
7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???).
8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads).

Berkeley-DB:
1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc).
2. Fix: Unable to pop blocks on reorganize due to transaction errors.
3. Patch: Large number of transaction aborts when running multi-threaded bulk queries.
4. Patch: Insufficient locks error when running full sync.
5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation.
6. Optim: Add bulk queries to get output global indices.
7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3)
8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key
9. Optim: Added thread-safe buffers used when multi-threading bulk queries.
10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details)
11. Mod: Added checkpoint thread and auto-remove-logs option.
12. *Now usable on 32-bit systems like RPI2.

LMDB:
1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect)
2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3)
3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key
4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details)
5. Mod: Auto resize to +1GB instead of multiplier x1.5

ETC:
1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete.
2. Fix: 32-bit saturation bug when computing next difficulty on large blocks.

[PENDING ISSUES]
1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization.
   This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD.
2. Berkeley db, possible bug "unable to allocate memory". TBD.

[NEW OPTIONS] (*Currently all enabled for testing purposes)
1. --fast-block-sync arg=[0:1] (default: 1)
	a. 0 = Compute long hash per block (may take a while depending on CPU)
	b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence)
2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000)
	a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions.
	b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache.
	Fast    - Write meta-data but defer data flush.
	Fastest - Defer meta-data and data flush.
	Sync    - Flush data after nblocks_per_sync and wait.
	Async   - Flush data after nblocks_per_sync but do not wait for the operation to finish.
3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower)
        Max number of threads to use when computing long-hash in groups.
4. --show-time-stats arg=[0:1] (default: 1)
	Show benchmark related time stats.
5. --db-auto-remove-logs arg=[0:1] (default: 1)
	For berkeley-db only. Auto remove logs if enabled.

**Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version.
	At the moment, you need a full resync to use this optimized version.

[PERFORMANCE COMPARISON]
**Some figures are approximations only.
Using a baseline machine of an i7-2600K+SSD+(with full pow computation):
1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain.
2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain.
3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain.

Averate procesing times (with full pow computation):
lmdb-optimized:
1. tx_ave = 2.5 ms / tx
2. block_ave = 5.87 ms / block
memory-official-repo:
1. tx_ave = 8.85 ms / tx
2. block_ave = 19.68 ms / block
lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e)
1. tx_ave = 47.8 ms / tx
2. block_ave = 64.2 ms / block

**Note: The following data denotes processing times only (does not include p2p download time)
lmdb-optimized processing times (with full pow computation):
1. Desktop,  Quad-core / 8-threads 2600k  (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000).
2. Laptop,   Dual-core / 4-threads U4200  (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000).
3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000).

lmdb-optimized processing times (with per-block-checkpoint)
1. Desktop,  Quad-core / 8-threads 2600k  (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000).

berkeley-db optimized processing times (with full pow computation)
1. Desktop, Quad-core / 8-threads 2600k  (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000).
2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).

berkeley-db optimized processing times (with per-block-checkpoint)
1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).

1 files changed, 162 insertions, 10 deletions

diff --git a/src/blockchain_db/berkeleydb/db_bdb.h b/src/blockchain_db/berkeleydb/db_bdb.h
index fec4ed24f..41f4bcb78 100644
--- a/src/blockchain_db/berkeleydb/db_bdb.h
+++ b/src/blockchain_db/berkeleydb/db_bdb.h
@@ -30,6 +30,11 @@
 #include "blockchain_db/blockchain_db.h"
 #include "cryptonote_protocol/blobdatatype.h" // for type blobdata
 
+#include <unordered_map>
+
+// ND: Enables multi-threaded bulk reads for when getting indices.
+//     TODO: Disabled for now, as it doesn't seem to provide noticeable improvements (??. Reason: TBD.
+// #define BDB_BULK_CAN_THREAD
 namespace cryptonote
 {
 
@@ -83,10 +88,145 @@ struct bdb_txn_safe
   {
     return &m_txn;
   }
-
+private:
   DbTxn* m_txn;
 };
 
+// ND: Class to handle buffer management when doing bulk queries
+// (DB_MULTIPLE). Allocates buffers then handles thread queuing
+// so a fixed set of buffers can be used (instead of allocating
+// every time a bulk query is needed).
+template <typename T>
+class bdb_safe_buffer
+{
+	// limit the number of buffers to 8
+	const size_t MaxAllowedBuffers = 8;
+public:
+    bdb_safe_buffer(size_t num_buffers, size_t count)
+    {
+    	if(num_buffers > MaxAllowedBuffers)
+    		num_buffers = MaxAllowedBuffers;
+    	
+        set_count(num_buffers);
+        for (size_t i = 0; i < num_buffers; i++)
+            m_buffers.push_back((T) malloc(sizeof(T) * count));
+        m_buffer_count = count;
+    }
+
+    ~bdb_safe_buffer()
+    {
+        for (size_t i = 0; i < m_buffers.size(); i++)
+        {
+            if (m_buffers[i])
+            {
+                free(m_buffers[i]);
+                m_buffers[i] = nullptr;
+            }
+        }
+
+        m_buffers.resize(0);
+    }
+
+    T acquire_buffer()
+    {
+        std::unique_lock<std::mutex> lock(m_lock);
+        m_cv.wait(lock, [&]{ return m_count > 0; });
+
+        --m_count;
+        size_t index = -1;
+        for (size_t i = 0; i < m_open_slot.size(); i++)
+        {
+            if (m_open_slot[i])
+            {
+                m_open_slot[i] = false;
+                index = i;
+                break;
+            }
+        }
+
+        assert(index >= 0);
+
+        T buffer = m_buffers[index];
+        m_buffer_map.emplace(buffer, index);
+        return buffer;
+    }
+
+    void release_buffer(T buffer)
+    {
+        std::unique_lock<std::mutex> lock(m_lock);
+
+        assert(buffer != nullptr);
+        auto it = m_buffer_map.find(buffer);
+        if (it != m_buffer_map.end())
+        {
+            auto index = it->second;
+
+            assert(index < m_open_slot.size());
+            assert(m_open_slot[index] == false);
+            assert(m_count < m_open_slot.size());
+
+            ++m_count;
+            m_open_slot[index] = true;
+            m_buffer_map.erase(it);
+            m_cv.notify_one();
+        }
+    }
+
+    size_t get_buffer_size() const
+    {
+        return m_buffer_count * sizeof(T);
+    }
+
+    size_t get_buffer_count() const
+    {
+        return m_buffer_count;
+    }
+
+    typedef T type;
+
+private:
+    void set_count(size_t count)
+    {
+        assert(count > 0);
+        m_open_slot.resize(count, true);
+        m_count = count;
+    }
+
+    std::vector<T> m_buffers;
+    std::unordered_map<T, size_t> m_buffer_map;
+
+    std::condition_variable m_cv;
+    std::vector<bool> m_open_slot;
+    size_t m_count;
+    std::mutex m_lock;
+
+    size_t m_buffer_count;
+};
+
+template <typename T>
+class bdb_safe_buffer_autolock
+{
+public:
+    bdb_safe_buffer_autolock(T &safe_buffer, typename T::type &buffer) :
+        m_safe_buffer(safe_buffer), m_buffer(nullptr)
+    {
+        m_buffer = m_safe_buffer.acquire_buffer();
+        buffer = m_buffer;
+    }
+
+    ~bdb_safe_buffer_autolock()
+    {
+        if (m_buffer != nullptr)
+        {
+            m_safe_buffer.release_buffer(m_buffer);
+            m_buffer = nullptr;
+        }
+    }
+private:
+    T &m_safe_buffer;
+    typename T::type m_buffer;
+};
+
 class BlockchainBDB : public BlockchainDB
 {
 public:
@@ -159,8 +299,9 @@ public:
 
   virtual uint64_t get_num_outputs(const uint64_t& amount) const;
 
-  virtual crypto::public_key get_output_key(const uint64_t& amount, const uint64_t& index) const;
-
+  virtual output_data_t get_output_key(const uint64_t& amount, const uint64_t& index);
+  virtual output_data_t get_output_key(const uint64_t& global_index) const;
+  virtual void get_output_key(const uint64_t &amount, const std::vector<uint64_t> &offsets, std::vector<output_data_t> &outputs);
   virtual tx_out get_output(const crypto::hash& h, const uint64_t& index) const;
 
   /**
@@ -175,9 +316,11 @@ public:
   tx_out get_output(const uint64_t& index) const;
 
   virtual tx_out_index get_output_tx_and_index_from_global(const uint64_t& index) const;
+  virtual void get_output_tx_and_index_from_global(const std::vector<uint64_t> &global_indices,
+          std::vector<tx_out_index> &tx_out_indices) const;
 
-  virtual tx_out_index get_output_tx_and_index(const uint64_t& amount, const uint64_t& index) const;
-  virtual void get_output_tx_and_index(const uint64_t& amount, std::vector<uint64_t> &offsets, std::vector<tx_out_index> &indices) const;
+  virtual tx_out_index get_output_tx_and_index(const uint64_t& amount, const uint64_t& index);
+  virtual void get_output_tx_and_index(const uint64_t& amount, const std::vector<uint64_t> &offsets, std::vector<tx_out_index> &indices);
 
   virtual std::vector<uint64_t> get_tx_output_indices(const crypto::hash& h) const;
   virtual std::vector<uint64_t> get_tx_amount_output_indices(const crypto::hash& h) const;
@@ -198,7 +341,12 @@ public:
   virtual void batch_abort();
 
   virtual void pop_block(block& blk, std::vector<transaction>& txs);
-  virtual bool has_bulk_indices() const { return true; }
+
+#if defined(BDB_BULK_CAN_THREAD)
+  virtual bool can_thread_bulk_indices() const { return true; }
+#else
+  virtual bool can_thread_bulk_indices() const { return false; }
+#endif
 
 private:
   virtual void add_block( const block& blk
@@ -214,7 +362,7 @@ private:
 
   virtual void remove_transaction_data(const crypto::hash& tx_hash, const transaction& tx);
 
-  virtual void add_output(const crypto::hash& tx_hash, const tx_out& tx_output, const uint64_t& local_index);
+  virtual void add_output(const crypto::hash& tx_hash, const tx_out& tx_output, const uint64_t& local_index, const uint64_t unlock_time);
 
   virtual void remove_output(const tx_out& tx_output);
 
@@ -227,6 +375,7 @@ private:
 
   virtual void remove_spent_key(const crypto::key_image& k_image);
 
+  void get_output_global_indices(const uint64_t& amount, const std::vector<uint64_t> &offsets, std::vector<uint64_t> &global_indices);
   /**
    * @brief convert a tx output to a blob for storage
    *
@@ -253,10 +402,13 @@ private:
    *
    * @return the global index of the desired output
    */
-  uint64_t get_output_global_index(const uint64_t& amount, const uint64_t& index) const;
-
+  uint64_t get_output_global_index(const uint64_t& amount, const uint64_t& index);
+  void checkpoint_worker() const;
   void check_open() const;
-  void *m_buffer;
+  bool m_run_checkpoint;
+  std::unique_ptr<boost::thread> m_checkpoint_thread;
+  typedef bdb_safe_buffer<void *> bdb_safe_buffer_t;
+  bdb_safe_buffer_t m_buffer;
 
   DbEnv* m_env;