Merge pull request #256

7d8acd1 explicitly disable AES on ARM (Riccardo Spagni)
4222b37 add arm6 target and flags (Riccardo Spagni)
5675325 new makefile targets for static builds (Riccardo Spagni)
9519526 Only compile BerkeleyDB as an option in non-static (Thomas Winget)
7b14d4a Steps toward multiple dbs available -- working (Thomas Winget)
dbdcf11 blockchain_converter: Add support for resume from last block (warptangent)
4880803 blockchain_import: lengthen string for line clear (warptangent)
7476d2e blockchain_export: show progress during export (warptangent)
4bedd68 Update Blockchain::get_db() to return reference instead of pointer (warptangent)
0386e99 Add README for blockchain converter, importer, and exporter utilities (warptangent)
ead7fad BerkeleyDB implementation of BlockchainDB seems to be working! (Thomas Winget)
ffadb65 blockchain_export: Add compile-time support for BlockchainDB (warptangent)
f6cbfb6 Add blockchain_export utility (warptangent)
275cbd4 Add support for database open with flags (warptangent)
cb862cb Add mdb_flags variable to LMDB database open (warptangent)
260cc56 Add blockchain_import utility (warptangent)
ca75b47 Blockchain: add get_db() accessor, needed for blockchain_import (warptangent)
a3dd9d1 blockchain_converter: Add support for batch transactions (warptangent)
acb5d29 Update and relocate comment that applies class wide (warptangent)
bd9da6d Merges #33 (Thomas Winget)
6e9e8ab Move db_drivers/ to external/ (Thomas Winget)
56a1bf5 fixed BUILD_64 option (Riccardo Spagni)
c97a685 fixed arch_width option (Riccardo Spagni)
3a3c07c fixed msys2 / mingw folders based on architecture, added license to unbound CMakeList as that is not part of standard Unbound (Riccardo Spagni)
3b3da86 moved 32-bit/64-bit detection into main CMakeLists (Riccardo Spagni)
a0cbf73 removed findlmdb.cmake, no longer required (Riccardo Spagni)
0e8bbdb 32-bit LMDB, removed check for in-system LMDB as it is consensus-critical (Riccardo Spagni)
11e8157 Revert "Moved db_drivers/ into external/ for consistency" (Thomas Winget)
daf2a8f Revert "Build fixed, goofed up some CMake" (Thomas Winget)
e146027 BlockchainBDB passes unit tests (Thomas Winget)
43477b7 BerkeleyDB Blockchain building, not working yet (Thomas Winget)
cade0da CMake wiring, minor cleanup, minor test addition (Thomas Winget)
1bc8939 BerkeleyDB BlockchainDB impl copy/paste/modify (Thomas Winget)
5112dc3 Try to not pollute cryptonote namespace (Thomas Winget)
edef0bb Initial commit of BDB BlockchainDB implementation (Thomas Winget)
eee3ee7 BlockchainDB implementations have names now (Thomas Winget)
5d2a2b7 Fixed includes in BlockchainDB unit tests (Thomas Winget)
8b82f3c Build fixed, goofed up some CMake (Thomas Winget)
b213356 Moved db_drivers/ into external/ for consistency (Thomas Winget)
5eab480 Moved BlockchainDB into its own src/ subfolder (Thomas Winget)
0ad0784 Changed log level of debug message -- too spammy (Thomas Winget)
6485dac BlockchainLMDB: Add profiling to tx_exists() (warptangent)
83fb6d8 BlockchainLMDB: Add batch transaction support to tx_exists() (warptangent)
8529c0e BlockchainDB, BlockchainLMDB: Add profiling for DB commits (warptangent)
7a66b8b BlockchainDB: Add virtual function declarations for batch transactions (warptangent)
b7a2d84 BlockchainLMDB: Add check for open database to two functions (warptangent)
58ecc58 BlockchainLMDB: Add support for batch transactions (warptangent)
8909d7d Improve block and tx processing efficiency by less repeat hashing (warptangent)
3676ac5 Add profiling to block and tx processing (warptangent)
ce71abd Move LMDB storage to subfolder (warptangent)
42f8fe5 Fix formatting (warptangent)
aa82f78 Fix log statement (warptangent)
26873db Remove unused variable (warptangent)
4b90fd3 Add log statement (warptangent)
2531aa3 Add and extend log statements (warptangent)
59305d3 Blockchain: match original function declaration from blockchain_storage (warptangent)
cd972bd Update year and formatting in license (warptangent)
b88ab64 Fix Blockchain::get_tail_id() to set parameter to last block number instead of height (warptangent)
963bc09 Revert "Bounds error, should fix #27" (warptangent)
6f1c4b4 Bounds error, should fix #27 (Thomas Winget)
84fe5fb Add compile-time support for both db implementations: in-memory and LMDB (warptangent)
8bd1983 Blockchain: reflect log updates from blockchain_storage (warptangent)
7f9b070 Blockchain: reflect log and assert updates from blockchain_storage (warptangent)
70342ec Blockchain: reflect log level of blockchain_storage (warptangent)
c8d27fb Blockchain: reflect assert behavior of blockchain_storage for get_tx_outputs_gindexs() (warptangent)
d00ee78 Update recently added log statement to fix possible null dereference (warptangent)
acd4c36 Should fix std::min issues related to size_t (Thomas Winget)
3488359 Add in-source lmdb to build process (Thomas Winget)
4e45fae Initial import of lmdb source (Thomas Winget)
800d9b9 Remove code previously made unused and marked unused (warptangent)
0840c2f Fix height assertion in Blockchain::handle_alternative_block() (warptangent)
63051be Fix comparison between main and alternate chain's cumulative difficulty. (warptangent)
909ea81 Remove a have_block() check so alternate block can be processed (warptangent)
4d0a94b Complete implementation of transaction removal (warptangent)
1701c26 Use block index when obtaining block's difficulty for log statement (warptangent)
4eba21f Fix transfers to support mixins (warptangent)
d045dfa Fix transfers (without mixins) (warptangent)
429a740 throw inline functions need to keep exception type (Thomas Winget)
14555ee Fixes segfault in Blockchain::handle_alternative_block (Thomas Winget)
5086ca1 add BlockchainDB tests to new cmake (Thomas Winget)
c5c100c Obtain tx hash and tx output index from amount and output offset (warptangent)
ad8200a db_lmdb: fix global index calculation off by 1 (moneromooo-monero)
57b80c5 db_lmdb: remove redundant checks (moneromooo-monero)
1362846 blockchain_converter: add --testnet for converting testnet blockchain (moneromooo-monero)
c50cd95 Fixes a bug with getting output metadata from BlockchainDB (Thomas Winget)
c3fa07b update comments to reflect changed code (Thomas Winget)
59d2b0e db_lmdb: do not give the group database write permissions (moneromooo-monero)
4c2a452 db_lmdb: catch attempt to remove block from an empty blockchain (moneromooo-monero)
3a3459d db_lmdb: factor all the log+throw code paths (moneromooo-monero)
3fcb8da db_lmdb: factor the MDB_val setup code (moneromooo-monero)
609cf7f blockchain_converter: a bit more user friendly output (moneromooo-monero)
2b9f737 blockchain_converter: only call data path function once (moneromooo-monero)
1860658 blockchain: do not append "testnet" to the data directory (moneromooo-monero)
1c578ad db_lmdb: remove block timestamp too when removing a block (moneromooo-monero)
c93a186 db_lmdb: do not cast const away (moneromooo-monero)
198368b blockchain: fix wallet syncing from scratch (moneromooo-monero)
1d23db2 db_lmdb: do not keep a dangling pointer to stack objects (moneromooo-monero)
29b5876 db_lmdb: make cursor internal members private (moneromooo-monero)
a3157d7 blockchain_storage: refactor genesis block creation (moneromooo-monero)
8e41b1e blockchain_storage: add consts where appropriate (moneromooo-monero)
98bdadc blockchain_converter: delete blockchain on succesful exit (moneromooo-monero)
256162f checkpoints: add consts where appropriate (moneromooo-monero)
b7270ab blockchain: add consts where appropriate (moneromooo-monero)
23f3cb4 blockchain_db: add consts where appropriate (moneromooo-monero)
10fd6ca blockchain_db: factor some exception code (moneromooo-monero)
11129b9 blockchain_converter: use the actual blockchain location (moneromooo-monero)
0886183 build: add liblmdb to the cmake autodetection system (moneromooo-monero)
215e63b extraneous semicolon in Blockchain::complete_timestamps_vector (Thomas Winget)
8e1b7e2 raised maximum mapsize for lmdb to ~16GB (Thomas Winget)
6c8b8ac more blockchain height-related fixes, syncing other nodes code this time (Thomas Winget)
9455e0c ~ didn't work, need hard path.  debug print. (Thomas Winget)
4af0918 very, VERY primitive blockchain converter (Thomas Winget)
26a7db3 add new checkpointing behavior to Blockchain class (Thomas Winget)
767aac2 Remove unused dependency (Thomas Winget)
006e106 Store output pubkeys separately, bug fixes (Thomas Winget)
ab7951d Minor bugfixes, redundancy removal (Thomas Winget)
71b18d7 moar bug fixes, removed debug prints (Thomas Winget)
0915913 BlockchainLMDB seems to be working*! (Thomas Winget)
1a546e3 some bug fixes, but still needs work (Thomas Winget)
006afe2 Minor bug fixes and debug prints (Thomas Winget)
90f402e minor fixes to Blockchain.cpp (Thomas Winget)
74a1a89 Integrate BlockchainDB into cryptonote_core (Thomas Winget)
d8c570b All LMDB BlockchainDB implemented, not tested (Thomas Winget)
e47e343 LMDB blockchain: remove outputs and spent keys (Thomas Winget)
a0af217 Adding block data to LMDB BlockchainDB coded (Thomas Winget)
db00ce0 Parts of LMDB impl of BlockchainDB done and working (Thomas Winget)
1240cf8 BlockchainDB unit tests, lmdb linker flag (Thomas Winget)
b98b964 Initial commit of lmdb BlockchainDB impl (Thomas Winget)
bc44bc1 Initial commit of BlockchainDB tests, other misc (Thomas Winget)
90d6f8b Adding libglim as an external library (Thomas Winget)
07733f9 update new blockchain to build with new changes (Thomas Winget)
1ffbeb2 stupid past me, fixing typos and shit... (Thomas Winget)
67515b8 missing typedef (Thomas Winget)
aba548c import of BlockchainDB files (Thomas Winget)

1 files changed, 229 insertions, 0 deletions

diff --git a/src/cryptonote_core/blockchain.h b/src/cryptonote_core/blockchain.h
new file mode 100644
index 000000000..bc13901d2
--- /dev/null
+++ b/src/cryptonote_core/blockchain.h
@@ -0,0 +1,229 @@
+// Copyright (c) 2014, The Monero Project
+// 
+// All rights reserved.
+// 
+// Redistribution and use in source and binary forms, with or without modification, are
+// permitted provided that the following conditions are met:
+// 
+// 1. Redistributions of source code must retain the above copyright notice, this list of
+//    conditions and the following disclaimer.
+// 
+// 2. Redistributions in binary form must reproduce the above copyright notice, this list
+//    of conditions and the following disclaimer in the documentation and/or other
+//    materials provided with the distribution.
+// 
+// 3. Neither the name of the copyright holder nor the names of its contributors may be
+//    used to endorse or promote products derived from this software without specific
+//    prior written permission.
+// 
+// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY
+// EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
+// MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
+// THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+// PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
+// STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
+// THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+// 
+// Parts of this file are originally copyright (c) 2012-2013 The Cryptonote developers
+
+#pragma once
+#include <boost/serialization/serialization.hpp>
+#include <boost/serialization/version.hpp>
+#include <boost/serialization/list.hpp>
+#include <boost/multi_index_container.hpp>
+#include <boost/multi_index/global_fun.hpp>
+#include <boost/multi_index/hashed_index.hpp>
+#include <boost/multi_index/member.hpp>
+#include <boost/foreach.hpp>
+#include <atomic>
+
+#include "syncobj.h"
+#include "string_tools.h"
+#include "cryptonote_basic.h"
+#include "common/util.h"
+#include "cryptonote_protocol/cryptonote_protocol_defs.h"
+#include "rpc/core_rpc_server_commands_defs.h"
+#include "difficulty.h"
+#include "cryptonote_core/cryptonote_format_utils.h"
+#include "verification_context.h"
+#include "crypto/hash.h"
+#include "checkpoints.h"
+#include "blockchain_db/blockchain_db.h"
+
+namespace cryptonote
+{
+  class tx_memory_pool;
+
+  /************************************************************************/
+  /*                                                                      */
+  /************************************************************************/
+  class Blockchain
+  {
+  public:
+    struct transaction_chain_entry
+    {
+      transaction tx;
+      uint64_t m_keeper_block_height;
+      size_t m_blob_size;
+      std::vector<uint64_t> m_global_output_indexes;
+    };
+
+    struct block_extended_info
+    {
+      block   bl;
+      uint64_t height;
+      size_t block_cumulative_size;
+      difficulty_type cumulative_difficulty;
+      uint64_t already_generated_coins;
+    };
+
+    Blockchain(tx_memory_pool& tx_pool);
+
+    bool init(BlockchainDB* db, const bool testnet = false);
+    bool deinit();
+
+    void set_checkpoints(checkpoints&& chk_pts) { m_checkpoints = chk_pts; }
+
+    //bool push_new_block();
+    bool get_blocks(uint64_t start_offset, size_t count, std::list<block>& blocks, std::list<transaction>& txs) const;
+    bool get_blocks(uint64_t start_offset, size_t count, std::list<block>& blocks) const;
+    bool get_alternative_blocks(std::list<block>& blocks) const;
+    size_t get_alternative_blocks_count() const;
+    crypto::hash get_block_id_by_height(uint64_t height) const;
+    bool get_block_by_hash(const crypto::hash &h, block &blk) const;
+    void get_all_known_block_ids(std::list<crypto::hash> &main, std::list<crypto::hash> &alt, std::list<crypto::hash> &invalid) const;
+
+    template<class archive_t>
+    void serialize(archive_t & ar, const unsigned int version);
+
+    bool have_tx(const crypto::hash &id) const;
+    bool have_tx_keyimges_as_spent(const transaction &tx) const;
+    bool have_tx_keyimg_as_spent(const crypto::key_image &key_im) const;
+
+    template<class visitor_t>
+    bool scan_outputkeys_for_indexes(const txin_to_key& tx_in_to_key, visitor_t& vis, uint64_t* pmax_related_block_height = NULL) const;
+
+    uint64_t get_current_blockchain_height() const;
+    crypto::hash get_tail_id() const;
+    crypto::hash get_tail_id(uint64_t& height) const;
+    difficulty_type get_difficulty_for_next_block() const;
+    bool add_new_block(const block& bl_, block_verification_context& bvc);
+    bool reset_and_set_genesis_block(const block& b);
+    bool create_block_template(block& b, const account_public_address& miner_address, difficulty_type& di, uint64_t& height, const blobdata& ex_nonce) const;
+    bool have_block(const crypto::hash& id) const;
+    size_t get_total_transactions() const;
+    bool get_short_chain_history(std::list<crypto::hash>& ids) const;
+    bool find_blockchain_supplement(const std::list<crypto::hash>& qblock_ids, NOTIFY_RESPONSE_CHAIN_ENTRY::request& resp) const;
+    bool find_blockchain_supplement(const std::list<crypto::hash>& qblock_ids, uint64_t& starter_offset) const;
+    bool find_blockchain_supplement(const uint64_t req_start_block, const std::list<crypto::hash>& qblock_ids, std::list<std::pair<block, std::list<transaction> > >& blocks, uint64_t& total_height, uint64_t& start_height, size_t max_count) const;
+    bool handle_get_objects(NOTIFY_REQUEST_GET_OBJECTS::request& arg, NOTIFY_RESPONSE_GET_OBJECTS::request& rsp);
+    bool handle_get_objects(const COMMAND_RPC_GET_RANDOM_OUTPUTS_FOR_AMOUNTS::request& req, COMMAND_RPC_GET_RANDOM_OUTPUTS_FOR_AMOUNTS::response& res);
+    bool get_random_outs_for_amounts(const COMMAND_RPC_GET_RANDOM_OUTPUTS_FOR_AMOUNTS::request& req, COMMAND_RPC_GET_RANDOM_OUTPUTS_FOR_AMOUNTS::response& res) const;
+    bool get_tx_outputs_gindexs(const crypto::hash& tx_id, std::vector<uint64_t>& indexs) const;
+    bool store_blockchain();
+    bool check_tx_input(const txin_to_key& txin, const crypto::hash& tx_prefix_hash, const std::vector<crypto::signature>& sig, uint64_t* pmax_related_block_height = NULL) const;
+    bool check_tx_inputs(const transaction& tx, uint64_t* pmax_used_block_height = NULL) const;
+    bool check_tx_inputs(const transaction& tx, uint64_t& pmax_used_block_height, crypto::hash& max_used_block_id) const;
+    uint64_t get_current_cumulative_blocksize_limit() const;
+    bool is_storing_blockchain()const{return m_is_blockchain_storing;}
+    uint64_t block_difficulty(uint64_t i) const;
+
+    template<class t_ids_container, class t_blocks_container, class t_missed_container>
+    bool get_blocks(const t_ids_container& block_ids, t_blocks_container& blocks, t_missed_container& missed_bs) const;
+
+    template<class t_ids_container, class t_tx_container, class t_missed_container>
+    bool get_transactions(const t_ids_container& txs_ids, t_tx_container& txs, t_missed_container& missed_txs) const;
+
+    //debug functions
+    void print_blockchain(uint64_t start_index, uint64_t end_index);
+    void print_blockchain_index();
+    void print_blockchain_outs(const std::string& file);
+
+    void check_against_checkpoints(const checkpoints& points, bool enforce);
+    void set_enforce_dns_checkpoints(bool enforce);
+    bool update_checkpoints(const std::string& file_path, bool check_dns);
+
+    BlockchainDB& get_db()
+    {
+      return *m_db;
+    }
+
+  private:
+    typedef std::unordered_map<crypto::hash, size_t> blocks_by_id_index;
+    typedef std::unordered_map<crypto::hash, transaction_chain_entry> transactions_container;
+    typedef std::unordered_set<crypto::key_image> key_images_container;
+    typedef std::vector<block_extended_info> blocks_container;
+    typedef std::unordered_map<crypto::hash, block_extended_info> blocks_ext_by_hash;
+    typedef std::unordered_map<crypto::hash, block> blocks_by_hash;
+    typedef std::map<uint64_t, std::vector<std::pair<crypto::hash, size_t>>> outputs_container; //crypto::hash - tx hash, size_t - index of out in transaction
+
+    BlockchainDB* m_db;
+
+    tx_memory_pool& m_tx_pool;
+    mutable epee::critical_section m_blockchain_lock; // TODO: add here reader/writer lock
+
+    // main chain
+    blocks_container m_blocks;               // height  -> block_extended_info
+    blocks_by_id_index m_blocks_index;       // crypto::hash -> height
+    transactions_container m_transactions;
+    key_images_container m_spent_keys;
+    size_t m_current_block_cumul_sz_limit;
+
+
+    // all alternative chains
+    blocks_ext_by_hash m_alternative_chains; // crypto::hash -> block_extended_info
+
+    // some invalid blocks
+    blocks_ext_by_hash m_invalid_blocks;     // crypto::hash -> block_extended_info
+    outputs_container m_outputs;
+
+
+    checkpoints m_checkpoints;
+    std::atomic<bool> m_is_in_checkpoint_zone;
+    std::atomic<bool> m_is_blockchain_storing;
+    bool m_enforce_dns_checkpoints;
+
+    bool switch_to_alternative_blockchain(std::list<blocks_ext_by_hash::iterator>& alt_chain, bool discard_disconnected_chain);
+    block pop_block_from_blockchain();
+    bool purge_transaction_from_blockchain(const crypto::hash& tx_id);
+    bool purge_transaction_keyimages_from_blockchain(const transaction& tx, bool strict_check);
+
+    bool handle_block_to_main_chain(const block& bl, block_verification_context& bvc);
+    bool handle_block_to_main_chain(const block& bl, const crypto::hash& id, block_verification_context& bvc);
+    bool handle_alternative_block(const block& b, const crypto::hash& id, block_verification_context& bvc);
+    difficulty_type get_next_difficulty_for_alternative_chain(const std::list<blocks_ext_by_hash::iterator>& alt_chain, block_extended_info& bei) const;
+    bool prevalidate_miner_transaction(const block& b, uint64_t height);
+    bool validate_miner_transaction(const block& b, size_t cumulative_block_size, uint64_t fee, uint64_t& base_reward, uint64_t already_generated_coins);
+    bool validate_transaction(const block& b, uint64_t height, const transaction& tx);
+    bool rollback_blockchain_switching(std::list<block>& original_chain, uint64_t rollback_height);
+    bool add_transaction_from_block(const transaction& tx, const crypto::hash& tx_id, const crypto::hash& bl_id, uint64_t bl_height);
+    bool push_transaction_to_global_outs_index(const transaction& tx, const crypto::hash& tx_id, std::vector<uint64_t>& global_indexes);
+    bool pop_transaction_from_global_index(const transaction& tx, const crypto::hash& tx_id);
+    void get_last_n_blocks_sizes(std::vector<size_t>& sz, size_t count) const;
+    void add_out_to_get_random_outs(COMMAND_RPC_GET_RANDOM_OUTPUTS_FOR_AMOUNTS::outs_for_amount& result_outs, uint64_t amount, size_t i) const;
+    bool is_tx_spendtime_unlocked(uint64_t unlock_time) const;
+    bool add_block_as_invalid(const block& bl, const crypto::hash& h);
+    bool add_block_as_invalid(const block_extended_info& bei, const crypto::hash& h);
+    bool check_block_timestamp(const block& b) const;
+    bool check_block_timestamp(std::vector<uint64_t>& timestamps, const block& b) const;
+    uint64_t get_adjusted_time() const;
+    bool complete_timestamps_vector(uint64_t start_height, std::vector<uint64_t>& timestamps);
+    bool update_next_cumulative_size_limit();
+
+    bool check_for_double_spend(const transaction& tx, key_images_container& keys_this_block) const;
+  };
+
+
+  /************************************************************************/
+  /*                                                                      */
+  /************************************************************************/
+
+  #define CURRENT_BLOCKCHAIN_STORAGE_ARCHIVE_VER    12
+
+  //------------------------------------------------------------------
+
+}  // namespace cryptonote
+
+BOOST_CLASS_VERSION(cryptonote::Blockchain, CURRENT_BLOCKCHAIN_STORAGE_ARCHIVE_VER)