diff options
74 files changed, 1422 insertions, 842 deletions
@@ -31,7 +31,7 @@ Our researchers are available on IRC in [#monero-research-lab on Freenode](https ## Build -### IMPORTANT +### IMPORTANT These builds are of the master branch, which is used for active development and can be either unstable or incompatible with release software. Please compile release branches. @@ -106,7 +106,7 @@ If you want to help out, see [CONTRIBUTING](CONTRIBUTING.md) for a set of guidel ## Scheduled software upgrades Monero uses a fixed-schedule software upgrade (hard fork) mechanism to implement new features. This means that users of Monero (end users and service providers) should run current versions and upgrade their software on a regular schedule. Software upgrades occur during the months of April and October. The required software for these upgrades will be available prior to the scheduled date. Please check the repository prior to this date for the proper Monero software version. Below is the historical schedule and the projected schedule for the next upgrade. -Dates are provided in the format YYYY-MM-DD. +Dates are provided in the format YYYY-MM-DD. | Software upgrade block height | Date | Fork version | Minimum Monero version | Recommended Monero version | Details | @@ -125,7 +125,7 @@ X's indicate that these details have not been determined as of commit date. ## Release staging schedule and protocol -Approximately three months prior to a scheduled software upgrade, a branch from Master will be created with the new release version tag. Pull requests that address bugs should then be made to both Master and the new release branch. Pull requests that require extensive review and testing (generally, optimizations and new features) should *not* be made to the release branch. +Approximately three months prior to a scheduled software upgrade, a branch from Master will be created with the new release version tag. Pull requests that address bugs should then be made to both Master and the new release branch. Pull requests that require extensive review and testing (generally, optimizations and new features) should *not* be made to the release branch. ## Compiling Monero from source @@ -200,12 +200,12 @@ invokes cmake commands as needed. *Note*: If cmake can not find zmq.hpp file on OS X, installing `zmq.hpp` from https://github.com/zeromq/cppzmq to `/usr/local/include` should fix that error. - + *Note*: The instructions above will compile the most stable release of the Monero software. If you would like to use and test the most recent software, use ```git checkout master```. The master branch may contain updates that are - both unstable and incompatible with release software, though testing is always - encouraged. + both unstable and incompatible with release software, though testing is always + encouraged. * The resulting executables can be found in `build/release/bin` @@ -235,14 +235,14 @@ Dependencies need to be built with -fPIC. Static libraries usually aren't, so yo #### On the Raspberry Pi -Tested on a Raspberry Pi Zero with a clean install of minimal Raspbian Stretch (2017-09-07 or later) from https://www.raspberrypi.org/downloads/raspbian/. If you are using Raspian Jessie, [please see note in the following section](#note-for-raspbian-jessie-users). +Tested on a Raspberry Pi Zero with a clean install of minimal Raspbian Stretch (2017-09-07 or later) from https://www.raspberrypi.org/downloads/raspbian/. If you are using Raspian Jessie, [please see note in the following section](#note-for-raspbian-jessie-users). * `apt-get update && apt-get upgrade` to install all of the latest software * Install the dependencies for Monero from the 'Debian' column in the table above. * Increase the system swap size: -``` +``` sudo /etc/init.d/dphys-swapfile stop sudo nano /etc/dphys-swapfile CONF_SWAPSIZE=2048 @@ -276,7 +276,7 @@ If you are using the older Raspbian Jessie image, compiling Monero is a bit more * As before, `apt-get update && apt-get upgrade` to install all of the latest software, and increase the system swap size -``` +``` sudo /etc/init.d/dphys-swapfile stop sudo nano /etc/dphys-swapfile CONF_SWAPSIZE=2048 @@ -316,13 +316,13 @@ application. * Open the MSYS shell via the `MSYS2 Shell` shortcut * Update packages using pacman: - pacman -Syuu + pacman -Syu * Exit the MSYS shell using Alt+F4 * Edit the properties for the `MSYS2 Shell` shortcut changing "msys2_shell.bat" to "msys2_shell.cmd -mingw64" for 64-bit builds or "msys2_shell.cmd -mingw32" for 32-bit builds * Restart MSYS shell via modified shortcut and update packages again using pacman: - pacman -Syuu + pacman -Syu * Install dependencies: @@ -332,7 +332,7 @@ application. pacman -S mingw-w64-x86_64-toolchain make mingw-w64-x86_64-cmake mingw-w64-x86_64-boost mingw-w64-x86_64-openssl mingw-w64-x86_64-zeromq mingw-w64-x86_64-libsodium mingw-w64-x86_64-hidapi To build for 32-bit Windows: - + pacman -S mingw-w64-i686-toolchain make mingw-w64-i686-cmake mingw-w64-i686-boost mingw-w64-i686-openssl mingw-w64-i686-zeromq mingw-w64-i686-libsodium mingw-w64-i686-hidapi * Open the MingW shell via `MinGW-w64-Win64 Shell` shortcut on 64-bit Windows @@ -348,11 +348,11 @@ application. **Building** * Change to the cloned directory, run: - + cd monero * If you would like a specific [version/tag](https://github.com/monero-project/monero/tags), do a git checkout for that version. eg. 'v0.13.0.0'. If you dont care about the version and just want binaries from master, skip this step: - + git checkout v0.13.0.4 * If you are on a 64-bit system, run: @@ -368,7 +368,7 @@ application. * **Optional**: to build Windows binaries suitable for debugging on a 64-bit system, run: make debug-static-win64 - + * **Optional**: to build Windows binaries suitable for debugging on a 32-bit system, run: make debug-static-win32 @@ -417,7 +417,7 @@ mkdir ~/boost cd ~/boost # Fetch boost source -ftp -o boost_1_64_0.tar.bz2 https://netcologne.dl.sourceforge.net/project/boost/boost/1.64.0/boost_1_64_0.tar.bz2 +ftp -o boost_1_64_0.tar.bz2 https://netcologne.dl.sourceforge.net/project/boost/boost/1.64.0/boost_1_64_0.tar.bz2 # MUST output: (SHA256) boost_1_64_0.tar.bz2: OK echo "7bcc5caace97baa948931d712ea5f37038dbb1c5d89b43ad4def4ed7cb683332 boost_1_64_0.tar.bz2" | sha256 -c @@ -651,7 +651,7 @@ Run the build. Once it stalls, enter the following command: ``` -gdb /path/to/monerod `pidof monerod` +gdb /path/to/monerod `pidof monerod` ``` Type `thread apply all bt` within gdb in order to obtain the stack trace diff --git a/contrib/depends/packages/boost.mk b/contrib/depends/packages/boost.mk index 89db5a415..e60a1c677 100644 --- a/contrib/depends/packages/boost.mk +++ b/contrib/depends/packages/boost.mk @@ -20,7 +20,7 @@ $(package)_archiver_$(host_os)=$($(package)_ar) $(package)_toolset_darwin=darwin $(package)_archiver_darwin=$($(package)_libtool) $(package)_config_libraries=chrono,filesystem,program_options,system,thread,test,date_time,regex,serialization,locale -$(package)_cxxflags=-std=c++11 -fvisibility=hidden +$(package)_cxxflags=-std=c++11 $(package)_cxxflags_linux=-fPIC endef diff --git a/contrib/depends/packages/icu4c.mk b/contrib/depends/packages/icu4c.mk index 7e092425e..370a02683 100644 --- a/contrib/depends/packages/icu4c.mk +++ b/contrib/depends/packages/icu4c.mk @@ -6,7 +6,7 @@ $(package)_sha256_hash=1f912c54035533fb4268809701d65c7468d00e292efbc31e644490845 $(package)_patches=icu-001-dont-build-static-dynamic-twice.patch define $(package)_set_vars - $(package)_build_opts=CFLAGS="$($(package)_cflags) $($(package)_cppflags) -DU_USING_ICU_NAMESPACE=0 --std=gnu++0x -DU_STATIC_IMPLEMENTATION -DU_COMBINED_IMPLEMENTATION -fPIC" + $(package)_build_opts=CFLAGS="$($(package)_cflags) $($(package)_cppflags) -DU_USING_ICU_NAMESPACE=0 --std=gnu++0x -DU_STATIC_IMPLEMENTATION -DU_COMBINED_IMPLEMENTATION -fPIC -DENABLE_STATIC=YES -DPGKDATA_MODE=static" endef define $(package)_config_cmds @@ -17,7 +17,7 @@ define $(package)_config_cmds sh ../source/runConfigureICU Linux &&\ make &&\ cd ../buildb &&\ - sh ../source/$($(package)_autoconf) --enable-static=yes --enable-shared=yes --disable-layoutex --prefix=$(host_prefix) --with-cross-build=`pwd`/../builda &&\ + sh ../source/$($(package)_autoconf) --enable-static=yes --disable-shared --disable-layout --disable-layoutex --disable-tests --disable-samples --prefix=$(host_prefix) --with-cross-build=`pwd`/../builda &&\ $(MAKE) $($(package)_build_opts) endef diff --git a/contrib/depends/packages/libevent.mk b/contrib/depends/packages/libevent.mk deleted file mode 100644 index 5f622f8e6..000000000 --- a/contrib/depends/packages/libevent.mk +++ /dev/null @@ -1,30 +0,0 @@ -package=libevent -$(package)_version=2.1.8-stable -$(package)_download_path=https://github.com/libevent/libevent/archive/ -$(package)_file_name=release-$($(package)_version).tar.gz -$(package)_sha256_hash=316ddb401745ac5d222d7c529ef1eada12f58f6376a66c1118eee803cb70f83d - -define $(package)_preprocess_cmds - ./autogen.sh -endef - -define $(package)_set_vars - $(package)_config_opts=--disable-shared --disable-openssl --disable-libevent-regress --disable-samples - $(package)_config_opts_release=--disable-debug-mode - $(package)_config_opts_linux=--with-pic -endef - -define $(package)_config_cmds - $($(package)_autoconf) -endef - -define $(package)_build_cmds - $(MAKE) -endef - -define $(package)_stage_cmds - $(MAKE) DESTDIR=$($(package)_staging_dir) install -endef - -define $(package)_postprocess_cmds -endef diff --git a/contrib/depends/packages/packages.mk b/contrib/depends/packages/packages.mk index b3a5992d0..4800c9936 100644 --- a/contrib/depends/packages/packages.mk +++ b/contrib/depends/packages/packages.mk @@ -1,4 +1,4 @@ -packages:=boost openssl libevent zeromq cppzmq zlib expat ldns cppzmq readline libiconv qt hidapi protobuf libusb +packages:=boost openssl zeromq cppzmq expat ldns cppzmq readline libiconv qt hidapi protobuf libusb native_packages := native_ccache native_protobuf darwin_native_packages = native_biplist native_ds_store native_mac_alias @@ -17,6 +17,5 @@ endif ifneq ($(build_os),darwin) darwin_native_packages += native_cctools native_cdrkit native_libdmg-hfsplus -packages += readline endif diff --git a/contrib/depends/packages/qt.mk b/contrib/depends/packages/qt.mk index bca2926cb..6a6f0e0f7 100644 --- a/contrib/depends/packages/qt.mk +++ b/contrib/depends/packages/qt.mk @@ -4,7 +4,6 @@ $(package)_download_path=https://download.qt.io/archive/qt/5.7/5.7.1/submodules $(package)_suffix=opensource-src-$($(package)_version).tar.gz $(package)_file_name=qtbase-$($(package)_suffix) $(package)_sha256_hash=95f83e532d23b3ddbde7973f380ecae1bac13230340557276f75f2e37984e410 -$(package)_dependencies=openssl zlib $(package)_build_subdir=qtbase $(package)_qt_libs=corelib $(package)_patches=pidlist_absolute.patch fix_qt_pkgconfig.patch qfixed-coretext.patch @@ -62,14 +61,14 @@ $(package)_config_opts += -no-xrender $(package)_config_opts += -nomake examples $(package)_config_opts += -nomake tests $(package)_config_opts += -opensource -$(package)_config_opts += -openssl-linked +$(package)_config_opts += -no-openssl $(package)_config_opts += -optimized-qmake $(package)_config_opts += -pch $(package)_config_opts += -pkg-config -$(package)_config_opts += -qt-libpng -$(package)_config_opts += -qt-libjpeg +$(package)_config_opts += -no-libpng +$(package)_config_opts += -no-libjpeg $(package)_config_opts += -qt-pcre -$(package)_config_opts += -system-zlib +$(package)_config_opts += -no-zlib $(package)_config_opts += -reduce-exports $(package)_config_opts += -static $(package)_config_opts += -silent @@ -124,7 +123,6 @@ define $(package)_config_cmds export PKG_CONFIG_LIBDIR=$(host_prefix)/lib/pkgconfig && \ export PKG_CONFIG_PATH=$(host_prefix)/share/pkgconfig && \ ./configure $($(package)_config_opts) && \ - echo "host_build: QT_CONFIG ~= s/system-zlib/zlib" >> mkspecs/qconfig.pri && \ echo "CONFIG += force_bootstrap" >> mkspecs/qconfig.pri && \ $(MAKE) sub-src-clean && \ cd ../qttranslations && ../qtbase/bin/qmake qttranslations.pro -o Makefile && \ diff --git a/contrib/depends/packages/zlib.mk b/contrib/depends/packages/zlib.mk deleted file mode 100644 index 1600b11a0..000000000 --- a/contrib/depends/packages/zlib.mk +++ /dev/null @@ -1,27 +0,0 @@ -package=zlib -$(package)_version=1.2.11 -$(package)_download_path=https://www.zlib.net -$(package)_file_name=$(package)-$($(package)_version).tar.gz -$(package)_sha256_hash=c3e5e9fdd5004dcb542feda5ee4f0ff0744628baf8ed2dd5d66f8ca1197cb1a1 - -define $(package)_set_vars -$(package)_build_opts= CC="$($(package)_cc)" -$(package)_build_opts+=CFLAGS="$($(package)_cflags) $($(package)_cppflags) -fPIC" -$(package)_build_opts+=RANLIB="$($(package)_ranlib)" -$(package)_build_opts+=AR="$($(package)_ar)" -$(package)_build_opts_darwin+=AR="$($(package)_libtool)" -$(package)_build_opts_darwin+=ARFLAGS="-o" -endef - -define $(package)_config_cmds - ./configure --static --prefix=$(host_prefix) -endef - -define $(package)_build_cmds - $(MAKE) $($(package)_build_opts) libz.a -endef - -define $(package)_stage_cmds - $(MAKE) DESTDIR=$($(package)_staging_dir) install $($(package)_build_opts) -endef - diff --git a/contrib/epee/include/net/buffer.h b/contrib/epee/include/net/buffer.h new file mode 100644 index 000000000..56c67f6bf --- /dev/null +++ b/contrib/epee/include/net/buffer.h @@ -0,0 +1,62 @@ +// Copyright (c) 2018, The Monero Project +// +// All rights reserved. +// +// Redistribution and use in source and binary forms, with or without modification, are +// permitted provided that the following conditions are met: +// +// 1. Redistributions of source code must retain the above copyright notice, this list of +// conditions and the following disclaimer. +// +// 2. Redistributions in binary form must reproduce the above copyright notice, this list +// of conditions and the following disclaimer in the documentation and/or other +// materials provided with the distribution. +// +// 3. Neither the name of the copyright holder nor the names of its contributors may be +// used to endorse or promote products derived from this software without specific +// prior written permission. +// +// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY +// EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF +// MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL +// THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, +// PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS +// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, +// STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF +// THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +#pragma once + +#include <vector> +#include "misc_log_ex.h" +#include "span.h" + +#undef MONERO_DEFAULT_LOG_CATEGORY +#define MONERO_DEFAULT_LOG_CATEGORY "net.buffer" + +//#define NET_BUFFER_LOG(x) MDEBUG(x) +#define NET_BUFFER_LOG(x) ((void)0) + +namespace epee +{ +namespace net_utils +{ +class buffer +{ +public: + buffer(size_t reserve = 0): offset(0) { storage.reserve(reserve); } + + void append(const void *data, size_t sz); + void erase(size_t sz) { NET_BUFFER_LOG("erasing " << sz << "/" << size()); CHECK_AND_ASSERT_THROW_MES(offset + sz <= storage.size(), "erase: sz too large"); offset += sz; if (offset == storage.size()) { storage.resize(0); offset = 0; } } + epee::span<const uint8_t> span(size_t sz) const { CHECK_AND_ASSERT_THROW_MES(sz <= size(), "span is too large"); return epee::span<const uint8_t>(storage.data() + offset, sz); } + // carve must keep the data in scope till next call, other API calls (such as append, erase) can invalidate the carved buffer + epee::span<const uint8_t> carve(size_t sz) { CHECK_AND_ASSERT_THROW_MES(sz <= size(), "span is too large"); offset += sz; return epee::span<const uint8_t>(storage.data() + offset - sz, sz); } + size_t size() const { return storage.size() - offset; } + +private: + std::vector<uint8_t> storage; + size_t offset; +}; +} +} diff --git a/contrib/epee/include/net/http_server_handlers_map2.h b/contrib/epee/include/net/http_server_handlers_map2.h index 997c801d1..64d035df9 100644 --- a/contrib/epee/include/net/http_server_handlers_map2.h +++ b/contrib/epee/include/net/http_server_handlers_map2.h @@ -92,7 +92,7 @@ handled = true; \ uint64_t ticks = misc_utils::get_tick_count(); \ boost::value_initialized<command_type::request> req; \ - bool parse_res = epee::serialization::load_t_from_binary(static_cast<command_type::request&>(req), query_info.m_body); \ + bool parse_res = epee::serialization::load_t_from_binary(static_cast<command_type::request&>(req), epee::strspan<uint8_t>(query_info.m_body)); \ CHECK_AND_ASSERT_MES(parse_res, false, "Failed to parse bin body data, body size=" << query_info.m_body.size()); \ uint64_t ticks1 = misc_utils::get_tick_count(); \ boost::value_initialized<command_type::response> resp;\ diff --git a/contrib/epee/include/net/levin_base.h b/contrib/epee/include/net/levin_base.h index 7d060f5ef..a88a1eb49 100644 --- a/contrib/epee/include/net/levin_base.h +++ b/contrib/epee/include/net/levin_base.h @@ -80,8 +80,8 @@ namespace levin template<class t_connection_context = net_utils::connection_context_base> struct levin_commands_handler { - virtual int invoke(int command, const std::string& in_buff, std::string& buff_out, t_connection_context& context)=0; - virtual int notify(int command, const std::string& in_buff, t_connection_context& context)=0; + virtual int invoke(int command, const epee::span<const uint8_t> in_buff, std::string& buff_out, t_connection_context& context)=0; + virtual int notify(int command, const epee::span<const uint8_t> in_buff, t_connection_context& context)=0; virtual void callback(t_connection_context& context){}; virtual void on_connection_new(t_connection_context& context){}; diff --git a/contrib/epee/include/net/levin_client.h b/contrib/epee/include/net/levin_client.h index 335f6ba02..76d528234 100644 --- a/contrib/epee/include/net/levin_client.h +++ b/contrib/epee/include/net/levin_client.h @@ -57,7 +57,7 @@ namespace levin bool is_connected(); bool disconnect(); - virtual int invoke(int command, const std::string& in_buff, std::string& buff_out); + virtual int invoke(int command, const epee::span<const uint8_t> in_buff, std::string& buff_out); virtual int notify(int command, const std::string& in_buff); protected: @@ -72,7 +72,7 @@ namespace levin { public: - int invoke(int command, const std::string& in_buff, std::string& buff_out); + int invoke(int command, const epee::span<const uint8_t> in_buff, std::string& buff_out); int notify(int command, const std::string& in_buff); }; diff --git a/contrib/epee/include/net/levin_client.inl b/contrib/epee/include/net/levin_client.inl index a580e81fd..2f048b027 100644 --- a/contrib/epee/include/net/levin_client.inl +++ b/contrib/epee/include/net/levin_client.inl @@ -74,7 +74,7 @@ levin_client_impl::~levin_client_impl() } //------------------------------------------------------------------------------ inline -int levin_client_impl::invoke(int command, const std::string& in_buff, std::string& buff_out) +int levin_client_impl::invoke(int command, const epee::span<const uint8_t> in_buff, std::string& buff_out) { if(!is_connected()) return -1; @@ -133,7 +133,7 @@ int levin_client_impl::notify(int command, const std::string& in_buff) //------------------------------------------------------------------------------ //------------------------------------------------------------------------------ inline - int levin_client_impl2::invoke(int command, const std::string& in_buff, std::string& buff_out) + int levin_client_impl2::invoke(int command, epee::span<const uint8_t>string& in_buff, std::string& buff_out) { if(!is_connected()) return -1; diff --git a/contrib/epee/include/net/levin_protocol_handler_async.h b/contrib/epee/include/net/levin_protocol_handler_async.h index 11649febe..a1ea3e680 100644 --- a/contrib/epee/include/net/levin_protocol_handler_async.h +++ b/contrib/epee/include/net/levin_protocol_handler_async.h @@ -34,6 +34,7 @@ #include <atomic> #include "levin_base.h" +#include "buffer.h" #include "misc_language.h" #include "syncobj.h" #include "misc_os_dependent.h" @@ -85,11 +86,11 @@ public: uint64_t m_max_packet_size; uint64_t m_invoke_timeout; - int invoke(int command, const std::string& in_buff, std::string& buff_out, boost::uuids::uuid connection_id); + int invoke(int command, const epee::span<const uint8_t> in_buff, std::string& buff_out, boost::uuids::uuid connection_id); template<class callback_t> - int invoke_async(int command, const std::string& in_buff, boost::uuids::uuid connection_id, const callback_t &cb, size_t timeout = LEVIN_DEFAULT_TIMEOUT_PRECONFIGURED); + int invoke_async(int command, const epee::span<const uint8_t> in_buff, boost::uuids::uuid connection_id, const callback_t &cb, size_t timeout = LEVIN_DEFAULT_TIMEOUT_PRECONFIGURED); - int notify(int command, const std::string& in_buff, boost::uuids::uuid connection_id); + int notify(int command, const epee::span<const uint8_t> in_buff, boost::uuids::uuid connection_id); bool close(boost::uuids::uuid connection_id); bool update_connection_context(const t_connection_context& contxt); bool request_callback(boost::uuids::uuid connection_id); @@ -143,7 +144,7 @@ public: config_type& m_config; t_connection_context& m_connection_context; - std::string m_cache_in_buffer; + net_utils::buffer m_cache_in_buffer; stream_state m_state; int32_t m_oponent_protocol_ver; @@ -151,7 +152,7 @@ public: struct invoke_response_handler_base { - virtual bool handle(int res, const std::string& buff, connection_context& context)=0; + virtual bool handle(int res, const epee::span<const uint8_t> buff, connection_context& context)=0; virtual bool is_timer_started() const=0; virtual void cancel()=0; virtual bool cancel_timer()=0; @@ -173,7 +174,7 @@ public: if(ec == boost::asio::error::operation_aborted) return; MINFO(con.get_context_ref() << "Timeout on invoke operation happened, command: " << command << " timeout: " << timeout); - std::string fake; + epee::span<const uint8_t> fake; cb(LEVIN_ERROR_CONNECTION_TIMEDOUT, fake, con.get_context_ref()); con.close(); con.finish_outer_call(); @@ -191,7 +192,7 @@ public: bool m_timer_cancelled; uint64_t m_timeout; int m_command; - virtual bool handle(int res, const std::string& buff, typename async_protocol_handler::connection_context& context) + virtual bool handle(int res, const epee::span<const uint8_t> buff, typename async_protocol_handler::connection_context& context) { if(!cancel_timer()) return false; @@ -207,7 +208,7 @@ public: { if(cancel_timer()) { - std::string fake; + epee::span<const uint8_t> fake; m_cb(LEVIN_ERROR_CONNECTION_DESTROYED, fake, m_con.get_context_ref()); m_con.finish_outer_call(); } @@ -237,7 +238,7 @@ public: if(ec == boost::asio::error::operation_aborted) return; MINFO(con.get_context_ref() << "Timeout on invoke operation happened, command: " << command << " timeout: " << timeout); - std::string fake; + epee::span<const uint8_t> fake; cb(LEVIN_ERROR_CONNECTION_TIMEDOUT, fake, con.get_context_ref()); con.close(); con.finish_outer_call(); @@ -265,6 +266,7 @@ public: m_pservice_endpoint(psnd_hndlr), m_config(config), m_connection_context(conn_context), + m_cache_in_buffer(256 * 1024), m_state(stream_state_head) { m_close_called = 0; @@ -405,14 +407,7 @@ public: break; } { - std::string buff_to_invoke; - if(m_cache_in_buffer.size() == m_current_head.m_cb) - buff_to_invoke.swap(m_cache_in_buffer); - else - { - buff_to_invoke.assign(m_cache_in_buffer, 0, (std::string::size_type)m_current_head.m_cb); - m_cache_in_buffer.erase(0, (std::string::size_type)m_current_head.m_cb); - } + epee::span<const uint8_t> buff_to_invoke = m_cache_in_buffer.carve((std::string::size_type)m_current_head.m_cb); bool is_response = (m_oponent_protocol_ver == LEVIN_PROTOCOL_VER_1 && m_current_head.m_flags&LEVIN_PACKET_RESPONSE); @@ -449,8 +444,8 @@ public: }else { CRITICAL_REGION_BEGIN(m_local_inv_buff_lock); - buff_to_invoke.swap(m_local_inv_buff); - buff_to_invoke.clear(); + m_local_inv_buff = std::string((const char*)buff_to_invoke.data(), buff_to_invoke.size()); + buff_to_invoke = epee::span<const uint8_t>((const uint8_t*)NULL, 0); m_invoke_result_code = m_current_head.m_return_code; CRITICAL_REGION_END(); boost::interprocess::ipcdetail::atomic_write32(&m_invoke_buf_ready, 1); @@ -503,7 +498,7 @@ public: { if(m_cache_in_buffer.size() < sizeof(bucket_head2)) { - if(m_cache_in_buffer.size() >= sizeof(uint64_t) && *((uint64_t*)m_cache_in_buffer.data()) != SWAP64LE(LEVIN_SIGNATURE)) + if(m_cache_in_buffer.size() >= sizeof(uint64_t) && *((uint64_t*)m_cache_in_buffer.span(8).data()) != SWAP64LE(LEVIN_SIGNATURE)) { MWARNING(m_connection_context << "Signature mismatch, connection will be closed"); return false; @@ -513,9 +508,9 @@ public: } #if BYTE_ORDER == LITTLE_ENDIAN - bucket_head2& phead = *(bucket_head2*)m_cache_in_buffer.data(); + bucket_head2& phead = *(bucket_head2*)m_cache_in_buffer.span(sizeof(bucket_head2)).data(); #else - bucket_head2 phead = *(bucket_head2*)m_cache_in_buffer.data(); + bucket_head2 phead = *(bucket_head2*)m_cache_in_buffer.span(sizeof(bucket_head2)).data(); phead.m_signature = SWAP64LE(phead.m_signature); phead.m_cb = SWAP64LE(phead.m_cb); phead.m_command = SWAP32LE(phead.m_command); @@ -530,7 +525,7 @@ public: } m_current_head = phead; - m_cache_in_buffer.erase(0, sizeof(bucket_head2)); + m_cache_in_buffer.erase(sizeof(bucket_head2)); m_state = stream_state_body; m_oponent_protocol_ver = m_current_head.m_protocol_version; if(m_current_head.m_cb > m_config.m_max_packet_size) @@ -562,7 +557,7 @@ public: } template<class callback_t> - bool async_invoke(int command, const std::string& in_buff, const callback_t &cb, size_t timeout = LEVIN_DEFAULT_TIMEOUT_PRECONFIGURED) + bool async_invoke(int command, const epee::span<const uint8_t> in_buff, const callback_t &cb, size_t timeout = LEVIN_DEFAULT_TIMEOUT_PRECONFIGURED) { misc_utils::auto_scope_leave_caller scope_exit_handler = misc_utils::create_scope_leave_handler( boost::bind(&async_protocol_handler::finish_outer_call, this)); @@ -606,7 +601,7 @@ public: break; } - if(!m_pservice_endpoint->do_send(in_buff.data(), (int)in_buff.size())) + if(!m_pservice_endpoint->do_send(in_buff.data(), in_buff.size())) { LOG_ERROR_CC(m_connection_context, "Failed to do_send"); err_code = LEVIN_ERROR_CONNECTION; @@ -623,7 +618,7 @@ public: if (LEVIN_OK != err_code) { - std::string stub_buff; + epee::span<const uint8_t> stub_buff{(const uint8_t*)"", 0}; // Never call callback inside critical section, that can cause deadlock cb(err_code, stub_buff, m_connection_context); return false; @@ -632,7 +627,7 @@ public: return true; } - int invoke(int command, const std::string& in_buff, std::string& buff_out) + int invoke(int command, const epee::span<const uint8_t> in_buff, std::string& buff_out) { misc_utils::auto_scope_leave_caller scope_exit_handler = misc_utils::create_scope_leave_handler( boost::bind(&async_protocol_handler::finish_outer_call, this)); @@ -662,7 +657,7 @@ public: return LEVIN_ERROR_CONNECTION; } - if(!m_pservice_endpoint->do_send(in_buff.data(), (int)in_buff.size())) + if(!m_pservice_endpoint->do_send(in_buff.data(), in_buff.size())) { LOG_ERROR_CC(m_connection_context, "Failed to do_send"); return LEVIN_ERROR_CONNECTION; @@ -706,7 +701,7 @@ public: return m_invoke_result_code; } - int notify(int command, const std::string& in_buff) + int notify(int command, const epee::span<const uint8_t> in_buff) { misc_utils::auto_scope_leave_caller scope_exit_handler = misc_utils::create_scope_leave_handler( boost::bind(&async_protocol_handler::finish_outer_call, this)); @@ -734,7 +729,7 @@ public: return -1; } - if(!m_pservice_endpoint->do_send(in_buff.data(), (int)in_buff.size())) + if(!m_pservice_endpoint->do_send(in_buff.data(), in_buff.size())) { LOG_ERROR_CC(m_connection_context, "Failed to do_send()"); return -1; @@ -839,7 +834,7 @@ int async_protocol_handler_config<t_connection_context>::find_and_lock_connectio } //------------------------------------------------------------------------------------------ template<class t_connection_context> -int async_protocol_handler_config<t_connection_context>::invoke(int command, const std::string& in_buff, std::string& buff_out, boost::uuids::uuid connection_id) +int async_protocol_handler_config<t_connection_context>::invoke(int command, const epee::span<const uint8_t> in_buff, std::string& buff_out, boost::uuids::uuid connection_id) { async_protocol_handler<t_connection_context>* aph; int r = find_and_lock_connection(connection_id, aph); @@ -847,7 +842,7 @@ int async_protocol_handler_config<t_connection_context>::invoke(int command, con } //------------------------------------------------------------------------------------------ template<class t_connection_context> template<class callback_t> -int async_protocol_handler_config<t_connection_context>::invoke_async(int command, const std::string& in_buff, boost::uuids::uuid connection_id, const callback_t &cb, size_t timeout) +int async_protocol_handler_config<t_connection_context>::invoke_async(int command, const epee::span<const uint8_t> in_buff, boost::uuids::uuid connection_id, const callback_t &cb, size_t timeout) { async_protocol_handler<t_connection_context>* aph; int r = find_and_lock_connection(connection_id, aph); @@ -896,7 +891,7 @@ void async_protocol_handler_config<t_connection_context>::set_handler(levin_comm } //------------------------------------------------------------------------------------------ template<class t_connection_context> -int async_protocol_handler_config<t_connection_context>::notify(int command, const std::string& in_buff, boost::uuids::uuid connection_id) +int async_protocol_handler_config<t_connection_context>::notify(int command, const epee::span<const uint8_t> in_buff, boost::uuids::uuid connection_id) { async_protocol_handler<t_connection_context>* aph; int r = find_and_lock_connection(connection_id, aph); diff --git a/contrib/epee/include/span.h b/contrib/epee/include/span.h index b1296a0b7..cfb5b1a17 100644 --- a/contrib/epee/include/span.h +++ b/contrib/epee/include/span.h @@ -163,4 +163,12 @@ namespace epee static_assert(!has_padding<T>(), "source type may have padding"); return {reinterpret_cast<std::uint8_t*>(std::addressof(src)), sizeof(T)}; } + + //! make a span from a std::string + template<typename T> + span<const T> strspan(const std::string &s) noexcept + { + static_assert(std::is_same<T, char>() || std::is_same<T, unsigned char>() || std::is_same<T, int8_t>() || std::is_same<T, uint8_t>(), "Unexpected type"); + return {reinterpret_cast<const T*>(s.data()), s.size()}; + } } diff --git a/contrib/epee/include/storages/http_abstract_invoke.h b/contrib/epee/include/storages/http_abstract_invoke.h index d93084ab0..18b7f10c1 100644 --- a/contrib/epee/include/storages/http_abstract_invoke.h +++ b/contrib/epee/include/storages/http_abstract_invoke.h @@ -97,7 +97,7 @@ namespace epee return false; } - return serialization::load_t_from_binary(result_struct, pri->m_body); + return serialization::load_t_from_binary(result_struct, epee::strspan<uint8_t>(pri->m_body)); } template<class t_request, class t_response, class t_transport> diff --git a/contrib/epee/include/storages/levin_abstract_invoke2.h b/contrib/epee/include/storages/levin_abstract_invoke2.h index d77e7a1f8..06eb9bdaf 100644 --- a/contrib/epee/include/storages/levin_abstract_invoke2.h +++ b/contrib/epee/include/storages/levin_abstract_invoke2.h @@ -28,6 +28,7 @@ #include "portable_storage_template_helper.h" #include <boost/utility/value_init.hpp> +#include "span.h" #include "net/levin_base.h" #undef MONERO_DEFAULT_LOG_CATEGORY @@ -114,7 +115,7 @@ namespace epee const_cast<t_arg&>(out_struct).store(stg);//TODO: add true const support to searilzation std::string buff_to_send; stg.store_to_binary(buff_to_send); - int res = transport.invoke_async(command, buff_to_send, conn_id, [cb, command](int code, const std::string& buff, typename t_transport::connection_context& context)->bool + int res = transport.invoke_async(command, epee::strspan<uint8_t>(buff_to_send), conn_id, [cb, command](int code, const epee::span<const uint8_t> buff, typename t_transport::connection_context& context)->bool { t_result result_struct = AUTO_VAL_INIT(result_struct); if( code <=0 ) @@ -156,7 +157,7 @@ namespace epee std::string buff_to_send; stg.store_to_binary(buff_to_send); - int res = transport.notify(command, buff_to_send, conn_id); + int res = transport.notify(command, epee::strspan<uint8_t>(buff_to_send), conn_id); if(res <=0 ) { MERROR("Failed to notify command " << command << " return code " << res); @@ -167,7 +168,7 @@ namespace epee //---------------------------------------------------------------------------------------------------- //---------------------------------------------------------------------------------------------------- template<class t_owner, class t_in_type, class t_out_type, class t_context, class callback_t> - int buff_to_t_adapter(int command, const std::string& in_buff, std::string& buff_out, callback_t cb, t_context& context ) + int buff_to_t_adapter(int command, const epee::span<const uint8_t> in_buff, std::string& buff_out, callback_t cb, t_context& context ) { serialization::portable_storage strg; if(!strg.load_from_binary(in_buff)) @@ -197,7 +198,7 @@ namespace epee } template<class t_owner, class t_in_type, class t_context, class callback_t> - int buff_to_t_adapter(t_owner* powner, int command, const std::string& in_buff, callback_t cb, t_context& context) + int buff_to_t_adapter(t_owner* powner, int command, const epee::span<const uint8_t> in_buff, callback_t cb, t_context& context) { serialization::portable_storage strg; if(!strg.load_from_binary(in_buff)) @@ -215,14 +216,14 @@ namespace epee } #define CHAIN_LEVIN_INVOKE_MAP2(context_type) \ - int invoke(int command, const std::string& in_buff, std::string& buff_out, context_type& context) \ + int invoke(int command, const epee::span<const uint8_t> in_buff, std::string& buff_out, context_type& context) \ { \ bool handled = false; \ return handle_invoke_map(false, command, in_buff, buff_out, context, handled); \ } #define CHAIN_LEVIN_NOTIFY_MAP2(context_type) \ - int notify(int command, const std::string& in_buff, context_type& context) \ + int notify(int command, const epee::span<const uint8_t> in_buff, context_type& context) \ { \ bool handled = false; std::string fake_str;\ return handle_invoke_map(true, command, in_buff, fake_str, context, handled); \ @@ -230,27 +231,27 @@ namespace epee #define CHAIN_LEVIN_INVOKE_MAP() \ - int invoke(int command, const std::string& in_buff, std::string& buff_out, epee::net_utils::connection_context_base& context) \ + int invoke(int command, const epee::span<const uint8_t> in_buff, std::string& buff_out, epee::net_utils::connection_context_base& context) \ { \ bool handled = false; \ return handle_invoke_map(false, command, in_buff, buff_out, context, handled); \ } #define CHAIN_LEVIN_NOTIFY_MAP() \ - int notify(int command, const std::string& in_buff, epee::net_utils::connection_context_base& context) \ + int notify(int command, const epee::span<const uint8_t> in_buff, epee::net_utils::connection_context_base& context) \ { \ bool handled = false; std::string fake_str;\ return handle_invoke_map(true, command, in_buff, fake_str, context, handled); \ } #define CHAIN_LEVIN_NOTIFY_STUB() \ - int notify(int command, const std::string& in_buff, epee::net_utils::connection_context_base& context) \ + int notify(int command, const epee::span<const uint8_t> in_buff, epee::net_utils::connection_context_base& context) \ { \ return -1; \ } #define BEGIN_INVOKE_MAP2(owner_type) \ - template <class t_context> int handle_invoke_map(bool is_notify, int command, const std::string& in_buff, std::string& buff_out, t_context& context, bool& handled) \ + template <class t_context> int handle_invoke_map(bool is_notify, int command, const epee::span<const uint8_t> in_buff, std::string& buff_out, t_context& context, bool& handled) \ { \ typedef owner_type internal_owner_type_name; diff --git a/contrib/epee/include/storages/parserse_base_utils.h b/contrib/epee/include/storages/parserse_base_utils.h index d73fbde3a..69b650cd4 100644 --- a/contrib/epee/include/storages/parserse_base_utils.h +++ b/contrib/epee/include/storages/parserse_base_utils.h @@ -29,6 +29,7 @@ #pragma once #include <algorithm> +#include <boost/utility/string_ref.hpp> namespace epee { @@ -36,6 +37,40 @@ namespace misc_utils { namespace parse { + // 1: digit + // 2: .eE (floating point) + // 4: alpha + // 8: whitespace + // 16: allowed in float but doesn't necessarily mean it's a float + static const constexpr uint8_t lut[256]={ + 0, 0, 0, 0, 0, 0, 0, 0, 0, 8, 8, 8, 8, 8, 0, 0, // 16 + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, // 32 + 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 16, 0, 16, 18, 0, // 48 + 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 0, 0, 0, 0, 0, 0, // 64 + 0, 4, 4, 4, 4, 22, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, // 80 + 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 0, 0, 0, 0, 0, // 96 + 0, 4, 4, 4, 4, 22, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, // 112 + 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 0, 0, 0, 0, 0, // 128 + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + }; + + inline bool isspace(char c) + { + return lut[(uint8_t)c] & 8; + } + + inline bool isdigit(char c) + { + return lut[(uint8_t)c] & 1; + } + inline std::string transform_to_escape_sequence(const std::string& src) { static const char escaped[] = "\b\f\n\r\t\v\"\\/"; @@ -159,25 +194,34 @@ namespace misc_utils return false; } } - inline void match_number2(std::string::const_iterator& star_end_string, std::string::const_iterator buf_end, std::string& val, bool& is_float_val, bool& is_signed_val) + inline void match_number2(std::string::const_iterator& star_end_string, std::string::const_iterator buf_end, boost::string_ref& val, bool& is_float_val, bool& is_signed_val) { val.clear(); - is_float_val = false; - for(std::string::const_iterator it = star_end_string;it != buf_end;it++) + uint8_t float_flag = 0; + is_signed_val = false; + size_t chars = 0; + std::string::const_iterator it = star_end_string; + if (it != buf_end && *it == '-') + { + is_signed_val = true; + ++chars; + ++it; + } + for(;it != buf_end;it++) { - if(isdigit(*it) || (it == star_end_string && *it == '-') || (val.size() && *it == '.' ) || (is_float_val && (*it == 'e' || *it == 'E' || *it == '-' || *it == '+' )) ) + const uint8_t flags = lut[(uint8_t)*it]; + if (flags & 16) { - if(!val.size() && *it == '-') - is_signed_val = true; - if(*it == '.' ) - is_float_val = true; - val.push_back(*it); + float_flag |= flags; + ++chars; } else { + val = boost::string_ref(&*star_end_string, chars); if(val.size()) { star_end_string = --it; + is_float_val = !!(float_flag & 2); return; } else @@ -186,7 +230,7 @@ namespace misc_utils } ASSERT_MES_AND_THROW("wrong number in json entry: " << std::string(star_end_string, buf_end)); } - inline bool match_number(std::string::const_iterator& star_end_string, std::string::const_iterator buf_end, std::string& val) + inline bool match_number(std::string::const_iterator& star_end_string, std::string::const_iterator buf_end, boost::string_ref& val) { try { @@ -199,15 +243,15 @@ namespace misc_utils return false; } } - inline void match_word2(std::string::const_iterator& star_end_string, std::string::const_iterator buf_end, std::string& val) + inline void match_word2(std::string::const_iterator& star_end_string, std::string::const_iterator buf_end, boost::string_ref& val) { val.clear(); for(std::string::const_iterator it = star_end_string;it != buf_end;it++) { - if(!isalpha(*it)) + if (!(lut[(uint8_t)*it] & 4)) { - val.assign(star_end_string, it); + val = boost::string_ref(&*star_end_string, std::distance(star_end_string, it)); if(val.size()) { star_end_string = --it; @@ -218,7 +262,7 @@ namespace misc_utils } ASSERT_MES_AND_THROW("failed to match word number in json entry: " << std::string(star_end_string, buf_end)); } - inline bool match_word(std::string::const_iterator& star_end_string, std::string::const_iterator buf_end, std::string& val) + inline bool match_word(std::string::const_iterator& star_end_string, std::string::const_iterator buf_end, boost::string_ref& val) { try { diff --git a/contrib/epee/include/storages/portable_storage.h b/contrib/epee/include/storages/portable_storage.h index 0f0c6210f..d0e40d606 100644 --- a/contrib/epee/include/storages/portable_storage.h +++ b/contrib/epee/include/storages/portable_storage.h @@ -35,6 +35,7 @@ #include "portable_storage_to_json.h" #include "portable_storage_from_json.h" #include "portable_storage_val_converters.h" +#include "span.h" #include "int-util.h" namespace epee @@ -81,7 +82,8 @@ namespace epee //------------------------------------------------------------------------------- bool store_to_binary(binarybuffer& target); - bool load_from_binary(const binarybuffer& target); + bool load_from_binary(const epee::span<const uint8_t> target); + bool load_from_binary(const std::string& target) { return load_from_binary(epee::strspan<uint8_t>(target)); } template<class trace_policy> bool dump_as_xml(std::string& targetObj, const std::string& root_name = ""); bool dump_as_json(std::string& targetObj, size_t indent = 0, bool insert_newlines = true); @@ -146,7 +148,7 @@ namespace epee CATCH_ENTRY("portable_storage::store_to_binary", false) } inline - bool portable_storage::load_from_binary(const binarybuffer& source) + bool portable_storage::load_from_binary(const epee::span<const uint8_t> source) { m_root.m_entries.clear(); if(source.size() < sizeof(storage_block_header)) diff --git a/contrib/epee/include/storages/portable_storage_base.h b/contrib/epee/include/storages/portable_storage_base.h index 93132548b..da84fd8ea 100644 --- a/contrib/epee/include/storages/portable_storage_base.h +++ b/contrib/epee/include/storages/portable_storage_base.h @@ -31,7 +31,8 @@ #include <boost/variant.hpp> #include <boost/any.hpp> #include <string> -#include <list> +#include <vector> +#include <deque> #define PORTABLE_STORAGE_SIGNATUREA 0x01011101 #define PORTABLE_STORAGE_SIGNATUREB 0x01020101 // bender's nightmare @@ -71,6 +72,9 @@ namespace epee { struct section; + template<typename T> struct entry_container { typedef std::vector<T> type; static void reserve(type &t, size_t n) { t.reserve(n); } }; + template<> struct entry_container<bool> { typedef std::deque<bool> type; static void reserve(type &t, size_t n) {} }; + /************************************************************************/ /* */ /************************************************************************/ @@ -119,8 +123,13 @@ namespace epee return m_array.back(); } - std::list<t_entry_type> m_array; - mutable typename std::list<t_entry_type>::const_iterator m_it; + void reserve(size_t n) + { + entry_container<t_entry_type>::reserve(m_array, n); + } + + typename entry_container<t_entry_type>::type m_array; + mutable typename entry_container<t_entry_type>::type::const_iterator m_it; }; diff --git a/contrib/epee/include/storages/portable_storage_from_bin.h b/contrib/epee/include/storages/portable_storage_from_bin.h index f9cc22d27..2884f8c5e 100644 --- a/contrib/epee/include/storages/portable_storage_from_bin.h +++ b/contrib/epee/include/storages/portable_storage_from_bin.h @@ -136,6 +136,7 @@ namespace epee //for pod types array_entry_t<type_name> sa; size_t size = read_varint(); + sa.reserve(size); //TODO: add some optimization here later while(size--) sa.m_array.push_back(read<type_name>()); diff --git a/contrib/epee/include/storages/portable_storage_from_json.h b/contrib/epee/include/storages/portable_storage_from_json.h index 0307b732c..3e3052541 100644 --- a/contrib/epee/include/storages/portable_storage_from_json.h +++ b/contrib/epee/include/storages/portable_storage_from_json.h @@ -39,7 +39,7 @@ namespace epee { namespace json { -#define CHECK_ISSPACE() if(!isspace(*it)){ ASSERT_MES_AND_THROW("Wrong JSON character at: " << std::string(it, buf_end));} +#define CHECK_ISSPACE() if(!epee::misc_utils::parse::isspace(*it)){ ASSERT_MES_AND_THROW("Wrong JSON character at: " << std::string(it, buf_end));} /*inline void parse_error() { @@ -114,11 +114,11 @@ namespace epee std::string val; match_string2(it, buf_end, val); //insert text value - stg.set_value(name, val, current_section); + stg.set_value(name, std::move(val), current_section); state = match_state_wonder_after_value; - }else if (isdigit(*it) || *it == '-') + }else if (epee::misc_utils::parse::isdigit(*it) || *it == '-') {//just a named number value started - std::string val; + boost::string_ref val; bool is_v_float = false;bool is_signed = false; match_number2(it, buf_end, val, is_v_float, is_signed); if(!is_v_float) @@ -126,27 +126,27 @@ namespace epee if(is_signed) { errno = 0; - int64_t nval = strtoll(val.c_str(), NULL, 10); - if (errno) throw std::runtime_error("Invalid number: " + val); + int64_t nval = strtoll(val.data(), NULL, 10); + if (errno) throw std::runtime_error("Invalid number: " + std::string(val)); stg.set_value(name, nval, current_section); }else { errno = 0; - uint64_t nval = strtoull(val.c_str(), NULL, 10); - if (errno) throw std::runtime_error("Invalid number: " + val); + uint64_t nval = strtoull(val.data(), NULL, 10); + if (errno) throw std::runtime_error("Invalid number: " + std::string(val)); stg.set_value(name, nval, current_section); } }else { errno = 0; - double nval = strtod(val.c_str(), NULL); - if (errno) throw std::runtime_error("Invalid number: " + val); + double nval = strtod(val.data(), NULL); + if (errno) throw std::runtime_error("Invalid number: " + std::string(val)); stg.set_value(name, nval, current_section); } state = match_state_wonder_after_value; }else if(isalpha(*it) ) {// could be null, true or false - std::string word; + boost::string_ref word; match_word2(it, buf_end, word); if(boost::iequals(word, "null")) { @@ -203,13 +203,13 @@ namespace epee //mean array of strings std::string val; match_string2(it, buf_end, val); - h_array = stg.insert_first_value(name, val, current_section); + h_array = stg.insert_first_value(name, std::move(val), current_section); CHECK_AND_ASSERT_THROW_MES(h_array, " failed to insert values entry"); state = match_state_array_after_value; array_md = array_mode_string; - }else if (isdigit(*it) || *it == '-') + }else if (epee::misc_utils::parse::isdigit(*it) || *it == '-') {//array of numbers value started - std::string val; + boost::string_ref val; bool is_v_float = false;bool is_signed_val = false; match_number2(it, buf_end, val, is_v_float, is_signed_val); if(!is_v_float) @@ -217,22 +217,22 @@ namespace epee if (is_signed_val) { errno = 0; - int64_t nval = strtoll(val.c_str(), NULL, 10); - if (errno) throw std::runtime_error("Invalid number: " + val); + int64_t nval = strtoll(val.data(), NULL, 10); + if (errno) throw std::runtime_error("Invalid number: " + std::string(val)); h_array = stg.insert_first_value(name, nval, current_section); }else { errno = 0; - uint64_t nval = strtoull(val.c_str(), NULL, 10); - if (errno) throw std::runtime_error("Invalid number: " + val); + uint64_t nval = strtoull(val.data(), NULL, 10); + if (errno) throw std::runtime_error("Invalid number: " + std::string(val)); h_array = stg.insert_first_value(name, nval, current_section); } CHECK_AND_ASSERT_THROW_MES(h_array, " failed to insert values section entry"); }else { errno = 0; - double nval = strtod(val.c_str(), NULL); - if (errno) throw std::runtime_error("Invalid number: " + val); + double nval = strtod(val.data(), NULL); + if (errno) throw std::runtime_error("Invalid number: " + std::string(val)); h_array = stg.insert_first_value(name, nval, current_section); CHECK_AND_ASSERT_THROW_MES(h_array, " failed to insert values section entry"); } @@ -245,7 +245,7 @@ namespace epee state = match_state_wonder_after_value; }else if(isalpha(*it) ) {// array of booleans - std::string word; + boost::string_ref word; match_word2(it, buf_end, word); if(boost::iequals(word, "true")) { @@ -291,15 +291,15 @@ namespace epee { std::string val; match_string2(it, buf_end, val); - bool res = stg.insert_next_value(h_array, val); + bool res = stg.insert_next_value(h_array, std::move(val)); CHECK_AND_ASSERT_THROW_MES(res, "failed to insert values"); state = match_state_array_after_value; }else CHECK_ISSPACE(); break; case array_mode_numbers: - if (isdigit(*it) || *it == '-') + if (epee::misc_utils::parse::isdigit(*it) || *it == '-') {//array of numbers value started - std::string val; + boost::string_ref val; bool is_v_float = false;bool is_signed_val = false; match_number2(it, buf_end, val, is_v_float, is_signed_val); bool insert_res = false; @@ -308,21 +308,21 @@ namespace epee if (is_signed_val) { errno = 0; - int64_t nval = strtoll(val.c_str(), NULL, 10); - if (errno) throw std::runtime_error("Invalid number: " + val); + int64_t nval = strtoll(val.data(), NULL, 10); + if (errno) throw std::runtime_error("Invalid number: " + std::string(val)); insert_res = stg.insert_next_value(h_array, nval); }else { errno = 0; - uint64_t nval = strtoull(val.c_str(), NULL, 10); - if (errno) throw std::runtime_error("Invalid number: " + val); + uint64_t nval = strtoull(val.data(), NULL, 10); + if (errno) throw std::runtime_error("Invalid number: " + std::string(val)); insert_res = stg.insert_next_value(h_array, nval); } }else { errno = 0; - double nval = strtod(val.c_str(), NULL); - if (errno) throw std::runtime_error("Invalid number: " + val); + double nval = strtod(val.data(), NULL); + if (errno) throw std::runtime_error("Invalid number: " + std::string(val)); insert_res = stg.insert_next_value(h_array, nval); } CHECK_AND_ASSERT_THROW_MES(insert_res, "Failed to insert next value"); @@ -333,7 +333,7 @@ namespace epee case array_mode_booleans: if(isalpha(*it) ) {// array of booleans - std::string word; + boost::string_ref word; match_word2(it, buf_end, word); if(boost::iequals(word, "true")) { diff --git a/contrib/epee/include/storages/portable_storage_template_helper.h b/contrib/epee/include/storages/portable_storage_template_helper.h index bbd8413fc..13c870d44 100644 --- a/contrib/epee/include/storages/portable_storage_template_helper.h +++ b/contrib/epee/include/storages/portable_storage_template_helper.h @@ -84,7 +84,7 @@ namespace epee } //----------------------------------------------------------------------------------------------------------- template<class t_struct> - bool load_t_from_binary(t_struct& out, const std::string& binary_buff) + bool load_t_from_binary(t_struct& out, const epee::span<const uint8_t> binary_buff) { portable_storage ps; bool rs = ps.load_from_binary(binary_buff); @@ -95,6 +95,12 @@ namespace epee } //----------------------------------------------------------------------------------------------------------- template<class t_struct> + bool load_t_from_binary(t_struct& out, const std::string& binary_buff) + { + return load_t_from_binary(out, epee::strspan<uint8_t>(binary_buff)); + } + //----------------------------------------------------------------------------------------------------------- + template<class t_struct> bool load_t_from_binary_file(t_struct& out, const std::string& binary_file) { std::string f_buff; diff --git a/contrib/epee/include/string_tools.h b/contrib/epee/include/string_tools.h index 0d8eca727..f7539fb6c 100644 --- a/contrib/epee/include/string_tools.h +++ b/contrib/epee/include/string_tools.h @@ -40,8 +40,6 @@ #include <cstdlib> #include <string> #include <type_traits> -#include <boost/uuid/uuid.hpp> -#include <boost/uuid/uuid_io.hpp> #include <boost/lexical_cast.hpp> #include <boost/algorithm/string/predicate.hpp> #include "hex.h" @@ -83,34 +81,6 @@ namespace epee { namespace string_tools { - //---------------------------------------------------------------------------- - inline std::string get_str_from_guid_a(const boost::uuids::uuid& rid) - { - return boost::lexical_cast<std::string>(rid); - } - //---------------------------------------------------------------------------- - inline bool get_guid_from_string(OUT boost::uuids::uuid& inetifer, const std::string& str_id) - { - std::string local_str_id = str_id; - if(local_str_id.size() < 36) - return false; - - if('{' == *local_str_id.begin()) - local_str_id.erase(0, 1); - - if('}' == *(--local_str_id.end())) - local_str_id.erase(--local_str_id.end()); - - try - { - inetifer = boost::lexical_cast<boost::uuids::uuid>(local_str_id); - return true; - } - catch(...) - { - return false; - } - } //---------------------------------------------------------------------------- inline std::string buff_to_hex_nodelimer(const std::string& src) { diff --git a/contrib/epee/src/CMakeLists.txt b/contrib/epee/src/CMakeLists.txt index bc437deb9..cea50c9dd 100644 --- a/contrib/epee/src/CMakeLists.txt +++ b/contrib/epee/src/CMakeLists.txt @@ -27,7 +27,7 @@ # THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. add_library(epee STATIC hex.cpp http_auth.cpp mlog.cpp net_utils_base.cpp string_tools.cpp wipeable_string.cpp memwipe.c - connection_basic.cpp network_throttle.cpp network_throttle-detail.cpp mlocker.cpp) + connection_basic.cpp network_throttle.cpp network_throttle-detail.cpp mlocker.cpp buffer.cpp) if (USE_READLINE AND GNU_READLINE_FOUND) add_library(epee_readline STATIC readline_buffer.cpp) endif() diff --git a/contrib/epee/src/buffer.cpp b/contrib/epee/src/buffer.cpp new file mode 100644 index 000000000..d637b905e --- /dev/null +++ b/contrib/epee/src/buffer.cpp @@ -0,0 +1,97 @@ +// Copyright (c) 2018, The Monero Project +// +// All rights reserved. +// +// Redistribution and use in source and binary forms, with or without modification, are +// permitted provided that the following conditions are met: +// +// 1. Redistributions of source code must retain the above copyright notice, this list of +// conditions and the following disclaimer. +// +// 2. Redistributions in binary form must reproduce the above copyright notice, this list +// of conditions and the following disclaimer in the documentation and/or other +// materials provided with the distribution. +// +// 3. Neither the name of the copyright holder nor the names of its contributors may be +// used to endorse or promote products derived from this software without specific +// prior written permission. +// +// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY +// EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF +// MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL +// THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, +// PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS +// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, +// STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF +// THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +#include <string.h> +#include "net/buffer.h" + +#undef MONERO_DEFAULT_LOG_CATEGORY +#define MONERO_DEFAULT_LOG_CATEGORY "net.buffer" + +namespace epee +{ +namespace net_utils +{ + +void buffer::append(const void *data, size_t sz) +{ + const size_t capacity = storage.capacity(); + const size_t avail = capacity - storage.size(); + + CHECK_AND_ASSERT_THROW_MES(storage.size() < std::numeric_limits<size_t>::max() - sz, "Too much data to append"); + + // decide when to move + if (sz > avail) + { + // we have to reallocate or move + const bool move = size() + sz <= capacity; + if (move) + { + const size_t bytes = storage.size() - offset; + NET_BUFFER_LOG("appending " << sz << " from " << size() << " by moving " << bytes << " from offset " << offset << " first (forced)"); + memmove(storage.data(), storage.data() + offset, bytes); + storage.resize(bytes); + offset = 0; + } + else + { + NET_BUFFER_LOG("appending " << sz << " from " << size() << " by reallocating"); + std::vector<uint8_t> new_storage; + size_t reserve = (((size() + sz) * 3 / 2) + 4095) & ~4095; + new_storage.reserve(reserve); + new_storage.resize(size()); + memcpy(new_storage.data(), storage.data() + offset, storage.size() - offset); + offset = 0; + std::swap(storage, new_storage); + } + } + else + { + // we have space already + if (size() <= 4096 && offset > 4096 * 16 && offset >= capacity / 2) + { + // we have little to move, and we're far enough into the buffer that it's probably a win to move anyway + const size_t pos = storage.size() - offset; + NET_BUFFER_LOG("appending " << sz << " from " << size() << " by moving " << pos << " from offset " << offset << " first (unforced)"); + memmove(storage.data(), storage.data() + offset, storage.size() - offset); + storage.resize(pos); + offset = 0; + } + else + { + NET_BUFFER_LOG("appending " << sz << " from " << size() << " by writing to existing capacity"); + } + } + + // add the new data + storage.insert(storage.end(), (const uint8_t*)data, (const uint8_t*)data + sz); + + NET_BUFFER_LOG("storage now " << offset << "/" << storage.size() << "/" << storage.capacity()); +} + +} +} diff --git a/contrib/epee/src/net_utils_base.cpp b/contrib/epee/src/net_utils_base.cpp index 354c3d2c3..263b344b4 100644 --- a/contrib/epee/src/net_utils_base.cpp +++ b/contrib/epee/src/net_utils_base.cpp @@ -1,7 +1,9 @@ #include "net/net_utils_base.h" -#include "string_tools.h" +#include <boost/uuid/uuid_io.hpp> + +#include "string_tools.h" #include "net/local_ip.h" namespace epee { namespace net_utils @@ -73,7 +75,7 @@ namespace epee { namespace net_utils std::string print_connection_context(const connection_context_base& ctx) { std::stringstream ss; - ss << ctx.m_remote_address.str() << " " << epee::string_tools::get_str_from_guid_a(ctx.m_connection_id) << (ctx.m_is_income ? " INC":" OUT"); + ss << ctx.m_remote_address.str() << " " << ctx.m_connection_id << (ctx.m_is_income ? " INC":" OUT"); return ss.str(); } diff --git a/src/blockchain_db/berkeleydb/db_bdb.cpp b/src/blockchain_db/berkeleydb/db_bdb.cpp index b27a00a69..60a7326f8 100644 --- a/src/blockchain_db/berkeleydb/db_bdb.cpp +++ b/src/blockchain_db/berkeleydb/db_bdb.cpp @@ -1632,7 +1632,7 @@ output_data_t BlockchainBDB::get_output_key(const uint64_t& global_index) const return v; } -output_data_t BlockchainBDB::get_output_key(const uint64_t& amount, const uint64_t& index) +output_data_t BlockchainBDB::get_output_key(const uint64_t& amount, const uint64_t& index) const { LOG_PRINT_L3("BlockchainBDB::" << __func__); check_open(); @@ -1641,7 +1641,7 @@ output_data_t BlockchainBDB::get_output_key(const uint64_t& amount, const uint64 return get_output_key(glob_index); } -tx_out_index BlockchainBDB::get_output_tx_and_index(const uint64_t& amount, const uint64_t& index) +tx_out_index BlockchainBDB::get_output_tx_and_index(const uint64_t& amount, const uint64_t& index) const { LOG_PRINT_L3("BlockchainBDB::" << __func__); std::vector < uint64_t > offsets; diff --git a/src/blockchain_db/blockchain_db.h b/src/blockchain_db/blockchain_db.h index f13aa0cae..fe61aabd8 100644 --- a/src/blockchain_db/blockchain_db.h +++ b/src/blockchain_db/blockchain_db.h @@ -1258,7 +1258,7 @@ public: * * @return the requested output data */ - virtual output_data_t get_output_key(const uint64_t& amount, const uint64_t& index) = 0; + virtual output_data_t get_output_key(const uint64_t& amount, const uint64_t& index, bool include_commitmemt = true) const = 0; /** * @brief gets an output's tx hash and index @@ -1310,7 +1310,7 @@ public: * @param offsets a list of amount-specific output indices * @param outputs return-by-reference a list of outputs' metadata */ - virtual void get_output_key(const epee::span<const uint64_t> &amounts, const std::vector<uint64_t> &offsets, std::vector<output_data_t> &outputs, bool allow_partial = false) = 0; + virtual void get_output_key(const epee::span<const uint64_t> &amounts, const std::vector<uint64_t> &offsets, std::vector<output_data_t> &outputs, bool allow_partial = false) const = 0; /* * FIXME: Need to check with git blame and ask what this does to @@ -1329,10 +1329,11 @@ public: * If an output cannot be found, the subclass should throw OUTPUT_DNE. * * @param tx_id a transaction ID + * @param n_txes how many txes to get data for, starting with tx_id * * @return a list of amount-specific output indices */ - virtual std::vector<uint64_t> get_tx_amount_output_indices(const uint64_t tx_id) const = 0; + virtual std::vector<std::vector<uint64_t>> get_tx_amount_output_indices(const uint64_t tx_id, size_t n_txes = 1) const = 0; /** * @brief check if a key image is stored as spent diff --git a/src/blockchain_db/lmdb/db_lmdb.cpp b/src/blockchain_db/lmdb/db_lmdb.cpp index acda777f9..2b5fa7fd9 100644 --- a/src/blockchain_db/lmdb/db_lmdb.cpp +++ b/src/blockchain_db/lmdb/db_lmdb.cpp @@ -1025,7 +1025,8 @@ void BlockchainLMDB::remove_tx_outputs(const uint64_t tx_id, const transaction& { LOG_PRINT_L3("BlockchainLMDB::" << __func__); - std::vector<uint64_t> amount_output_indices = get_tx_amount_output_indices(tx_id); + std::vector<std::vector<uint64_t>> amount_output_indices_set = get_tx_amount_output_indices(tx_id, 1); + const std::vector<uint64_t> &amount_output_indices = amount_output_indices_set.front(); if (amount_output_indices.empty()) { @@ -2535,7 +2536,7 @@ uint64_t BlockchainLMDB::get_num_outputs(const uint64_t& amount) const return num_elems; } -output_data_t BlockchainLMDB::get_output_key(const uint64_t& amount, const uint64_t& index) +output_data_t BlockchainLMDB::get_output_key(const uint64_t& amount, const uint64_t& index, bool include_commitmemt) const { LOG_PRINT_L3("BlockchainLMDB::" << __func__); check_open(); @@ -2562,7 +2563,8 @@ output_data_t BlockchainLMDB::get_output_key(const uint64_t& amount, const uint6 { const pre_rct_outkey *okp = (const pre_rct_outkey *)v.mv_data; memcpy(&ret, &okp->data, sizeof(pre_rct_output_data_t));; - ret.commitment = rct::zeroCommit(amount); + if (include_commitmemt) + ret.commitment = rct::zeroCommit(amount); } TXN_POSTFIX_RDONLY(); return ret; @@ -2604,7 +2606,7 @@ tx_out_index BlockchainLMDB::get_output_tx_and_index(const uint64_t& amount, con return indices[0]; } -std::vector<uint64_t> BlockchainLMDB::get_tx_amount_output_indices(const uint64_t tx_id) const +std::vector<std::vector<uint64_t>> BlockchainLMDB::get_tx_amount_output_indices(uint64_t tx_id, size_t n_txes) const { LOG_PRINT_L3("BlockchainLMDB::" << __func__); @@ -2613,35 +2615,40 @@ std::vector<uint64_t> BlockchainLMDB::get_tx_amount_output_indices(const uint64_ TXN_PREFIX_RDONLY(); RCURSOR(tx_outputs); - int result = 0; MDB_val_set(k_tx_id, tx_id); MDB_val v; - std::vector<uint64_t> amount_output_indices; + std::vector<std::vector<uint64_t>> amount_output_indices_set; + amount_output_indices_set.reserve(n_txes); - result = mdb_cursor_get(m_cur_tx_outputs, &k_tx_id, &v, MDB_SET); - if (result == MDB_NOTFOUND) - LOG_PRINT_L0("WARNING: Unexpected: tx has no amount indices stored in " - "tx_outputs, but it should have an empty entry even if it's a tx without " - "outputs"); - else if (result) - throw0(DB_ERROR(lmdb_error("DB error attempting to get data for tx_outputs[tx_index]", result).c_str())); + MDB_cursor_op op = MDB_SET; + while (n_txes-- > 0) + { + int result = mdb_cursor_get(m_cur_tx_outputs, &k_tx_id, &v, op); + if (result == MDB_NOTFOUND) + LOG_PRINT_L0("WARNING: Unexpected: tx has no amount indices stored in " + "tx_outputs, but it should have an empty entry even if it's a tx without " + "outputs"); + else if (result) + throw0(DB_ERROR(lmdb_error("DB error attempting to get data for tx_outputs[tx_index]", result).c_str())); - const uint64_t* indices = (const uint64_t*)v.mv_data; - int num_outputs = v.mv_size / sizeof(uint64_t); + op = MDB_NEXT; - amount_output_indices.reserve(num_outputs); - for (int i = 0; i < num_outputs; ++i) - { - // LOG_PRINT_L0("amount output index[" << 2*i << "]" << ": " << paired_indices[2*i] << " global output index: " << paired_indices[2*i+1]); - amount_output_indices.push_back(indices[i]); + const uint64_t* indices = (const uint64_t*)v.mv_data; + size_t num_outputs = v.mv_size / sizeof(uint64_t); + + amount_output_indices_set.resize(amount_output_indices_set.size() + 1); + std::vector<uint64_t> &amount_output_indices = amount_output_indices_set.back(); + amount_output_indices.reserve(num_outputs); + for (size_t i = 0; i < num_outputs; ++i) + { + amount_output_indices.push_back(indices[i]); + } } - indices = nullptr; TXN_POSTFIX_RDONLY(); - return amount_output_indices; + return amount_output_indices_set; } - bool BlockchainLMDB::has_key_image(const crypto::key_image& img) const { LOG_PRINT_L3("BlockchainLMDB::" << __func__); @@ -3242,7 +3249,7 @@ void BlockchainLMDB::get_output_tx_and_index_from_global(const std::vector<uint6 TXN_POSTFIX_RDONLY(); } -void BlockchainLMDB::get_output_key(const epee::span<const uint64_t> &amounts, const std::vector<uint64_t> &offsets, std::vector<output_data_t> &outputs, bool allow_partial) +void BlockchainLMDB::get_output_key(const epee::span<const uint64_t> &amounts, const std::vector<uint64_t> &offsets, std::vector<output_data_t> &outputs, bool allow_partial) const { if (amounts.size() != 1 && amounts.size() != offsets.size()) throw0(DB_ERROR("Invalid sizes of amounts and offets")); diff --git a/src/blockchain_db/lmdb/db_lmdb.h b/src/blockchain_db/lmdb/db_lmdb.h index b7b1b04d5..a60956ab1 100644 --- a/src/blockchain_db/lmdb/db_lmdb.h +++ b/src/blockchain_db/lmdb/db_lmdb.h @@ -242,8 +242,8 @@ public: virtual uint64_t get_num_outputs(const uint64_t& amount) const; - virtual output_data_t get_output_key(const uint64_t& amount, const uint64_t& index); - virtual void get_output_key(const epee::span<const uint64_t> &amounts, const std::vector<uint64_t> &offsets, std::vector<output_data_t> &outputs, bool allow_partial = false); + virtual output_data_t get_output_key(const uint64_t& amount, const uint64_t& index, bool include_commitmemt) const; + virtual void get_output_key(const epee::span<const uint64_t> &amounts, const std::vector<uint64_t> &offsets, std::vector<output_data_t> &outputs, bool allow_partial = false) const; virtual tx_out_index get_output_tx_and_index_from_global(const uint64_t& index) const; virtual void get_output_tx_and_index_from_global(const std::vector<uint64_t> &global_indices, @@ -252,7 +252,7 @@ public: virtual tx_out_index get_output_tx_and_index(const uint64_t& amount, const uint64_t& index) const; virtual void get_output_tx_and_index(const uint64_t& amount, const std::vector<uint64_t> &offsets, std::vector<tx_out_index> &indices) const; - virtual std::vector<uint64_t> get_tx_amount_output_indices(const uint64_t tx_id) const; + virtual std::vector<std::vector<uint64_t>> get_tx_amount_output_indices(const uint64_t tx_id, size_t n_txes) const; virtual bool has_key_image(const crypto::key_image& img) const; diff --git a/src/blockchain_utilities/blockchain_ancestry.cpp b/src/blockchain_utilities/blockchain_ancestry.cpp index e01a8892c..a64ce160a 100644 --- a/src/blockchain_utilities/blockchain_ancestry.cpp +++ b/src/blockchain_utilities/blockchain_ancestry.cpp @@ -51,6 +51,8 @@ using namespace epee; using namespace cryptonote; static bool stop_requested = false; +static uint64_t cached_txes = 0, cached_blocks = 0, cached_outputs = 0, total_txes = 0, total_blocks = 0, total_outputs = 0; +static bool opt_cache_outputs = false, opt_cache_txes = false, opt_cache_blocks = false; struct ancestor { @@ -137,6 +139,8 @@ struct ancestry_state_t std::unordered_map<crypto::hash, ::tx_data_t> tx_cache; std::vector<cryptonote::block> block_cache; + ancestry_state_t(): height(0) {} + template <typename t_archive> void serialize(t_archive &a, const unsigned int ver) { a & height; @@ -219,6 +223,113 @@ static std::unordered_set<ancestor> get_ancestry(const std::unordered_map<crypto return i->second; } +static bool get_block_from_height(ancestry_state_t &state, BlockchainDB *db, uint64_t height, cryptonote::block &b) +{ + ++total_blocks; + if (state.block_cache.size() > height && !state.block_cache[height].miner_tx.vin.empty()) + { + ++cached_blocks; + b = state.block_cache[height]; + return true; + } + cryptonote::blobdata bd = db->get_block_blob_from_height(height); + if (!cryptonote::parse_and_validate_block_from_blob(bd, b)) + { + LOG_PRINT_L0("Bad block from db"); + return false; + } + if (opt_cache_blocks) + { + state.block_cache.resize(height + 1); + state.block_cache[height] = b; + } + return true; +} + +static bool get_transaction(ancestry_state_t &state, BlockchainDB *db, const crypto::hash &txid, ::tx_data_t &tx_data) +{ + std::unordered_map<crypto::hash, ::tx_data_t>::const_iterator i = state.tx_cache.find(txid); + ++total_txes; + if (i != state.tx_cache.end()) + { + ++cached_txes; + tx_data = i->second; + return true; + } + + cryptonote::blobdata bd; + if (!db->get_pruned_tx_blob(txid, bd)) + { + LOG_PRINT_L0("Failed to get txid " << txid << " from db"); + return false; + } + cryptonote::transaction tx; + if (!cryptonote::parse_and_validate_tx_base_from_blob(bd, tx)) + { + LOG_PRINT_L0("Bad tx: " << txid); + return false; + } + tx_data = ::tx_data_t(tx); + if (opt_cache_txes) + state.tx_cache.insert(std::make_pair(txid, tx_data)); + return true; +} + +static bool get_output_txid(ancestry_state_t &state, BlockchainDB *db, uint64_t amount, uint64_t offset, crypto::hash &txid) +{ + ++total_outputs; + std::unordered_map<ancestor, crypto::hash>::const_iterator i = state.output_cache.find({amount, offset}); + if (i != state.output_cache.end()) + { + ++cached_outputs; + txid = i->second; + return true; + } + + const output_data_t od = db->get_output_key(amount, offset, false); + cryptonote::block b; + if (!get_block_from_height(state, db, od.height, b)) + return false; + + for (size_t out = 0; out < b.miner_tx.vout.size(); ++out) + { + if (b.miner_tx.vout[out].target.type() == typeid(cryptonote::txout_to_key)) + { + const auto &txout = boost::get<cryptonote::txout_to_key>(b.miner_tx.vout[out].target); + if (txout.key == od.pubkey) + { + txid = cryptonote::get_transaction_hash(b.miner_tx); + if (opt_cache_outputs) + state.output_cache.insert(std::make_pair(ancestor{amount, offset}, txid)); + return true; + } + } + else + { + LOG_PRINT_L0("Bad vout type in txid " << cryptonote::get_transaction_hash(b.miner_tx)); + return false; + } + } + for (const crypto::hash &block_txid: b.tx_hashes) + { + ::tx_data_t tx_data3; + if (!get_transaction(state, db, block_txid, tx_data3)) + return false; + + for (size_t out = 0; out < tx_data3.vout.size(); ++out) + { + if (tx_data3.vout[out] == od.pubkey) + { + txid = block_txid; + if (opt_cache_outputs) + state.output_cache.insert(std::make_pair(ancestor{amount, offset}, txid)); + return true; + } + } + } + return false; +} + int main(int argc, char* argv[]) { TRY_ENTRY(); @@ -243,12 +354,13 @@ int main(int argc, char* argv[]) "database", available_dbs.c_str(), default_db_type }; const command_line::arg_descriptor<std::string> arg_txid = {"txid", "Get ancestry for this txid", ""}; + const command_line::arg_descriptor<std::string> arg_output = {"output", "Get ancestry for this output (amount/offset format)", ""}; const command_line::arg_descriptor<uint64_t> arg_height = {"height", "Get ancestry for all txes at this height", 0}; - const command_line::arg_descriptor<bool> arg_all = {"all", "Include the whole chain", false}; + const command_line::arg_descriptor<bool> arg_refresh = {"refresh", "Refresh the whole chain first", false}; const command_line::arg_descriptor<bool> arg_cache_outputs = {"cache-outputs", "Cache outputs (memory hungry)", false}; const command_line::arg_descriptor<bool> arg_cache_txes = {"cache-txes", "Cache txes (memory hungry)", false}; const command_line::arg_descriptor<bool> arg_cache_blocks = {"cache-blocks", "Cache blocks (memory hungry)", false}; - const command_line::arg_descriptor<bool> arg_include_coinbase = {"include-coinbase", "Including coinbase tx", false}; + const command_line::arg_descriptor<bool> arg_include_coinbase = {"include-coinbase", "Including coinbase tx in per height average", false}; const command_line::arg_descriptor<bool> arg_show_cache_stats = {"show-cache-stats", "Show cache statistics", false}; command_line::add_arg(desc_cmd_sett, cryptonote::arg_data_dir); @@ -257,8 +369,9 @@ int main(int argc, char* argv[]) command_line::add_arg(desc_cmd_sett, arg_log_level); command_line::add_arg(desc_cmd_sett, arg_database); command_line::add_arg(desc_cmd_sett, arg_txid); + command_line::add_arg(desc_cmd_sett, arg_output); command_line::add_arg(desc_cmd_sett, arg_height); - command_line::add_arg(desc_cmd_sett, arg_all); + command_line::add_arg(desc_cmd_sett, arg_refresh); command_line::add_arg(desc_cmd_sett, arg_cache_outputs); command_line::add_arg(desc_cmd_sett, arg_cache_txes); command_line::add_arg(desc_cmd_sett, arg_cache_blocks); @@ -300,20 +413,22 @@ int main(int argc, char* argv[]) bool opt_stagenet = command_line::get_arg(vm, cryptonote::arg_stagenet_on); network_type net_type = opt_testnet ? TESTNET : opt_stagenet ? STAGENET : MAINNET; std::string opt_txid_string = command_line::get_arg(vm, arg_txid); + std::string opt_output_string = command_line::get_arg(vm, arg_output); uint64_t opt_height = command_line::get_arg(vm, arg_height); - bool opt_all = command_line::get_arg(vm, arg_all); - bool opt_cache_outputs = command_line::get_arg(vm, arg_cache_outputs); - bool opt_cache_txes = command_line::get_arg(vm, arg_cache_txes); - bool opt_cache_blocks = command_line::get_arg(vm, arg_cache_blocks); + bool opt_refresh = command_line::get_arg(vm, arg_refresh); + opt_cache_outputs = command_line::get_arg(vm, arg_cache_outputs); + opt_cache_txes = command_line::get_arg(vm, arg_cache_txes); + opt_cache_blocks = command_line::get_arg(vm, arg_cache_blocks); bool opt_include_coinbase = command_line::get_arg(vm, arg_include_coinbase); bool opt_show_cache_stats = command_line::get_arg(vm, arg_show_cache_stats); - if ((!opt_txid_string.empty()) + !!opt_height + !!opt_all > 1) + if ((!opt_txid_string.empty()) + !!opt_height + !opt_output_string.empty() > 1) { - std::cerr << "Only one of --txid, --height and --all can be given" << std::endl; + std::cerr << "Only one of --txid, --height, --output can be given" << std::endl; return 1; } crypto::hash opt_txid = crypto::null_hash; + uint64_t output_amount = 0, output_offset = 0; if (!opt_txid_string.empty()) { if (!epee::string_tools::hex_to_pod(opt_txid_string, opt_txid)) @@ -322,6 +437,14 @@ int main(int argc, char* argv[]) return 1; } } + else if (!opt_output_string.empty()) + { + if (sscanf(opt_output_string.c_str(), "%" SCNu64 "/%" SCNu64, &output_amount, &output_offset) != 2) + { + std::cerr << "Invalid output" << std::endl; + return 1; + } + } std::string db_type = command_line::get_arg(vm, arg_database); if (!cryptonote::blockchain_valid_db_type(db_type)) @@ -372,37 +495,36 @@ int main(int argc, char* argv[]) std::vector<crypto::hash> start_txids; - // forward method - if (opt_all) - { - uint64_t cached_txes = 0, cached_blocks = 0, cached_outputs = 0, total_txes = 0, total_blocks = 0, total_outputs = 0; - ancestry_state_t state; + ancestry_state_t state; - const std::string state_file_path = (boost::filesystem::path(opt_data_dir) / "ancestry-state.bin").string(); - LOG_PRINT_L0("Loading state data from " << state_file_path); - std::ifstream state_data_in; - state_data_in.open(state_file_path, std::ios_base::binary | std::ios_base::in); - if (!state_data_in.fail()) + const std::string state_file_path = (boost::filesystem::path(opt_data_dir) / "ancestry-state.bin").string(); + LOG_PRINT_L0("Loading state data from " << state_file_path); + std::ifstream state_data_in; + state_data_in.open(state_file_path, std::ios_base::binary | std::ios_base::in); + if (!state_data_in.fail()) + { + try { - try - { - boost::archive::portable_binary_iarchive a(state_data_in); - a >> state; - } - catch (const std::exception &e) - { - MERROR("Failed to load state data from " << state_file_path << ", restarting from scratch"); - state = ancestry_state_t(); - } - state_data_in.close(); + boost::archive::portable_binary_iarchive a(state_data_in); + a >> state; + } + catch (const std::exception &e) + { + MERROR("Failed to load state data from " << state_file_path << ", restarting from scratch"); + state = ancestry_state_t(); } + state_data_in.close(); + } - tools::signal_handler::install([](int type) { - stop_requested = true; - }); + tools::signal_handler::install([](int type) { + stop_requested = true; + }); + // forward method + const uint64_t db_height = db->height(); + if (opt_refresh) + { MINFO("Starting from height " << state.height); - const uint64_t db_height = db->height(); state.block_cache.reserve(db_height); for (uint64_t h = state.height; h < db_height; ++h) { @@ -464,113 +586,20 @@ int main(int argc, char* argv[]) { for (size_t ring = 0; ring < tx_data.vin.size(); ++ring) { - if (1) + const uint64_t amount = tx_data.vin[ring].first; + const std::vector<uint64_t> &absolute_offsets = tx_data.vin[ring].second; + for (uint64_t offset: absolute_offsets) { - const uint64_t amount = tx_data.vin[ring].first; - const std::vector<uint64_t> &absolute_offsets = tx_data.vin[ring].second; - for (uint64_t offset: absolute_offsets) + add_ancestry(state.ancestry, txid, ancestor{amount, offset}); + // find the tx which created this output + bool found = false; + crypto::hash output_txid; + if (!get_output_txid(state, db, amount, offset, output_txid)) { - const output_data_t od = db->get_output_key(amount, offset); - add_ancestry(state.ancestry, txid, ancestor{amount, offset}); - cryptonote::block b; - ++total_blocks; - if (state.block_cache.size() > od.height && !state.block_cache[od.height].miner_tx.vin.empty()) - { - ++cached_blocks; - b = state.block_cache[od.height]; - } - else - { - cryptonote::blobdata bd = db->get_block_blob_from_height(od.height); - if (!cryptonote::parse_and_validate_block_from_blob(bd, b)) - { - LOG_PRINT_L0("Bad block from db"); - return 1; - } - if (opt_cache_blocks) - { - state.block_cache.resize(od.height + 1); - state.block_cache[od.height] = b; - } - } - // find the tx which created this output - bool found = false; - std::unordered_map<ancestor, crypto::hash>::const_iterator i = state.output_cache.find({amount, offset}); - ++total_outputs; - if (i != state.output_cache.end()) - { - ++cached_outputs; - add_ancestry(state.ancestry, txid, get_ancestry(state.ancestry, i->second)); - found = true; - } - else for (size_t out = 0; out < b.miner_tx.vout.size(); ++out) - { - if (b.miner_tx.vout[out].target.type() == typeid(cryptonote::txout_to_key)) - { - const auto &txout = boost::get<cryptonote::txout_to_key>(b.miner_tx.vout[out].target); - if (txout.key == od.pubkey) - { - found = true; - add_ancestry(state.ancestry, txid, get_ancestry(state.ancestry, cryptonote::get_transaction_hash(b.miner_tx))); - if (opt_cache_outputs) - state.output_cache.insert(std::make_pair(ancestor{amount, offset}, cryptonote::get_transaction_hash(b.miner_tx))); - break; - } - } - else - { - LOG_PRINT_L0("Bad vout type in txid " << cryptonote::get_transaction_hash(b.miner_tx)); - return 1; - } - } - for (const crypto::hash &block_txid: b.tx_hashes) - { - if (found) - break; - ::tx_data_t tx_data2; - std::unordered_map<crypto::hash, ::tx_data_t>::const_iterator i = state.tx_cache.find(block_txid); - ++total_txes; - if (i != state.tx_cache.end()) - { - ++cached_txes; - tx_data2 = i->second; - } - else - { - cryptonote::blobdata bd; - if (!db->get_pruned_tx_blob(block_txid, bd)) - { - LOG_PRINT_L0("Failed to get txid " << block_txid << " from db"); - return 1; - } - cryptonote::transaction tx; - if (!cryptonote::parse_and_validate_tx_base_from_blob(bd, tx)) - { - LOG_PRINT_L0("Bad tx: " << block_txid); - return 1; - } - tx_data2 = ::tx_data_t(tx); - if (opt_cache_txes) - state.tx_cache.insert(std::make_pair(block_txid, tx_data2)); - } - for (size_t out = 0; out < tx_data2.vout.size(); ++out) - { - if (tx_data2.vout[out] == od.pubkey) - { - found = true; - add_ancestry(state.ancestry, txid, get_ancestry(state.ancestry, block_txid)); - if (opt_cache_outputs) - state.output_cache.insert(std::make_pair(ancestor{amount, offset}, block_txid)); - break; - } - } - } - if (!found) - { - LOG_PRINT_L0("Output originating transaction not found"); - return 1; - } + LOG_PRINT_L0("Output originating transaction not found"); + return 1; } + add_ancestry(state.ancestry, txid, get_ancestry(state.ancestry, output_txid)); } } } @@ -581,10 +610,6 @@ int main(int argc, char* argv[]) if (!txids.empty()) { std::string stats_msg; - if (opt_show_cache_stats) - stats_msg = std::string(", cache: txes ") + std::to_string(cached_txes*100./total_txes) - + ", blocks " + std::to_string(cached_blocks*100./total_blocks) + ", outputs " - + std::to_string(cached_outputs*100./total_outputs); MINFO("Height " << h << ": " << (block_ancestry_size / txids.size()) << " average over " << txids.size() << stats_msg); } state.height = h; @@ -608,14 +633,30 @@ int main(int argc, char* argv[]) } state_data_out.close(); } - - goto done; + } + else + { + if (state.height < db_height) + { + MWARNING("The state file is only built up to height " << state.height << ", but the blockchain reached height " << db_height); + MWARNING("You may want to run with --refresh if you want to get ancestry for newer data"); + } } if (!opt_txid_string.empty()) { start_txids.push_back(opt_txid); } + else if (!opt_output_string.empty()) + { + crypto::hash txid; + if (!get_output_txid(state, db, output_amount, output_offset, txid)) + { + LOG_PRINT_L0("Output not found in db"); + return 1; + } + start_txids.push_back(txid); + } else { const cryptonote::blobdata bd = db->get_block_blob_from_height(opt_height); @@ -648,108 +689,40 @@ int main(int argc, char* argv[]) const crypto::hash txid = txids.front(); txids.pop_front(); - cryptonote::blobdata bd; - if (!db->get_pruned_tx_blob(txid, bd)) - { - LOG_PRINT_L0("Failed to get txid " << txid << " from db"); - return 1; - } - cryptonote::transaction tx; - if (!cryptonote::parse_and_validate_tx_base_from_blob(bd, tx)) - { - LOG_PRINT_L0("Bad tx: " << txid); + if (stop_requested) + goto done; + + ::tx_data_t tx_data2; + if (!get_transaction(state, db, txid, tx_data2)) return 1; - } - const bool coinbase = tx.vin.size() == 1 && tx.vin[0].type() == typeid(cryptonote::txin_gen); + + const bool coinbase = tx_data2.coinbase; if (coinbase) continue; - for (size_t ring = 0; ring < tx.vin.size(); ++ring) + for (size_t ring = 0; ring < tx_data2.vin.size(); ++ring) { - if (tx.vin[ring].type() == typeid(cryptonote::txin_to_key)) { - const cryptonote::txin_to_key &txin = boost::get<cryptonote::txin_to_key>(tx.vin[ring]); - const uint64_t amount = txin.amount; - auto absolute_offsets = cryptonote::relative_output_offsets_to_absolute(txin.key_offsets); + const uint64_t amount = tx_data2.vin[ring].first; + auto absolute_offsets = tx_data2.vin[ring].second; for (uint64_t offset: absolute_offsets) { add_ancestor(ancestry, amount, offset); - const output_data_t od = db->get_output_key(amount, offset); - bd = db->get_block_blob_from_height(od.height); - cryptonote::block b; - if (!cryptonote::parse_and_validate_block_from_blob(bd, b)) - { - LOG_PRINT_L0("Bad block from db"); - return 1; - } + // find the tx which created this output bool found = false; - for (size_t out = 0; out < b.miner_tx.vout.size(); ++out) - { - if (b.miner_tx.vout[out].target.type() == typeid(cryptonote::txout_to_key)) - { - const auto &txout = boost::get<cryptonote::txout_to_key>(b.miner_tx.vout[out].target); - if (txout.key == od.pubkey) - { - found = true; - txids.push_back(cryptonote::get_transaction_hash(b.miner_tx)); - MDEBUG("adding txid: " << cryptonote::get_transaction_hash(b.miner_tx)); - break; - } - } - else - { - LOG_PRINT_L0("Bad vout type in txid " << cryptonote::get_transaction_hash(b.miner_tx)); - return 1; - } - } - for (const crypto::hash &block_txid: b.tx_hashes) - { - if (found) - break; - if (!db->get_pruned_tx_blob(block_txid, bd)) - { - LOG_PRINT_L0("Failed to get txid " << block_txid << " from db"); - return 1; - } - cryptonote::transaction tx2; - if (!cryptonote::parse_and_validate_tx_base_from_blob(bd, tx2)) - { - LOG_PRINT_L0("Bad tx: " << block_txid); - return 1; - } - for (size_t out = 0; out < tx2.vout.size(); ++out) - { - if (tx2.vout[out].target.type() == typeid(cryptonote::txout_to_key)) - { - const auto &txout = boost::get<cryptonote::txout_to_key>(tx2.vout[out].target); - if (txout.key == od.pubkey) - { - found = true; - txids.push_back(block_txid); - MDEBUG("adding txid: " << block_txid); - break; - } - } - else - { - LOG_PRINT_L0("Bad vout type in txid " << block_txid); - return 1; - } - } - } - if (!found) + crypto::hash output_txid; + if (!get_output_txid(state, db, amount, offset, output_txid)) { LOG_PRINT_L0("Output originating transaction not found"); return 1; } + + add_ancestry(state.ancestry, txid, get_ancestry(state.ancestry, output_txid)); + txids.push_back(output_txid); + MDEBUG("adding txid: " << output_txid); } } - else - { - LOG_PRINT_L0("Bad vin type in txid " << txid); - return 1; - } } } @@ -762,6 +735,13 @@ int main(int argc, char* argv[]) done: core_storage->deinit(); + + if (opt_show_cache_stats) + MINFO("cache: txes " << std::to_string(cached_txes*100./total_txes) + << "%, blocks " << std::to_string(cached_blocks*100./total_blocks) + << "%, outputs " << std::to_string(cached_outputs*100./total_outputs) + << "%"); + return 0; CATCH_ENTRY("Depth query error", 1); diff --git a/src/blockchain_utilities/blockchain_blackball.cpp b/src/blockchain_utilities/blockchain_blackball.cpp index 73819bd25..8b007e901 100644 --- a/src/blockchain_utilities/blockchain_blackball.cpp +++ b/src/blockchain_utilities/blockchain_blackball.cpp @@ -1131,7 +1131,7 @@ int main(int argc, char* argv[]) return 1; } - mlog_configure(mlog_get_default_log_path("monero-blockchain-find-spent-outputs.log"), true); + mlog_configure(mlog_get_default_log_path("monero-blockchain-mark-spent-outputs.log"), true); if (!command_line::is_arg_defaulted(vm, arg_log_level)) mlog_set_log(command_line::get_arg(vm, arg_log_level).c_str()); else diff --git a/src/blocks/CMakeLists.txt b/src/blocks/CMakeLists.txt index 30d85adbf..ff48af6dc 100644 --- a/src/blocks/CMakeLists.txt +++ b/src/blocks/CMakeLists.txt @@ -39,7 +39,7 @@ foreach(BLOB_NAME checkpoints testnet_blocks stagenet_blocks) cd ${CMAKE_CURRENT_BINARY_DIR} && echo "'#include\t<stddef.h>'" > ${OUTPUT_C_SOURCE} && echo "'const\tunsigned\tchar\t${BLOB_NAME}[]={'" >> ${OUTPUT_C_SOURCE} && - od -v -An -tu1 ${CMAKE_CURRENT_SOURCE_DIR}/${INPUT_DAT_FILE} | sed -e "'s/[0-9]\\{1,\\}/&,/g'" -e "'$$s/.$$//'" >> ${OUTPUT_C_SOURCE} && + od -v -An -tx1 ${CMAKE_CURRENT_SOURCE_DIR}/${INPUT_DAT_FILE} | sed -e "'s/[0-9a-fA-F]\\{1,\\}/0x&,/g'" -e "'$$s/.$$//'" >> ${OUTPUT_C_SOURCE} && echo "'};'" >> ${OUTPUT_C_SOURCE} && echo "'const\tsize_t\t${BLOB_NAME}_len\t=\tsizeof(${BLOB_NAME});'" >> ${OUTPUT_C_SOURCE} ) diff --git a/src/cryptonote_basic/hardfork.cpp b/src/cryptonote_basic/hardfork.cpp index 87a394918..447d79aee 100644 --- a/src/cryptonote_basic/hardfork.cpp +++ b/src/cryptonote_basic/hardfork.cpp @@ -222,7 +222,6 @@ bool HardFork::reorganize_from_block_height(uint64_t height) if (height >= db.height()) return false; - db.set_batch_transactions(true); bool stop_batch = db.batch_start(); versions.clear(); @@ -306,6 +305,29 @@ bool HardFork::rescan_from_chain_height(uint64_t height) return rescan_from_block_height(height - 1); } +void HardFork::on_block_popped(uint64_t nblocks) +{ + CHECK_AND_ASSERT_THROW_MES(nblocks > 0, "nblocks must be greater than 0"); + + CRITICAL_REGION_LOCAL(lock); + + const uint64_t new_chain_height = db.height(); + const uint64_t old_chain_height = new_chain_height + nblocks; + uint8_t version; + uint64_t height; + for (height = old_chain_height - 1; height >= new_chain_height; --height) + { + versions.pop_back(); + version = db.get_hard_fork_version(height); + versions.push_front(version); + } + + // does not take voting into account + for (current_fork_index = heights.size() - 1; current_fork_index > 0; --current_fork_index) + if (height >= heights[current_fork_index].height) + break; +} + int HardFork::get_voted_fork_index(uint64_t height) const { CRITICAL_REGION_LOCAL(lock); diff --git a/src/cryptonote_basic/hardfork.h b/src/cryptonote_basic/hardfork.h index a63a66976..a3fc25dfa 100644 --- a/src/cryptonote_basic/hardfork.h +++ b/src/cryptonote_basic/hardfork.h @@ -150,6 +150,16 @@ namespace cryptonote bool reorganize_from_chain_height(uint64_t height); /** + * @brief called when one or more blocks are popped from the blockchain + * + * The current fork will be updated by looking up the db, + * which is much cheaper than recomputing everything + * + * @param new_chain_height the height of the chain after popping + */ + void on_block_popped(uint64_t new_chain_height); + + /** * @brief returns current state at the given time * * Based on the approximate time of the last known hard fork, diff --git a/src/cryptonote_core/blockchain.cpp b/src/cryptonote_core/blockchain.cpp index bbac20eaa..a108124a8 100644 --- a/src/cryptonote_core/blockchain.cpp +++ b/src/cryptonote_core/blockchain.cpp @@ -173,7 +173,8 @@ Blockchain::Blockchain(tx_memory_pool& tx_pool) : //------------------------------------------------------------------ Blockchain::~Blockchain() { - deinit(); + try { deinit(); } + catch (const std::exception &e) { /* ignore */ } } //------------------------------------------------------------------ bool Blockchain::have_tx(const crypto::hash &id) const @@ -641,6 +642,9 @@ block Blockchain::pop_block_from_blockchain() throw; } + // make sure the hard fork object updates its current version + m_hardfork->on_block_popped(1); + // return transactions from popped block to the tx_pool for (transaction& tx : popped_txs) { @@ -651,12 +655,7 @@ block Blockchain::pop_block_from_blockchain() // FIXME: HardFork // Besides the below, popping a block should also remove the last entry // in hf_versions. - // - // FIXME: HardFork - // This is not quite correct, as we really want to add the txes - // to the pool based on the version determined after all blocks - // are popped. - uint8_t version = get_current_hard_fork_version(); + uint8_t version = get_ideal_hard_fork_version(m_db->height()); // We assume that if they were in a block, the transactions are already // known to the network as a whole. However, if we had mined that block, @@ -921,8 +920,10 @@ difficulty_type Blockchain::get_difficulty_for_next_block() //------------------------------------------------------------------ std::vector<time_t> Blockchain::get_last_block_timestamps(unsigned int blocks) const { - std::vector<time_t> timestamps(blocks); uint64_t height = m_db->height(); + if (blocks > height) + blocks = height; + std::vector<time_t> timestamps(blocks); while (blocks--) timestamps[blocks] = m_db->get_block_timestamp(height - blocks - 1); return timestamps; @@ -1199,7 +1200,7 @@ bool Blockchain::validate_miner_transaction(const block& b, size_t cumulative_bl return false; } // From hard fork 2, we allow a miner to claim less block reward than is allowed, in case a miner wants less dust - if (m_hardfork->get_current_version() < 2) + if (version < 2) { if(base_reward + fee != money_in_use) { @@ -2298,7 +2299,7 @@ bool Blockchain::check_for_double_spend(const transaction& tx, key_images_contai return true; } //------------------------------------------------------------------ -bool Blockchain::get_tx_outputs_gindexs(const crypto::hash& tx_id, std::vector<uint64_t>& indexs) const +bool Blockchain::get_tx_outputs_gindexs(const crypto::hash& tx_id, size_t n_txes, std::vector<std::vector<uint64_t>>& indexs) const { LOG_PRINT_L3("Blockchain::" << __func__); CRITICAL_REGION_LOCAL(m_blockchain_lock); @@ -2308,16 +2309,25 @@ bool Blockchain::get_tx_outputs_gindexs(const crypto::hash& tx_id, std::vector<u MERROR_VER("get_tx_outputs_gindexs failed to find transaction with id = " << tx_id); return false; } + indexs = m_db->get_tx_amount_output_indices(tx_index, n_txes); + CHECK_AND_ASSERT_MES(n_txes == indexs.size(), false, "Wrong indexs size"); - // get amount output indexes, currently referred to in parts as "output global indices", but they are actually specific to amounts - indexs = m_db->get_tx_amount_output_indices(tx_index); - if (indexs.empty()) + return true; +} +//------------------------------------------------------------------ +bool Blockchain::get_tx_outputs_gindexs(const crypto::hash& tx_id, std::vector<uint64_t>& indexs) const +{ + LOG_PRINT_L3("Blockchain::" << __func__); + CRITICAL_REGION_LOCAL(m_blockchain_lock); + uint64_t tx_index; + if (!m_db->tx_exists(tx_id, tx_index)) { - // empty indexs is only valid if the vout is empty, which is legal but rare - cryptonote::transaction tx = m_db->get_tx(tx_id); - CHECK_AND_ASSERT_MES(tx.vout.empty(), false, "internal error: global indexes for transaction " << tx_id << " is empty, and tx vout is not"); + MERROR_VER("get_tx_outputs_gindexs failed to find transaction with id = " << tx_id); + return false; } - + std::vector<std::vector<uint64_t>> indices = m_db->get_tx_amount_output_indices(tx_index, 1); + CHECK_AND_ASSERT_MES(indices.size() == 1, false, "Wrong indices size"); + indexs = indices.front(); return true; } //------------------------------------------------------------------ @@ -3842,33 +3852,11 @@ bool Blockchain::cleanup_handle_incoming_blocks(bool force_sync) } //------------------------------------------------------------------ -void Blockchain::output_scan_worker(const uint64_t amount, const std::vector<uint64_t> &offsets, std::vector<output_data_t> &outputs, const std::vector<output_data_t> &extra_tx_map) const +void Blockchain::output_scan_worker(const uint64_t amount, const std::vector<uint64_t> &offsets, std::vector<output_data_t> &outputs) const { try { m_db->get_output_key(epee::span<const uint64_t>(&amount, 1), offsets, outputs, true); - if (outputs.size() < offsets.size()) - { - const uint64_t n_outputs = m_db->get_num_outputs(amount); - for (size_t i = outputs.size(); i < offsets.size(); ++i) - { - uint64_t idx = offsets[i]; - if (idx < n_outputs) - { - MWARNING("Index " << idx << " not found in db for amount " << amount << ", but it is less than the number of entries"); - break; - } - else if (idx < n_outputs + extra_tx_map.size()) - { - outputs.push_back(extra_tx_map[idx - n_outputs]); - } - else - { - MWARNING("missed " << amount << "/" << idx << " in " << extra_tx_map.size() << " (chain " << n_outputs << ")"); - break; - } - } - } } catch (const std::exception& e) { @@ -3983,34 +3971,6 @@ uint64_t Blockchain::prevalidate_block_hashes(uint64_t height, const std::vector // vs [k_image, output_keys] (m_scan_table). This is faster because it takes advantage of bulk queries // and is threaded if possible. The table (m_scan_table) will be used later when querying output // keys. -static bool update_output_map(std::map<uint64_t, std::vector<output_data_t>> &extra_tx_map, const transaction &tx, uint64_t height, bool miner) -{ - MTRACE("Blockchain::" << __func__); - for (size_t i = 0; i < tx.vout.size(); ++i) - { - const auto &out = tx.vout[i]; - if (out.target.type() != typeid(txout_to_key)) - continue; - const txout_to_key &out_to_key = boost::get<txout_to_key>(out.target); - rct::key commitment; - uint64_t amount = out.amount; - if (miner && tx.version == 2) - { - commitment = rct::zeroCommit(amount); - amount = 0; - } - else if (tx.version > 1) - { - CHECK_AND_ASSERT_MES(i < tx.rct_signatures.outPk.size(), false, "Invalid outPk size"); - commitment = tx.rct_signatures.outPk[i].mask; - } - else - commitment = rct::zero(); - extra_tx_map[amount].push_back(output_data_t{out_to_key.key, tx.unlock_time, height, commitment}); - } - return true; -} - bool Blockchain::prepare_handle_incoming_blocks(const std::vector<block_complete_entry> &blocks_entry) { MTRACE("Blockchain::" << __func__); @@ -4179,7 +4139,7 @@ bool Blockchain::prepare_handle_incoming_blocks(const std::vector<block_complete // [input] stores all absolute_offsets for each amount std::map<uint64_t, std::vector<uint64_t>> offset_map; // [output] stores all output_data_t for each absolute_offset - std::map<uint64_t, std::vector<output_data_t>> tx_map, extra_tx_map; + std::map<uint64_t, std::vector<output_data_t>> tx_map; std::vector<std::pair<cryptonote::transaction, crypto::hash>> txes(total_txs); #define SCAN_TABLE_QUIT(m) \ @@ -4196,8 +4156,6 @@ bool Blockchain::prepare_handle_incoming_blocks(const std::vector<block_complete if (m_cancel) return false; - if (!update_output_map(extra_tx_map, blocks[block_index].miner_tx, height + block_index, true)) - SCAN_TABLE_QUIT("Error building extra tx map."); for (const auto &tx_blob : entry.txs) { if (tx_index >= txes.size()) @@ -4256,8 +4214,6 @@ bool Blockchain::prepare_handle_incoming_blocks(const std::vector<block_complete offset_map[in_to_key.amount].push_back(offset); } - if (!update_output_map(extra_tx_map, tx, height + block_index, false)) - SCAN_TABLE_QUIT("Error building extra tx map."); } ++block_index; } @@ -4282,7 +4238,7 @@ bool Blockchain::prepare_handle_incoming_blocks(const std::vector<block_complete for (size_t i = 0; i < amounts.size(); i++) { uint64_t amount = amounts[i]; - tpool.submit(&waiter, boost::bind(&Blockchain::output_scan_worker, this, amount, std::cref(offset_map[amount]), std::ref(tx_map[amount]), std::cref(extra_tx_map[amount])), true); + tpool.submit(&waiter, boost::bind(&Blockchain::output_scan_worker, this, amount, std::cref(offset_map[amount]), std::ref(tx_map[amount])), true); } waiter.wait(&tpool); } @@ -4291,7 +4247,7 @@ bool Blockchain::prepare_handle_incoming_blocks(const std::vector<block_complete for (size_t i = 0; i < amounts.size(); i++) { uint64_t amount = amounts[i]; - output_scan_worker(amount, offset_map[amount], tx_map[amount], extra_tx_map[amount]); + output_scan_worker(amount, offset_map[amount], tx_map[amount]); } } diff --git a/src/cryptonote_core/blockchain.h b/src/cryptonote_core/blockchain.h index 67bccc6c6..5a1c4b9ad 100644 --- a/src/cryptonote_core/blockchain.h +++ b/src/cryptonote_core/blockchain.h @@ -521,10 +521,12 @@ namespace cryptonote * * @param tx_id the hash of the transaction to fetch indices for * @param indexs return-by-reference the global indices for the transaction's outputs + * @param n_txes how many txes in a row to get results for * * @return false if the transaction does not exist, or if no indices are found, otherwise true */ bool get_tx_outputs_gindexs(const crypto::hash& tx_id, std::vector<uint64_t>& indexs) const; + bool get_tx_outputs_gindexs(const crypto::hash& tx_id, size_t n_txes, std::vector<std::vector<uint64_t>>& indexs) const; /** * @brief stores the blockchain @@ -923,7 +925,7 @@ namespace cryptonote * @param outputs return-by-reference the outputs collected */ void output_scan_worker(const uint64_t amount,const std::vector<uint64_t> &offsets, - std::vector<output_data_t> &outputs, const std::vector<output_data_t> &extra_tx_map) const; + std::vector<output_data_t> &outputs) const; /** * @brief computes the "short" and "long" hashes for a set of blocks diff --git a/src/cryptonote_core/cryptonote_core.cpp b/src/cryptonote_core/cryptonote_core.cpp index f3249ea92..1fa6969a6 100644 --- a/src/cryptonote_core/cryptonote_core.cpp +++ b/src/cryptonote_core/cryptonote_core.cpp @@ -1220,6 +1220,11 @@ namespace cryptonote return m_blockchain_storage.get_tx_outputs_gindexs(tx_id, indexs); } //----------------------------------------------------------------------------------------------- + bool core::get_tx_outputs_gindexs(const crypto::hash& tx_id, size_t n_txes, std::vector<std::vector<uint64_t>>& indexs) const + { + return m_blockchain_storage.get_tx_outputs_gindexs(tx_id, n_txes, indexs); + } + //----------------------------------------------------------------------------------------------- void core::pause_mine() { m_miner.pause(); diff --git a/src/cryptonote_core/cryptonote_core.h b/src/cryptonote_core/cryptonote_core.h index cc53fce58..fe86f8d39 100644 --- a/src/cryptonote_core/cryptonote_core.h +++ b/src/cryptonote_core/cryptonote_core.h @@ -534,6 +534,7 @@ namespace cryptonote * @note see Blockchain::get_tx_outputs_gindexs */ bool get_tx_outputs_gindexs(const crypto::hash& tx_id, std::vector<uint64_t>& indexs) const; + bool get_tx_outputs_gindexs(const crypto::hash& tx_id, size_t n_txes, std::vector<std::vector<uint64_t>>& indexs) const; /** * @copydoc Blockchain::get_tail_id diff --git a/src/cryptonote_protocol/block_queue.cpp b/src/cryptonote_protocol/block_queue.cpp index 05f4189fb..c1989f093 100644 --- a/src/cryptonote_protocol/block_queue.cpp +++ b/src/cryptonote_protocol/block_queue.cpp @@ -31,6 +31,7 @@ #include <vector> #include <unordered_map> #include <boost/uuid/nil_generator.hpp> +#include <boost/uuid/uuid_io.hpp> #include "string_tools.h" #include "cryptonote_protocol_defs.h" #include "block_queue.h" diff --git a/src/cryptonote_protocol/cryptonote_protocol_handler.h b/src/cryptonote_protocol/cryptonote_protocol_handler.h index 618b635cc..3c5b22b4a 100644 --- a/src/cryptonote_protocol/cryptonote_protocol_handler.h +++ b/src/cryptonote_protocol/cryptonote_protocol_handler.h @@ -157,7 +157,7 @@ namespace cryptonote std::string blob; epee::serialization::store_t_to_binary(arg, blob); //handler_response_blocks_now(blob.size()); // XXX - return m_p2p->invoke_notify_to_peer(t_parameter::ID, blob, context); + return m_p2p->invoke_notify_to_peer(t_parameter::ID, epee::strspan<uint8_t>(blob), context); } template<class t_parameter> @@ -166,7 +166,7 @@ namespace cryptonote LOG_PRINT_L2("[" << epee::net_utils::print_connection_context_short(exclude_context) << "] post relay " << typeid(t_parameter).name() << " -->"); std::string arg_buff; epee::serialization::store_t_to_binary(arg, arg_buff); - return m_p2p->relay_notify_to_all(t_parameter::ID, arg_buff, exclude_context); + return m_p2p->relay_notify_to_all(t_parameter::ID, epee::strspan<uint8_t>(arg_buff), exclude_context); } }; diff --git a/src/cryptonote_protocol/cryptonote_protocol_handler.inl b/src/cryptonote_protocol/cryptonote_protocol_handler.inl index 999ec5650..01f70cba1 100644 --- a/src/cryptonote_protocol/cryptonote_protocol_handler.inl +++ b/src/cryptonote_protocol/cryptonote_protocol_handler.inl @@ -1720,13 +1720,13 @@ skip: { std::string fluffyBlob; epee::serialization::store_t_to_binary(fluffy_arg, fluffyBlob); - m_p2p->relay_notify_to_list(NOTIFY_NEW_FLUFFY_BLOCK::ID, fluffyBlob, fluffyConnections); + m_p2p->relay_notify_to_list(NOTIFY_NEW_FLUFFY_BLOCK::ID, epee::strspan<uint8_t>(fluffyBlob), fluffyConnections); } if (!fullConnections.empty()) { std::string fullBlob; epee::serialization::store_t_to_binary(arg, fullBlob); - m_p2p->relay_notify_to_list(NOTIFY_NEW_BLOCK::ID, fullBlob, fullConnections); + m_p2p->relay_notify_to_list(NOTIFY_NEW_BLOCK::ID, epee::strspan<uint8_t>(fullBlob), fullConnections); } return true; diff --git a/src/daemon/command_parser_executor.cpp b/src/daemon/command_parser_executor.cpp index 853cde9c3..b5b747e97 100644 --- a/src/daemon/command_parser_executor.cpp +++ b/src/daemon/command_parser_executor.cpp @@ -163,9 +163,21 @@ bool t_command_parser_executor::print_height(const std::vector<std::string>& arg bool t_command_parser_executor::print_block(const std::vector<std::string>& args) { + bool include_hex = false; + + // Assumes that optional flags come after mandatory argument <transaction_hash> + for (unsigned int i = 1; i < args.size(); ++i) { + if (args[i] == "+hex") + include_hex = true; + else + { + std::cout << "unexpected argument: " << args[i] << std::endl; + return true; + } + } if (args.empty()) { - std::cout << "expected: print_block (<block_hash> | <block_height>)" << std::endl; + std::cout << "expected: print_block (<block_hash> | <block_height>) [+hex]" << std::endl; return false; } @@ -173,14 +185,14 @@ bool t_command_parser_executor::print_block(const std::vector<std::string>& args try { uint64_t height = boost::lexical_cast<uint64_t>(arg); - return m_executor.print_block_by_height(height); + return m_executor.print_block_by_height(height, include_hex); } catch (const boost::bad_lexical_cast&) { crypto::hash block_hash; if (parse_hash256(arg, block_hash)) { - return m_executor.print_block_by_hash(block_hash); + return m_executor.print_block_by_hash(block_hash, include_hex); } } diff --git a/src/daemon/rpc_command_executor.cpp b/src/daemon/rpc_command_executor.cpp index 015e1e1f9..4c7d68686 100644 --- a/src/daemon/rpc_command_executor.cpp +++ b/src/daemon/rpc_command_executor.cpp @@ -66,7 +66,7 @@ namespace { void print_block_header(cryptonote::block_header_response const & header) { tools::success_msg_writer() - << "timestamp: " << boost::lexical_cast<std::string>(header.timestamp) << std::endl + << "timestamp: " << boost::lexical_cast<std::string>(header.timestamp) << " (" << tools::get_human_readable_timestamp(header.timestamp) << ")" << std::endl << "previous hash: " << header.prev_hash << std::endl << "nonce: " << boost::lexical_cast<std::string>(header.nonce) << std::endl << "is orphan: " << header.orphan_status << std::endl @@ -569,7 +569,7 @@ bool t_rpc_command_executor::print_blockchain_info(uint64_t start_block_index, u if (!first) tools::msg_writer() << "" << std::endl; tools::msg_writer() - << "height: " << header.height << ", timestamp: " << header.timestamp + << "height: " << header.height << ", timestamp: " << header.timestamp << " (" << tools::get_human_readable_timestamp(header.timestamp) << ")" << ", size: " << header.block_size << ", weight: " << header.block_weight << ", transactions: " << header.num_txes << std::endl << "major version: " << (unsigned)header.major_version << ", minor version: " << (unsigned)header.minor_version << std::endl << "block id: " << header.hash << ", previous block id: " << header.prev_hash << std::endl @@ -663,7 +663,7 @@ bool t_rpc_command_executor::print_height() { return true; } -bool t_rpc_command_executor::print_block_by_hash(crypto::hash block_hash) { +bool t_rpc_command_executor::print_block_by_hash(crypto::hash block_hash, bool include_hex) { cryptonote::COMMAND_RPC_GET_BLOCK::request req; cryptonote::COMMAND_RPC_GET_BLOCK::response res; epee::json_rpc::error error_resp; @@ -689,13 +689,15 @@ bool t_rpc_command_executor::print_block_by_hash(crypto::hash block_hash) { } } + if (include_hex) + tools::success_msg_writer() << res.blob << std::endl; print_block_header(res.block_header); tools::success_msg_writer() << res.json << ENDL; return true; } -bool t_rpc_command_executor::print_block_by_height(uint64_t height) { +bool t_rpc_command_executor::print_block_by_height(uint64_t height, bool include_hex) { cryptonote::COMMAND_RPC_GET_BLOCK::request req; cryptonote::COMMAND_RPC_GET_BLOCK::response res; epee::json_rpc::error error_resp; @@ -721,6 +723,8 @@ bool t_rpc_command_executor::print_block_by_height(uint64_t height) { } } + if (include_hex) + tools::success_msg_writer() << res.blob << std::endl; print_block_header(res.block_header); tools::success_msg_writer() << res.json << ENDL; diff --git a/src/daemon/rpc_command_executor.h b/src/daemon/rpc_command_executor.h index 592584a5f..1541a1a8e 100644 --- a/src/daemon/rpc_command_executor.h +++ b/src/daemon/rpc_command_executor.h @@ -91,9 +91,9 @@ public: bool print_height(); - bool print_block_by_hash(crypto::hash block_hash); + bool print_block_by_hash(crypto::hash block_hash, bool include_hex); - bool print_block_by_height(uint64_t height); + bool print_block_by_height(uint64_t height, bool include_hex); bool print_transaction(crypto::hash transaction_hash, bool include_hex, bool include_json); diff --git a/src/device_trezor/trezor/protocol.cpp b/src/device_trezor/trezor/protocol.cpp index c4a92426c..13506a67f 100644 --- a/src/device_trezor/trezor/protocol.cpp +++ b/src/device_trezor/trezor/protocol.cpp @@ -877,6 +877,9 @@ namespace tx { valueS.SetString(m_ct.enc_salt2.c_str(), m_ct.enc_salt2.size()); json.AddMember("salt2", valueS, json.GetAllocator()); + valueS.SetString(m_ct.tx_prefix_hash.c_str(), m_ct.tx_prefix_hash.size()); + json.AddMember("tx_prefix_hash", valueS, json.GetAllocator()); + valueS.SetString(m_ct.enc_keys.c_str(), m_ct.enc_keys.size()); json.AddMember("enc_keys", valueS, json.GetAllocator()); diff --git a/src/p2p/net_node.h b/src/p2p/net_node.h index 4db0a6cb7..0ef2dbb30 100644 --- a/src/p2p/net_node.h +++ b/src/p2p/net_node.h @@ -180,10 +180,10 @@ namespace nodetool virtual void on_connection_close(p2p_connection_context& context); virtual void callback(p2p_connection_context& context); //----------------- i_p2p_endpoint ------------------------------------------------------------- - virtual bool relay_notify_to_list(int command, const std::string& data_buff, const std::list<boost::uuids::uuid> &connections); - virtual bool relay_notify_to_all(int command, const std::string& data_buff, const epee::net_utils::connection_context_base& context); - virtual bool invoke_command_to_peer(int command, const std::string& req_buff, std::string& resp_buff, const epee::net_utils::connection_context_base& context); - virtual bool invoke_notify_to_peer(int command, const std::string& req_buff, const epee::net_utils::connection_context_base& context); + virtual bool relay_notify_to_list(int command, const epee::span<const uint8_t> data_buff, const std::list<boost::uuids::uuid> &connections); + virtual bool relay_notify_to_all(int command, const epee::span<const uint8_t> data_buff, const epee::net_utils::connection_context_base& context); + virtual bool invoke_command_to_peer(int command, const epee::span<const uint8_t> req_buff, std::string& resp_buff, const epee::net_utils::connection_context_base& context); + virtual bool invoke_notify_to_peer(int command, const epee::span<const uint8_t> req_buff, const epee::net_utils::connection_context_base& context); virtual bool drop_connection(const epee::net_utils::connection_context_base& context); virtual void request_callback(const epee::net_utils::connection_context_base& context); virtual void for_each_connection(std::function<bool(typename t_payload_net_handler::connection_context&, peerid_type, uint32_t)> f); diff --git a/src/p2p/net_node.inl b/src/p2p/net_node.inl index 5b845fe15..25ac1ba18 100644 --- a/src/p2p/net_node.inl +++ b/src/p2p/net_node.inl @@ -33,6 +33,7 @@ #include <algorithm> #include <boost/date_time/posix_time/posix_time.hpp> #include <boost/thread/thread.hpp> +#include <boost/uuid/uuid_io.hpp> #include <boost/bind.hpp> #include <atomic> @@ -740,7 +741,7 @@ namespace nodetool if(rsp.node_data.network_id != m_network_id) { - LOG_WARNING_CC(context, "COMMAND_HANDSHAKE Failed, wrong network! (" << epee::string_tools::get_str_from_guid_a(rsp.node_data.network_id) << "), closing connection."); + LOG_WARNING_CC(context, "COMMAND_HANDSHAKE Failed, wrong network! (" << rsp.node_data.network_id << "), closing connection."); return; } @@ -1516,7 +1517,7 @@ namespace nodetool } //----------------------------------------------------------------------------------- template<class t_payload_net_handler> - bool node_server<t_payload_net_handler>::relay_notify_to_list(int command, const std::string& data_buff, const std::list<boost::uuids::uuid> &connections) + bool node_server<t_payload_net_handler>::relay_notify_to_list(int command, const epee::span<const uint8_t> data_buff, const std::list<boost::uuids::uuid> &connections) { for(const auto& c_id: connections) { @@ -1526,7 +1527,7 @@ namespace nodetool } //----------------------------------------------------------------------------------- template<class t_payload_net_handler> - bool node_server<t_payload_net_handler>::relay_notify_to_all(int command, const std::string& data_buff, const epee::net_utils::connection_context_base& context) + bool node_server<t_payload_net_handler>::relay_notify_to_all(int command, const epee::span<const uint8_t> data_buff, const epee::net_utils::connection_context_base& context) { std::list<boost::uuids::uuid> connections; m_net_server.get_config_object().foreach_connection([&](const p2p_connection_context& cntxt) @@ -1545,14 +1546,14 @@ namespace nodetool } //----------------------------------------------------------------------------------- template<class t_payload_net_handler> - bool node_server<t_payload_net_handler>::invoke_notify_to_peer(int command, const std::string& req_buff, const epee::net_utils::connection_context_base& context) + bool node_server<t_payload_net_handler>::invoke_notify_to_peer(int command, const epee::span<const uint8_t> req_buff, const epee::net_utils::connection_context_base& context) { int res = m_net_server.get_config_object().notify(command, req_buff, context.m_connection_id); return res > 0; } //----------------------------------------------------------------------------------- template<class t_payload_net_handler> - bool node_server<t_payload_net_handler>::invoke_command_to_peer(int command, const std::string& req_buff, std::string& resp_buff, const epee::net_utils::connection_context_base& context) + bool node_server<t_payload_net_handler>::invoke_command_to_peer(int command, const epee::span<const uint8_t> req_buff, std::string& resp_buff, const epee::net_utils::connection_context_base& context) { int res = m_net_server.get_config_object().invoke(command, req_buff, resp_buff, context.m_connection_id); return res > 0; @@ -1686,7 +1687,7 @@ namespace nodetool if(arg.node_data.network_id != m_network_id) { - LOG_INFO_CC(context, "WRONG NETWORK AGENT CONNECTED! id=" << epee::string_tools::get_str_from_guid_a(arg.node_data.network_id)); + LOG_INFO_CC(context, "WRONG NETWORK AGENT CONNECTED! id=" << arg.node_data.network_id); drop_connection(context); add_host_fail(context.m_remote_address); return 1; @@ -1802,7 +1803,7 @@ namespace nodetool { ss << cntxt.m_remote_address.str() << " \t\tpeer_id " << cntxt.peer_id - << " \t\tconn_id " << epee::string_tools::get_str_from_guid_a(cntxt.m_connection_id) << (cntxt.m_is_income ? " INC":" OUT") + << " \t\tconn_id " << cntxt.m_connection_id << (cntxt.m_is_income ? " INC":" OUT") << std::endl; return true; }); @@ -2016,7 +2017,7 @@ namespace nodetool return false; if (!m_peerlist.get_random_gray_peer(pe)) { - return false; + return true; } bool success = check_connection_and_handshake_with_peer(pe.adr, pe.last_seen); diff --git a/src/p2p/net_node_common.h b/src/p2p/net_node_common.h index 218250efa..656c6155b 100644 --- a/src/p2p/net_node_common.h +++ b/src/p2p/net_node_common.h @@ -43,10 +43,10 @@ namespace nodetool template<class t_connection_context> struct i_p2p_endpoint { - virtual bool relay_notify_to_list(int command, const std::string& data_buff, const std::list<boost::uuids::uuid>& connections)=0; - virtual bool relay_notify_to_all(int command, const std::string& data_buff, const epee::net_utils::connection_context_base& context)=0; - virtual bool invoke_command_to_peer(int command, const std::string& req_buff, std::string& resp_buff, const epee::net_utils::connection_context_base& context)=0; - virtual bool invoke_notify_to_peer(int command, const std::string& req_buff, const epee::net_utils::connection_context_base& context)=0; + virtual bool relay_notify_to_list(int command, const epee::span<const uint8_t> data_buff, const std::list<boost::uuids::uuid>& connections)=0; + virtual bool relay_notify_to_all(int command, const epee::span<const uint8_t> data_buff, const epee::net_utils::connection_context_base& context)=0; + virtual bool invoke_command_to_peer(int command, const epee::span<const uint8_t> req_buff, std::string& resp_buff, const epee::net_utils::connection_context_base& context)=0; + virtual bool invoke_notify_to_peer(int command, const epee::span<const uint8_t> req_buff, const epee::net_utils::connection_context_base& context)=0; virtual bool drop_connection(const epee::net_utils::connection_context_base& context)=0; virtual void request_callback(const epee::net_utils::connection_context_base& context)=0; virtual uint64_t get_connections_count()=0; @@ -61,19 +61,19 @@ namespace nodetool template<class t_connection_context> struct p2p_endpoint_stub: public i_p2p_endpoint<t_connection_context> { - virtual bool relay_notify_to_list(int command, const std::string& data_buff, const std::list<boost::uuids::uuid>& connections) + virtual bool relay_notify_to_list(int command, const epee::span<const uint8_t> data_buff, const std::list<boost::uuids::uuid>& connections) { return false; } - virtual bool relay_notify_to_all(int command, const std::string& data_buff, const epee::net_utils::connection_context_base& context) + virtual bool relay_notify_to_all(int command, const epee::span<const uint8_t> data_buff, const epee::net_utils::connection_context_base& context) { return false; } - virtual bool invoke_command_to_peer(int command, const std::string& req_buff, std::string& resp_buff, const epee::net_utils::connection_context_base& context) + virtual bool invoke_command_to_peer(int command, const epee::span<const uint8_t> req_buff, std::string& resp_buff, const epee::net_utils::connection_context_base& context) { return false; } - virtual bool invoke_notify_to_peer(int command, const std::string& req_buff, const epee::net_utils::connection_context_base& context) + virtual bool invoke_notify_to_peer(int command, const epee::span<const uint8_t> req_buff, const epee::net_utils::connection_context_base& context) { return true; } diff --git a/src/ringct/rctSigs.cpp b/src/ringct/rctSigs.cpp index baa649f82..316f0e5e8 100644 --- a/src/ringct/rctSigs.cpp +++ b/src/ringct/rctSigs.cpp @@ -605,10 +605,19 @@ namespace rct { keyV tmp(rows + 1); size_t i; keyM M(cols, tmp); + ge_p3 Cp3; + CHECK_AND_ASSERT_MES_L1(ge_frombytes_vartime(&Cp3, C.bytes) == 0, false, "point conv failed"); + ge_cached Ccached; + ge_p3_to_cached(&Ccached, &Cp3); + ge_p1p1 p1; //create the matrix to mg sig for (i = 0; i < cols; i++) { M[i][0] = pubs[i].dest; - subKeys(M[i][1], pubs[i].mask, C); + ge_p3 p3; + CHECK_AND_ASSERT_MES_L1(ge_frombytes_vartime(&p3, pubs[i].mask.bytes) == 0, false, "point conv failed"); + ge_sub(&p1, &p3, &Ccached); + ge_p1p1_to_p3(&p3, &p1); + ge_p3_tobytes(M[i][1].bytes, &p3); } //DP(C); return MLSAG_Ver(message, M, mg, rows); diff --git a/src/rpc/core_rpc_server.cpp b/src/rpc/core_rpc_server.cpp index 0aa25bda7..d20000a53 100644 --- a/src/rpc/core_rpc_server.cpp +++ b/src/rpc/core_rpc_server.cpp @@ -252,19 +252,11 @@ namespace cryptonote pruned_size += bd.first.first.size(); unpruned_size += bd.first.first.size(); res.output_indices.push_back(COMMAND_RPC_GET_BLOCKS_FAST::block_output_indices()); - res.output_indices.back().indices.push_back(COMMAND_RPC_GET_BLOCKS_FAST::tx_output_indices()); - if (!req.no_miner_tx) - { - bool r = m_core.get_tx_outputs_gindexs(bd.first.second, res.output_indices.back().indices.back().indices); - if (!r) - { - res.status = "Failed"; - return false; - } - } ntxes += bd.second.size(); + res.output_indices.back().indices.reserve(1 + bd.second.size()); + if (req.no_miner_tx) + res.output_indices.back().indices.push_back(COMMAND_RPC_GET_BLOCKS_FAST::tx_output_indices()); res.blocks.back().txs.reserve(bd.second.size()); - res.output_indices.back().indices.reserve(bd.second.size()); for (std::vector<std::pair<crypto::hash, cryptonote::blobdata>>::iterator i = bd.second.begin(); i != bd.second.end(); ++i) { unpruned_size += i->second.size(); @@ -272,14 +264,25 @@ namespace cryptonote i->second.clear(); i->second.shrink_to_fit(); pruned_size += res.blocks.back().txs.back().size(); + } - res.output_indices.back().indices.push_back(COMMAND_RPC_GET_BLOCKS_FAST::tx_output_indices()); - bool r = m_core.get_tx_outputs_gindexs(i->first, res.output_indices.back().indices.back().indices); + const size_t n_txes_to_lookup = bd.second.size() + (req.no_miner_tx ? 0 : 1); + if (n_txes_to_lookup > 0) + { + std::vector<std::vector<uint64_t>> indices; + bool r = m_core.get_tx_outputs_gindexs(req.no_miner_tx ? bd.second.front().first : bd.first.second, n_txes_to_lookup, indices); if (!r) { res.status = "Failed"; return false; } + if (indices.size() != n_txes_to_lookup || res.output_indices.back().indices.size() != (req.no_miner_tx ? 1 : 0)) + { + res.status = "Failed"; + return false; + } + for (size_t i = 0; i < indices.size(); ++i) + res.output_indices.back().indices.push_back({std::move(indices[i])}); } } @@ -703,31 +706,31 @@ namespace cryptonote if(!m_core.handle_incoming_tx(tx_blob, tvc, false, false, req.do_not_relay) || tvc.m_verifivation_failed) { res.status = "Failed"; - res.reason = ""; + std::string reason = ""; if ((res.low_mixin = tvc.m_low_mixin)) - add_reason(res.reason, "bad ring size"); + add_reason(reason, "bad ring size"); if ((res.double_spend = tvc.m_double_spend)) - add_reason(res.reason, "double spend"); + add_reason(reason, "double spend"); if ((res.invalid_input = tvc.m_invalid_input)) - add_reason(res.reason, "invalid input"); + add_reason(reason, "invalid input"); if ((res.invalid_output = tvc.m_invalid_output)) - add_reason(res.reason, "invalid output"); + add_reason(reason, "invalid output"); if ((res.too_big = tvc.m_too_big)) - add_reason(res.reason, "too big"); + add_reason(reason, "too big"); if ((res.overspend = tvc.m_overspend)) - add_reason(res.reason, "overspend"); + add_reason(reason, "overspend"); if ((res.fee_too_low = tvc.m_fee_too_low)) - add_reason(res.reason, "fee too low"); + add_reason(reason, "fee too low"); if ((res.not_rct = tvc.m_not_rct)) - add_reason(res.reason, "tx is not ringct"); - const std::string punctuation = res.reason.empty() ? "" : ": "; + add_reason(reason, "tx is not ringct"); + const std::string punctuation = reason.empty() ? "" : ": "; if (tvc.m_verifivation_failed) { - LOG_PRINT_L0("[on_send_raw_tx]: tx verification failed" << punctuation << res.reason); + LOG_PRINT_L0("[on_send_raw_tx]: tx verification failed" << punctuation << reason); } else { - LOG_PRINT_L0("[on_send_raw_tx]: Failed to process tx" << punctuation << res.reason); + LOG_PRINT_L0("[on_send_raw_tx]: Failed to process tx" << punctuation << reason); } return true; } diff --git a/src/simplewallet/simplewallet.cpp b/src/simplewallet/simplewallet.cpp index bdb6d2bfe..c3f06e831 100644 --- a/src/simplewallet/simplewallet.cpp +++ b/src/simplewallet/simplewallet.cpp @@ -113,6 +113,14 @@ typedef cryptonote::simple_wallet sw; #define PRINT_USAGE(usage_help) fail_msg_writer() << boost::format(tr("usage: %s")) % usage_help; +#define LONG_PAYMENT_ID_SUPPORT_CHECK() \ + do { \ + if (!m_long_payment_id_support) { \ + fail_msg_writer() << tr("Long payment IDs are obsolete. Use --long-payment-id-support if you really must use one."); \ + return true; \ + } \ + } while(0) + enum TransferType { Transfer, TransferLocked, @@ -136,10 +144,12 @@ namespace const command_line::arg_descriptor<bool> arg_non_deterministic = {"non-deterministic", sw::tr("Generate non-deterministic view and spend keys"), false}; const command_line::arg_descriptor<bool> arg_allow_mismatched_daemon_version = {"allow-mismatched-daemon-version", sw::tr("Allow communicating with a daemon that uses a different RPC version"), false}; const command_line::arg_descriptor<uint64_t> arg_restore_height = {"restore-height", sw::tr("Restore from specific blockchain height"), 0}; + const command_line::arg_descriptor<std::string> arg_restore_date = {"restore-date", sw::tr("Restore from estimated blockchain height on specified date"), ""}; const command_line::arg_descriptor<bool> arg_do_not_relay = {"do-not-relay", sw::tr("The newly created transaction will not be relayed to the monero network"), false}; const command_line::arg_descriptor<bool> arg_create_address_file = {"create-address-file", sw::tr("Create an address file for new wallets"), false}; const command_line::arg_descriptor<std::string> arg_subaddress_lookahead = {"subaddress-lookahead", tools::wallet2::tr("Set subaddress lookahead sizes to <major>:<minor>"), ""}; const command_line::arg_descriptor<bool> arg_use_english_language_names = {"use-english-language-names", sw::tr("Display English language names"), false}; + const command_line::arg_descriptor<bool> arg_long_payment_id_support = {"long-payment-id-support", sw::tr("Support obsolete long (unencrypted) payment ids"), false}; const command_line::arg_descriptor< std::vector<std::string> > arg_command = {"command", ""}; @@ -150,12 +160,12 @@ namespace const char* USAGE_PAYMENTS("payments <PID_1> [<PID_2> ... <PID_N>]"); const char* USAGE_PAYMENT_ID("payment_id"); const char* USAGE_TRANSFER("transfer [index=<N1>[,<N2>,...]] [<priority>] [<ring_size>] (<URI> | <address> <amount>) [<payment_id>]"); - const char* USAGE_LOCKED_TRANSFER("locked_transfer [index=<N1>[,<N2>,...]] [<priority>] [<ring_size>] (<URI> | <addr> <amount>) <lockblocks> [<payment_id>]"); - const char* USAGE_LOCKED_SWEEP_ALL("locked_sweep_all [index=<N1>[,<N2>,...]] [<priority>] [<ring_size>] <address> <lockblocks> [<payment_id>]"); - const char* USAGE_SWEEP_ALL("sweep_all [index=<N1>[,<N2>,...]] [<priority>] [<ring_size>] [outputs=<N>] <address> [<payment_id>]"); - const char* USAGE_SWEEP_BELOW("sweep_below <amount_threshold> [index=<N1>[,<N2>,...]] [<priority>] [<ring_size>] <address> [<payment_id>]"); - const char* USAGE_SWEEP_SINGLE("sweep_single [<priority>] [<ring_size>] [outputs=<N>] <key_image> <address> [<payment_id>]"); - const char* USAGE_DONATE("donate [index=<N1>[,<N2>,...]] [<priority>] [<ring_size>] <amount> [<payment_id>]"); + const char* USAGE_LOCKED_TRANSFER("locked_transfer [index=<N1>[,<N2>,...]] [<priority>] [<ring_size>] (<URI> | <addr> <amount>) <lockblocks> [<payment_id (obsolete)>]"); + const char* USAGE_LOCKED_SWEEP_ALL("locked_sweep_all [index=<N1>[,<N2>,...]] [<priority>] [<ring_size>] <address> <lockblocks> [<payment_id (obsolete)>]"); + const char* USAGE_SWEEP_ALL("sweep_all [index=<N1>[,<N2>,...]] [<priority>] [<ring_size>] [outputs=<N>] <address> [<payment_id (obsolete)>]"); + const char* USAGE_SWEEP_BELOW("sweep_below <amount_threshold> [index=<N1>[,<N2>,...]] [<priority>] [<ring_size>] <address> [<payment_id (obsolete)>]"); + const char* USAGE_SWEEP_SINGLE("sweep_single [<priority>] [<ring_size>] [outputs=<N>] <key_image> <address> [<payment_id (obsolete)>]"); + const char* USAGE_DONATE("donate [index=<N1>[,<N2>,...]] [<priority>] [<ring_size>] <amount> [<payment_id (obsolete)>]"); const char* USAGE_SIGN_TRANSFER("sign_transfer [export_raw]"); const char* USAGE_SET_LOG("set_log <level>|{+,-,}<categories>"); const char* USAGE_ACCOUNT("account\n" @@ -230,12 +240,15 @@ namespace const char* USAGE_VERSION("version"); const char* USAGE_HELP("help [<command>]"); - std::string input_line(const std::string& prompt) + std::string input_line(const std::string& prompt, bool yesno = false) { #ifdef HAVE_READLINE rdln::suspend_readline pause_readline; #endif std::cout << prompt; + if (yesno) + std::cout << " (Y/Yes/N/No)"; + std::cout << ": " << std::flush; std::string buf; #ifdef _WIN32 @@ -247,12 +260,12 @@ namespace return epee::string_tools::trim(buf); } - epee::wipeable_string input_secure_line(const std::string& prompt) + epee::wipeable_string input_secure_line(const char *prompt) { #ifdef HAVE_READLINE rdln::suspend_readline pause_readline; #endif - auto pwd_container = tools::password_container::prompt(false, prompt.c_str(), false); + auto pwd_container = tools::password_container::prompt(false, prompt, false); if (!pwd_container) { MERROR("Failed to read secure line"); @@ -425,10 +438,10 @@ namespace << ", " << dnssec_str << std::endl << sw::tr(" Monero Address = ") << addresses[0] << std::endl - << sw::tr("Is this OK? (Y/n) ") + << sw::tr("Is this OK?") ; // prompt the user for confirmation given the dns query and dnssec status - std::string confirm_dns_ok = input_line(prompt.str()); + std::string confirm_dns_ok = input_line(prompt.str(), true); if (std::cin.eof()) { return {}; @@ -540,7 +553,7 @@ namespace } catch (const tools::error::tx_rejected& e) { - fail_msg_writer() << (boost::format(sw::tr("transaction %s was rejected by daemon with status: ")) % get_transaction_hash(e.tx())) << e.status(); + fail_msg_writer() << (boost::format(sw::tr("transaction %s was rejected by daemon")) % get_transaction_hash(e.tx())); std::string reason = e.reason(); if (!reason.empty()) fail_msg_writer() << sw::tr("Reason: ") << reason; @@ -596,7 +609,7 @@ namespace fail_msg_writer() << boost::format(sw::tr("File %s likely stores wallet private keys! Use a different file name.")) % filename; return false; } - return command_line::is_yes(input_line((boost::format(sw::tr("File %s already exists. Are you sure to overwrite it? (Y/Yes/N/No): ")) % filename).str())); + return command_line::is_yes(input_line((boost::format(sw::tr("File %s already exists. Are you sure to overwrite it?")) % filename).str(), true)); } return true; } @@ -860,6 +873,8 @@ bool simple_wallet::change_password(const std::vector<std::string> &args) bool simple_wallet::payment_id(const std::vector<std::string> &args/* = std::vector<std::string>()*/) { + LONG_PAYMENT_ID_SUPPORT_CHECK(); + crypto::hash payment_id; if (args.size() > 0) { @@ -1165,6 +1180,7 @@ bool simple_wallet::exchange_multisig_keys_main(const std::vector<std::string> & uint32_t threshold, total; m_wallet->multisig(NULL, &threshold, &total); success_msg_writer() << tr("Multisig wallet has been successfully created. Current wallet type: ") << threshold << "/" << total; + success_msg_writer() << tr("Multisig address: ") << m_wallet->get_account().get_public_address_str(m_wallet->nettype()); } } catch (const std::exception &e) @@ -2221,6 +2237,8 @@ bool simple_wallet::set_refresh_type(const std::vector<std::string> &args/* = st bool simple_wallet::set_confirm_missing_payment_id(const std::vector<std::string> &args/* = std::vector<std::string>()*/) { + LONG_PAYMENT_ID_SUPPORT_CHECK(); + const auto pwd_container = get_and_verify_password(); if (pwd_container) { @@ -2781,7 +2799,7 @@ simple_wallet::simple_wallet() m_cmd_binder.set_handler("rescan_bc", boost::bind(&simple_wallet::rescan_blockchain, this, _1), tr(USAGE_RESCAN_BC), - tr("Rescan the blockchain from scratch, losing any information which can not be recovered from the blockchain itself.")); + tr("Rescan the blockchain from scratch. If \"hard\" is specified, you will lose any information which can not be recovered from the blockchain itself.")); m_cmd_binder.set_handler("set_tx_note", boost::bind(&simple_wallet::set_tx_note, this, _1), tr(USAGE_SET_TX_NOTE), @@ -2846,7 +2864,7 @@ simple_wallet::simple_wallet() m_cmd_binder.set_handler("payment_id", boost::bind(&simple_wallet::payment_id, this, _1), tr(USAGE_PAYMENT_ID), - tr("Generate a new random full size payment id. These will be unencrypted on the blockchain, see integrated_address for encrypted short payment ids.")); + tr("Generate a new random full size payment id (obsolete). These will be unencrypted on the blockchain, see integrated_address for encrypted short payment ids.")); m_cmd_binder.set_handler("fee", boost::bind(&simple_wallet::print_fee_info, this, _1), tr("Print the information about the current fee and transaction backlog.")); @@ -3152,9 +3170,9 @@ bool simple_wallet::ask_wallet_create_if_needed() LOG_PRINT_L3("User asked to specify wallet file name."); wallet_path = input_line( tr(m_restoring ? "Specify a new wallet file name for your restored wallet (e.g., MyWallet).\n" - "Wallet file name (or Ctrl-C to quit): " : + "Wallet file name (or Ctrl-C to quit)" : "Specify wallet file name (e.g., MyWallet). If the wallet doesn't exist, it will be created.\n" - "Wallet file name (or Ctrl-C to quit): ") + "Wallet file name (or Ctrl-C to quit)") ); if(std::cin.eof()) { @@ -3201,7 +3219,7 @@ bool simple_wallet::ask_wallet_create_if_needed() if (!m_restoring) { message_writer() << tr("No wallet found with that name. Confirm creation of new wallet named: ") << wallet_path; - confirm_creation = input_line(tr("(Y/Yes/N/No): ")); + confirm_creation = input_line("", true); if(std::cin.eof()) { LOG_ERROR("Unexpected std::cin.eof() - Exited simple_wallet::ask_wallet_create_if_needed()"); @@ -3260,6 +3278,28 @@ static bool might_be_partial_seed(const epee::wipeable_string &words) return seed.size() < 24; } //---------------------------------------------------------------------------------------------------- +static bool datestr_to_int(const std::string &heightstr, uint16_t &year, uint8_t &month, uint8_t &day) +{ + if (heightstr.size() != 10 || heightstr[4] != '-' || heightstr[7] != '-') + { + fail_msg_writer() << tr("date format must be YYYY-MM-DD"); + return false; + } + try + { + year = boost::lexical_cast<uint16_t>(heightstr.substr(0,4)); + // lexical_cast<uint8_t> won't work because uint8_t is treated as character type + month = boost::lexical_cast<uint16_t>(heightstr.substr(5,2)); + day = boost::lexical_cast<uint16_t>(heightstr.substr(8,2)); + } + catch (const boost::bad_lexical_cast &) + { + fail_msg_writer() << tr("bad height parameter: ") << heightstr; + return false; + } + return true; +} +//---------------------------------------------------------------------------------------------------- bool simple_wallet::init(const boost::program_options::variables_map& vm) { epee::misc_utils::auto_scope_leave_caller scope_exit_handler = epee::misc_utils::create_scope_leave_handler([&](){ @@ -3388,7 +3428,7 @@ bool simple_wallet::init(const boost::program_options::variables_map& vm) { m_wallet_file = m_generate_from_view_key; // parse address - std::string address_string = input_line("Standard address: "); + std::string address_string = input_line("Standard address"); if (std::cin.eof()) return false; if (address_string.empty()) { @@ -3408,7 +3448,7 @@ bool simple_wallet::init(const boost::program_options::variables_map& vm) } // parse view secret key - epee::wipeable_string viewkey_string = input_secure_line("Secret view key: "); + epee::wipeable_string viewkey_string = input_secure_line("Secret view key"); if (std::cin.eof()) return false; if (viewkey_string.empty()) { @@ -3443,7 +3483,7 @@ bool simple_wallet::init(const boost::program_options::variables_map& vm) { m_wallet_file = m_generate_from_spend_key; // parse spend secret key - epee::wipeable_string spendkey_string = input_secure_line("Secret spend key: "); + epee::wipeable_string spendkey_string = input_secure_line("Secret spend key"); if (std::cin.eof()) return false; if (spendkey_string.empty()) { @@ -3463,7 +3503,7 @@ bool simple_wallet::init(const boost::program_options::variables_map& vm) { m_wallet_file = m_generate_from_keys; // parse address - std::string address_string = input_line("Standard address: "); + std::string address_string = input_line("Standard address"); if (std::cin.eof()) return false; if (address_string.empty()) { @@ -3483,7 +3523,7 @@ bool simple_wallet::init(const boost::program_options::variables_map& vm) } // parse spend secret key - epee::wipeable_string spendkey_string = input_secure_line("Secret spend key: "); + epee::wipeable_string spendkey_string = input_secure_line("Secret spend key"); if (std::cin.eof()) return false; if (spendkey_string.empty()) { @@ -3498,7 +3538,7 @@ bool simple_wallet::init(const boost::program_options::variables_map& vm) } // parse view secret key - epee::wipeable_string viewkey_string = input_secure_line("Secret view key: "); + epee::wipeable_string viewkey_string = input_secure_line("Secret view key"); if (std::cin.eof()) return false; if (viewkey_string.empty()) { @@ -3545,7 +3585,7 @@ bool simple_wallet::init(const boost::program_options::variables_map& vm) unsigned int multisig_n; // parse multisig type - std::string multisig_type_string = input_line("Multisig type (input as M/N with M <= N and M > 1): "); + std::string multisig_type_string = input_line("Multisig type (input as M/N with M <= N and M > 1)"); if (std::cin.eof()) return false; if (multisig_type_string.empty()) @@ -3571,7 +3611,7 @@ bool simple_wallet::init(const boost::program_options::variables_map& vm) message_writer() << boost::format(tr("Generating master wallet from %u of %u multisig wallet keys")) % multisig_m % multisig_n; // parse multisig address - std::string address_string = input_line("Multisig wallet address: "); + std::string address_string = input_line("Multisig wallet address"); if (std::cin.eof()) return false; if (address_string.empty()) { @@ -3586,7 +3626,7 @@ bool simple_wallet::init(const boost::program_options::variables_map& vm) } // parse secret view key - epee::wipeable_string viewkey_string = input_secure_line("Secret view key: "); + epee::wipeable_string viewkey_string = input_secure_line("Secret view key"); if (std::cin.eof()) return false; if (viewkey_string.empty()) @@ -3625,7 +3665,7 @@ bool simple_wallet::init(const boost::program_options::variables_map& vm) // get N secret spend keys from user for(unsigned int i=0; i<multisig_n; ++i) { - spendkey_string = input_secure_line(tr((boost::format(tr("Secret spend key (%u of %u):")) % (i+1) % multisig_m).str().c_str())); + spendkey_string = input_secure_line(tr((boost::format(tr("Secret spend key (%u of %u)")) % (i+1) % multisig_m).str().c_str())); if (std::cin.eof()) return false; if (spendkey_string.empty()) @@ -3698,15 +3738,15 @@ bool simple_wallet::init(const boost::program_options::variables_map& vm) if(m_wallet->get_refresh_from_block_height() == 0) { { tools::scoped_message_writer wrt = tools::msg_writer(); - wrt << tr("No restore height is specified."); - wrt << tr("Assumed you are creating a new account, restore will be done from current estimated blockchain height."); - wrt << tr("Use --restore-height if you want to restore an already setup account from a specific height"); + wrt << tr("No restore height is specified.") << " "; + wrt << tr("Assumed you are creating a new account, restore will be done from current estimated blockchain height.") << " "; + wrt << tr("Use --restore-height or --restore-date if you want to restore an already setup account from a specific height."); } - std::string confirm = input_line(tr("Is this okay? (Y/Yes/N/No): ")); + std::string confirm = input_line(tr("Is this okay?"), true); if (std::cin.eof() || !command_line::is_yes(confirm)) CHECK_AND_ASSERT_MES(false, false, tr("account creation aborted")); - m_wallet->set_refresh_from_block_height(m_wallet->estimate_blockchain_height()-1); + m_wallet->set_refresh_from_block_height(m_wallet->estimate_blockchain_height()); m_wallet->explicit_refresh_from_block_height(true); m_restore_height = m_wallet->get_refresh_from_block_height(); } @@ -3729,7 +3769,26 @@ bool simple_wallet::init(const boost::program_options::variables_map& vm) if (m_restoring && m_generate_from_json.empty() && m_generate_from_device.empty()) { - m_wallet->explicit_refresh_from_block_height(!command_line::is_arg_defaulted(vm, arg_restore_height)); + m_wallet->explicit_refresh_from_block_height(!(command_line::is_arg_defaulted(vm, arg_restore_height) || + command_line::is_arg_defaulted(vm, arg_restore_date))); + if (command_line::is_arg_defaulted(vm, arg_restore_height) && !command_line::is_arg_defaulted(vm, arg_restore_date)) + { + uint16_t year; + uint8_t month; + uint8_t day; + if (!datestr_to_int(m_restore_date, year, month, day)) + return false; + try + { + m_restore_height = m_wallet->get_blockchain_height_by_date(year, month, day); + success_msg_writer() << tr("Restore height is: ") << m_restore_height; + } + catch (const std::runtime_error& e) + { + fail_msg_writer() << e.what(); + return false; + } + } } if (!m_wallet->explicit_refresh_from_block_height() && m_restoring) { @@ -3739,9 +3798,9 @@ bool simple_wallet::init(const boost::program_options::variables_map& vm) { std::string heightstr; if (!connected || version < MAKE_CORE_RPC_VERSION(1, 6)) - heightstr = input_line("Restore from specific blockchain height (optional, default 0): "); + heightstr = input_line("Restore from specific blockchain height (optional, default 0)"); else - heightstr = input_line("Restore from specific blockchain height (optional, default 0),\nor alternatively from specific date (YYYY-MM-DD): "); + heightstr = input_line("Restore from specific blockchain height (optional, default 0),\nor alternatively from specific date (YYYY-MM-DD)"); if (std::cin.eof()) return false; if (heightstr.empty()) @@ -3761,23 +3820,16 @@ bool simple_wallet::init(const boost::program_options::variables_map& vm) fail_msg_writer() << tr("bad m_restore_height parameter: ") << heightstr; continue; } - if (heightstr.size() != 10 || heightstr[4] != '-' || heightstr[7] != '-') - { - fail_msg_writer() << tr("date format must be YYYY-MM-DD"); - continue; - } uint16_t year; uint8_t month; // 1, 2, ..., 12 uint8_t day; // 1, 2, ..., 31 try { - year = boost::lexical_cast<uint16_t>(heightstr.substr(0,4)); - // lexical_cast<uint8_t> won't work because uint8_t is treated as character type - month = boost::lexical_cast<uint16_t>(heightstr.substr(5,2)); - day = boost::lexical_cast<uint16_t>(heightstr.substr(8,2)); + if (!datestr_to_int(heightstr, year, month, day)) + return false; m_restore_height = m_wallet->get_blockchain_height_by_date(year, month, day); success_msg_writer() << tr("Restore height is: ") << m_restore_height; - std::string confirm = input_line(tr("Is this okay? (Y/Yes/N/No): ")); + std::string confirm = input_line(tr("Is this okay?"), true); if (std::cin.eof()) return false; if(command_line::is_yes(confirm)) @@ -3800,7 +3852,7 @@ bool simple_wallet::init(const boost::program_options::variables_map& vm) if (m_restore_height >= estimate_height) { success_msg_writer() << tr("Restore height ") << m_restore_height << (" is not yet reached. The current estimated height is ") << estimate_height; - std::string confirm = input_line(tr("Still apply restore height? (Y/Yes/N/No): ")); + std::string confirm = input_line(tr("Still apply restore height?"), true); if (std::cin.eof() || command_line::is_no(confirm)) m_restore_height = 0; } @@ -3861,9 +3913,11 @@ bool simple_wallet::handle_command_line(const boost::program_options::variables_ m_non_deterministic = command_line::get_arg(vm, arg_non_deterministic); m_allow_mismatched_daemon_version = command_line::get_arg(vm, arg_allow_mismatched_daemon_version); m_restore_height = command_line::get_arg(vm, arg_restore_height); + m_restore_date = command_line::get_arg(vm, arg_restore_date); m_do_not_relay = command_line::get_arg(vm, arg_do_not_relay); m_subaddress_lookahead = command_line::get_arg(vm, arg_subaddress_lookahead); m_use_english_language_names = command_line::get_arg(vm, arg_use_english_language_names); + m_long_payment_id_support = command_line::get_arg(vm, arg_long_payment_id_support); m_restoring = !m_generate_from_view_key.empty() || !m_generate_from_spend_key.empty() || !m_generate_from_keys.empty() || @@ -3873,6 +3927,14 @@ bool simple_wallet::handle_command_line(const boost::program_options::variables_ m_restore_deterministic_wallet || m_restore_multisig_wallet; + if (!command_line::is_arg_defaulted(vm, arg_restore_date)) + { + uint16_t year; + uint8_t month, day; + if (!datestr_to_int(m_restore_date, year, month, day)) + return false; + } + return true; } //---------------------------------------------------------------------------------------------------- @@ -3923,7 +3985,7 @@ std::string simple_wallet::get_mnemonic_language() } while (language_number < 0) { - language_choice = input_line(tr("Enter the number corresponding to the language of your choice: ")); + language_choice = input_line(tr("Enter the number corresponding to the language of your choice")); if (std::cin.eof()) return std::string(); try @@ -4569,7 +4631,7 @@ void simple_wallet::on_money_received(uint64_t height, const crypto::hash &txid, tr("NOTE: this transaction uses an encrypted payment ID: consider using subaddresses instead"); else if (get_payment_id_from_tx_extra_nonce(extra_nonce.nonce, payment_id)) message_writer(console_color_red, false) << - tr("WARNING: this transaction uses an unencrypted payment ID: consider using subaddresses instead"); + (m_long_payment_id_support ? tr("WARNING: this transaction uses an unencrypted payment ID: consider using subaddresses instead.") : tr("WARNING: this transaction uses an unencrypted payment ID: these are obsolete. Support will be withdrawn in the future. Use subaddresses instead.")); } } if (m_auto_refresh_refreshing) @@ -5268,6 +5330,8 @@ bool simple_wallet::transfer_main(int transfer_type, const std::vector<std::stri bool r = true; if (tools::wallet2::parse_long_payment_id(payment_id_str, payment_id)) { + LONG_PAYMENT_ID_SUPPORT_CHECK(); + std::string extra_nonce; set_payment_id_to_tx_extra_nonce(extra_nonce, payment_id); r = add_extra_nonce_to_tx_extra(extra, extra_nonce); @@ -5275,19 +5339,6 @@ bool simple_wallet::transfer_main(int transfer_type, const std::vector<std::stri payment_id_seen = true; message_writer() << tr("Unencrypted payment IDs are bad for privacy: ask the recipient to use subaddresses instead"); } - else - { - crypto::hash8 payment_id8; - if (tools::wallet2::parse_short_payment_id(payment_id_str, payment_id8)) - { - std::string extra_nonce; - set_encrypted_payment_id_to_tx_extra_nonce(extra_nonce, payment_id8); - r = add_extra_nonce_to_tx_extra(extra, extra_nonce); - local_args.pop_back(); - payment_id_seen = true; - } - } - if(!r) { fail_msg_writer() << tr("payment id failed to encode"); @@ -5391,6 +5442,7 @@ bool simple_wallet::transfer_main(int transfer_type, const std::vector<std::stri } else if (tools::wallet2::parse_payment_id(payment_id_uri, payment_id)) { + LONG_PAYMENT_ID_SUPPORT_CHECK(); set_payment_id_to_tx_extra_nonce(extra_nonce, payment_id); message_writer() << tr("Unencrypted payment IDs are bad for privacy: ask the recipient to use subaddresses instead"); } @@ -5412,9 +5464,9 @@ bool simple_wallet::transfer_main(int transfer_type, const std::vector<std::stri } // prompt is there is no payment id and confirmation is required - if (!payment_id_seen && m_wallet->confirm_missing_payment_id() && dsts.size() > num_subaddresses) + if (m_long_payment_id_support && !payment_id_seen && m_wallet->confirm_missing_payment_id() && dsts.size() > num_subaddresses) { - std::string accepted = input_line(tr("No payment id is included with this transaction. Is this okay? (Y/Yes/N/No): ")); + std::string accepted = input_line(tr("No payment id is included with this transaction. Is this okay?"), true); if (std::cin.eof()) return false; if (!command_line::is_yes(accepted)) @@ -5478,23 +5530,23 @@ bool simple_wallet::transfer_main(int transfer_type, const std::vector<std::stri std::vector<std::pair<uint64_t, uint64_t>> nblocks = m_wallet->estimate_backlog({std::make_pair(worst_fee_per_byte, worst_fee_per_byte)}); if (nblocks.size() != 1) { - prompt << "Internal error checking for backlog. " << tr("Is this okay anyway? (Y/Yes/N/No): "); + prompt << "Internal error checking for backlog. " << tr("Is this okay anyway?"); } else { if (nblocks[0].first > m_wallet->get_confirm_backlog_threshold()) - prompt << (boost::format(tr("There is currently a %u block backlog at that fee level. Is this okay? (Y/Yes/N/No): ")) % nblocks[0].first).str(); + prompt << (boost::format(tr("There is currently a %u block backlog at that fee level. Is this okay?")) % nblocks[0].first).str(); } } catch (const std::exception &e) { - prompt << tr("Failed to check for backlog: ") << e.what() << ENDL << tr("Is this okay anyway? (Y/Yes/N/No): "); + prompt << tr("Failed to check for backlog: ") << e.what() << ENDL << tr("Is this okay anyway?"); } std::string prompt_str = prompt.str(); if (!prompt_str.empty()) { - std::string accepted = input_line(prompt_str); + std::string accepted = input_line(prompt_str, true); if (std::cin.eof()) return false; if (!command_line::is_yes(accepted)) @@ -5580,9 +5632,9 @@ bool simple_wallet::transfer_main(int transfer_type, const std::vector<std::stri { prompt << tr("WARNING: this is a non default ring size, which may harm your privacy. Default is recommended."); } - prompt << ENDL << tr("Is this okay? (Y/Yes/N/No): "); + prompt << ENDL << tr("Is this okay?"); - std::string accepted = input_line(prompt.str()); + std::string accepted = input_line(prompt.str(), true); if (std::cin.eof()) return false; if (!command_line::is_yes(accepted)) @@ -5720,17 +5772,17 @@ bool simple_wallet::sweep_unmixable(const std::vector<std::string> &args_) std::string prompt_str = tr("Sweeping ") + print_money(total_unmixable); if (ptx_vector.size() > 1) { - prompt_str = (boost::format(tr("Sweeping %s in %llu transactions for a total fee of %s. Is this okay? (Y/Yes/N/No): ")) % + prompt_str = (boost::format(tr("Sweeping %s in %llu transactions for a total fee of %s. Is this okay?")) % print_money(total_unmixable) % ((unsigned long long)ptx_vector.size()) % print_money(total_fee)).str(); } else { - prompt_str = (boost::format(tr("Sweeping %s for a total fee of %s. Is this okay? (Y/Yes/N/No): ")) % + prompt_str = (boost::format(tr("Sweeping %s for a total fee of %s. Is this okay?")) % print_money(total_unmixable) % print_money(total_fee)).str(); } - std::string accepted = input_line(prompt_str); + std::string accepted = input_line(prompt_str, true); if (std::cin.eof()) return true; if (!command_line::is_yes(accepted)) @@ -5773,7 +5825,7 @@ bool simple_wallet::sweep_unmixable(const std::vector<std::string> &args_) catch (const tools::error::not_enough_unlocked_money& e) { fail_msg_writer() << tr("Not enough money in unlocked balance"); - std::string accepted = input_line((boost::format(tr("Discarding %s of unmixable outputs that cannot be spent, which can be undone by \"rescan_spent\". Is this okay? (Y/Yes/N/No): ")) % print_money(e.available())).str()); + std::string accepted = input_line((boost::format(tr("Discarding %s of unmixable outputs that cannot be spent, which can be undone by \"rescan_spent\". Is this okay?")) % print_money(e.available())).str(), true); if (std::cin.eof()) return true; if (command_line::is_yes(accepted)) @@ -5935,23 +5987,13 @@ bool simple_wallet::sweep_main(uint64_t below, bool locked, const std::vector<st bool r = tools::wallet2::parse_long_payment_id(payment_id_str, payment_id); if(r) { + LONG_PAYMENT_ID_SUPPORT_CHECK(); + std::string extra_nonce; set_payment_id_to_tx_extra_nonce(extra_nonce, payment_id); r = add_extra_nonce_to_tx_extra(extra, extra_nonce); payment_id_seen = true; } - else - { - crypto::hash8 payment_id8; - r = tools::wallet2::parse_short_payment_id(payment_id_str, payment_id8); - if(r) - { - std::string extra_nonce; - set_encrypted_payment_id_to_tx_extra_nonce(extra_nonce, payment_id8); - r = add_extra_nonce_to_tx_extra(extra, extra_nonce); - payment_id_seen = true; - } - } if(!r && local_args.size() == 3) { @@ -5991,9 +6033,9 @@ bool simple_wallet::sweep_main(uint64_t below, bool locked, const std::vector<st } // prompt is there is no payment id and confirmation is required - if (!payment_id_seen && m_wallet->confirm_missing_payment_id() && !info.is_subaddress) + if (m_long_payment_id_support && !payment_id_seen && m_wallet->confirm_missing_payment_id() && !info.is_subaddress) { - std::string accepted = input_line(tr("No payment id is included with this transaction. Is this okay? (Y/Yes/N/No): ")); + std::string accepted = input_line(tr("No payment id is included with this transaction. Is this okay?"), true); if (std::cin.eof()) return true; if (!command_line::is_yes(accepted)) @@ -6041,17 +6083,17 @@ bool simple_wallet::sweep_main(uint64_t below, bool locked, const std::vector<st if (m_wallet->print_ring_members() && !print_ring_members(ptx_vector, prompt)) return true; if (ptx_vector.size() > 1) { - prompt << boost::format(tr("Sweeping %s in %llu transactions for a total fee of %s. Is this okay? (Y/Yes/N/No): ")) % + prompt << boost::format(tr("Sweeping %s in %llu transactions for a total fee of %s. Is this okay?")) % print_money(total_sent) % ((unsigned long long)ptx_vector.size()) % print_money(total_fee); } else { - prompt << boost::format(tr("Sweeping %s for a total fee of %s. Is this okay? (Y/Yes/N/No): ")) % + prompt << boost::format(tr("Sweeping %s for a total fee of %s. Is this okay?")) % print_money(total_sent) % print_money(total_fee); } - std::string accepted = input_line(prompt.str()); + std::string accepted = input_line(prompt.str(), true); if (std::cin.eof()) return true; if (!command_line::is_yes(accepted)) @@ -6202,12 +6244,9 @@ bool simple_wallet::sweep_single(const std::vector<std::string> &args_) std::string extra_nonce; if (tools::wallet2::parse_long_payment_id(local_args.back(), payment_id)) { + LONG_PAYMENT_ID_SUPPORT_CHECK(); set_payment_id_to_tx_extra_nonce(extra_nonce, payment_id); } - else if(tools::wallet2::parse_short_payment_id(local_args.back(), payment_id8)) - { - set_encrypted_payment_id_to_tx_extra_nonce(extra_nonce, payment_id8); - } else { fail_msg_writer() << tr("failed to parse Payment ID"); @@ -6263,9 +6302,9 @@ bool simple_wallet::sweep_single(const std::vector<std::string> &args_) } // prompt if there is no payment id and confirmation is required - if (!payment_id_seen && m_wallet->confirm_missing_payment_id() && !info.is_subaddress) + if (m_long_payment_id_support && !payment_id_seen && m_wallet->confirm_missing_payment_id() && !info.is_subaddress) { - std::string accepted = input_line(tr("No payment id is included with this transaction. Is this okay? (Y/Yes/N/No): ")); + std::string accepted = input_line(tr("No payment id is included with this transaction. Is this okay?"), true); if (std::cin.eof()) return true; if (!command_line::is_yes(accepted)) @@ -6307,10 +6346,10 @@ bool simple_wallet::sweep_single(const std::vector<std::string> &args_) std::ostringstream prompt; if (!print_ring_members(ptx_vector, prompt)) return true; - prompt << boost::format(tr("Sweeping %s for a total fee of %s. Is this okay? (Y/Yes/N/No): ")) % + prompt << boost::format(tr("Sweeping %s for a total fee of %s. Is this okay?")) % print_money(total_sent) % print_money(total_fee); - std::string accepted = input_line(prompt.str()); + std::string accepted = input_line(prompt.str(), true); if (std::cin.eof()) return true; if (!command_line::is_yes(accepted)) @@ -6479,6 +6518,7 @@ bool simple_wallet::accept_loaded_tx(const std::function<size_t()> get_num_txes, if (!payment_id_string.empty()) payment_id_string += ", "; payment_id_string = std::string("unencrypted payment ID ") + epee::string_tools::pod_to_hex(payment_id); + payment_id_string += " (OBSOLETE)"; } } } @@ -6574,8 +6614,8 @@ bool simple_wallet::accept_loaded_tx(const std::function<size_t()> get_num_txes, change_string += tr("no change"); uint64_t fee = amount - amount_to_dests; - std::string prompt_str = (boost::format(tr("Loaded %lu transactions, for %s, fee %s, %s, %s, with min ring size %lu, %s. %sIs this okay? (Y/Yes/N/No): ")) % (unsigned long)get_num_txes() % print_money(amount) % print_money(fee) % dest_string % change_string % (unsigned long)min_ring_size % payment_id_string % extra_message).str(); - return command_line::is_yes(input_line(prompt_str)); + std::string prompt_str = (boost::format(tr("Loaded %lu transactions, for %s, fee %s, %s, %s, with min ring size %lu, %s. %sIs this okay?")) % (unsigned long)get_num_txes() % print_money(amount) % print_money(fee) % dest_string % change_string % (unsigned long)min_ring_size % payment_id_string % extra_message).str(); + return command_line::is_yes(input_line(prompt_str, true)); } //---------------------------------------------------------------------------------------------------- bool simple_wallet::accept_loaded_tx(const tools::wallet2::unsigned_tx_set &txs) @@ -7296,6 +7336,7 @@ bool simple_wallet::get_transfers(std::vector<std::string>& local_args, std::vec const std::string type = pd.m_coinbase ? tr("block") : tr("in"); const bool unlocked = m_wallet->is_transfer_unlocked(pd.m_unlock_time, pd.m_block_height); transfers.push_back({ + type, pd.m_block_height, pd.m_timestamp, type, @@ -7328,6 +7369,7 @@ bool simple_wallet::get_transfers(std::vector<std::string>& local_args, std::vec payment_id = payment_id.substr(0,16); std::string note = m_wallet->get_tx_note(i->first); transfers.push_back({ + "out", pd.m_block_height, pd.m_timestamp, "out", @@ -7365,6 +7407,7 @@ bool simple_wallet::get_transfers(std::vector<std::string>& local_args, std::vec double_spend_note = tr("[Double spend seen on the network: this transaction may or may not end up being mined] "); transfers.push_back({ "pool", + "pool", pd.m_timestamp, "in", false, @@ -7405,6 +7448,7 @@ bool simple_wallet::get_transfers(std::vector<std::string>& local_args, std::vec if ((failed && is_failed) || (!is_failed && pending)) { transfers.push_back({ (is_failed ? "failed" : "pending"), + (is_failed ? "failed" : "pending"), pd.m_timestamp, "out", false, @@ -7452,7 +7496,7 @@ bool simple_wallet::show_transfers(const std::vector<std::string> &args_) for (const auto& transfer : all_transfers) { - const auto color = transfer.confirmed ? ((transfer.direction == "in" || transfer.direction == "block") ? console_color_green : console_color_magenta) : console_color_white; + const auto color = transfer.type == "failed" ? console_color_red : transfer.confirmed ? ((transfer.direction == "in" || transfer.direction == "block") ? console_color_green : console_color_magenta) : console_color_default; std::string destinations = "-"; if (!transfer.outputs.empty()) @@ -7729,7 +7773,7 @@ bool simple_wallet::rescan_blockchain(const std::vector<std::string> &args_) { message_writer() << tr("Warning: this will lose any information which can not be recovered from the blockchain."); message_writer() << tr("This includes destination addresses, tx secret keys, tx notes, etc"); - std::string confirm = input_line(tr("Rescan anyway ? (Y/Yes/N/No): ")); + std::string confirm = input_line(tr("Rescan anyway?"), true); if(!std::cin.eof()) { if (!command_line::is_yes(confirm)) @@ -8203,12 +8247,13 @@ bool simple_wallet::address_book(const std::vector<std::string> &args/* = std::v { if (tools::wallet2::parse_long_payment_id(args[3], payment_id)) { + LONG_PAYMENT_ID_SUPPORT_CHECK(); description_start += 2; } else if (tools::wallet2::parse_short_payment_id(args[3], info.payment_id)) { - memcpy(payment_id.data, info.payment_id.data, 8); - description_start += 2; + fail_msg_writer() << tr("Short payment IDs are to be used within an integrated address only"); + return true; } else { @@ -8246,7 +8291,7 @@ bool simple_wallet::address_book(const std::vector<std::string> &args/* = std::v auto& row = address_book[i]; success_msg_writer() << tr("Index: ") << i; success_msg_writer() << tr("Address: ") << get_account_address_as_str(m_wallet->nettype(), row.m_is_subaddress, row.m_address); - success_msg_writer() << tr("Payment ID: ") << row.m_payment_id; + success_msg_writer() << tr("Payment ID: ") << row.m_payment_id << " (OBSOLETE)"; success_msg_writer() << tr("Description: ") << row.m_description << "\n"; } } @@ -8924,10 +8969,12 @@ int main(int argc, char* argv[]) command_line::add_arg(desc_params, arg_electrum_seed ); command_line::add_arg(desc_params, arg_allow_mismatched_daemon_version); command_line::add_arg(desc_params, arg_restore_height); + command_line::add_arg(desc_params, arg_restore_date); command_line::add_arg(desc_params, arg_do_not_relay); command_line::add_arg(desc_params, arg_create_address_file); command_line::add_arg(desc_params, arg_subaddress_lookahead); command_line::add_arg(desc_params, arg_use_english_language_names); + command_line::add_arg(desc_params, arg_long_payment_id_support); po::positional_options_description positional_options; positional_options.add(arg_command.name, -1); diff --git a/src/simplewallet/simplewallet.h b/src/simplewallet/simplewallet.h index e49da8c18..c3dc16d96 100644 --- a/src/simplewallet/simplewallet.h +++ b/src/simplewallet/simplewallet.h @@ -256,6 +256,7 @@ namespace cryptonote struct transfer_view { + std::string type; boost::variant<uint64_t, std::string> block; uint64_t timestamp; std::string direction; @@ -369,6 +370,7 @@ namespace cryptonote std::string m_mnemonic_language; std::string m_import_path; std::string m_subaddress_lookahead; + std::string m_restore_date; // optional - converted to m_restore_height epee::wipeable_string m_electrum_seed; // electrum-style seed parameter @@ -396,6 +398,8 @@ namespace cryptonote bool m_auto_refresh_refreshing; std::atomic<bool> m_in_manual_refresh; uint32_t m_current_subaddress_account; + + bool m_long_payment_id_support; // MMS mms::message_store& get_message_store() const { return m_wallet->get_message_store(); }; diff --git a/src/wallet/api/wallet.cpp b/src/wallet/api/wallet.cpp index 7cd3b65bb..595dbac5e 100644 --- a/src/wallet/api/wallet.cpp +++ b/src/wallet/api/wallet.cpp @@ -942,6 +942,12 @@ uint64_t WalletImpl::approximateBlockChainHeight() const { return m_wallet->get_approximate_blockchain_height(); } + +uint64_t WalletImpl::estimateBlockChainHeight() const +{ + return m_wallet->estimate_blockchain_height(); +} + uint64_t WalletImpl::daemonBlockChainHeight() const { if(m_wallet->light_wallet()) { @@ -1149,7 +1155,7 @@ std::string WalletImpl::getSubaddressLabel(uint32_t accountIndex, uint32_t addre } catch (const std::exception &e) { - LOG_ERROR("Error getting subaddress label: ") << e.what(); + LOG_ERROR("Error getting subaddress label: " << e.what()); setStatusError(string(tr("Failed to get subaddress label: ")) + e.what()); return ""; } @@ -1162,7 +1168,7 @@ void WalletImpl::setSubaddressLabel(uint32_t accountIndex, uint32_t addressIndex } catch (const std::exception &e) { - LOG_ERROR("Error setting subaddress label: ") << e.what(); + LOG_ERROR("Error setting subaddress label: " << e.what()); setStatusError(string(tr("Failed to set subaddress label: ")) + e.what()); } } @@ -1179,7 +1185,7 @@ string WalletImpl::getMultisigInfo() const { clearStatus(); return m_wallet->get_multisig_info(); } catch (const exception& e) { - LOG_ERROR("Error on generating multisig info: ") << e.what(); + LOG_ERROR("Error on generating multisig info: " << e.what()); setStatusError(string(tr("Failed to get multisig info: ")) + e.what()); } @@ -1196,7 +1202,7 @@ string WalletImpl::makeMultisig(const vector<string>& info, uint32_t threshold) return m_wallet->make_multisig(epee::wipeable_string(m_password), info, threshold); } catch (const exception& e) { - LOG_ERROR("Error on making multisig wallet: ") << e.what(); + LOG_ERROR("Error on making multisig wallet: " << e.what()); setStatusError(string(tr("Failed to make multisig: ")) + e.what()); } @@ -1210,7 +1216,7 @@ std::string WalletImpl::exchangeMultisigKeys(const std::vector<std::string> &inf return m_wallet->exchange_multisig_keys(epee::wipeable_string(m_password), info); } catch (const exception& e) { - LOG_ERROR("Error on exchanging multisig keys: ") << e.what(); + LOG_ERROR("Error on exchanging multisig keys: " << e.what()); setStatusError(string(tr("Failed to make multisig: ")) + e.what()); } @@ -1228,7 +1234,7 @@ bool WalletImpl::finalizeMultisig(const vector<string>& extraMultisigInfo) { setStatusError(tr("Failed to finalize multisig wallet creation")); } catch (const exception& e) { - LOG_ERROR("Error on finalizing multisig wallet creation: ") << e.what(); + LOG_ERROR("Error on finalizing multisig wallet creation: " << e.what()); setStatusError(string(tr("Failed to finalize multisig wallet creation: ")) + e.what()); } @@ -1244,7 +1250,7 @@ bool WalletImpl::exportMultisigImages(string& images) { images = epee::string_tools::buff_to_hex_nodelimer(blob); return true; } catch (const exception& e) { - LOG_ERROR("Error on exporting multisig images: ") << e.what(); + LOG_ERROR("Error on exporting multisig images: " << e.what()); setStatusError(string(tr("Failed to export multisig images: ")) + e.what()); } @@ -1272,7 +1278,7 @@ size_t WalletImpl::importMultisigImages(const vector<string>& images) { return m_wallet->import_multisig(blobs); } catch (const exception& e) { - LOG_ERROR("Error on importing multisig images: ") << e.what(); + LOG_ERROR("Error on importing multisig images: " << e.what()); setStatusError(string(tr("Failed to import multisig images: ")) + e.what()); } @@ -1286,7 +1292,7 @@ bool WalletImpl::hasMultisigPartialKeyImages() const { return m_wallet->has_multisig_partial_key_images(); } catch (const exception& e) { - LOG_ERROR("Error on checking for partial multisig key images: ") << e.what(); + LOG_ERROR("Error on checking for partial multisig key images: " << e.what()); setStatusError(string(tr("Failed to check for partial multisig key images: ")) + e.what()); } @@ -1314,7 +1320,7 @@ PendingTransaction* WalletImpl::restoreMultisigTransaction(const string& signDat return ptx; } catch (exception& e) { - LOG_ERROR("Error on restoring multisig transaction: ") << e.what(); + LOG_ERROR("Error on restoring multisig transaction: " << e.what()); setStatusError(string(tr("Failed to restore multisig transaction: ")) + e.what()); } diff --git a/src/wallet/api/wallet.h b/src/wallet/api/wallet.h index b4637b8e6..55240d64f 100644 --- a/src/wallet/api/wallet.h +++ b/src/wallet/api/wallet.h @@ -109,6 +109,7 @@ public: uint64_t unlockedBalance(uint32_t accountIndex = 0) const override; uint64_t blockChainHeight() const override; uint64_t approximateBlockChainHeight() const override; + uint64_t estimateBlockChainHeight() const override; uint64_t daemonBlockChainHeight() const override; uint64_t daemonBlockChainTargetHeight() const override; bool synchronized() const override; diff --git a/src/wallet/api/wallet2_api.h b/src/wallet/api/wallet2_api.h index 82627de29..5c301974f 100644 --- a/src/wallet/api/wallet2_api.h +++ b/src/wallet/api/wallet2_api.h @@ -575,6 +575,12 @@ struct Wallet virtual uint64_t approximateBlockChainHeight() const = 0; /** + * @brief estimateBlockChainHeight - returns estimate blockchain height. More accurate than approximateBlockChainHeight, + * uses daemon height and falls back to calculation from date/time + * @return + **/ + virtual uint64_t estimateBlockChainHeight() const = 0; + /** * @brief daemonBlockChainHeight - returns daemon blockchain height * @return 0 - in case error communicating with the daemon. * status() will return Status_Error and errorString() will return verbose error description diff --git a/src/wallet/api/wallet_manager.cpp b/src/wallet/api/wallet_manager.cpp index 5b262f1b7..89fe01c0d 100644 --- a/src/wallet/api/wallet_manager.cpp +++ b/src/wallet/api/wallet_manager.cpp @@ -127,6 +127,8 @@ Wallet *WalletManagerImpl::createWalletFromDevice(const std::string &path, WalletImpl * wallet = new WalletImpl(nettype, kdf_rounds); if(restoreHeight > 0){ wallet->setRefreshFromBlockHeight(restoreHeight); + } else { + wallet->setRefreshFromBlockHeight(wallet->estimateBlockChainHeight()); } auto lookahead = tools::parse_subaddress_lookahead(subaddressLookahead); if (lookahead) diff --git a/src/wallet/message_store.cpp b/src/wallet/message_store.cpp index ce6f4f52b..7381005c1 100644 --- a/src/wallet/message_store.cpp +++ b/src/wallet/message_store.cpp @@ -125,7 +125,7 @@ void message_store::set_signer(const multisig_wallet_state &state, const boost::optional<std::string> &transport_address, const boost::optional<cryptonote::account_public_address> monero_address) { - THROW_WALLET_EXCEPTION_IF(index >= m_num_authorized_signers, tools::error::wallet_internal_error, "Invalid signer index " + index); + THROW_WALLET_EXCEPTION_IF(index >= m_num_authorized_signers, tools::error::wallet_internal_error, "Invalid signer index " + std::to_string(index)); authorized_signer &m = m_signers[index]; if (label) { @@ -146,7 +146,7 @@ void message_store::set_signer(const multisig_wallet_state &state, const authorized_signer &message_store::get_signer(uint32_t index) const { - THROW_WALLET_EXCEPTION_IF(index >= m_num_authorized_signers, tools::error::wallet_internal_error, "Invalid signer index " + index); + THROW_WALLET_EXCEPTION_IF(index >= m_num_authorized_signers, tools::error::wallet_internal_error, "Invalid signer index " + std::to_string(index)); return m_signers[index]; } @@ -201,7 +201,7 @@ void message_store::unpack_signer_config(const multisig_wallet_state &state, con THROW_WALLET_EXCEPTION_IF(true, tools::error::wallet_internal_error, "Invalid structure of signer config"); } uint32_t num_signers = (uint32_t)signers.size(); - THROW_WALLET_EXCEPTION_IF(num_signers != m_num_authorized_signers, tools::error::wallet_internal_error, "Wrong number of signers in config: " + num_signers); + THROW_WALLET_EXCEPTION_IF(num_signers != m_num_authorized_signers, tools::error::wallet_internal_error, "Wrong number of signers in config: " + std::to_string(num_signers)); } void message_store::process_signer_config(const multisig_wallet_state &state, const std::string &signer_config) @@ -424,7 +424,7 @@ void message_store::setup_signer_for_auto_config(uint32_t index, const std::stri // auto-config parameters. In the wallet of somebody using the token to send auto-config // data the auto-config parameters are stored in the "me" signer and taken from there // to send that data. - THROW_WALLET_EXCEPTION_IF(index >= m_num_authorized_signers, tools::error::wallet_internal_error, "Invalid signer index " + index); + THROW_WALLET_EXCEPTION_IF(index >= m_num_authorized_signers, tools::error::wallet_internal_error, "Invalid signer index " + std::to_string(index)); authorized_signer &m = m_signers[index]; m.auto_config_token = token; crypto::hash_to_scalar(token.data(), token.size(), m.auto_config_secret_key); @@ -506,7 +506,7 @@ void message_store::process_wallet_created_data(const multisig_wallet_state &sta break; default: - THROW_WALLET_EXCEPTION(tools::error::wallet_internal_error, "Illegal message type " + (uint32_t)type); + THROW_WALLET_EXCEPTION(tools::error::wallet_internal_error, "Illegal message type " + std::to_string((uint32_t)type)); break; } } @@ -573,7 +573,7 @@ size_t message_store::get_message_index_by_id(uint32_t id) const { size_t index; bool found = get_message_index_by_id(id, index); - THROW_WALLET_EXCEPTION_IF(!found, tools::error::wallet_internal_error, "Invalid message id " + id); + THROW_WALLET_EXCEPTION_IF(!found, tools::error::wallet_internal_error, "Invalid message id " + std::to_string(id)); return index; } @@ -601,7 +601,7 @@ message message_store::get_message_by_id(uint32_t id) const { message m; bool found = get_message_by_id(id, m); - THROW_WALLET_EXCEPTION_IF(!found, tools::error::wallet_internal_error, "Invalid message id " + id); + THROW_WALLET_EXCEPTION_IF(!found, tools::error::wallet_internal_error, "Invalid message id " + std::to_string(id)); return m; } diff --git a/src/wallet/wallet2.cpp b/src/wallet/wallet2.cpp index 2080c1832..c6b70ee2e 100644 --- a/src/wallet/wallet2.cpp +++ b/src/wallet/wallet2.cpp @@ -192,6 +192,37 @@ namespace return false; } + + void add_reason(std::string &reasons, const char *reason) + { + if (!reasons.empty()) + reasons += ", "; + reasons += reason; + } + + std::string get_text_reason(const cryptonote::COMMAND_RPC_SEND_RAW_TX::response &res) + { + std::string reason; + if (res.low_mixin) + add_reason(reason, "bad ring size"); + if (res.double_spend) + add_reason(reason, "double spend"); + if (res.invalid_input) + add_reason(reason, "invalid input"); + if (res.invalid_output) + add_reason(reason, "invalid output"); + if (res.too_big) + add_reason(reason, "too big"); + if (res.overspend) + add_reason(reason, "overspend"); + if (res.fee_too_low) + add_reason(reason, "fee too low"); + if (res.not_rct) + add_reason(reason, "tx is not ringct"); + if (res.not_relayed) + add_reason(reason, "tx was not relayed"); + return reason; + } } namespace @@ -583,19 +614,6 @@ std::pair<std::unique_ptr<tools::wallet2>, tools::password_container> generate_f return {nullptr, tools::password_container{}}; } -static void throw_on_rpc_response_error(const boost::optional<std::string> &status, const char *method) -{ - // no error - if (!status) - return; - - // empty string -> not connection - THROW_WALLET_EXCEPTION_IF(status->empty(), tools::error::no_connection_to_daemon, method); - - THROW_WALLET_EXCEPTION_IF(*status == CORE_RPC_STATUS_BUSY, tools::error::daemon_busy, method); - THROW_WALLET_EXCEPTION_IF(*status != CORE_RPC_STATUS_OK, tools::error::wallet_generic_rpc_error, method, *status); -} - std::string strjoin(const std::vector<size_t> &V, const char *sep) { std::stringstream ss; @@ -902,6 +920,7 @@ wallet2::wallet2(network_type nettype, uint64_t kdf_rounds, bool unattended): m_multisig(false), m_multisig_threshold(0), m_node_rpc_proxy(m_http_client, m_daemon_rpc_mutex), + m_account_public_address{crypto::null_pkey, crypto::null_pkey}, m_subaddress_lookahead_major(SUBADDRESS_LOOKAHEAD_MAJOR), m_subaddress_lookahead_minor(SUBADDRESS_LOOKAHEAD_MINOR), m_light_wallet(false), @@ -918,6 +937,7 @@ wallet2::wallet2(network_type nettype, uint64_t kdf_rounds, bool unattended): m_last_block_reward(0), m_encrypt_keys_after_refresh(boost::none), m_unattended(unattended), + m_devices_registered(false), m_device_last_key_image_sync(0) { } @@ -1661,7 +1681,25 @@ void wallet2::process_new_transaction(const crypto::hash &txid, const cryptonote td.m_txid = txid; td.m_key_image = tx_scan_info[o].ki; td.m_key_image_known = !m_watch_only && !m_multisig; - td.m_key_image_requested = false; + if (!td.m_key_image_known) + { + // we might have cold signed, and have a mapping to key images + std::unordered_map<crypto::public_key, crypto::key_image>::const_iterator i = m_cold_key_images.find(tx_scan_info[o].in_ephemeral.pub); + if (i != m_cold_key_images.end()) + { + td.m_key_image = i->second; + td.m_key_image_known = true; + } + } + if (m_watch_only) + { + // for view wallets, that flag means "we want to request it" + td.m_key_image_request = true; + } + else + { + td.m_key_image_request = false; + } td.m_key_image_partial = m_multisig; td.m_amount = amount; td.m_pk_index = pk_index - 1; @@ -1683,7 +1721,7 @@ void wallet2::process_new_transaction(const crypto::hash &txid, const cryptonote td.m_rct = false; } set_unspent(m_transfers.size()-1); - if (!m_multisig && !m_watch_only) + if (td.m_key_image_known) m_key_images[td.m_key_image] = m_transfers.size()-1; m_pub_keys[tx_scan_info[o].in_ephemeral.pub] = m_transfers.size()-1; if (output_tracker_cache) @@ -2124,7 +2162,7 @@ void wallet2::pull_blocks(uint64_t start_height, uint64_t &blocks_start_height, m_daemon_rpc_mutex.unlock(); THROW_WALLET_EXCEPTION_IF(!r, error::no_connection_to_daemon, "getblocks.bin"); THROW_WALLET_EXCEPTION_IF(res.status == CORE_RPC_STATUS_BUSY, error::daemon_busy, "getblocks.bin"); - THROW_WALLET_EXCEPTION_IF(res.status != CORE_RPC_STATUS_OK, error::get_blocks_error, res.status); + THROW_WALLET_EXCEPTION_IF(res.status != CORE_RPC_STATUS_OK, error::get_blocks_error, get_rpc_status(res.status)); THROW_WALLET_EXCEPTION_IF(res.blocks.size() != res.output_indices.size(), error::wallet_internal_error, "mismatched blocks (" + boost::lexical_cast<std::string>(res.blocks.size()) + ") and output_indices (" + boost::lexical_cast<std::string>(res.output_indices.size()) + ") sizes from daemon"); @@ -2146,7 +2184,7 @@ void wallet2::pull_hashes(uint64_t start_height, uint64_t &blocks_start_height, m_daemon_rpc_mutex.unlock(); THROW_WALLET_EXCEPTION_IF(!r, error::no_connection_to_daemon, "gethashes.bin"); THROW_WALLET_EXCEPTION_IF(res.status == CORE_RPC_STATUS_BUSY, error::daemon_busy, "gethashes.bin"); - THROW_WALLET_EXCEPTION_IF(res.status != CORE_RPC_STATUS_OK, error::get_hashes_error, res.status); + THROW_WALLET_EXCEPTION_IF(res.status != CORE_RPC_STATUS_OK, error::get_hashes_error, get_rpc_status(res.status)); blocks_start_height = res.start_height; hashes = std::move(res.m_block_ids); @@ -2191,7 +2229,6 @@ void wallet2::process_parsed_blocks(uint64_t start_height, const std::vector<cry const cryptonote::account_keys &keys = m_account.get_keys(); auto gender = [&](wallet2::is_out_data &iod) { - boost::unique_lock<hw::device> hwdev_lock(hwdev); if (!hwdev.generate_key_derivation(iod.pkey, keys.m_view_secret_key, iod.derivation)) { MWARNING("Failed to generate key derivation from tx pubkey, skipping"); @@ -2200,12 +2237,16 @@ void wallet2::process_parsed_blocks(uint64_t start_height, const std::vector<cry } }; - for (auto &slot: tx_cache_data) + for (size_t i = 0; i < tx_cache_data.size(); ++i) { - for (auto &iod: slot.primary) - tpool.submit(&waiter, [&gender, &iod]() { gender(iod); }, true); - for (auto &iod: slot.additional) - tpool.submit(&waiter, [&gender, &iod]() { gender(iod); }, true); + tpool.submit(&waiter, [&hwdev, &gender, &tx_cache_data, i]() { + auto &slot = tx_cache_data[i]; + boost::unique_lock<hw::device> hwdev_lock(hwdev); + for (auto &iod: slot.primary) + gender(iod); + for (auto &iod: slot.additional) + gender(iod); + }, true); } waiter.wait(&tpool); @@ -2305,11 +2346,10 @@ void wallet2::pull_and_parse_next_blocks(uint64_t start_height, uint64_t &blocks THROW_WALLET_EXCEPTION_IF(prev_blocks.size() != prev_parsed_blocks.size(), error::wallet_internal_error, "size mismatch"); // prepend the last 3 blocks, should be enough to guard against a block or two's reorg - std::vector<parsed_block>::const_reverse_iterator i = prev_parsed_blocks.rbegin(); - for (size_t n = 0; n < std::min((size_t)3, prev_parsed_blocks.size()); ++n) + auto s = std::next(prev_parsed_blocks.rbegin(), std::min((size_t)3, prev_parsed_blocks.size())).base(); + for (; s != prev_parsed_blocks.end(); ++s) { - short_chain_history.push_front(i->hash); - ++i; + short_chain_history.push_front(s->hash); } // pull the new blocks @@ -2608,7 +2648,7 @@ void wallet2::update_pool_state(bool refreshed) } else { - LOG_PRINT_L0("Error calling gettransactions daemon RPC: r " << r << ", status " << res.status); + LOG_PRINT_L0("Error calling gettransactions daemon RPC: r " << r << ", status " << get_rpc_status(res.status)); } } MTRACE("update_pool_state end"); @@ -2796,13 +2836,16 @@ void wallet2::refresh(bool trusted_daemon, uint64_t start_height, uint64_t & blo bool first = true; while(m_run.load(std::memory_order_relaxed)) { + uint64_t next_blocks_start_height; + std::vector<cryptonote::block_complete_entry> next_blocks; + std::vector<parsed_block> next_parsed_blocks; + bool error; try { // pull the next set of blocks while we're processing the current one - uint64_t next_blocks_start_height; - std::vector<cryptonote::block_complete_entry> next_blocks; - std::vector<parsed_block> next_parsed_blocks; - bool error = false; + error = false; + next_blocks.clear(); + next_parsed_blocks.clear(); added_blocks = 0; if (!first && blocks.empty()) { @@ -2840,6 +2883,11 @@ void wallet2::refresh(bool trusted_daemon, uint64_t start_height, uint64_t & blo start_height = stop_height; throw std::runtime_error(""); // loop again } + catch (const std::exception &e) + { + MERROR("Error parsing blocks: " << e.what()); + error = true; + } blocks_fetched += added_blocks; } waiter.wait(&tpool); @@ -4481,6 +4529,23 @@ std::string wallet2::make_multisig(const epee::wipeable_string &password, bool wallet2::finalize_multisig(const epee::wipeable_string &password, const std::unordered_set<crypto::public_key> &pkeys, std::vector<crypto::public_key> signers) { + bool ready; + uint32_t threshold, total; + if (!multisig(&ready, &threshold, &total)) + { + MERROR("This is not a multisig wallet"); + return false; + } + if (ready) + { + MERROR("This multisig wallet is already finalized"); + return false; + } + if (threshold + 1 != total) + { + MERROR("finalize_multisig should only be used for N-1/N wallets, use exchange_multisig_keys instead"); + return false; + } exchange_multisig_keys(password, pkeys, signers); return true; } @@ -5280,7 +5345,7 @@ void wallet2::rescan_spent() m_daemon_rpc_mutex.unlock(); THROW_WALLET_EXCEPTION_IF(!r, error::no_connection_to_daemon, "is_key_image_spent"); THROW_WALLET_EXCEPTION_IF(daemon_resp.status == CORE_RPC_STATUS_BUSY, error::daemon_busy, "is_key_image_spent"); - THROW_WALLET_EXCEPTION_IF(daemon_resp.status != CORE_RPC_STATUS_OK, error::is_key_image_spent_error, daemon_resp.status); + THROW_WALLET_EXCEPTION_IF(daemon_resp.status != CORE_RPC_STATUS_OK, error::is_key_image_spent_error, get_rpc_status(daemon_resp.status)); THROW_WALLET_EXCEPTION_IF(daemon_resp.spent_status.size() != n_outputs, error::wallet_internal_error, "daemon returned wrong response for is_key_image_spent, wrong amounts count = " + std::to_string(daemon_resp.spent_status.size()) + ", expected " + std::to_string(n_outputs)); @@ -5613,7 +5678,7 @@ void wallet2::commit_tx(pending_tx& ptx) m_daemon_rpc_mutex.unlock(); THROW_WALLET_EXCEPTION_IF(!r, error::no_connection_to_daemon, "submit_raw_tx"); // MyMonero and OpenMonero use different status strings - THROW_WALLET_EXCEPTION_IF(ores.status != "OK" && ores.status != "success" , error::tx_rejected, ptx.tx, ores.status, ores.error); + THROW_WALLET_EXCEPTION_IF(ores.status != "OK" && ores.status != "success" , error::tx_rejected, ptx.tx, get_rpc_status(ores.status), ores.error); } else { @@ -5627,7 +5692,7 @@ void wallet2::commit_tx(pending_tx& ptx) m_daemon_rpc_mutex.unlock(); THROW_WALLET_EXCEPTION_IF(!r, error::no_connection_to_daemon, "sendrawtransaction"); THROW_WALLET_EXCEPTION_IF(daemon_send_resp.status == CORE_RPC_STATUS_BUSY, error::daemon_busy, "sendrawtransaction"); - THROW_WALLET_EXCEPTION_IF(daemon_send_resp.status != CORE_RPC_STATUS_OK, error::tx_rejected, ptx.tx, daemon_send_resp.status, daemon_send_resp.reason); + THROW_WALLET_EXCEPTION_IF(daemon_send_resp.status != CORE_RPC_STATUS_OK, error::tx_rejected, ptx.tx, get_rpc_status(daemon_send_resp.status), get_text_reason(daemon_send_resp)); // sanity checks for (size_t idx: ptx.selected_transfers) { @@ -5877,6 +5942,61 @@ bool wallet2::sign_tx(unsigned_tx_set &exported_txs, std::vector<wallet2::pendin txs.back().additional_tx_keys = additional_tx_keys; } + // add key image mapping for these txes + const account_keys &keys = get_account().get_keys(); + hw::device &hwdev = m_account.get_device(); + for (size_t n = 0; n < exported_txs.txes.size(); ++n) + { + const cryptonote::transaction &tx = signed_txes.ptx[n].tx; + + crypto::key_derivation derivation; + std::vector<crypto::key_derivation> additional_derivations; + + // compute public keys from out secret keys + crypto::public_key tx_pub_key; + crypto::secret_key_to_public_key(txs[n].tx_key, tx_pub_key); + std::vector<crypto::public_key> additional_tx_pub_keys; + for (const crypto::secret_key &skey: txs[n].additional_tx_keys) + { + additional_tx_pub_keys.resize(additional_tx_pub_keys.size() + 1); + crypto::secret_key_to_public_key(skey, additional_tx_pub_keys.back()); + } + + // compute derivations + hwdev.set_mode(hw::device::TRANSACTION_PARSE); + if (!hwdev.generate_key_derivation(tx_pub_key, keys.m_view_secret_key, derivation)) + { + MWARNING("Failed to generate key derivation from tx pubkey in " << cryptonote::get_transaction_hash(tx) << ", skipping"); + static_assert(sizeof(derivation) == sizeof(rct::key), "Mismatched sizes of key_derivation and rct::key"); + memcpy(&derivation, rct::identity().bytes, sizeof(derivation)); + } + for (size_t i = 0; i < additional_tx_pub_keys.size(); ++i) + { + additional_derivations.push_back({}); + if (!hwdev.generate_key_derivation(additional_tx_pub_keys[i], keys.m_view_secret_key, additional_derivations.back())) + { + MWARNING("Failed to generate key derivation from additional tx pubkey in " << cryptonote::get_transaction_hash(tx) << ", skipping"); + memcpy(&additional_derivations.back(), rct::identity().bytes, sizeof(crypto::key_derivation)); + } + } + + for (size_t i = 0; i < tx.vout.size(); ++i) + { + if (tx.vout[i].target.type() != typeid(cryptonote::txout_to_key)) + continue; + const cryptonote::txout_to_key &out = boost::get<cryptonote::txout_to_key>(tx.vout[i].target); + // if this output is back to this wallet, we can calculate its key image already + if (!is_out_to_acc_precomp(m_subaddresses, out.key, derivation, additional_derivations, i, hwdev)) + continue; + crypto::key_image ki; + cryptonote::keypair in_ephemeral; + if (generate_key_image_helper(keys, m_subaddresses, out.key, tx_pub_key, additional_tx_pub_keys, i, in_ephemeral, ki, hwdev)) + signed_txes.tx_key_images[out.key] = ki; + else + MERROR("Failed to calculate key image"); + } + } + // add key images signed_txes.key_images.resize(m_transfers.size()); for (size_t i = 0; i < m_transfers.size(); ++i) @@ -6039,6 +6159,10 @@ bool wallet2::parse_tx_from_str(const std::string &signed_tx_st, std::vector<too bool r = import_key_images(signed_txs.key_images); if (!r) return false; + // remember key images for this tx, for when we get those txes from the blockchain + for (const auto &e: signed_txs.tx_key_images) + m_cold_key_images.insert(e); + ptx = signed_txs.ptx; return true; @@ -6518,7 +6642,7 @@ uint32_t wallet2::adjust_priority(uint32_t priority) m_daemon_rpc_mutex.unlock(); THROW_WALLET_EXCEPTION_IF(!r, error::no_connection_to_daemon, "getblockheadersrange"); THROW_WALLET_EXCEPTION_IF(getbh_res.status == CORE_RPC_STATUS_BUSY, error::daemon_busy, "getblockheadersrange"); - THROW_WALLET_EXCEPTION_IF(getbh_res.status != CORE_RPC_STATUS_OK, error::get_blocks_error, getbh_res.status); + THROW_WALLET_EXCEPTION_IF(getbh_res.status != CORE_RPC_STATUS_OK, error::get_blocks_error, get_rpc_status(getbh_res.status)); if (getbh_res.headers.size() != N) { MERROR("Bad blockheaders size"); @@ -6980,7 +7104,7 @@ void wallet2::get_outs(std::vector<std::vector<tools::wallet2::get_outs_entry>> m_daemon_rpc_mutex.unlock(); THROW_WALLET_EXCEPTION_IF(!r, error::no_connection_to_daemon, "transfer_selected"); THROW_WALLET_EXCEPTION_IF(resp_t.status == CORE_RPC_STATUS_BUSY, error::daemon_busy, "get_output_histogram"); - THROW_WALLET_EXCEPTION_IF(resp_t.status != CORE_RPC_STATUS_OK, error::get_histogram_error, resp_t.status); + THROW_WALLET_EXCEPTION_IF(resp_t.status != CORE_RPC_STATUS_OK, error::get_histogram_error, get_rpc_status(resp_t.status)); } // if we want to segregate fake outs pre or post fork, get distribution @@ -7003,7 +7127,7 @@ void wallet2::get_outs(std::vector<std::vector<tools::wallet2::get_outs_entry>> m_daemon_rpc_mutex.unlock(); THROW_WALLET_EXCEPTION_IF(!r, error::no_connection_to_daemon, "transfer_selected"); THROW_WALLET_EXCEPTION_IF(resp_t.status == CORE_RPC_STATUS_BUSY, error::daemon_busy, "get_output_distribution"); - THROW_WALLET_EXCEPTION_IF(resp_t.status != CORE_RPC_STATUS_OK, error::get_output_distribution, resp_t.status); + THROW_WALLET_EXCEPTION_IF(resp_t.status != CORE_RPC_STATUS_OK, error::get_output_distribution, get_rpc_status(resp_t.status)); // check we got all data for(size_t idx: selected_transfers) @@ -7402,7 +7526,7 @@ void wallet2::get_outs(std::vector<std::vector<tools::wallet2::get_outs_entry>> m_daemon_rpc_mutex.unlock(); THROW_WALLET_EXCEPTION_IF(!r, error::no_connection_to_daemon, "get_outs.bin"); THROW_WALLET_EXCEPTION_IF(daemon_resp.status == CORE_RPC_STATUS_BUSY, error::daemon_busy, "get_outs.bin"); - THROW_WALLET_EXCEPTION_IF(daemon_resp.status != CORE_RPC_STATUS_OK, error::get_outs_error, daemon_resp.status); + THROW_WALLET_EXCEPTION_IF(daemon_resp.status != CORE_RPC_STATUS_OK, error::get_outs_error, get_rpc_status(daemon_resp.status)); THROW_WALLET_EXCEPTION_IF(daemon_resp.outs.size() != req.outputs.size(), error::wallet_internal_error, "daemon returned wrong response for get_outs.bin, wrong amounts count = " + std::to_string(daemon_resp.outs.size()) + ", expected " + std::to_string(req.outputs.size())); @@ -8237,7 +8361,7 @@ void wallet2::light_wallet_get_unspent_outs() td.m_key_image = unspent_key_image; td.m_key_image_known = !m_watch_only && !m_multisig; - td.m_key_image_requested = false; + td.m_key_image_request = false; td.m_key_image_partial = m_multisig; td.m_amount = o.amount; td.m_pk_index = 0; @@ -10550,7 +10674,10 @@ uint64_t wallet2::get_daemon_blockchain_height(string &err) const boost::optional<std::string> result = m_node_rpc_proxy.get_height(height); if (result) { - err = *result; + if (m_trusted_daemon) + err = *result; + else + err = "daemon error"; return 0; } @@ -10565,7 +10692,10 @@ uint64_t wallet2::get_daemon_blockchain_target_height(string &err) const auto result = m_node_rpc_proxy.get_target_height(target_height); if (result && *result != CORE_RPC_STATUS_OK) { - err= *result; + if (m_trusted_daemon) + err = *result; + else + err = "daemon error"; return 0; } return target_height; @@ -10833,23 +10963,23 @@ bool wallet2::export_key_images(const std::string &filename) const } //---------------------------------------------------------------------------------------------------- -std::pair<size_t, std::vector<std::pair<crypto::key_image, crypto::signature>>> wallet2::export_key_images() const +std::pair<size_t, std::vector<std::pair<crypto::key_image, crypto::signature>>> wallet2::export_key_images(bool all) const { PERF_TIMER(export_key_images_raw); std::vector<std::pair<crypto::key_image, crypto::signature>> ski; size_t offset = 0; - while (offset < m_transfers.size() && !m_transfers[offset].m_key_image_requested) - ++offset; + if (!all) + { + while (offset < m_transfers.size() && !m_transfers[offset].m_key_image_request) + ++offset; + } ski.reserve(m_transfers.size() - offset); for (size_t n = offset; n < m_transfers.size(); ++n) { const transfer_details &td = m_transfers[n]; - crypto::hash hash; - crypto::cn_fast_hash(&td.m_key_image, sizeof(td.m_key_image), hash); - // get ephemeral public key const cryptonote::tx_out &out = td.m_tx.vout[td.m_internal_output_index]; THROW_WALLET_EXCEPTION_IF(out.target.type() != typeid(txout_to_key), error::wallet_internal_error, @@ -11001,7 +11131,7 @@ uint64_t wallet2::import_key_images(const std::vector<std::pair<crypto::key_imag m_transfers[n + offset].m_key_image = signed_key_images[n].first; m_key_images[m_transfers[n + offset].m_key_image] = n + offset; m_transfers[n + offset].m_key_image_known = true; - m_transfers[n + offset].m_key_image_requested = false; + m_transfers[n + offset].m_key_image_request = false; m_transfers[n + offset].m_key_image_partial = false; } PERF_TIMER_STOP(import_key_images_B); @@ -11220,7 +11350,7 @@ bool wallet2::import_key_images(std::vector<crypto::key_image> key_images) td.m_key_image = key_images[i]; m_key_images[m_transfers[i].m_key_image] = i; td.m_key_image_known = true; - td.m_key_image_requested = false; + td.m_key_image_request = false; td.m_key_image_partial = false; m_pub_keys[m_transfers[i].get_public_key()] = i; } @@ -11292,7 +11422,7 @@ std::pair<size_t, std::vector<tools::wallet2::transfer_details>> wallet2::export std::vector<tools::wallet2::transfer_details> outs; size_t offset = 0; - while (offset < m_transfers.size() && m_transfers[offset].m_key_image_known) + while (offset < m_transfers.size() && (m_transfers[offset].m_key_image_known && !m_transfers[offset].m_key_image_request)) ++offset; outs.reserve(m_transfers.size() - offset); @@ -11336,7 +11466,7 @@ size_t wallet2::import_outputs(const std::pair<size_t, std::vector<tools::wallet const size_t original_size = m_transfers.size(); m_transfers.resize(offset + outputs.second.size()); for (size_t i = 0; i < offset; ++i) - m_transfers[i].m_key_image_requested = false; + m_transfers[i].m_key_image_request = false; for (size_t i = 0; i < outputs.second.size(); ++i) { transfer_details td = outputs.second[i]; @@ -11377,7 +11507,7 @@ process: THROW_WALLET_EXCEPTION_IF(!r, error::wallet_internal_error, "Failed to generate key image"); expand_subaddresses(td.m_subaddr_index); td.m_key_image_known = true; - td.m_key_image_requested = true; + td.m_key_image_request = true; td.m_key_image_partial = false; THROW_WALLET_EXCEPTION_IF(in_ephemeral.pub != out_key, error::wallet_internal_error, "key_image generated ephemeral public key not matched with output_key at index " + boost::lexical_cast<std::string>(i + offset)); @@ -11623,7 +11753,7 @@ void wallet2::update_multisig_rescan_info(const std::vector<std::vector<rct::key m_key_images.erase(td.m_key_image); td.m_key_image = get_multisig_composite_key_image(n); td.m_key_image_known = true; - td.m_key_image_requested = false; + td.m_key_image_request = false; td.m_key_image_partial = false; td.m_multisig_k = multisig_k[n]; m_key_images[td.m_key_image] = n; @@ -12006,7 +12136,7 @@ uint64_t wallet2::get_blockchain_height_by_date(uint16_t year, uint8_t month, ui else if (res.status == CORE_RPC_STATUS_BUSY) oss << "daemon is busy"; else - oss << res.status; + oss << get_rpc_status(res.status); throw std::runtime_error(oss.str()); } cryptonote::block blk_min, blk_mid, blk_max; @@ -12225,4 +12355,27 @@ void wallet2::on_passphrase_request(bool on_device, epee::wipeable_string & pass if (0 != m_callback) m_callback->on_passphrase_request(on_device, passphrase); } +//---------------------------------------------------------------------------------------------------- +std::string wallet2::get_rpc_status(const std::string &s) const +{ + if (m_trusted_daemon) + return s; + return "<error>"; +} +//---------------------------------------------------------------------------------------------------- +void wallet2::throw_on_rpc_response_error(const boost::optional<std::string> &status, const char *method) const +{ + // no error + if (!status) + return; + + MERROR("RPC error: " << method << ": status " << *status); + + // empty string -> not connection + THROW_WALLET_EXCEPTION_IF(status->empty(), tools::error::no_connection_to_daemon, method); + + THROW_WALLET_EXCEPTION_IF(*status == CORE_RPC_STATUS_BUSY, tools::error::daemon_busy, method); + THROW_WALLET_EXCEPTION_IF(*status != CORE_RPC_STATUS_OK, tools::error::wallet_generic_rpc_error, method, m_trusted_daemon ? *status : "daemon error"); +} + } diff --git a/src/wallet/wallet2.h b/src/wallet/wallet2.h index 5b1988080..ed92aec19 100644 --- a/src/wallet/wallet2.h +++ b/src/wallet/wallet2.h @@ -267,7 +267,7 @@ namespace tools uint64_t m_amount; bool m_rct; bool m_key_image_known; - bool m_key_image_requested; + bool m_key_image_request; // view wallets: we want to request it; cold wallets: it was requested size_t m_pk_index; cryptonote::subaddress_index m_subaddr_index; bool m_key_image_partial; @@ -292,7 +292,7 @@ namespace tools FIELD(m_amount) FIELD(m_rct) FIELD(m_key_image_known) - FIELD(m_key_image_requested) + FIELD(m_key_image_request) FIELD(m_pk_index) FIELD(m_subaddr_index) FIELD(m_key_image_partial) @@ -448,6 +448,7 @@ namespace tools { std::vector<pending_tx> ptx; std::vector<crypto::key_image> key_images; + std::unordered_map<crypto::public_key, crypto::key_image> tx_key_images; }; struct multisig_tx_set @@ -927,6 +928,9 @@ namespace tools if(ver < 27) return; a & m_device_last_key_image_sync; + if(ver < 28) + return; + a & m_cold_key_images; } /*! @@ -1113,7 +1117,7 @@ namespace tools std::tuple<size_t, crypto::hash, std::vector<crypto::hash>> export_blockchain() const; void import_blockchain(const std::tuple<size_t, crypto::hash, std::vector<crypto::hash>> &bc); bool export_key_images(const std::string &filename) const; - std::pair<size_t, std::vector<std::pair<crypto::key_image, crypto::signature>>> export_key_images() const; + std::pair<size_t, std::vector<std::pair<crypto::key_image, crypto::signature>>> export_key_images(bool all = false) const; uint64_t import_key_images(const std::vector<std::pair<crypto::key_image, crypto::signature>> &signed_key_images, size_t offset, uint64_t &spent, uint64_t &unspent, bool check_spent = true); uint64_t import_key_images(const std::string &filename, uint64_t &spent, uint64_t &unspent); bool import_key_images(std::vector<crypto::key_image> key_images); @@ -1326,6 +1330,9 @@ namespace tools void on_pin_request(epee::wipeable_string & pin); void on_passphrase_request(bool on_device, epee::wipeable_string & passphrase); + std::string get_rpc_status(const std::string &s) const; + void throw_on_rpc_response_error(const boost::optional<std::string> &status, const char *method) const; + cryptonote::account_base m_account; boost::optional<epee::net_utils::http::login> m_daemon_login; std::string m_daemon_address; @@ -1355,6 +1362,7 @@ namespace tools uint64_t m_upper_transaction_weight_limit; //TODO: auto-calc this value or request from daemon, now use some fixed value const std::vector<std::vector<tools::wallet2::multisig_info>> *m_multisig_rescan_info; const std::vector<std::vector<rct::key>> *m_multisig_rescan_k; + std::unordered_map<crypto::public_key, crypto::key_image> m_cold_key_images; std::atomic<bool> m_run; @@ -1449,7 +1457,7 @@ namespace tools std::unique_ptr<wallet_device_callback> m_device_callback; }; } -BOOST_CLASS_VERSION(tools::wallet2, 27) +BOOST_CLASS_VERSION(tools::wallet2, 28) BOOST_CLASS_VERSION(tools::wallet2::transfer_details, 11) BOOST_CLASS_VERSION(tools::wallet2::multisig_info, 1) BOOST_CLASS_VERSION(tools::wallet2::multisig_info::LR, 0) @@ -1461,7 +1469,7 @@ BOOST_CLASS_VERSION(tools::wallet2::confirmed_transfer_details, 6) BOOST_CLASS_VERSION(tools::wallet2::address_book_row, 17) BOOST_CLASS_VERSION(tools::wallet2::reserve_proof_entry, 0) BOOST_CLASS_VERSION(tools::wallet2::unsigned_tx_set, 0) -BOOST_CLASS_VERSION(tools::wallet2::signed_tx_set, 0) +BOOST_CLASS_VERSION(tools::wallet2::signed_tx_set, 1) BOOST_CLASS_VERSION(tools::wallet2::tx_construction_data, 3) BOOST_CLASS_VERSION(tools::wallet2::pending_tx, 3) BOOST_CLASS_VERSION(tools::wallet2::multisig_sig, 0) @@ -1510,7 +1518,7 @@ namespace boost } if (ver < 10) { - x.m_key_image_requested = false; + x.m_key_image_request = false; } } @@ -1598,7 +1606,7 @@ namespace boost initialize_transfer_details(a, x, ver); return; } - a & x.m_key_image_requested; + a & x.m_key_image_request; if (ver < 11) return; a & x.m_uses; @@ -1798,6 +1806,9 @@ namespace boost { a & x.ptx; a & x.key_images; + if (ver < 1) + return; + a & x.tx_key_images; } template <class Archive> diff --git a/src/wallet/wallet_rpc_server.cpp b/src/wallet/wallet_rpc_server.cpp index d7dc2914e..dd65ee7fe 100644 --- a/src/wallet/wallet_rpc_server.cpp +++ b/src/wallet/wallet_rpc_server.cpp @@ -2495,7 +2495,7 @@ namespace tools if (!m_wallet) return not_open(er); try { - std::pair<size_t, std::vector<std::pair<crypto::key_image, crypto::signature>>> ski = m_wallet->export_key_images(); + std::pair<size_t, std::vector<std::pair<crypto::key_image, crypto::signature>>> ski = m_wallet->export_key_images(req.all); res.offset = ski.first; res.signed_key_images.resize(ski.second.size()); for (size_t n = 0; n < ski.second.size(); ++n) diff --git a/src/wallet/wallet_rpc_server_commands_defs.h b/src/wallet/wallet_rpc_server_commands_defs.h index afb8c6e91..f0c1a4e9d 100644 --- a/src/wallet/wallet_rpc_server_commands_defs.h +++ b/src/wallet/wallet_rpc_server_commands_defs.h @@ -47,7 +47,7 @@ // advance which version they will stop working with // Don't go over 32767 for any of these #define WALLET_RPC_VERSION_MAJOR 1 -#define WALLET_RPC_VERSION_MINOR 6 +#define WALLET_RPC_VERSION_MINOR 7 #define MAKE_WALLET_RPC_VERSION(major,minor) (((major)<<16)|(minor)) #define WALLET_RPC_VERSION MAKE_WALLET_RPC_VERSION(WALLET_RPC_VERSION_MAJOR, WALLET_RPC_VERSION_MINOR) namespace tools @@ -1565,7 +1565,10 @@ namespace wallet_rpc { struct request { + bool all; + BEGIN_KV_SERIALIZE_MAP() + KV_SERIALIZE_OPT(all, false); END_KV_SERIALIZE_MAP() }; diff --git a/tests/README.md b/tests/README.md index 0bf097254..001ab6154 100644 --- a/tests/README.md +++ b/tests/README.md @@ -27,7 +27,7 @@ To run the same tests on a release build, replace `debug` with `release`. # Crypto Tests -Crypto tests are located under the `tests/crypto` directory. +Crypto tests are located under the `tests/crypto` directory. - `crypto-tests.h` contains test harness headers - `main.cpp` implements the driver for the crypto tests @@ -50,7 +50,7 @@ To run the same tests on a release build, replace `debug` with `release`. # Functional tests [TODO] -Functional tests are located under the `tests/functional` directory. +Functional tests are located under the `tests/functional` directory. First, run a regtest daemon in the offline mode and with a fixed difficulty: ``` diff --git a/tests/core_tests/chaingen.h b/tests/core_tests/chaingen.h index b380aca01..907b32bcd 100644 --- a/tests/core_tests/chaingen.h +++ b/tests/core_tests/chaingen.h @@ -669,7 +669,9 @@ inline bool do_replay_file(const std::string& filename) } #define GENERATE_AND_PLAY(genclass) \ - if (filter.empty() || boost::regex_match(std::string(#genclass), match, boost::regex(filter))) \ + if (list_tests) \ + std::cout << #genclass << std::endl; \ + else if (filter.empty() || boost::regex_match(std::string(#genclass), match, boost::regex(filter))) \ { \ std::vector<test_event_entry> events; \ ++tests_count; \ diff --git a/tests/core_tests/chaingen_main.cpp b/tests/core_tests/chaingen_main.cpp index 84254cfdb..71b8c4463 100644 --- a/tests/core_tests/chaingen_main.cpp +++ b/tests/core_tests/chaingen_main.cpp @@ -44,6 +44,7 @@ namespace const command_line::arg_descriptor<bool> arg_generate_and_play_test_data = {"generate_and_play_test_data", ""}; const command_line::arg_descriptor<bool> arg_test_transactions = {"test_transactions", ""}; const command_line::arg_descriptor<std::string> arg_filter = { "filter", "Regular expression filter for which tests to run" }; + const command_line::arg_descriptor<bool> arg_list_tests = {"list_tests", ""}; } int main(int argc, char* argv[]) @@ -64,6 +65,7 @@ int main(int argc, char* argv[]) command_line::add_arg(desc_options, arg_generate_and_play_test_data); command_line::add_arg(desc_options, arg_test_transactions); command_line::add_arg(desc_options, arg_filter); + command_line::add_arg(desc_options, arg_list_tests); po::variables_map vm; bool r = command_line::handle_error_helper(desc_options, [&]() @@ -87,6 +89,7 @@ int main(int argc, char* argv[]) size_t tests_count = 0; std::vector<std::string> failed_tests; std::string tests_folder = command_line::get_arg(vm, arg_test_data_path); + bool list_tests = false; if (command_line::get_arg(vm, arg_generate_test_data)) { GENERATE("chain001.dat", gen_simple_chain_001); @@ -95,7 +98,7 @@ int main(int argc, char* argv[]) { PLAY("chain001.dat", gen_simple_chain_001); } - else if (command_line::get_arg(vm, arg_generate_and_play_test_data)) + else if (command_line::get_arg(vm, arg_generate_and_play_test_data) || (list_tests = command_line::get_arg(vm, arg_list_tests))) { GENERATE_AND_PLAY(gen_simple_chain_001); GENERATE_AND_PLAY(gen_simple_chain_split_1); @@ -229,10 +232,10 @@ int main(int argc, char* argv[]) GENERATE_AND_PLAY(gen_multisig_tx_valid_25_1_2_many_inputs); GENERATE_AND_PLAY(gen_multisig_tx_valid_48_1_234); GENERATE_AND_PLAY(gen_multisig_tx_valid_48_1_234_many_inputs); - GENERATE_AND_PLAY(gen_multisig_tx_valid_24_1_no_signers); - GENERATE_AND_PLAY(gen_multisig_tx_valid_25_1_no_signers); - GENERATE_AND_PLAY(gen_multisig_tx_valid_48_1_no_signers); - GENERATE_AND_PLAY(gen_multisig_tx_valid_48_1_23_no_threshold); + GENERATE_AND_PLAY(gen_multisig_tx_invalid_24_1_no_signers); + GENERATE_AND_PLAY(gen_multisig_tx_invalid_25_1_no_signers); + GENERATE_AND_PLAY(gen_multisig_tx_invalid_48_1_no_signers); + GENERATE_AND_PLAY(gen_multisig_tx_invalid_48_1_23_no_threshold); GENERATE_AND_PLAY(gen_bp_tx_valid_1); GENERATE_AND_PLAY(gen_bp_tx_invalid_1_1); @@ -251,9 +254,12 @@ int main(int argc, char* argv[]) GENERATE_AND_PLAY(gen_bp_tx_invalid_borromean_type); el::Level level = (failed_tests.empty() ? el::Level::Info : el::Level::Error); - MLOG(level, "\nREPORT:"); - MLOG(level, " Test run: " << tests_count); - MLOG(level, " Failures: " << failed_tests.size()); + if (!list_tests) + { + MLOG(level, "\nREPORT:"); + MLOG(level, " Test run: " << tests_count); + MLOG(level, " Failures: " << failed_tests.size()); + } if (!failed_tests.empty()) { MLOG(level, "FAILED TESTS:"); diff --git a/tests/core_tests/multisig.cpp b/tests/core_tests/multisig.cpp index fe5feb942..46ba997be 100644 --- a/tests/core_tests/multisig.cpp +++ b/tests/core_tests/multisig.cpp @@ -644,28 +644,28 @@ bool gen_multisig_tx_invalid_45_5_23_no_threshold::generate(std::vector<test_eve return generate_with(events, 2, mixin, amount_paid, false, 4, 5, 5, {2, 3}, NULL, NULL); } -bool gen_multisig_tx_valid_24_1_no_signers::generate(std::vector<test_event_entry>& events) const +bool gen_multisig_tx_invalid_24_1_no_signers::generate(std::vector<test_event_entry>& events) const { const size_t mixin = 4; const uint64_t amount_paid = 10000; return generate_with(events, 2, mixin, amount_paid, false, 2, 4, 1, {}, NULL, NULL); } -bool gen_multisig_tx_valid_25_1_no_signers::generate(std::vector<test_event_entry>& events) const +bool gen_multisig_tx_invalid_25_1_no_signers::generate(std::vector<test_event_entry>& events) const { const size_t mixin = 4; const uint64_t amount_paid = 10000; return generate_with(events, 2, mixin, amount_paid, false, 2, 5, 1, {}, NULL, NULL); } -bool gen_multisig_tx_valid_48_1_no_signers::generate(std::vector<test_event_entry>& events) const +bool gen_multisig_tx_invalid_48_1_no_signers::generate(std::vector<test_event_entry>& events) const { const size_t mixin = 4; const uint64_t amount_paid = 10000; return generate_with(events, 2, mixin, amount_paid, false, 4, 8, 1, {}, NULL, NULL); } -bool gen_multisig_tx_valid_48_1_23_no_threshold::generate(std::vector<test_event_entry>& events) const +bool gen_multisig_tx_invalid_48_1_23_no_threshold::generate(std::vector<test_event_entry>& events) const { const size_t mixin = 4; const uint64_t amount_paid = 10000; diff --git a/tests/core_tests/multisig.h b/tests/core_tests/multisig.h index 9e4cb3a23..e0d227840 100644 --- a/tests/core_tests/multisig.h +++ b/tests/core_tests/multisig.h @@ -234,26 +234,26 @@ struct gen_multisig_tx_invalid_45_5_23_no_threshold: public gen_multisig_tx_vali }; template<> struct get_test_options<gen_multisig_tx_invalid_45_5_23_no_threshold>: public get_test_options<gen_multisig_tx_validation_base> {}; -struct gen_multisig_tx_valid_24_1_no_signers: public gen_multisig_tx_validation_base +struct gen_multisig_tx_invalid_24_1_no_signers: public gen_multisig_tx_validation_base { bool generate(std::vector<test_event_entry>& events) const; }; -template<> struct get_test_options<gen_multisig_tx_valid_24_1_no_signers>: public get_test_options<gen_multisig_tx_validation_base> {}; +template<> struct get_test_options<gen_multisig_tx_invalid_24_1_no_signers>: public get_test_options<gen_multisig_tx_validation_base> {}; -struct gen_multisig_tx_valid_25_1_no_signers: public gen_multisig_tx_validation_base +struct gen_multisig_tx_invalid_25_1_no_signers: public gen_multisig_tx_validation_base { bool generate(std::vector<test_event_entry>& events) const; }; -template<> struct get_test_options<gen_multisig_tx_valid_25_1_no_signers>: public get_test_options<gen_multisig_tx_validation_base> {}; +template<> struct get_test_options<gen_multisig_tx_invalid_25_1_no_signers>: public get_test_options<gen_multisig_tx_validation_base> {}; -struct gen_multisig_tx_valid_48_1_no_signers: public gen_multisig_tx_validation_base +struct gen_multisig_tx_invalid_48_1_no_signers: public gen_multisig_tx_validation_base { bool generate(std::vector<test_event_entry>& events) const; }; -template<> struct get_test_options<gen_multisig_tx_valid_48_1_no_signers>: public get_test_options<gen_multisig_tx_validation_base> {}; +template<> struct get_test_options<gen_multisig_tx_invalid_48_1_no_signers>: public get_test_options<gen_multisig_tx_validation_base> {}; -struct gen_multisig_tx_valid_48_1_23_no_threshold: public gen_multisig_tx_validation_base +struct gen_multisig_tx_invalid_48_1_23_no_threshold: public gen_multisig_tx_validation_base { bool generate(std::vector<test_event_entry>& events) const; }; -template<> struct get_test_options<gen_multisig_tx_valid_48_1_23_no_threshold>: public get_test_options<gen_multisig_tx_validation_base> {}; +template<> struct get_test_options<gen_multisig_tx_invalid_48_1_23_no_threshold>: public get_test_options<gen_multisig_tx_validation_base> {}; diff --git a/tests/functional_tests/transactions_flow_test.cpp b/tests/functional_tests/transactions_flow_test.cpp index ffe500d21..7e5b73415 100644 --- a/tests/functional_tests/transactions_flow_test.cpp +++ b/tests/functional_tests/transactions_flow_test.cpp @@ -29,6 +29,7 @@ // Parts of this file are originally copyright (c) 2012-2013 The Cryptonote developers #include <boost/uuid/uuid.hpp> +#include <boost/uuid/uuid_io.hpp> #include <boost/uuid/random_generator.hpp> #include <unordered_map> diff --git a/tests/fuzz/levin.cpp b/tests/fuzz/levin.cpp index d0c5803f5..573abd1d3 100644 --- a/tests/fuzz/levin.cpp +++ b/tests/fuzz/levin.cpp @@ -65,22 +65,22 @@ namespace { } - virtual int invoke(int command, const std::string& in_buff, std::string& buff_out, test_levin_connection_context& context) + virtual int invoke(int command, const epee::span<const uint8_t> in_buff, std::string& buff_out, test_levin_connection_context& context) { m_invoke_counter.inc(); boost::unique_lock<boost::mutex> lock(m_mutex); m_last_command = command; - m_last_in_buf = in_buff; + m_last_in_buf = std::string((const char*)in_buff.data(), in_buff.size()); buff_out = m_invoke_out_buf; return m_return_code; } - virtual int notify(int command, const std::string& in_buff, test_levin_connection_context& context) + virtual int notify(int command, const epee::span<const uint8_t> in_buff, test_levin_connection_context& context) { m_notify_counter.inc(); boost::unique_lock<boost::mutex> lock(m_mutex); m_last_command = command; - m_last_in_buf = in_buff; + m_last_in_buf = std::string((const char*)in_buff.data(), in_buff.size()); return m_return_code; } diff --git a/tests/net_load_tests/net_load_tests.h b/tests/net_load_tests/net_load_tests.h index 7e92c21b9..6c4f7cb76 100644 --- a/tests/net_load_tests/net_load_tests.h +++ b/tests/net_load_tests/net_load_tests.h @@ -33,6 +33,7 @@ #include <atomic> #include <boost/asio/io_service.hpp> +#include <boost/uuid/uuid_io.hpp> #include "include_base_utils.h" #include "string_tools.h" @@ -62,7 +63,7 @@ namespace net_load_tests { } - virtual int invoke(int command, const std::string& in_buff, std::string& buff_out, test_connection_context& context) + virtual int invoke(int command, const epee::span<const uint8_t> in_buff, std::string& buff_out, test_connection_context& context) { //m_invoke_counter.inc(); //std::unique_lock<std::mutex> lock(m_mutex); @@ -73,7 +74,7 @@ namespace net_load_tests return LEVIN_OK; } - virtual int notify(int command, const std::string& in_buff, test_connection_context& context) + virtual int notify(int command, const epee::span<const uint8_t> in_buff, test_connection_context& context) { //m_notify_counter.inc(); //std::unique_lock<std::mutex> lock(m_mutex); diff --git a/tests/unit_tests/epee_levin_protocol_handler_async.cpp b/tests/unit_tests/epee_levin_protocol_handler_async.cpp index 10e62c167..9ea71875b 100644 --- a/tests/unit_tests/epee_levin_protocol_handler_async.cpp +++ b/tests/unit_tests/epee_levin_protocol_handler_async.cpp @@ -56,22 +56,22 @@ namespace { } - virtual int invoke(int command, const std::string& in_buff, std::string& buff_out, test_levin_connection_context& context) + virtual int invoke(int command, const epee::span<const uint8_t> in_buff, std::string& buff_out, test_levin_connection_context& context) { m_invoke_counter.inc(); boost::unique_lock<boost::mutex> lock(m_mutex); m_last_command = command; - m_last_in_buf = in_buff; + m_last_in_buf = std::string((const char*)in_buff.data(), in_buff.size()); buff_out = m_invoke_out_buf; return m_return_code; } - virtual int notify(int command, const std::string& in_buff, test_levin_connection_context& context) + virtual int notify(int command, const epee::span<const uint8_t> in_buff, test_levin_connection_context& context) { m_notify_counter.inc(); boost::unique_lock<boost::mutex> lock(m_mutex); m_last_command = command; - m_last_in_buf = in_buff; + m_last_in_buf = std::string((const char*)in_buff.data(), in_buff.size()); return m_return_code; } diff --git a/tests/unit_tests/epee_utils.cpp b/tests/unit_tests/epee_utils.cpp index c384ce9a5..3d5882d7d 100644 --- a/tests/unit_tests/epee_utils.cpp +++ b/tests/unit_tests/epee_utils.cpp @@ -46,9 +46,11 @@ #include "hex.h" #include "net/net_utils_base.h" #include "net/local_ip.h" +#include "net/buffer.h" #include "p2p/net_peerlist_boost_serialization.h" #include "span.h" #include "string_tools.h" +#include "storages/parserse_base_utils.h" namespace { @@ -764,3 +766,154 @@ TEST(NetUtils, PrivateRanges) ASSERT_EQ(is_local("0.0.30.172"), false); ASSERT_EQ(is_local("0.0.30.127"), false); } + +TEST(net_buffer, basic) +{ + epee::net_utils::buffer buf; + + ASSERT_EQ(buf.size(), 0); + EXPECT_THROW(buf.span(1), std::runtime_error); + buf.append("a", 1); + epee::span<const uint8_t> span = buf.span(1); + ASSERT_EQ(span.size(), 1); + ASSERT_EQ(span.data()[0], 'a'); + EXPECT_THROW(buf.span(2), std::runtime_error); + buf.append("bc", 2); + buf.erase(1); + EXPECT_THROW(buf.span(3), std::runtime_error); + span = buf.span(2); + ASSERT_EQ(span.size(), 2); + ASSERT_EQ(span.data()[0], 'b'); + ASSERT_EQ(span.data()[1], 'c'); + buf.erase(1); + EXPECT_THROW(buf.span(2), std::runtime_error); + span = buf.span(1); + ASSERT_EQ(span.size(), 1); + ASSERT_EQ(span.data()[0], 'c'); + EXPECT_THROW(buf.erase(2), std::runtime_error); + buf.erase(1); + EXPECT_EQ(buf.size(), 0); + EXPECT_THROW(buf.span(1), std::runtime_error); +} + +TEST(net_buffer, existing_capacity) +{ + epee::net_utils::buffer buf; + + buf.append("123456789", 9); + buf.erase(9); + buf.append("abc", 3); + buf.append("def", 3); + ASSERT_EQ(buf.size(), 6); + epee::span<const uint8_t> span = buf.span(6); + ASSERT_TRUE(!memcmp(span.data(), "abcdef", 6)); +} + +TEST(net_buffer, reallocate) +{ + epee::net_utils::buffer buf; + + buf.append(std::string(4000, ' ').c_str(), 4000); + buf.append(std::string(8000, '0').c_str(), 8000); + ASSERT_EQ(buf.size(), 12000); + epee::span<const uint8_t> span = buf.span(12000); + ASSERT_TRUE(!memcmp(span.data(), std::string(4000, ' ').c_str(), 4000)); + ASSERT_TRUE(!memcmp(span.data() + 4000, std::string(8000, '0').c_str(), 8000)); +} + +TEST(net_buffer, move) +{ + epee::net_utils::buffer buf; + + buf.append(std::string(400, ' ').c_str(), 400); + buf.erase(399); + buf.append(std::string(4000, '0').c_str(), 4000); + ASSERT_EQ(buf.size(), 4001); + epee::span<const uint8_t> span = buf.span(4001); + ASSERT_TRUE(!memcmp(span.data(), std::string(1, ' ').c_str(), 1)); + ASSERT_TRUE(!memcmp(span.data() + 1, std::string(4000, '0').c_str(), 4000)); +} + +TEST(parsing, isspace) +{ + ASSERT_FALSE(epee::misc_utils::parse::isspace(0)); + for (int c = 1; c < 256; ++c) + { + ASSERT_EQ(epee::misc_utils::parse::isspace(c), strchr("\r\n\t\f\v ", c) != NULL); + } +} + +TEST(parsing, isdigit) +{ + ASSERT_FALSE(epee::misc_utils::parse::isdigit(0)); + for (int c = 1; c < 256; ++c) + { + ASSERT_EQ(epee::misc_utils::parse::isdigit(c), strchr("0123456789", c) != NULL); + } +} + +TEST(parsing, number) +{ + boost::string_ref val; + std::string s; + std::string::const_iterator i; + + // the parser expects another character to end the number, and accepts things + // that aren't numbers, as it's meant as a pre-filter for strto* functions, + // so we just check that numbers get accepted, but don't test non numbers + + s = "0 "; + i = s.begin(); + epee::misc_utils::parse::match_number(i, s.end(), val); + ASSERT_EQ(val, "0"); + + s = "000 "; + i = s.begin(); + epee::misc_utils::parse::match_number(i, s.end(), val); + ASSERT_EQ(val, "000"); + + s = "10x"; + i = s.begin(); + epee::misc_utils::parse::match_number(i, s.end(), val); + ASSERT_EQ(val, "10"); + + s = "10.09/"; + i = s.begin(); + epee::misc_utils::parse::match_number(i, s.end(), val); + ASSERT_EQ(val, "10.09"); + + s = "-1.r"; + i = s.begin(); + epee::misc_utils::parse::match_number(i, s.end(), val); + ASSERT_EQ(val, "-1."); + + s = "-49.;"; + i = s.begin(); + epee::misc_utils::parse::match_number(i, s.end(), val); + ASSERT_EQ(val, "-49."); + + s = "0.78/"; + i = s.begin(); + epee::misc_utils::parse::match_number(i, s.end(), val); + ASSERT_EQ(val, "0.78"); + + s = "33E9$"; + i = s.begin(); + epee::misc_utils::parse::match_number(i, s.end(), val); + ASSERT_EQ(val, "33E9"); + + s = ".34e2="; + i = s.begin(); + epee::misc_utils::parse::match_number(i, s.end(), val); + ASSERT_EQ(val, ".34e2"); + + s = "-9.34e-2="; + i = s.begin(); + epee::misc_utils::parse::match_number(i, s.end(), val); + ASSERT_EQ(val, "-9.34e-2"); + + s = "+9.34e+03="; + i = s.begin(); + epee::misc_utils::parse::match_number(i, s.end(), val); + ASSERT_EQ(val, "+9.34e+03"); +} diff --git a/tests/unit_tests/testdb.h b/tests/unit_tests/testdb.h index 5d9ba5833..8f5cf70e8 100644 --- a/tests/unit_tests/testdb.h +++ b/tests/unit_tests/testdb.h @@ -89,14 +89,14 @@ public: virtual uint64_t get_tx_block_height(const crypto::hash& h) const { return 0; } virtual uint64_t get_num_outputs(const uint64_t& amount) const { return 1; } virtual uint64_t get_indexing_base() const { return 0; } - virtual cryptonote::output_data_t get_output_key(const uint64_t& amount, const uint64_t& index) { return cryptonote::output_data_t(); } + virtual cryptonote::output_data_t get_output_key(const uint64_t& amount, const uint64_t& index, bool include_commitmemt) const { return cryptonote::output_data_t(); } virtual cryptonote::tx_out_index get_output_tx_and_index_from_global(const uint64_t& index) const { return cryptonote::tx_out_index(); } virtual cryptonote::tx_out_index get_output_tx_and_index(const uint64_t& amount, const uint64_t& index) const { return cryptonote::tx_out_index(); } virtual void get_output_tx_and_index(const uint64_t& amount, const std::vector<uint64_t> &offsets, std::vector<cryptonote::tx_out_index> &indices) const {} - virtual void get_output_key(const epee::span<const uint64_t> &amounts, const std::vector<uint64_t> &offsets, std::vector<cryptonote::output_data_t> &outputs, bool allow_partial = false) {} + virtual void get_output_key(const epee::span<const uint64_t> &amounts, const std::vector<uint64_t> &offsets, std::vector<cryptonote::output_data_t> &outputs, bool allow_partial = false) const {} virtual bool can_thread_bulk_indices() const { return false; } virtual std::vector<uint64_t> get_tx_output_indices(const crypto::hash& h) const { return std::vector<uint64_t>(); } - virtual std::vector<uint64_t> get_tx_amount_output_indices(const uint64_t tx_index) const { return std::vector<uint64_t>(); } + virtual std::vector<std::vector<uint64_t>> get_tx_amount_output_indices(const uint64_t tx_index, size_t n_txes) const { return std::vector<std::vector<uint64_t>>(); } virtual bool has_key_image(const crypto::key_image& img) const { return false; } virtual void remove_block() { } virtual uint64_t add_transaction_data(const crypto::hash& blk_hash, const cryptonote::transaction& tx, const crypto::hash& tx_hash, const crypto::hash& tx_prunable_hash) {return 0;} diff --git a/utils/gpg_keys/erciccione.asc b/utils/gpg_keys/erciccione.asc new file mode 100644 index 000000000..109941d55 --- /dev/null +++ b/utils/gpg_keys/erciccione.asc @@ -0,0 +1,30 @@ +-----BEGIN PGP PUBLIC KEY BLOCK----- + +mQENBFohQ30BCADktKHeTg48Dm4oU9JPrfHXzY3sZtTQsqsQGmYiFT8nQjRnSzic +dZe4sh2W4UYQSLFw7pEb7a51ZYkz5+gFKcj3NmIaEpNra7+SEtszoS3IALHUyGGQ ++nFDxUHwTV5OeIUXPHs2AnCVZa4yEWmavXFiPL35+dcFkwyUA5psd97dFIbtkPQ4 +nT2RCc2emy6NHVE5L7pzlBe9CeFkFyovs847d3WDa/96TT4JMRm34qvUrSvOkqti +xrui8hGwNedAw64ObiJhQLa3rvQ8bCtTLNHhpdc+zhBi3L5nIdwquMd5Bd6zHfmW +HHyXEoRTZYMRabNBSR9FrsjGBiHG9wuEK2B7ABEBAAG0NWVyY2ljY2lvbmVAcHJv +dG9ubWFpbC5jb20gPGVyY2ljY2lvbmVAcHJvdG9ubWFpbC5jb20+iQE1BBABCAAp +BQJaIUN9BgsJBwgDAgkQdir4xgjlbN8EFQgKAgMWAgECGQECGwMCHgEAAFc+CAC2 +7BDCYPWfXCKtaqP/jzay46nai66hSyuIJ2sd4RSbQlVNCwFUQ2NtUwDxtzdzB3bk +AfsqrkdSS7V82bfZPOoYapWikjA3LGd2GHvClKNmIgIg3CvrAByD9okhLYOmxChU +5/FzMgSgDdbez4Z+XXzUothirHcHR7J+PtzqMZWusuyNqfXlWgPNPbAHG/hm5RLw +s6uGOLDqhJDiqi0HjC/p9ponPAJ3g0jfECnXFp7R32EgbBHExT/g8e3L4J3Q8y/E +WrZiQsIc89srANw5YIcjlr/i3PY8tyrelpLDVBj05802WYamYx10Zd2epM4NglBs ++YjiWpSM9fTi9QKFCHyAuQENBFohQ30BCADe4FJcus4e3FzsagKdAM/iFeZ4BE2l +gVgSg8CVWIYlTAmeK+SBPfsxvwOWxmx/KkirWn3hOSA2U6mp9+g6hr9rLqWNwatM +OsitrJlWG2aPcuAmlzLFvtH0Vbilh0m4B98g2XUa94HtrlGdPAoYPquS1HsfL284 +CHpQXXOypOS+/spTK/OlpdUaPaVyqVfE3kzXYqt24RbDcsOvUzSzVRCA8kBp6azm +1qoqDboP112Ax7OkV0FGc/kd5PNMPTOnVR+4wVXWhl6gAI1I2JFvo3EJlxtddyTB +Rkx06jbYRyPj9KCEC4CBx9qsNlo7TUHgObAr+CLX4I5hkEWW0pS98vlpABEBAAGJ +AR8EGAEIABMFAlohQ30JEHYq+MYI5WzfAhsMAAD1twgAxjbETzOUeq/bWwVoGzmh +SMJajZs2c5Pv0SAwi5QZCoRefp2HYfpWcBsfZG84B0EmJb9Wwu3wrXWBf96xMTFM +neU0+L2+RcH8uH7/C2MlVaZjFv5dPPJAtwbY2BQl1bX5DrEwZ8QH7IUXZmWfjmZ3 +ww5llfCAyVCxMn+tnIJnk/7HODY0t1LmusFvZsAVsWIgdndKImXM4CTzirfmDks8 +TxpXfaJzU3B3KTIzMgEvwtTLJfgf07Foy4ITIl+1zS7gLL0fXldtx6fJJMvANsLf +JrRjafh21qfJb3I9TUH8G7C0Ln3LspBGsWZ4f+fjq4LEg7SCOAJSeBwfOtMJeCrU +/Q== +=a70Z +-----END PGP PUBLIC KEY BLOCK----- |