This is an alpha release of the EOSIO History Tools. It includes database fillers
fill-rocksdb) which pull data from nodeos's State History Plugin, and a new
query engine (
wasm-ql-rocksdb) which supports queries defined by wasm, along
with an emulation of the legacy
/v1/ RPC API.
This alpha release is designed to solicit community feedback. There are several potential directions this toolset may take; we'd like feedback on which direction(s) may be most useful. Please create issues about changes you'd like to see going forward.
Since this is an alpha release, it will likely have incompatible changes in the future. Some of these may be driven by community feedback.
This release includes the following:
This release upgrades
fill-pg to support
nodeos v2.1.0. The remaining tools are still disabled and have not been upgraded.
The support for RocksDB is likely to be dropped entirely now that
nodeos can use it as a backend store. The schema used by
fill-pg has changed. At present no data migration tool is available, so data may be manually migrated, regenerated by replaying
from the desired block number, or exist side by side in two schemas, with older blocks in the v0.3.0 schema and newer blocks in
the new schema. Use of the
--pg-schema option will facilitate the transition. Schema differences include:
- addition of version numbers in column names for fields which are part of versioned structures, e.g. in the
- the addition of an action_trace_v1 table, where all new action traces are written
- new columns in tables with fields from versioned structures indicating which version is populated for the row, named with
- addition of a key_value table for storing data from the new nodeos storage mechanism of the same name.
Full details of the differences can be found via diff of plain text backups of each schema using
pg_dump, not included here
This release adds temporary workarounds to
fill-pg to support Nodeos 2.0. It also disables the remaining tools. If you would
like to test rocksdb support or wasm-ql support, stick with Nodeos 1.8 and the Alpha 0.2.0 release of History Tools.
- Removed the
- Removed the
- Temporarily disabled building everything except
- There are now 2 self-contained demonstrations in public Docker images. See container-demos for details.
- Talk: this demonstrates using wasm-ql to provide messages from on-chain conversations to clients in threaded order.
- Partial history: this demonstrates some of wasm-ql's chain and token queries on data drawn from one of the public EOSIO networks.
- Added RocksDB and removed LMDB. This has the following advantages:
- Filling outperforms both PostgreSQL and LMDB by considerable margins, both for partial history and for full history on large well-known chains.
- Database size for full history is much smaller than PostgreSQL.
- Database fillers have a new option
--fill-trxto filter transaction traces.
- Database fillers no longer need
--fill-skip-towhen starting from partial history.
- Database fillers now automatically reconnect to the State History Plugin.
- wasm-ql now uses a thread pool to handle queries.
--wql-threadscontrols the thread pool size.
- wasm-ql now uses eos-vm instead of SpiderMonkey. This simplifies the build process.
- wasm-ql can now serve static files. Enabled by the new
- SHiP connection handling moved to
state_history_connection.hpp. This file may aid users needing to write custom solutions which connect to the State History Plugin.