This tool was created to aggregate the EOS ERC20 token distribution in it’s entirety, acknowledge various scenarios, run validation on data and provide files representing balances in various states.
This tool can be used to generate snapshots for any period in a deterministic fashion until the EOS ERC20 token has been frozen. Once frozen, the tool will only produce deterministic final snapshots that represent the final state of the EOS ERC20 token distribution.
config.default.jsand rename the copy to
config.js. The parameters are described in the file, but for your convenience they are also listed below in greater detail. DO NOT REMOVE OR RENAME config.default.js
MySQL stores all the intricate details of the distribution and the contract. These instructions assume you already have MySQL configured locally or have a remote MySQL instance.
schema.sqllocated in the
./bindirectory of this project.
The instructions below are for IPC because it’s the recommended connection method. HTTP/WS has issues with multi-threading, and so you will be limited to a single thread for public key sync.
On most linux systems, the default IPC defined in config.default.js should work for most linux platforms.
parity --no-warp --mode active --tracing off --pruning fast --db-compaction ssd --jsonrpc-apis all --chain mainnet --cache-size 2048
If you’re on Mac, it’s suggested you pass a flag to parity to generate an IPC file, like below
parity --mode active --tracing off --pruning fast --db-compaction ssd --jsonrpc-apis all --chain mainnet --no-warp --cache-size 2048 --ipc-path /Users/youruser/jsonrpc.ipc
And then set that path in your config.js
it’s imperitive that you start parity with –no-warp. If you have a pre-existing parity configuration file you’ve modified, you’ll want to create a new configuration file for the snapshot.
Important: Since Parity v1.7.8
--warp is enabled by default. If you fail to configure with
no-warp you will have issues.
--db-compactionparameter for parity to
hdd, like so:
--cache-sizeas needed, this could provide some sync-speed improvements.
Please view Parity Documentation if you have issues with Parity.
If you’re taking a “final” snapshot, much of the configuration is automatic.
period. Block ranges for the snapshot will be determined by the opening block (first transaction from crowdsale contract) and the deterministic freeze block. This is to prevent user error.
If you’re taking an “ongoing” snapshot, change “period” to the period for which you would like to generate a snapshot. Note If you put in a period that hasn’t yet completed, the “last closed period” will override your choice.
From the root directory of this project directory, run the following:
npm install(first time)
node snapshot.js --load_config
Note See other useful configuration options, such as
datadirectory, will be a directory named after the period number you generated, for example:
./data/123.If it’s a final snapshot, into the
snapshot.csv- This file contains the Et um Address, EOS Key and EOS Balance of every address that registered correctly with the contract up to the configured period, and has a balance greater than the value set by
snapshot-unregistered.csv- This file contains the Ethereum Address and EOS Balance of every address that either failed to register or registered incorrectly with the crowdsale contract up to the configured period.
distribution.csv- This file includes the Ethereum Address and EOS balance of the entire distribution up to the configured period, with no rules or validation imposed.
snapshot.json- This file contains information about the snapshot-session that created the above files. This file should not be used for verification, but can be used for debugging and to help identify indeterminism.
overwrite_snapshot is set to true, all the above files will be put into the root directory of the project. If you are doing any development on this project, it’s suggested that you change this to
false (it’s enabled by default in the
There are three methods for configuration
period(integer, default: last closed period) The period to sync to, it will sync to the last block of any given period.
snapshot_minimum_balance(integer, default: 1) Minimum balance required for inclusion in snapshots.
overwrite_snapshot(boolean, default: true) Overwrites snapshots in root directory.
poll(boolean) Will finish the configured period, and then try the next one. If the next one isn’t ready (or tokens aren’t frozen) it will continue to try every 10 seconds until there’s a new period to process.
resume(boolean) Will resume from last synced data. If you successfully run period 1, and the run period 3 with resume, it will only sync contract data between 1 and 3 (instead of starting fresh) and will only update wallets with changes between 1 and 3. Important This functionality isn’t perfect, there are situations where this flag could produce a failed test and force a resync (run without resume)
only_produce_final_snapshot(boolean) Should be used in conjunction with poll. It will sync contract data, but will only calculate wallets after the tokens are frozen.
author(string, default: “Anonymous”) Optional meta to identify snapshotter
eth_node_type(http, ipc , ws) [default: http] Based on performance testing,
ipcis recommended for local configurations and
httpis recommended for cloud configurations.
eth_node_path(valid host/path) Relative to type as defined above
All of the above options can be set either in your config.js or as startup parameters, simply prepend the option with
-- so for example
--period=111. Startup parameters will override config.js parameters.
Misuse of these parameters without understanding the implications can result in inaccurate snapshot and/or result in error
recalculate_wallets(boolean, default: false) Recalculates wallet balances without syncing contracts. Useful for development when resyncing the contracts is unnecessary (for example, you’ve already synced the contracts for period 1, and simply need to recalculate wallet balances for period 1)
skip_web3_sync(boolean, default: false) Skips web3 sync requirement. Useful for development if you know that Parity is caught up to the period you want to sync to (for example, you want to run a snapshot on period 2, but ran period 300 last night, so you know Parity is caught up to Period 2.)
Because you need to sync the entire blockchain.
Because this distribution requires an ETL mechanism in order to determinstically aggregate the distribution.
Unclaimed tokens are attributed to the contributing address.
Tokens accidentally sent to one of the contracts are attributed to the sending address.
It is exposed to fallback registeration. Script will sync all ethereum public keys in block range. If it can locate a public key belonging to an unregsitered address, it will generate an EOS Public Key from the Ethereum Public Key. The Ethereum Private Key converted to EOS WIF should then match the EOS Public Key (NOTE: This is highly experimental, but has been tested to work with high confidence)
Addresses not exposed to fallback registrations that remain invalid are exported to
snapshot-unregistered.csv file, which could prove useful.
Determinstic index is the order of all wallets with respect to when they were seen by either of the contract’s (EOSCrowdsale and EOSTokens)
Account names the deterministic index encoded to byte32, and then padded with “genesis11111” up to 12 chars.
Because of this unresolved issue
Registration Fallback will attempt to find a public key for the address, and fallback register it.
NO! That sounds like a recipe for indeterminism. The script will choose the deterministic end block for final snapshot, it’s picks that block by detecting when the tokens were frozen. For ongoing snapshots the block ranges are determined by period, from the first block where the crowdsale had a transaction, to the last block of the defined period.
Data is corrupted, try running with
--recalculate_wallets, if that doesn’t work, run without
npm update npm install
You probably started your Parity node without
--no-warp (add –no-warp to parity startup), if using geth make sure
syncmode is set to “full” (For example:
First, run snapshot with
--verbose_mt, this will display the stdout of the child processes for public key sync. If there’s an error, look into it. Most likely you just need to
npm install. You can also add
BLUEBIRD_LONG_STACK_TRACES=1 to your snapshot startup like…
BLUEBIRD_LONG_STACK_TRACES=1 node snapshot...
This is related to the web3 fork mentioned in FAQ. Web3.js team runs lerna before releasing a package, this package is not listed on NPM, so your system will have to run lerna. I’m not doing the package process to keep the forked repo as close to origin as possible.
npm update npm install -g lerna npm install
node snapshot --load_config --poll --resume --period=350
If period 350 isn’t closed yet, the script will reset your period to the
last_closed_period, and then when finished try to sync again.
node snapshot --load_config --poll --resume --recalculate_wallets --period=350
This will recalculate the wallets, but resume on the contract data.
node snapshot --load_config --period=350 --recalculate_wallets
This is a nuke, it will truncate your contract tables and wallet table, recalculating wallets.
Note: Public Keys table is not deleted, if you need to delete it for whatever reason, it must be done manually
--verbose_mt to startup
Verbose MT will display stdout from child processes, it’s noisy, so it’s disabled by default.
There are some differences between “ongoing” and “final” snapshots that need to be mentioned.
balanceOf()method provided by EOSCrowdsale contract
balanceOf()function provided by Token Contract.
key()public constant from crowdsale contract, registrations are not possible after 23 hours after distribution finishes.
The snapshot parameters this software proposes are as follows
snapshot_minimum_balanceto be considered valid for snapshot export.
minimum_snapshot_balanceare exported to
snapshot.csvfile (ethereum address, EOS public key, balance)
minimum_snapshot_balance, will be exported to
snapshot_unregistered.csv(ethereum address, balance)
distribution.csvfile without any rules or validation applied (ethereum address, balance)
The script employs strict patterns to encourage predictable output, often at the expense of performance. The pattern is aggregate - calculate - validate and closely resembles an ETL or extract, transform and load pattern. This decision came after numerous iterations and determining that debugging from state was more efficient than debugging from logs.
Below is the script transposed to plain english.
state.block_stateand the deterministic freeze block (block tokens were frozen)
order by first_seen, addressin batches of 10000.
snapshottable, ordered by balance DESC.
snapshot_unregisteredtable, ordered by balance DESC.
walletstable without any rules of validation imposed to distribution.csv
snapshot.json - Snapshot meta data
minimum_snapshot_balancethat has correctly registered their ethereum address with the EOSCrowdsale contract using a valid EOS Public Key.
minimum_snapshot_balancethat has either incorrectly registered or failed to register their address with an EOS Public Key.