Relay all block data + statediffs over full node websocket subscription #8

Closed
i-norden wants to merge 17 commits from rpc_statediffs_at_head into staging

17 Commits

Author SHA1 Message Date
Ian Norden
edf001e1d2 edit to fix tests after rebase + linting + a semi-mock service test 2019-04-14 23:06:58 -05:00
Ian Norden
7b017e1cba adjust state diffs to form closer to the eth-ipld objects we need to publish + pass their proofs and paths along with them; adjust tests 2019-04-14 19:36:26 -05:00
Ian Norden
e158ac9d2f websocket stream of block + state diff data for seed node 2019-04-14 19:36:26 -05:00
Elizabeth Engelman
f74be7a644 Address PR comments 2019-04-14 19:36:26 -05:00
Elizabeth Engelman
a8fa37b73b Address PR comments 2019-04-14 19:35:50 -05:00
Elizabeth Engelman
71382c5b64 Add a sleep to the blockchain test 2019-04-14 19:35:50 -05:00
Elizabeth Engelman
33f9be20e4 Apply go fmt changes 2019-04-14 19:35:50 -05:00
Elizabeth Engelman
dfcdd64e2d Clean up log lines and remove unnecessary fields from builder 2019-04-14 19:35:50 -05:00
Elizabeth Engelman
bb46f6265e Use blockchain stateCache.OpenTrie for storage diffs 2019-04-14 19:35:50 -05:00
Elizabeth Engelman
b07eb751bd Make sure that statediff has been processed before pruning 2019-04-14 19:35:50 -05:00
Elizabeth Engelman
41ac23ab8a Use a node's LeafKey as an identifier instead of the address
It was proving difficult to find look the address up from a given path
with a full node (sometimes the value wouldn't exist in the disk db).
So, instead, for now we are using the node's LeafKey with is a Keccak256
hash of the address, so if we know the address we can figure out which
LeafKey it matches up to.
2019-04-14 19:34:34 -05:00
Elizabeth Engelman
6611de7801 Open a trie from the in-memory database 2019-04-14 19:34:34 -05:00
Elizabeth
824a363fe0 Gracefully exit geth command(#4) 2019-04-14 19:34:34 -05:00
Elizabeth Engelman
2765285d2c Apply gosimple changes to statediff 2019-04-14 19:34:34 -05:00
Elizabeth Engelman
3564c3c8d9 Apply goimports to statediff 2019-04-14 19:34:02 -05:00
Elizabeth Engelman
cc0676f9a5 Apply go fmt changes to statediff 2019-04-14 19:34:02 -05:00
Elizabeth
4b1ea6b8d4 Write state diff to CSV (#2)
* port statediff from 9b7fd9af80/statediff/statediff.go; minor fixes

* integrating state diff extracting, building, and persisting into geth processes

* work towards persisting created statediffs in ipfs; based off github.com/vulcanize/eth-block-extractor

* Add a state diff service

* Remove diff extractor from blockchain

* Update imports

* Move statediff on/off check to geth cmd config

* Update starting state diff service

* Add debugging logs for creating diff

* Add statediff extractor and builder tests and small refactoring

* Start to write statediff to a CSV

* Restructure statediff directory

* Pull CSV publishing methods into their own file

* Reformatting due to go fmt

* Add gomega to vendor dir

* Remove testing focuses

* Update statediff tests to use golang test pkg

instead of ginkgo

- builder_test
- extractor_test
- publisher_test

* Use hexutil.Encode instead of deprecated common.ToHex

* Remove OldValue from DiffBigInt and DiffUint64 fields

* Update builder test

* Remove old storage value from updated accounts

* Remove old values from created/deleted accounts

* Update publisher to account for only storing current account values

* Update service loop and fetching previous block

* Update testing

- remove statediff ginkgo test suite file
- move mocks to their own dir

* Updates per go fmt

* Updates to tests

* Pass statediff mode and path in through cli

* Return filename from publisher

* Remove some duplication in builder

* Remove code field from state diff output

this is the contract byte code, and it can still be obtained by querying
the db by the codeHash

* Consolidate acct diff structs for updated & updated/deleted accts

* Include block number in csv filename

* Clean up error logging

* Cleanup formatting, spelling, etc

* Address PR comments

* Add contract address and storage value to csv

* Refactor accumulating account row in csv publisher

* Add DiffStorage struct

* Add storage key to csv

* Address PR comments

* Fix publisher to include rows for accounts that don't have store updates

* Update builder test after merging in release/1.8

* Update test contract to include storage on contract intialization

- so that we're able to test that storage diffing works for created and
deleted accounts (not just updated accounts).

* Factor out a common trie iterator method in builder
2019-04-14 19:34:02 -05:00