Commit Graph

11548 Commits

Author SHA1 Message Date
Ian Norden
b7048a3d66 fix builder so that we handle account deletions properly and properly diff storage when an account is moved to a new path; update params 2020-05-13 12:40:19 -05:00
Ian Norden
b7c412d303 diff type for representing deleted accounts 2020-05-10 15:44:21 -05:00
Ian Norden
5c3e0d6437 complete tests for early mainnet blocks 2020-05-08 16:41:11 -05:00
Ian Norden
5598d67872 common.BytesToHash(path) => crypto.Keaccak256(hash) in builder; BytesToHash produces same hash for e.g. []byte{} and []byte{\x00} - prefix \x00 steps are inconsequential to the hash result 2020-05-08 11:58:22 -05:00
Ian Norden
b5a9ea52e0 builder test on mainnet blocks 2020-05-07 22:48:55 -05:00
Ian Norden
0e0ebd7cb3 rlp files for mainnet blocks 0-3, for tests 2020-05-07 22:48:55 -05:00
Ian Norden
8c9ac32245 add genesis block to test; handle block 0 in StateDiffAt 2020-05-07 22:48:49 -05:00
Ian Norden
d07dc803d2 adjust statediff builder tests to changes and extend to test intermediate nodes; golint 2020-03-11 13:25:36 -05:00
Ian Norden
c2c80f32d9 fix state diff builder: emit actual leaf nodes instead of value nodes; diff on the leaf not on the value; emit correct path for intermediate nodes 2020-03-11 13:25:27 -05:00
Ian Norden
c7bfb7eda3 include total difficulty to the payload 2020-03-05 10:12:09 -06:00
Ian Norden
6e05b78bd0 fix linter issues 2020-03-05 10:12:09 -06:00
Ian Norden
2395e70a61 statediff verison meta 2020-03-05 10:12:09 -06:00
Ian Norden
2364e07975 fixes after rebase 2020-03-05 10:11:31 -06:00
Ian Norden
8eb2e64435 review fixes 2020-03-05 10:11:31 -06:00
Ian Norden
a8b456883f rpc method to get statediff at specific block; requires archival node or the block be within the pruning range 2020-03-05 10:11:31 -06:00
Ian Norden
b3f10c9e69 relay receipts with the rest of the data + review fixes/changes 2020-03-05 10:11:31 -06:00
Ian Norden
f3e4e85cd7 adjust buffering to improve stability; doc.go; fix notifier
err handling
2020-03-05 10:11:31 -06:00
Ian Norden
ad87bad4de review fixes; proper method signature for api; adjust service so that statediff processing is halted/paused until there is at least one subscriber listening for the results 2020-03-05 10:11:31 -06:00
Ian Norden
8543581b5d review fixes and fixes for issues ran into in integration 2020-03-05 10:11:31 -06:00
Ian Norden
35538aea05 cli parameter to limit statediffing to select account addresses + test 2020-03-05 10:11:31 -06:00
Ian Norden
19489d3b34 make state diff rlp serializable 2020-03-05 10:11:31 -06:00
Ian Norden
e7487003f2 option to process intermediate nodes 2020-03-05 10:11:31 -06:00
Ian Norden
5c62d0bcb1 make proofs and paths optional + compress service loop into single for loop (may be missing something here) 2020-03-05 10:11:31 -06:00
Ian Norden
71b41b5c77 refactoring state diff service and adding api which allows for streaming state diff payloads over an rpc websocket subscription 2020-03-05 10:11:31 -06:00
Ian Norden
bf5a25478e refactoring statediff builder and types and adjusted to relay proofs and paths (still need to make this optional) 2020-03-05 10:11:31 -06:00
Ian Norden
3e2450bac4 refactoring/reorganizing packages 2020-03-05 10:11:31 -06:00
Elizabeth
4c284ea5c1 Statediff for full node (#6)
* Open a trie from the in-memory database

* Use a node's LeafKey as an identifier instead of the address

It was proving difficult to find look the address up from a given path
with a full node (sometimes the value wouldn't exist in the disk db).
So, instead, for now we are using the node's LeafKey with is a Keccak256
hash of the address, so if we know the address we can figure out which
LeafKey it matches up to.

* Make sure that statediff has been processed before pruning

* Use blockchain stateCache.OpenTrie for storage diffs

* Clean up log lines and remove unnecessary fields from builder

* Apply go fmt changes

* Add a sleep to the blockchain test

* Address PR comments

* Address PR comments
2020-03-05 10:11:31 -06:00
Elizabeth
18c0083e10 Gracefully exit geth command(#4) 2020-03-05 10:10:50 -06:00
Elizabeth Engelman
8c01c9a6b8 Apply gosimple changes to statediff 2020-03-05 10:10:50 -06:00
Elizabeth Engelman
95f16994fa Apply goimports to statediff 2020-03-05 10:10:50 -06:00
Elizabeth Engelman
6b93154c3d Apply go fmt changes to statediff 2020-03-05 10:10:50 -06:00
Elizabeth
39c2a55942 Write state diff to CSV (#2)
* port statediff from 9b7fd9af80/statediff/statediff.go; minor fixes

* integrating state diff extracting, building, and persisting into geth processes

* work towards persisting created statediffs in ipfs; based off github.com/vulcanize/eth-block-extractor

* Add a state diff service

* Remove diff extractor from blockchain

* Update imports

* Move statediff on/off check to geth cmd config

* Update starting state diff service

* Add debugging logs for creating diff

* Add statediff extractor and builder tests and small refactoring

* Start to write statediff to a CSV

* Restructure statediff directory

* Pull CSV publishing methods into their own file

* Reformatting due to go fmt

* Add gomega to vendor dir

* Remove testing focuses

* Update statediff tests to use golang test pkg

instead of ginkgo

- builder_test
- extractor_test
- publisher_test

* Use hexutil.Encode instead of deprecated common.ToHex

* Remove OldValue from DiffBigInt and DiffUint64 fields

* Update builder test

* Remove old storage value from updated accounts

* Remove old values from created/deleted accounts

* Update publisher to account for only storing current account values

* Update service loop and fetching previous block

* Update testing

- remove statediff ginkgo test suite file
- move mocks to their own dir

* Updates per go fmt

* Updates to tests

* Pass statediff mode and path in through cli

* Return filename from publisher

* Remove some duplication in builder

* Remove code field from state diff output

this is the contract byte code, and it can still be obtained by querying
the db by the codeHash

* Consolidate acct diff structs for updated & updated/deleted accts

* Include block number in csv filename

* Clean up error logging

* Cleanup formatting, spelling, etc

* Address PR comments

* Add contract address and storage value to csv

* Refactor accumulating account row in csv publisher

* Add DiffStorage struct

* Add storage key to csv

* Address PR comments

* Fix publisher to include rows for accounts that don't have store updates

* Update builder test after merging in release/1.8

* Update test contract to include storage on contract intialization

- so that we're able to test that storage diffing works for created and
deleted accounts (not just updated accounts).

* Factor out a common trie iterator method in builder
2020-03-05 10:10:50 -06:00
Péter Szilágyi
6a62fe399b
params: release Geth v1.9.11 stable 2020-02-18 13:26:00 +02:00
Felix Lange
dae3aee5ff
les: add bootstrap nodes as initial discoveries (#20688) 2020-02-18 13:24:05 +02:00
Péter Szilágyi
05ccbb5edd
Merge pull request #20687 from karalabe/cht-1.9.11
params: update CHTs for the v1.9.11 release
2020-02-18 10:57:05 +02:00
Péter Szilágyi
4f55e24c02
params: update CHTs for the v1.9.11 release 2020-02-18 10:55:44 +02:00
Felix Lange
91b228966e
rpc: remove startup error for invalid modules, log it instead (#20684)
This removes the error added in #20597 in favor of a log message at
error level. Failing to start broke a bunch of people's setups and is
probably not the right thing to do for this check.
2020-02-17 18:33:32 +02:00
Boqin Qin
1b9c5b393b
all: fix goroutine leaks in unit tests by adding 1-elem channel buffer (#20666)
This fixes a bunch of cases where a timeout in the test would leak
a goroutine.
2020-02-17 17:33:11 +01:00
Felix Lange
57d4898e29
p2p/dnsdisc: re-check tree root when leaf resolution fails (#20682)
This adds additional logic to re-resolve the root name of a tree when a
couple of leaf requests have failed. We need this change to avoid
getting into a failure state where leaf requests keep failing for half
an hour when the tree has been updated.
2020-02-17 15:23:25 +01:00
Péter Szilágyi
c2117982b8
Merge pull request #20678 from karalabe/broadcast-sqrt-proper
eth: don't enforce minimum broadcast, fix broadcast test
2020-02-17 14:43:30 +02:00
Felix Lange
1c4c486a85
cmd/ethkey: speed up test by using weaker scrypt parameters (#20680) 2020-02-17 13:22:52 +02:00
Felix Lange
ac72787768
p2p: remove MeteredPeerEvent (#20679)
This event was added for the dashboard, but we don't need it anymore
since the dashboard is gone.
2020-02-17 13:22:14 +02:00
Péter Szilágyi
26284ec3cc
Merge pull request #20681 from karalabe/go1.13.8
travis, appveyor, build: bump builder Go to 1.13.8
2020-02-17 13:15:17 +02:00
Péter Szilágyi
fef8c985bc
travis, appveyor, build: bump builder Go to 1.13.8 2020-02-17 13:13:24 +02:00
Péter Szilágyi
36a1e0b67d
eth: don't enforce minimum broadcast, fix broadcast test 2020-02-17 12:01:03 +02:00
Boqin Qin
37531b1884
cmd/faucet: protect f.reqs with Rlock to prevent data race (#20669)
* cmd/faucet: add Rlock to protect f.reqs in apiHandler

* cmd/faucet: make a locked copy of f.reqs
2020-02-15 20:14:29 +02:00
Martin Holst Swende
855690523a
core: ensure state exists for prefetcher (#20627) 2020-02-14 10:54:02 +02:00
Felix Lange
38d1b0cba2
cmd/geth: enable DNS discovery by default (#20660)
* node: expose config in service context

* eth: integrate p2p/dnsdisc

* cmd/geth: add some DNS flags

* eth: remove DNS URLs

* cmd/utils: configure DNS names for testnets

* params: update DNS URLs

* cmd/geth: configure mainnet DNS

* cmd/utils: rename DNS flag and fix flag processing

* cmd/utils: remove debug print

* node: fix test
2020-02-13 15:38:30 +02:00
Péter Szilágyi
eddcecc160
Merge pull request #20234 from rjl493456442/newtxhashes_2
core, eth: announce based transaction propagation
2020-02-13 15:28:34 +02:00
Péter Szilágyi
9938d954c8
eth: rework tx fetcher to use O(1) ops + manage network requests 2020-02-13 15:27:15 +02:00