gary rong
7088f1e814
core, eth: faster snapshot generation ( #22504 )
...
* eth/protocols: persist received state segments
* core: initial implementation
* core/state/snapshot: add tests
* core, eth: updates
* eth/protocols/snapshot: count flat state size
* core/state: add metrics
* core/state/snapshot: skip unnecessary deletion
* core/state/snapshot: rename
* core/state/snapshot: use the global batch
* core/state/snapshot: add logs and fix wiping
* core/state/snapshot: fix
* core/state/snapshot: save generation progress even if the batch is empty
* core/state/snapshot: fixes
* core/state/snapshot: fix initial account range length
* core/state/snapshot: fix initial account range
* eth/protocols/snap: store flat states during the healing
* eth/protocols/snap: print logs
* core/state/snapshot: refactor (#4 )
* core/state/snapshot: refactor
* core/state/snapshot: tiny fix and polish
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
* core, eth: fixes
* core, eth: fix healing writer
* core, trie, eth: fix paths
* eth/protocols/snap: fix encoding
* eth, core: add debug log
* core/state/generate: release iterator asap (#5 )
core/state/snapshot: less copy
core/state/snapshot: revert split loop
core/state/snapshot: handle storage becoming empty, improve test robustness
core/state: test modified codehash
core/state/snapshot: polish
* core/state/snapshot: optimize stats counter
* core, eth: add metric
* core/state/snapshot: update comments
* core/state/snapshot: improve tests
* core/state/snapshot: replace secure trie with standard trie
* core/state/snapshot: wrap return as the struct
* core/state/snapshot: skip wiping correct states
* core/state/snapshot: updates
* core/state/snapshot: fixes
* core/state/snapshot: fix panic due to reference flaw in closure
* core/state/snapshot: fix errors in state generation logic + fix log output
* core/state/snapshot: remove an error case
* core/state/snapshot: fix condition-check for exhausted snap state
* core/state/snapshot: use stackTrie for small tries
* core/state/snapshot: don't resolve small storage tries in vain
* core/state/snapshot: properly clean up storage of deleted accounts
* core/state/snapshot: avoid RLP-encoding in some cases + minor nitpicks
* core/state/snapshot: fix error (+testcase)
* core/state/snapshot: clean up tests a bit
* core/state/snapshot: work in progress on better tests
* core/state/snapshot: polish code
* core/state/snapshot: fix trie iteration abortion trigger
* core/state/snapshot: fixes flaws
* core/state/snapshot: remove panic
* core/state/snapshot: fix abort
* core/state/snapshot: more tests (plus failing testcase)
* core/state/snapshot: more testcases + fix for failing test
* core/state/snapshot: testcase for malformed data
* core/state/snapshot: some test nitpicks
* core/state/snapshot: improvements to logging
* core/state/snapshot: testcase to demo error in abortion
* core/state/snapshot: fix abortion
* cmd/geth: make verify-state report the root
* trie: fix failing test
* core/state/snapshot: add timer metrics
* core/state/snapshot: fix metrics
* core/state/snapshot: udpate tests
* eth/protocols/snap: write snapshot account even if code or state is needed
* core/state/snapshot: fix diskmore check
* core/state/snapshot: review fixes
* core/state/snapshot: improve error message
* cmd/geth: rename 'error' to 'err' in logs
* core/state/snapshot: fix some review concerns
* core/state/snapshot, eth/protocols/snap: clear snapshot marker when starting/resuming snap sync
* core: add error log
* core/state/snapshot: use proper timers for metrics collection
* core/state/snapshot: address some review concerns
* eth/protocols/snap: improved log message
* eth/protocols/snap: fix heal logs to condense infos
* core/state/snapshot: wait for generator termination before restarting
* core/state/snapshot: revert timers to counters to track total time
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
2021-04-14 23:23:11 +03:00
gary rong
5e9f5ca5d3
core/state/snapshot: write snapshot generator in batch ( #22163 )
...
* core/state/snapshot: write snapshot generator in batch
* core: refactor the tests
* core: update tests
* core: update tests
2021-01-18 14:39:43 +01:00
Péter Szilágyi
c4deebbf1e
core/state/snapshot: add generation logs to storage too
2021-01-15 12:26:46 +02:00
Melvin Junhee Woo
167ff563d1
core/state/snapshot: gethring -> gathering typo ( #22104 )
2021-01-04 10:07:43 +02:00
Péter Szilágyi
017831dd5b
core, eth: split eth package, implement snap protocol ( #21482 )
...
This commit splits the eth package, separating the handling of eth and snap protocols. It also includes the capability to run snap sync (https://github.com/ethereum/devp2p/blob/master/caps/snap.md ) , but does not enable it by default.
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
Co-authored-by: Martin Holst Swende <martin@swende.se>
2020-12-14 10:27:15 +01:00
Péter Szilágyi
7b7b327ff2
core/state/snapshot: update generator marker in sync with flushes
2020-11-09 16:03:58 +02:00
gary rong
b63e3c37a6
core: improve snapshot journal recovery ( #21594 )
...
* core/state/snapshot: introduce snapshot journal version
* core: update the disk layer in an atomic way
* core: persist the disk layer generator periodically
* core/state/snapshot: improve logging
* core/state/snapshot: forcibly ensure the legacy snapshot is matched
* core/state/snapshot: add debug logs
* core, tests: fix tests and special recovery case
* core: polish
* core: add more blockchain tests for snapshot recovery
* core/state: fix comment
* core: add recovery flag for snapshot
* core: add restart after start-after-crash tests
* core/rawdb: fix imports
* core: fix tests
* core: remove log
* core/state/snapshot: fix snapshot
* core: avoid callbacks in SetHead
* core: fix setHead cornercase where the threshold root has state
* core: small docs for the test cases
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
2020-10-29 21:01:58 +02:00
Péter Szilágyi
a15d71a255
core/state/snapshot: stop generator if it hits missing trie nodes ( #21649 )
...
* core/state/snapshot: exit Geth if generator hits missing trie nodes
* core/state/snapshot: error instead of hard die on generator fault
* core/state/snapshot: don't enable logging on the tests
2020-10-05 11:52:36 +03:00
Péter Szilágyi
3ee91b9f2e
core/state/snapshot: reduce disk layer depth during generation
2020-08-24 13:22:36 +03:00
gary rong
26d271dfbb
core/state/snapshot: implement storage iterator ( #20971 )
...
* core/state/snapshot: implement storage iterator
* core/state/snapshot, tests: implement helper function
* core/state/snapshot: fix storage issue
If an account is deleted in the tx_1 but recreated in the tx_2,
the it can happen that in this diff layer, both destructedSet
and storageData records this account. In this case, the storage
iterator should be able to iterate the slots belong to new account
but disable further iteration in deeper layers(belong to old account)
* core/state/snapshot: address peter and martin's comment
* core/state: address comments
* core/state/snapshot: fix test
2020-04-29 12:53:08 +03:00
Péter Szilágyi
6e05ccd845
core/state/snapshot, tests: sync snap gen + snaps in consensus tests
2020-03-03 09:17:13 +02:00
Péter Szilágyi
06d4470b41
core: fix broken tests due to API changes + linter
2020-02-25 12:51:16 +02:00
Péter Szilágyi
351a5903b0
core/rawdb, core/state/snapshot: runtime snapshot generation
2020-02-25 12:51:08 +02:00
Martin Holst Swende
f300c0df01
core/state/snapshot: replace bigcache with fastcache
2020-02-25 12:51:08 +02:00
Péter Szilágyi
d754091a87
core/state/snapshot: unlink snapshots from blocks, quad->linear cleanup
2020-02-25 12:51:07 +02:00
Martin Holst Swende
e146fbe4e7
core/state: lazy sorting, snapshot invalidation
2020-02-25 12:51:05 +02:00
Péter Szilágyi
542df8898e
core: initial version of state snapshots
2020-02-25 12:51:04 +02:00
