Martin Holst Swende
c375ee91e9
cmd/geth, core/state/snapshot: rework journal loading, implement account-check ( #24765 )
...
* cmd/geth, core/state/snapshot: rework journal loading, implement account-check
* core/state/snapshot, cmd/geth: polish code (#37 )
* core/state/snapshot: minor nits
* core/state/snapshot: simplify error logic
* cmd/geth: go format
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
2022-06-06 17:09:39 +02:00
rjl493456442
59ac229f87
core/state/snapshot: detect and clean up dangling storage snapshot in generation ( #24811 )
...
* core/state/snapshot: check dangling storages when generating snapshot
* core/state/snapshot: polish
* core/state/snapshot: wipe the last part of the dangling storages
* core/state/snapshot: fix and add tests
* core/state/snapshot: fix comment
* README: remove mentions of fast sync (#24656 )
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
* core, cmd: expose dangling storage detector for wider usage
* core/state/snapshot: rename variable
* core, ethdb: use global iterators for snapshot generation
* core/state/snapshot: polish
* cmd, core/state/snapshot: polish
* core/state/snapshot: polish
* Update core/state/snapshot/generate.go
Co-authored-by: Martin Holst Swende <martin@swende.se>
* ethdb: extend db test suite and fix memorydb iterator
* ethdb/dbtest: rollback changes
* ethdb/memorydb: simplify iteration
* core/state/snapshot: update dangling counter
* core/state/snapshot: release iterators
* core/state/snapshot: update metrics
* core/state/snapshot: update time metrics
* metrics/influxdb: temp solution to present counter meaningfully, remove it
* add debug log, revert later
* core/state/snapshot: fix iterator panic
* all: customized snapshot iterator for backward iteration
* core, ethdb: polish
* core/state/snapshot: remove debug log
* core/state/snapshot: address comments from peter
* core/state/snapshot: reopen the iterator at the next position
* ethdb, core/state/snapshot: address comment from peter
* core/state/snapshot: reopen exhausted iterators
Co-authored-by: Tbnoapi <63448616+nuoomnoy02@users.noreply.github.com>
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
Co-authored-by: Martin Holst Swende <martin@swende.se>
2022-05-23 13:26:22 +03:00
Martin Holst Swende
0914234d10
cmd/geth, core/state/snapshot: fix flaw in dangling-storage check + inspect difflayers ( #24677 )
...
This PR fixes the flaw that @rjl493456442 found in https://github.com/ethereum/go-ethereum/pull/#issuecomment-1093817551 , namely, that the snapshot iterator uses the combined (disk + difflayers) 'view', wheres the raw iterator uses only the disk 'view'.
This PR instead splits up the work: one phase is iterating the disk layer data, another phase is loading the journalled difflayers and performing the same check there.
2022-04-26 09:08:43 +02:00
Martin Holst Swende
51eb5f8ca8
cmd/geth: add db cmd to show metadata ( #23900 )
...
* cmd/geth: add db cmd to show metadata
* cmd/geth: better output generator status
Co-authored-by: Sina Mahmoodi <1591639+s1na@users.noreply.github.com>
* cmd: minor
Co-authored-by: Sina Mahmoodi <1591639+s1na@users.noreply.github.com>
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
2022-01-18 12:30:41 +02:00
Péter Szilágyi
745757ac6b
core, eth: abort snapshot generation on snap sync and resume later
2021-04-30 17:03:10 +03:00
gary rong
d6ffa14035
core: nuke legacy snapshot supporting ( #22663 )
2021-04-20 08:27:46 +03:00
gary rong
7088f1e814
core, eth: faster snapshot generation ( #22504 )
...
* eth/protocols: persist received state segments
* core: initial implementation
* core/state/snapshot: add tests
* core, eth: updates
* eth/protocols/snapshot: count flat state size
* core/state: add metrics
* core/state/snapshot: skip unnecessary deletion
* core/state/snapshot: rename
* core/state/snapshot: use the global batch
* core/state/snapshot: add logs and fix wiping
* core/state/snapshot: fix
* core/state/snapshot: save generation progress even if the batch is empty
* core/state/snapshot: fixes
* core/state/snapshot: fix initial account range length
* core/state/snapshot: fix initial account range
* eth/protocols/snap: store flat states during the healing
* eth/protocols/snap: print logs
* core/state/snapshot: refactor (#4 )
* core/state/snapshot: refactor
* core/state/snapshot: tiny fix and polish
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
* core, eth: fixes
* core, eth: fix healing writer
* core, trie, eth: fix paths
* eth/protocols/snap: fix encoding
* eth, core: add debug log
* core/state/generate: release iterator asap (#5 )
core/state/snapshot: less copy
core/state/snapshot: revert split loop
core/state/snapshot: handle storage becoming empty, improve test robustness
core/state: test modified codehash
core/state/snapshot: polish
* core/state/snapshot: optimize stats counter
* core, eth: add metric
* core/state/snapshot: update comments
* core/state/snapshot: improve tests
* core/state/snapshot: replace secure trie with standard trie
* core/state/snapshot: wrap return as the struct
* core/state/snapshot: skip wiping correct states
* core/state/snapshot: updates
* core/state/snapshot: fixes
* core/state/snapshot: fix panic due to reference flaw in closure
* core/state/snapshot: fix errors in state generation logic + fix log output
* core/state/snapshot: remove an error case
* core/state/snapshot: fix condition-check for exhausted snap state
* core/state/snapshot: use stackTrie for small tries
* core/state/snapshot: don't resolve small storage tries in vain
* core/state/snapshot: properly clean up storage of deleted accounts
* core/state/snapshot: avoid RLP-encoding in some cases + minor nitpicks
* core/state/snapshot: fix error (+testcase)
* core/state/snapshot: clean up tests a bit
* core/state/snapshot: work in progress on better tests
* core/state/snapshot: polish code
* core/state/snapshot: fix trie iteration abortion trigger
* core/state/snapshot: fixes flaws
* core/state/snapshot: remove panic
* core/state/snapshot: fix abort
* core/state/snapshot: more tests (plus failing testcase)
* core/state/snapshot: more testcases + fix for failing test
* core/state/snapshot: testcase for malformed data
* core/state/snapshot: some test nitpicks
* core/state/snapshot: improvements to logging
* core/state/snapshot: testcase to demo error in abortion
* core/state/snapshot: fix abortion
* cmd/geth: make verify-state report the root
* trie: fix failing test
* core/state/snapshot: add timer metrics
* core/state/snapshot: fix metrics
* core/state/snapshot: udpate tests
* eth/protocols/snap: write snapshot account even if code or state is needed
* core/state/snapshot: fix diskmore check
* core/state/snapshot: review fixes
* core/state/snapshot: improve error message
* cmd/geth: rename 'error' to 'err' in logs
* core/state/snapshot: fix some review concerns
* core/state/snapshot, eth/protocols/snap: clear snapshot marker when starting/resuming snap sync
* core: add error log
* core/state/snapshot: use proper timers for metrics collection
* core/state/snapshot: address some review concerns
* eth/protocols/snap: improved log message
* eth/protocols/snap: fix heal logs to condense infos
* core/state/snapshot: wait for generator termination before restarting
* core/state/snapshot: revert timers to counters to track total time
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
2021-04-14 23:23:11 +03:00
gary rong
5e9f5ca5d3
core/state/snapshot: write snapshot generator in batch ( #22163 )
...
* core/state/snapshot: write snapshot generator in batch
* core: refactor the tests
* core: update tests
* core: update tests
2021-01-18 14:39:43 +01:00
Péter Szilágyi
7b7b327ff2
core/state/snapshot: update generator marker in sync with flushes
2020-11-09 16:03:58 +02:00
gary rong
e6402677c2
core/state/snapshot: fix journal recovery from generating old journal ( #21775 )
...
* core/state/snapshot: print warning if failed to resolve journal
* core/state/snapshot: fix snapshot recovery
When we meet the snapshot journal consisted with:
- disk layer generator with new-format
- diff layer journal with old-format
The base layer should be returned without error.
The broken diff layer can be reconstructed later
but we definitely don't want to reconstruct the
huge diff layer.
* core: add tests
2020-11-04 13:41:46 +02:00
gary rong
b63e3c37a6
core: improve snapshot journal recovery ( #21594 )
...
* core/state/snapshot: introduce snapshot journal version
* core: update the disk layer in an atomic way
* core: persist the disk layer generator periodically
* core/state/snapshot: improve logging
* core/state/snapshot: forcibly ensure the legacy snapshot is matched
* core/state/snapshot: add debug logs
* core, tests: fix tests and special recovery case
* core: polish
* core: add more blockchain tests for snapshot recovery
* core/state: fix comment
* core: add recovery flag for snapshot
* core: add restart after start-after-crash tests
* core/rawdb: fix imports
* core: fix tests
* core: remove log
* core/state/snapshot: fix snapshot
* core: avoid callbacks in SetHead
* core: fix setHead cornercase where the threshold root has state
* core: small docs for the test cases
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
2020-10-29 21:01:58 +02:00
Péter Szilágyi
3ee91b9f2e
core/state/snapshot: reduce disk layer depth during generation
2020-08-24 13:22:36 +03:00
Péter Szilágyi
4b6f6ffe23
core/state/snapshot: fix journal nil deserialziation
2020-04-29 18:00:29 +03:00
Péter Szilágyi
a4cf279494
core/state: extend snapshotter to handle account resurrections
2020-03-03 15:52:00 +02:00
Péter Szilágyi
6e05ccd845
core/state/snapshot, tests: sync snap gen + snaps in consensus tests
2020-03-03 09:17:13 +02:00
Péter Szilágyi
06d4470b41
core: fix broken tests due to API changes + linter
2020-02-25 12:51:16 +02:00
Martin Holst Swende
19099421dc
core/state/snapshot: faster account iteration, CLI integration
2020-02-25 12:51:15 +02:00
Péter Szilágyi
fd39f722a3
core: journal the snapshot inside leveldb, not a flat file
2020-02-25 12:51:10 +02:00
Péter Szilágyi
351a5903b0
core/rawdb, core/state/snapshot: runtime snapshot generation
2020-02-25 12:51:08 +02:00
