Commit Graph

28 Commits

Author SHA1 Message Date
Guillaume Ballet
443afc975c
core/state: move state account to core/types + abstracted "write account to trie" (#23567)
* core/state: abstracted "write account to trie" method

* fix appveyor build

* Apply suggestions from code review

Co-authored-by: Martin Holst Swende <martin@swende.se>

* review feedback

* core/state/accounts: move Account to core/types

* core/types: rename Account -> StateAccount

* core/state: restore EncodeRLP for stateObject

* core/types: add the missing file

* more review feedback

* more review feedback

Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Felix Lange <fjl@twurst.com>
2021-09-28 10:48:07 +02:00
gary rong
7088f1e814
core, eth: faster snapshot generation (#22504)
* eth/protocols: persist received state segments

* core: initial implementation

* core/state/snapshot: add tests

* core, eth: updates

* eth/protocols/snapshot: count flat state size

* core/state: add metrics

* core/state/snapshot: skip unnecessary deletion

* core/state/snapshot: rename

* core/state/snapshot: use the global batch

* core/state/snapshot: add logs and fix wiping

* core/state/snapshot: fix

* core/state/snapshot: save generation progress even if the batch is empty

* core/state/snapshot: fixes

* core/state/snapshot: fix initial account range length

* core/state/snapshot: fix initial account range

* eth/protocols/snap: store flat states during the healing

* eth/protocols/snap: print logs

* core/state/snapshot: refactor (#4)

* core/state/snapshot: refactor

* core/state/snapshot: tiny fix and polish

Co-authored-by: rjl493456442 <garyrong0905@gmail.com>

* core, eth: fixes

* core, eth: fix healing writer

* core, trie, eth: fix paths

* eth/protocols/snap: fix encoding

* eth, core: add debug log

* core/state/generate: release iterator asap (#5)

core/state/snapshot: less copy

core/state/snapshot: revert split loop

core/state/snapshot: handle storage becoming empty, improve test robustness

core/state: test modified codehash

core/state/snapshot: polish

* core/state/snapshot: optimize stats counter

* core, eth: add metric

* core/state/snapshot: update comments

* core/state/snapshot: improve tests

* core/state/snapshot: replace secure trie with standard trie

* core/state/snapshot: wrap return as the struct

* core/state/snapshot: skip wiping correct states

* core/state/snapshot: updates

* core/state/snapshot: fixes

* core/state/snapshot: fix panic due to reference flaw in closure

* core/state/snapshot: fix errors in state generation logic + fix log output

* core/state/snapshot: remove an error case

* core/state/snapshot: fix condition-check for exhausted snap state

* core/state/snapshot: use stackTrie for small tries

* core/state/snapshot: don't resolve small storage tries in vain

* core/state/snapshot: properly clean up storage of deleted accounts

* core/state/snapshot: avoid RLP-encoding in some cases + minor nitpicks

* core/state/snapshot: fix error (+testcase)

* core/state/snapshot: clean up tests a bit

* core/state/snapshot: work in progress on better tests

* core/state/snapshot: polish code

* core/state/snapshot: fix trie iteration abortion trigger

* core/state/snapshot: fixes flaws

* core/state/snapshot: remove panic

* core/state/snapshot: fix abort

* core/state/snapshot: more tests (plus failing testcase)

* core/state/snapshot: more testcases + fix for failing test

* core/state/snapshot: testcase for malformed data

* core/state/snapshot: some test nitpicks

* core/state/snapshot: improvements to logging

* core/state/snapshot: testcase to demo error in abortion

* core/state/snapshot: fix abortion

* cmd/geth: make verify-state report the root

* trie: fix failing test

* core/state/snapshot: add timer metrics

* core/state/snapshot: fix metrics

* core/state/snapshot: udpate tests

* eth/protocols/snap: write snapshot account even if code or state is needed

* core/state/snapshot: fix diskmore check

* core/state/snapshot: review fixes

* core/state/snapshot: improve error message

* cmd/geth: rename 'error' to 'err' in logs

* core/state/snapshot: fix some review concerns

* core/state/snapshot, eth/protocols/snap: clear snapshot marker when starting/resuming snap sync

* core: add error log

* core/state/snapshot: use proper timers for metrics collection

* core/state/snapshot: address some review concerns

* eth/protocols/snap: improved log message

* eth/protocols/snap: fix heal logs to condense infos

* core/state/snapshot: wait for generator termination before restarting

* core/state/snapshot: revert timers to counters to track total time

Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
2021-04-14 23:23:11 +03:00
Martin Holst Swende
b44f24e3e6
core, trie: speed up some tests with quadratic processing flaw (#21987)
This commit fixes a flaw in two testcases, and brings down the exec-time from ~40s to ~8s for trie/TestIncompleteSync.

The checkConsistency was performed over and over again on the complete set of nodes, not just the recently added, turning it into a quadratic runtime.
2020-12-10 14:48:32 +01:00
Péter Szilágyi
eeaf191633
core, eth, trie: prepare trie sync for path based operation 2020-09-02 13:21:32 +03:00
gary rong
87c0ba9213
core, eth, les, trie: add a prefix to contract code (#21080) 2020-08-21 15:10:40 +03:00
Péter Szilágyi
06d4470b41
core: fix broken tests due to API changes + linter 2020-02-25 12:51:16 +02:00
gary rong
ecdbb402ee trie: remove node ordering slice in sync batch (#19929)
When we flush a batch of trie nodes into database during the state
sync, we should guarantee that all children should be flushed before
parent.

Actually the trie nodes commit order is strict by: children -> parent.
But when we flush all ready nodes into db, we don't need the order
anymore since

    (1) they are all ready nodes (no more dependency)
    (2) underlying database provides write atomicity
2019-10-28 18:50:11 +01:00
Péter Szilágyi
9effd64290
core, eth, trie: bloom filter for trie node dedup during fast sync (#19489)
* core, eth, trie: bloom filter for trie node dedup during fast sync

* eth/downloader, trie: address review comments

* core, ethdb, trie: restart fast-sync bloom construction now and again

* eth/downloader: initialize fast sync bloom on startup

* eth: reenable eth/62 until we properly remove it
2019-05-13 15:28:01 +03:00
Péter Szilágyi
054412e335
all: clean up and proerly abstract database access 2019-03-06 13:35:03 +02:00
gary rong
7beccb29be all: get rid of error when creating memory database (#16716)
* all: get rid of error when create mdb

* core: clean up variables definition

* all: inline mdb definition
2018-05-09 15:24:25 +03:00
Péter Szilágyi
55599ee95d core, trie: intermediate mempool between trie and database (#15857)
This commit reduces database I/O by not writing every state trie to disk.
2018-02-05 17:40:32 +01:00
Felix Lange
9e5f03b6c4 core/state: access trie through Database interface, track errors (#14589)
With this commit, core/state's access to the underlying key/value database is
mediated through an interface. Database errors are tracked in StateDB and
returned by CommitTo or the new Error method.

Motivation for this change: We can remove the light client's duplicated copy of
core/state. The light client now supports node iteration, so tracing and storage
enumeration can work with the light client (not implemented in this commit).
2017-06-27 15:57:06 +02:00
Felix Lange
0042f13d47 eth/downloader: separate state sync from queue (#14460)
* eth/downloader: separate state sync from queue

Scheduling of state node downloads hogged the downloader queue lock when
new requests were scheduled. This caused timeouts for other requests.
With this change, state sync is fully independent of all other downloads
and doesn't involve the queue at all.

State sync is started and checked on in processContent. This is slightly
awkward because processContent doesn't have a select loop. Instead, the
queue is closed by an auxiliary goroutine when state sync fails. We
tried several alternatives to this but settled on the current approach
because it's the least amount of change overall.

Handling of the pivot block has changed slightly: the queue previously
prevented import of pivot block receipts before the state of the pivot
block was available. In this commit, the receipt will be imported before
the state. This causes an annoyance where the pivot block is committed
as fast block head even when state downloads fail. Stay tuned for more
updates in this area ;)

* eth/downloader: remove cancelTimeout channel

* eth/downloader: retry state requests on timeout

* eth/downloader: improve comment

* eth/downloader: mark peers idle when state sync is done

* eth/downloader: move pivot block splitting to processContent

This change also ensures that pivot block receipts aren't imported
before the pivot block itself.

* eth/downloader: limit state node retries

* eth/downloader: improve state node error handling and retry check

* eth/downloader: remove maxStateNodeRetries

It fails the sync too much.

* eth/downloader: remove last use of cancelCh in statesync.go

Fixes TestDeliverHeadersHang*Fast and (hopefully)
the weird cancellation behaviour at the end of fast sync.

* eth/downloader: fix leak in runStateSync

* eth/downloader: don't run processFullSyncContent in LightSync mode

* eth/downloader: improve comments

* eth/downloader: fix vet, megacheck

* eth/downloader: remove unrequested tasks anyway

* eth/downloader, trie: various polishes around duplicate items

This commit explicitly tracks duplicate and unexpected state
delieveries done against a trie Sync structure, also adding
there to import info logs.

The commit moves the db batch used to commit trie changes one
level deeper so its flushed after every node insertion. This
is needed to avoid a lot of duplicate retrievals caused by
inconsistencies between Sync internals and database. A better
approach is to track not-yet-written states in trie.Sync and
flush on commit, but I'm focuing on correctness first now.

The commit fixes a regression around pivot block fail count.
The counter previously was reset to 1 if and only if a sync
cycle progressed (inserted at least 1 entry to the database).
The current code reset it already if a node was delivered,
which is not stong enough, because unless it ends up written
to disk, an attacker can just loop and attack ad infinitum.

The commit also fixes a regression around state deliveries
and timeouts. The old downloader tracked if a delivery is
stale (none of the deliveries were requestedt), in which
case it didn't mark the node idle and did not send further
requests, since it signals a past timeout. The current code
did mark it idle even on stale deliveries, which eventually
caused two requests to be in flight at the same time, making
the deliveries always stale and mass duplicating retrievals
between multiple peers.

* eth/downloader: fix state request leak

This commit fixes the hang seen sometimes while doing the state
sync. The cause of the hang was a rare combination of events:
request state data from peer, peer drops and reconnects almost
immediately. This caused a new download task to be assigned to
the peer, overwriting the old one still waiting for a timeout,
which in turned leaked the requests out, never to be retried.
The fix is to ensure that a task assignment moves any pending
one back into the retry queue.

The commit also fixes a regression with peer dropping due to
stalls. The current code considered a peer stalling if they
timed out delivering 1 item. However, the downloader never
requests only one, the minimum is 2 (attempt to fine tune
estimated latency/bandwidth). The fix is simply to drop if
a timeout is detected at 2 items.

Apart from the above bugfixes, the commit contains some code
polishes I made while debugging the hang.

* core, eth, trie: support batched trie sync db writes

* trie: rename SyncMemCache to syncMemBatch
2017-06-22 15:26:03 +03:00
Felix Lange
f2da6581ba all: fix issues reported by honnef.co/go/simple/cmd/gosimple 2017-01-06 18:18:07 +01:00
Felix Lange
35a7dcb162 all: gofmt -w -s 2017-01-06 15:52:03 +01:00
Felix Lange
d3b751e4d9 trie: remove dependency on ethdb
This removes the core/types -> leveldb dependency.
2017-01-06 14:15:22 +01:00
Jeffrey Wilcke
445feaeef5 core, core/state, trie: EIP158, reprice & skip empty account write
This commit implements EIP158 part 1, 2, 3 & 4

1. If an account is empty it's no longer written to the trie. An empty
  account is defined as (balance=0, nonce=0, storage=0, code=0).
2. Delete an empty account if it's touched
3. An empty account is redefined as either non-existent or empty.
4. Zero value calls and zero value suicides no longer consume the 25k
  reation costs.

params: moved core/config to params

Signed-off-by: Jeffrey Wilcke <jeffrey@ethereum.org>
2016-11-13 10:44:04 +01:00
Péter Szilágyi
90b16a3e85
core/state, eth/downloader, trie: reset fast-failure on progress 2016-10-31 14:19:14 +02:00
Felix Lange
1f1ea18b54 core/state: implement reverts by journaling all changes
This commit replaces the deep-copy based state revert mechanism with a
linear complexity journal. This commit also hides several internal
StateDB methods to limit the number of ways in which calling code can
use the journal incorrectly.

As usual consultation and bug fixes to the initial implementation were
provided by @karalabe, @obscuren and @Arachnid. Thank you!
2016-10-06 15:32:16 +02:00
Péter Szilágyi
cb84e3f029 cmd, core, internal, light, tests: avoid hashing the code in the VM 2016-10-01 16:01:58 +03:00
Felix Lange
cd791bd855 core, trie: replace state caches with trie journal 2016-09-28 11:27:28 +03:00
Felix Lange
6fdd0893c3 all: fix go vet warnings 2016-04-15 11:17:27 +02:00
Péter Szilágyi
b8d59d9c98 core/state, trie: switch iterator panics to error fields 2016-02-16 12:37:00 +02:00
Péter Szilágyi
5a057a8ded core/state, trie: surface iterator entry hashes 2016-02-16 12:21:08 +02:00
Péter Szilágyi
7e29b0b5b4 core/state, trie: add node iterator, test state/trie sync consistency 2016-02-16 12:21:08 +02:00
Péter Szilágyi
5b0ee8ec30 core, eth, trie: fix data races and merge/review issues 2015-10-21 16:49:55 +03:00
Péter Szilágyi
b97e34a8e4 eth/downloader: concurrent receipt and state processing 2015-10-19 10:03:10 +03:00
Péter Szilágyi
ab27bee25a core, eth, trie: direct state trie synchronization 2015-10-19 10:03:09 +03:00