Commit Graph

84 Commits

Author SHA1 Message Date
Péter Szilágyi
be65b47645
all: update golang/x/ext and fix slice sorting fallout (#27909)
The Go authors updated golang/x/ext to change the function signature of the slices sort method. 
It's an entire shitshow now because x/ext is not tagged, so everyone's codebase just 
picked a new version that some other dep depends on, causing our code to fail building.

This PR updates the dep on our code too and does all the refactorings to follow upstream...
2023-08-12 00:04:12 +02:00
rjl493456442
503f1f7ada
all: activate pbss as experimental feature (#26274)
* all: activate pbss

* core/rawdb: fix compilation error

* cma, core, eth, les, trie: address comments

* cmd, core, eth, trie: polish code

* core, cmd, eth: address comments

* cmd, core, eth, les, light, tests: address comment

* cmd/utils: shorten log message

* trie/triedb/pathdb: limit node buffer size to 1gb

* cmd/utils: fix opening non-existing db

* cmd/utils: rename flag name

* cmd, core: group chain history flags and fix tests

* core, eth, trie: fix memory leak in snapshot generation

* cmd, eth, internal: deprecate flags

* all: enable state tests for pathdb, fixes

* cmd, core: polish code

* trie/triedb/pathdb: limit the node buffer size to 256mb

---------

Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
2023-08-10 22:21:36 +03:00
Péter Szilágyi
6e934f40f9
eth/protocols/snap: fix batch writer when resuming an aborted sync (#27842) 2023-08-03 14:51:02 +03:00
rjl493456442
88f3d61468
all: expose block number information to statedb (#27753)
* core/state: clean up

* all: add block number infomration to statedb

* core, trie: rename blockNumber to block
2023-07-24 13:22:09 +03:00
rjl493456442
4b06e4f25e
core/state: value diff tracking in StateDB (#27349)
This change makes the StateDB track the state key value diff of a block transition.
We already tracked current account and storage values for the purpose of updating
the state snapshot. With this PR, we now also track the original (pre-transition) values
of accounts and storage slots.
2023-07-11 15:43:23 +02:00
lightclient
cbf2579691
p2p, p2p/discover: add dial metrics (#27621)
This PR adds metrics for p2p dialing, which gives us visibility into the quality of the dial 
candidates  returned by our discovery methods.
2023-07-06 16:20:31 +02:00
Guillaume Ballet
c7b099b2ea
trie, core/state: revert error removal in (*state.Trie).Commit (#27544)
* trie, core/state: revert error removal in (*state.Trie).Commit

* Gary's nitpick :)

Co-Authored-By:  rjl493456442 <garyrong0905@gmail.com>

---------

Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
2023-06-27 15:36:38 +03:00
rjl493456442
6d2aeb43d5
cmd, core/state, eth, tests, trie: improve state reader (#27428)
The state availability is checked during the creation of a state reader.

-    In hash-based database, if the specified root node does not exist on disk disk, then
    the state reader won't be created and an error will be returned.

-    In path-based database, if the specified state layer is not available, then the
    state reader won't be created and an error will be returned.

This change also contains a stricter semantics regarding the `Commit` operation: once it has been performed, the trie is no longer usable, and certain operations will return an error.
2023-06-20 15:31:45 -04:00
Dan Laine
b1ef0bfe03
eth: use slices package for sorting (#27490)
Also adds Hash.Less method for sorting purposes.

---------

Co-authored-by: Felix Lange <fjl@twurst.com>
2023-06-19 23:38:57 +02:00
rjl493456442
ceca4578ca
trie: remove parameter 'fromLevel' in Prove (#27512)
This removes the feature where top nodes of the proof can be elided.
It was intended to be used by the LES server, to save bandwidth 
when the client had already fetched parts of the state and only needed
some extra nodes to complete the proof. Alas, it never got implemented
in the client.
2023-06-19 16:28:40 +02:00
rjl493456442
0e5d2c7c53
core/state/snapshot, core/types, eth: move account definition to type (#27323)
* core/state/snapshot, core/types, eth: move account definition to type

* core, eth: revert snapshot Account API change
2023-06-06 11:17:39 +03:00
rjl493456442
a14301823e
all: new empty trie with types.EmptyRootHash instead of null (#27230) 2023-05-11 10:19:42 +03:00
rjl493456442
5021d36d35
all: port boring changes from pbss (#27176)
* all: port boring changes from pbss

* core, trie: address comments from martin

* trie: minor fixes

* core/rawdb: update comment

* core, eth, tests, trie: address comments

* tests, trie: add extra check when update trie database

* trie/triedb/hashdb: degrade the error to warning
2023-05-09 10:11:04 +03:00
s7v7nislands
f541cad272
eth: use new atomic types (#27137) 2023-04-25 12:06:50 +02:00
rjl493456442
99f81d2724
all: refactor trie API (#26995)
In this PR, all TryXXX(e.g. TryGet) APIs of trie are renamed to XXX(e.g. Get) with an error returned.

The original XXX(e.g. Get) APIs are renamed to MustXXX(e.g. MustGet) and does not return any error -- they print a log output. A future PR will change the behaviour to panic on errorrs.
2023-04-20 06:57:24 -04:00
Guillaume Ballet
41f89ca944
core/state, trie: remove Try prefix in Trie accessors (#26975)
This change renames StateTrie methods to remove the Try* prefix. 

We added the Trie methods with prefix 'Try' a long time ago, working
around the problem that most existing methods of Trie did not return the
database error. This weird naming convention has persisted until now.

Co-authored-by: Gary Rong <garyrong0905@gmail.com>
2023-03-27 10:48:46 +02:00
rjl493456442
fe01a2f63b
all: use unified emptyRootHash and emptyCodeHash (#26718)
The EmptyRootHash and EmptyCodeHash are defined everywhere in the codebase, this PR replaces all of them with unified one defined in core/types package, and also defines constants for TxRoot, WithdrawalsRoot and UncleRoot
2023-02-21 06:12:27 -05:00
Martin Holst Swende
22c3ad1d12
core/state, trie: remove unused error-return from trie Commit operation (#26641) 2023-02-09 08:56:59 -05:00
Martin Holst Swende
8860b39754
all: prepare for path-based trie storage (#26603)
This PR moves some trie-related db accessor methods to a different file, and also removes the schema type. Instead of the schema type, a string is used to distinguish between hashbased/pathbased db accessors.
This also moves some code from trie package to rawdb package.

This PR is intended to be a no-functionality-change prep PR for #25963 .

---------

Co-authored-by: Gary Rong <garyrong0905@gmail.com>
2023-02-06 10:28:40 -05:00
rjl493456442
6c149fd4ad
core, eth, trie, light: clean up trie interface (#26388)
* all: cleanup trie interface

* eth, trie: address comments
2023-01-03 15:41:40 +02:00
rjl493456442
743e404906
core, eth, les, tests, trie: abstract node scheme (#25532)
This PR introduces a node scheme abstraction. The interface is only implemented by `hashScheme` at the moment, but will be extended by `pathScheme` very soon.

Apart from that, a few changes are also included which is worth mentioning:

-  port the changes in the stacktrie, tracking the path prefix of nodes during commit
-  use ethdb.Database for constructing trie.Database. This is not necessary right now, but it is required for path-based used to open reverse diff freezer
2022-11-28 14:31:28 +01:00
Martin Holst Swende
5a02b2d6d0
all: fix spelling mistakes (#25961) 2022-10-11 09:37:00 +02:00
Péter Szilágyi
4f7a425aa8
Merge pull request #25924 from holiman/comments_fix
all: fix docstrings
2022-10-04 13:30:00 +03:00
Martin Holst Swende
ee301c750b
all: fix docstrings 2022-10-04 09:18:02 +02:00
Martin Holst Swende
f61b50b1e8
eth/protocols/snap: serve snap requests when possible (#25644)
This PR makes it so that the snap server responds to trie heal requests when possible, even if the snapshot does not exist. The idea being that it might prolong the lifetime of a state root, so we don't have to pivot quite as often.
2022-10-03 13:37:17 +02:00
Martin Holst Swende
456b187892
trie, eth/protocols/snap: less tiny db-writes during trie-healing (#25681)
This PR prevent making tiny writes during state healing, by only performing the batch-write if the accumulated data is large enough to be meaningful.
2022-09-28 08:08:18 +02:00
rjl493456442
bff84a99fe
cmd, core, eth, les, light: track deleted nodes (#25757)
* cmd, core, eth, les, light: track deleted nodes

* trie: add docs

* trie: address comments

* cmd, core, eth, les, light, trie: trie id

* trie: add tests

* trie, core: updates

* trie: fix imports

* trie: add utility print-method for nodeset

* trie: import err

* trie: fix go vet warnings

Co-authored-by: Martin Holst Swende <martin@swende.se>
2022-09-27 10:01:02 +02:00
Martin Holst Swende
fb500d12d5
eth/protocols/snap: make log messages more clear that sync is ongoing (#25837)
* eth/protocols/snap: make log messages more clear that sync is ongoing

* Update sync.go

Co-authored-by: Felix Lange <fjl@twurst.com>
2022-09-23 11:11:41 +03:00
Felix Lange
b628d72766
build: upgrade to go 1.19 (#25726)
This changes the CI / release builds to use the latest Go version. It also
upgrades golangci-lint to a newer version compatible with Go 1.19.

In Go 1.19, godoc has gained official support for links and lists. The
syntax for code blocks in doc comments has changed and now requires a
leading tab character. gofmt adapts comments to the new syntax
automatically, so there are a lot of comment re-formatting changes in this
PR. We need to apply the new format in order to pass the CI lint stage with
Go 1.19.

With the linter upgrade, I have decided to disable 'gosec' - it produces
too many false-positive warnings. The 'deadcode' and 'varcheck' linters
have also been removed because golangci-lint warns about them being
unmaintained. 'unused' provides similar coverage and we already have it
enabled, so we don't lose much with this change.
2022-09-10 13:25:40 +02:00
Péter Szilágyi
de8d5fa042
eth/protocols/snap: throttle trie heal requests when peers DoS us (#25666)
* eth/protocols/snap: throttle trie heal requests when peers DoS us

* eth/protocols/snap: lower heal throttle log to debug

Co-authored-by: Martin Holst Swende <martin@swende.se>

* eth/protocols/snap: fix comment

Co-authored-by: Martin Holst Swende <martin@swende.se>
2022-09-09 11:42:57 +03:00
Martin Holst Swende
dafa40e7a7
eth/protocols/snap: fix problems due to idle-but-busy peers 2022-08-31 17:58:18 +02:00
Justin Traglia
2c5648d891
all: fix some typos (#25551)
* Fix some typos

* Fix some mistakes

* Revert 4byte.json

* Fix an incorrect fix

* Change files to fails
2022-08-19 09:00:21 +03:00
Guillaume Ballet
f67e54c92f
core: use TryGetAccount to read what TryUpdateAccount has written (#25458)
* core: use TryGetAccount to read where TryUpdateAccount has been used to write

* Gary's review feedback

* implement Gary's suggestion

* fix bug + rename NewSecure into NewStateTrie

* trie: add backwards-compatibility aliases for SecureTrie

* Update database.go

* make the linter happy

Co-authored-by: Felix Lange <fjl@twurst.com>
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
2022-08-04 16:13:18 +02:00
rjl493456442
8b53b92eb4
core, trie: rework trie committer (#25320)
* all: rework trie and trie committer

* all: get rid of internal cache in trie

* all: fixes

* trie: polish

* core, trie: address comments

* trie: fix imports

* core/state: address comments

* core/state/snapshot: polish

* trie: remove unused code

* trie: update tests

* trie: don't set db as nil

* trie: address comments

* trie: unskip test
2022-08-04 11:03:20 +03:00
rjl493456442
1657e43931
core, les, eth: port snap sync changes (#24898)
core, eth, les, trie: rework snap sync
2022-07-15 14:55:51 +03:00
Martin Holst Swende
a907d7e81a
all: more linters (#24783)
This enables the following linters

- typecheck
- unused
- staticcheck
- bidichk
- durationcheck
- exportloopref
- gosec

WIth a few exceptions.

- We use a deprecated protobuf in trezor. I didn't want to mess with that, since I cannot meaningfully test any changes there.
- The deprecated TypeMux is used in a few places still, so the warning for it is silenced for now.
- Using string type in context.WithValue is apparently wrong, one should use a custom type, to prevent collisions between different places in the hierarchy of callers. That should be fixed at some point, but may require some attention.
- The warnings for using weak random generator are squashed, since we use a lot of random without need for cryptographic guarantees.
2022-06-13 16:24:45 +02:00
rjl493456442
22defa5af7
all: introduce trie owner notion (#24750)
* cmd, core/state, light, trie, eth: add trie owner notion

* all: refactor

* tests: fix goimports

* core/state/snapshot: fix ineffasigns

Co-authored-by: Martin Holst Swende <martin@swende.se>
2022-06-06 17:14:55 +02:00
Felix Lange
9244d5cd61
all: update license headers and AUTHORS from git history (#24947) 2022-05-24 20:39:40 +02:00
rjl493456442
39fb82bcfb
eth: fix flaky test, don't attach empty slots/proofs (#24885)
* eth/protocols/snap: don't include empty snapshot slot slice

This PR fixes the snapshot storage serving handler. In snap protocol
the response is capped by the response size. Server can cutdown the
response if the accumulated byte size exceeds the local hard limit.

It means we can meet a special scenario that there is no storage slot
included for a requested account, but we attach the proof for this
account by mistake.

So in the prover side, when it meets a empty storage response but with
a valid proof proves there are some more slots left in the trie, then
requestor will reject this response and disconnect with server.

In this PR, if there is no storage slot served for the requested account,
then no proof should be attached as well.

* eth/protocols/snap: loosen restrictions for flaky tests

* eth/catalyst: fix flaky test in catalyst
2022-05-17 11:19:51 +03:00
Martin Holst Swende
646503208e
eth/protocols/snap: sort trienode heal requests by path (#24779)
* sort snap trienode heal requests

* eth/protocols/snap: remove debug code

* eth/protocols/snap: simplify sort, generate pathsets later

* eth/protocols/snap: review concern

* eth/protocols/snap: renamings

* eth/protocols/snap: add comments in Merge

* eth/protocols/snap: remove variable 'last' in Merge

* eth/protocols/snap: fix lint flaws in test

Co-authored-by: Felix Lange <fjl@twurst.com>
2022-05-10 17:37:24 +03:00
Marius van der Wijden
86d5477079
core/state/snapshot: fix race condition (#24685)
Fixes three race conditions found through fuzzing by David Theodore
2022-05-06 17:20:41 +02:00
Péter Szilágyi
1b58e42802
Merge pull request #24529 from holiman/fix_inf
eth/protocols/snap: avoid estimating infinite percentage
2022-03-11 10:40:55 +02:00
沉风
7d3ecca451
eth/protocols/snap: a little typo Merkel -> Merkle (#24530) 2022-03-11 10:32:08 +02:00
Martin Holst Swende
658415960e
eth/protocols/snap: avoid estimating infinite percentage 2022-03-10 12:46:48 +01:00
Felix Lange
d6f49bf764
core/types: faster RLP encoding of Header, StateAcccount, ReceiptForStorage (#24420)
This change makes use of the new code generator rlp/rlpgen to improve the
performance of RLP encoding for Header and StateAccount. It also speeds up
encoding of ReceiptForStorage using the new rlp.EncoderBuffer API.

The change is much less transparent than I wanted it to be, because Header and
StateAccount now have an EncodeRLP method defined with pointer receiver. It
used to be possible to encode non-pointer values of these types, but the new
method prevents that and attempting to encode unadressable values (even if
part of another value) will return an error. The error can be surprising and may
pop up in places that previously didn't expect any errors.

To make things work, I also needed to update all code paths (mostly in unit tests)
that lead to encoding of non-pointer values, and pass a pointer instead.

Benchmark results:

    name                             old time/op    new time/op    delta
    EncodeRLP/legacy-header-8           328ns ± 0%     237ns ± 1%   -27.63%  (p=0.000 n=8+8)
    EncodeRLP/london-header-8           353ns ± 0%     247ns ± 1%   -30.06%  (p=0.000 n=8+8)
    EncodeRLP/receipt-for-storage-8     237ns ± 0%     123ns ± 0%   -47.86%  (p=0.000 n=8+7)
    EncodeRLP/receipt-full-8            297ns ± 0%     301ns ± 1%    +1.39%  (p=0.000 n=8+8)

    name                             old speed      new speed      delta
    EncodeRLP/legacy-header-8        1.66GB/s ± 0%  2.29GB/s ± 1%   +38.19%  (p=0.000 n=8+8)
    EncodeRLP/london-header-8        1.55GB/s ± 0%  2.22GB/s ± 1%   +42.99%  (p=0.000 n=8+8)
    EncodeRLP/receipt-for-storage-8  38.0MB/s ± 0%  64.8MB/s ± 0%   +70.48%  (p=0.000 n=8+7)
    EncodeRLP/receipt-full-8          910MB/s ± 0%   897MB/s ± 1%    -1.37%  (p=0.000 n=8+8)

    name                             old alloc/op   new alloc/op   delta
    EncodeRLP/legacy-header-8           0.00B          0.00B           ~     (all equal)
    EncodeRLP/london-header-8           0.00B          0.00B           ~     (all equal)
    EncodeRLP/receipt-for-storage-8     64.0B ± 0%      0.0B       -100.00%  (p=0.000 n=8+8)
    EncodeRLP/receipt-full-8             320B ± 0%      320B ± 0%      ~     (all equal)
2022-02-18 08:10:26 +01:00
Martin Holst Swende
6ce4670bc0
cmd/devp2p: implement snap protocol testing (#24276)
This also contains some changes to the protocol handler to
make the tests pass.
2022-02-04 15:24:32 +01:00
Martin Holst Swende
893502e561
trie, core, eth: use db.has over db.get where possible 2021-12-15 16:16:45 +01:00
Péter Szilágyi
58d1988349
core, eth, les, trie: remove the sync bloom, used by fast sync 2021-12-03 12:32:41 +02:00
Martin Holst Swende
1988b47e02
tests/fuzzzers. eth/protocols/snap: add snap protocol fuzzers (#23957) 2021-12-01 10:17:18 +01:00
Péter Szilágyi
c10a0a62c3
eth: request id dispatcher and direct req/reply APIs (#23576)
* eth: request ID based message dispatcher

* eth: fix dispatcher cancellation, rework fetchers idleness tracker

* eth/downloader: drop peers who refuse to serve advertised chains
2021-11-26 13:26:03 +02:00