Commit Graph

278 Commits

Author SHA1 Message Date
Martin Holst Swende
5a02b2d6d0
all: fix spelling mistakes (#25961) 2022-10-11 09:37:00 +02:00
Martin Holst Swende
456b187892
trie, eth/protocols/snap: less tiny db-writes during trie-healing (#25681)
This PR prevent making tiny writes during state healing, by only performing the batch-write if the accumulated data is large enough to be meaningful.
2022-09-28 08:08:18 +02:00
rjl493456442
bff84a99fe
cmd, core, eth, les, light: track deleted nodes (#25757)
* cmd, core, eth, les, light: track deleted nodes

* trie: add docs

* trie: address comments

* cmd, core, eth, les, light, trie: trie id

* trie: add tests

* trie, core: updates

* trie: fix imports

* trie: add utility print-method for nodeset

* trie: import err

* trie: fix go vet warnings

Co-authored-by: Martin Holst Swende <martin@swende.se>
2022-09-27 10:01:02 +02:00
Amir Hossein
57a65f00c9
trie: handle more batch commit errors in Database (#25674) 2022-09-15 23:35:53 +02:00
Felix Lange
b628d72766
build: upgrade to go 1.19 (#25726)
This changes the CI / release builds to use the latest Go version. It also
upgrades golangci-lint to a newer version compatible with Go 1.19.

In Go 1.19, godoc has gained official support for links and lists. The
syntax for code blocks in doc comments has changed and now requires a
leading tab character. gofmt adapts comments to the new syntax
automatically, so there are a lot of comment re-formatting changes in this
PR. We need to apply the new format in order to pass the CI lint stage with
Go 1.19.

With the linter upgrade, I have decided to disable 'gosec' - it produces
too many false-positive warnings. The 'deadcode' and 'varcheck' linters
have also been removed because golangci-lint warns about them being
unmaintained. 'unused' provides similar coverage and we already have it
enabled, so we don't lose much with this change.
2022-09-10 13:25:40 +02:00
Darioush Jalali
8363f79f8f
trie: update comments + err check for preimages (#25672)
This PR includes minor updates to comments in trie/committer that reference insertion to the db, and adds an err != nil check for the return value of preimages.commit.
2022-09-08 12:36:07 +02:00
rjl493456442
5ddedd2f83
core, light, trie: remove DiskDB function from trie database (#25690) 2022-09-07 09:08:56 +02:00
Péter Szilágyi
a9ec2ab2e6
trie: check childrens' existence concurrently for snap heal 2022-09-06 12:57:03 +03:00
Amir Hossein
4b9c307d26
trie: fix unhandled error in test (#25628) 2022-09-02 18:07:55 +02:00
Seungbae Yu
0d68b6bf10
trie: fix typo in comment (#25667) 2022-09-02 17:48:26 +02:00
Martin Holst Swende
d79bd2f0f6
trie: better error reporting (#25645) 2022-09-01 08:41:10 +02:00
rjl493456442
3f79afb599
trie: cleanup stateTrie (#25640)
It's a trivial PR to hide the error log when the trie node is not found in the database. The idea for this change is for all TryXXX functions, the error is already returned and we don't need to fire a log explicitly.

Recently there are a few tickets #25613 #25589 reporting that the trie nodes are missing because of debug.SetHead. The root cause is after resetting, the chain rewinds to a historical point and re-imports the blocks on top.

Since the node is already synced and started to accept transactions previously, these transactions are still kept in the txpool and verified by txpool with a live state. This live state is constructed based on the live trie database, which is changed fast by node referencing and de-referencing.

Unfortunately, when we construct a live state(like the state in txpool), we don't reference the state we have. The blockchain will garbage collect the intermediate version nodes in another thread which leads the broken live state.

The best solution for this is to forcibly obtain a reference for all live states we create and call release function once it's used up. But it might end up with more junks persisted into disk. Will try to find an elegant solution later in the following PR.
2022-08-31 18:18:18 +02:00
aaronbuchwald
3d68bb03c3
trie: fix grammar in comment (#25648) 2022-08-31 16:26:39 +02:00
Péter Szilágyi
5758d1fb11
core/state, trie: fix trie flush order for proper pruning 2022-08-23 21:17:12 +03:00
Justin Traglia
2c5648d891
all: fix some typos (#25551)
* Fix some typos

* Fix some mistakes

* Revert 4byte.json

* Fix an incorrect fix

* Change files to fails
2022-08-19 09:00:21 +03:00
rjl493456442
a1b8892384
trie: improve node rlp decoding performance (#25357)
This avoids copying the input []byte while decoding trie nodes. In most
cases, particularly when the input slice is provided by the underlying
database, this optimization is safe to use.

For cases where the origin of the input slice is unclear, the copying version
is retained. The new code performs better even when the input must be
copied, because it is now only copied once in decodeNode.
2022-08-19 00:39:47 +02:00
Guillaume Ballet
6da5c1644d
core/state, trie, light: add a TryDeleteAccount method (#25531)
* core/state, trie, light: Add a DeleteAccount method

* review feedback

* Update database.go

* pr triage feedback

Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
2022-08-17 14:14:49 +03:00
Guillaume Ballet
12185e40e0
core, trie: flush preimages to db on blockchain close (#25533)
* core, trie: flush preimages to db on database close

Co-authored-by: rjl493456442 <garyrong0905@gmail.com>

* rename Close to CommitPreimages for clarity

* core, trie: nitpick fixes

Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
2022-08-17 14:12:10 +03:00
Guillaume Ballet
f67e54c92f
core: use TryGetAccount to read what TryUpdateAccount has written (#25458)
* core: use TryGetAccount to read where TryUpdateAccount has been used to write

* Gary's review feedback

* implement Gary's suggestion

* fix bug + rename NewSecure into NewStateTrie

* trie: add backwards-compatibility aliases for SecureTrie

* Update database.go

* make the linter happy

Co-authored-by: Felix Lange <fjl@twurst.com>
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
2022-08-04 16:13:18 +02:00
rjl493456442
8b53b92eb4
core, trie: rework trie committer (#25320)
* all: rework trie and trie committer

* all: get rid of internal cache in trie

* all: fixes

* trie: polish

* core, trie: address comments

* trie: fix imports

* core/state: address comments

* core/state/snapshot: polish

* trie: remove unused code

* trie: update tests

* trie: don't set db as nil

* trie: address comments

* trie: unskip test
2022-08-04 11:03:20 +03:00
rjl493456442
9d76a9b94f
core, trie, eth, cmd: rework preimage store (#25287)
* core, trie, eth, cmd: rework preimage store

* trie: address comment
2022-07-27 20:37:04 +02:00
Delweng
b196ad1c16
all: add whitespace linter (#25312)
* golangci: typo

Signed-off-by: Delweng <delweng@gmail.com>

* golangci: add whietspace

Signed-off-by: Delweng <delweng@gmail.com>

* *: rm whitesapce using golangci-lint

Signed-off-by: Delweng <delweng@gmail.com>

* cmd/puppeth: revert accidental resurrection

Co-authored-by: Péter Szilágyi <peterke@gmail.com>
2022-07-25 13:14:03 +03:00
Felix Lange
e3df3d34cf
trie: fix 'gosimple' lint issue (#25309) 2022-07-15 18:36:05 +02:00
rjl493456442
1657e43931
core, les, eth: port snap sync changes (#24898)
core, eth, les, trie: rework snap sync
2022-07-15 14:55:51 +03:00
aaronbuchwald
926b3e08ba
trie: fix typo in comment (#25241)
paralallel -> parallel
2022-07-06 08:49:09 +02:00
Seungbae.yu
2697e44d81
all: change format 0x%x to %#x (#25221) 2022-07-04 11:03:32 +03:00
aaronbuchwald
241dd27300
trie: fix size accounting in cleaner (#25007)
Decrease children size instead of dirties size when marking dirties as cleaned up in trie cleaner
2022-06-21 12:00:37 +03:00
Martin Holst Swende
a907d7e81a
all: more linters (#24783)
This enables the following linters

- typecheck
- unused
- staticcheck
- bidichk
- durationcheck
- exportloopref
- gosec

WIth a few exceptions.

- We use a deprecated protobuf in trezor. I didn't want to mess with that, since I cannot meaningfully test any changes there.
- The deprecated TypeMux is used in a few places still, so the warning for it is silenced for now.
- Using string type in context.WithValue is apparently wrong, one should use a custom type, to prevent collisions between different places in the hierarchy of callers. That should be fixed at some point, but may require some attention.
- The warnings for using weak random generator are squashed, since we use a lot of random without need for cryptographic guarantees.
2022-06-13 16:24:45 +02:00
aaronbuchwald
6b3e6cb2ab
trie: move locking into trieDB insert method (#25030)
Move locking into trieDB insert function
2022-06-07 08:02:04 +02:00
rjl493456442
22defa5af7
all: introduce trie owner notion (#24750)
* cmd, core/state, light, trie, eth: add trie owner notion

* all: refactor

* tests: fix goimports

* core/state/snapshot: fix ineffasigns

Co-authored-by: Martin Holst Swende <martin@swende.se>
2022-06-06 17:14:55 +02:00
Felix Lange
9244d5cd61
all: update license headers and AUTHORS from git history (#24947) 2022-05-24 20:39:40 +02:00
Martin Holst Swende
646503208e
eth/protocols/snap: sort trienode heal requests by path (#24779)
* sort snap trienode heal requests

* eth/protocols/snap: remove debug code

* eth/protocols/snap: simplify sort, generate pathsets later

* eth/protocols/snap: review concern

* eth/protocols/snap: renamings

* eth/protocols/snap: add comments in Merge

* eth/protocols/snap: remove variable 'last' in Merge

* eth/protocols/snap: fix lint flaws in test

Co-authored-by: Felix Lange <fjl@twurst.com>
2022-05-10 17:37:24 +03:00
Darioush Jalali
8bf0565ebb
trie: remove unused makeHashNode (#24702) 2022-04-20 16:12:06 +02:00
Eng Zer Jun
8d066f1f42
all: use T.TempDir to create temporary test directories (#24633)
This commit replaces ioutil.TempDir with t.TempDir in tests. The
directory created by t.TempDir is automatically removed when the test
and all its subtests complete.

Prior to this commit, temporary directory created using ioutil.TempDir
had to be removed manually by calling os.RemoveAll, which is omitted in
some tests. The error handling boilerplate e.g.

	defer func() {
		if err := os.RemoveAll(dir); err != nil {
			t.Fatal(err)
		}
	}

is also tedious, but t.TempDir handles this for us nicely.

Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2022-04-08 15:44:55 +02:00
rjl493456442
da16d089c0
trie, les, tests, core: implement trie tracer (#24403)
Trie tracer is an auxiliary tool to capture all deleted nodes
which can't be captured by trie.Committer. The deleted nodes
can be removed from the disk later.
2022-03-31 09:28:32 +02:00
rjl493456442
fb2ae8e995
trie: fix two issues in trie iterator (#24539)
* trie: fix memory leak in trie iterator

In the trie iterator, live nodes are tracked in a stack while iterating.
Popped node states should be explictly set to nil in order to get
garbage-collected.

* trie: fix empty trie iterator
2022-03-15 10:23:37 +01:00
rjl493456442
8c8a9e5ca1
core, ethdb, tests, trie: introduce database snapshot (#24486) 2022-03-10 09:35:22 +01:00
Qian Bin
65ed1a6871
rlp, trie: faster trie node encoding (#24126)
This change speeds up trie hashing and all other activities that require
RLP encoding of trie nodes by approximately 20%. The speedup is achieved by
avoiding reflection overhead during node encoding.

The interface type trie.node now contains a method 'encode' that works with
rlp.EncoderBuffer. Management of EncoderBuffers is left to calling code.
trie.hasher, which is pooled to avoid allocations, now maintains an
EncoderBuffer. This means memory resources related to trie node encoding
are tied to the hasher pool.

Co-authored-by: Felix Lange <fjl@twurst.com>
2022-03-09 14:45:17 +01:00
Péter Szilágyi
fc8ad1b70d
Merge pull request #24391 from rjl493456442/trie-iterator
trie: implement NodeBlob API for trie iterator
2022-02-15 15:35:47 +02:00
rjl493456442
4d086430bd
core, ethdb, tests, trie: implement NewBatchWithSize API for batcher (#24392)
This PR adds an addtional API called `NewBatchWithSize` for db
batcher. It turns out that leveldb batch memory allocation is
super inefficient. The main reason is the allocation step of
leveldb Batch is too small when the batch size is large. It can
take a few second to build a leveldb batch with 100MB size.

Luckily, leveldb also offers another API called MakeBatch which can
pre-allocate the memory area. So if the approximate size of batch is
known in advance, this API can be used in this case.

It's needed in new state scheme PR which needs to commit a batch of
trie nodes in a single batch. Implement the feature in a seperate PR.
2022-02-15 15:15:13 +02:00
Gary Rong
55430b6ea2 trie: implement NodeBlob API for trie iterator
This functionality is needed in new path-based storage scheme, but
can be implemented in a seperate PR though.

When an account is deleted, then all the storage slots should be
nuked out from the disk as well. In hash-based storage scheme they
are still left in the disk but in new scheme, they will be iterated
and marked as deleted.

But why the NodeBlob API is needed in this scenario? Because when
the node is marked deleted, the previous value is also required to
be recorded to construct the reverse diff.
2022-02-15 16:12:17 +08:00
Darioush Jalali
2dfa4bcf6c
trie: test for edgecase in VerifyRangeProof (#24257)
* trie/proof: edge case for VerifyRangeProof

* more consistency with other tests in the file

* trie: fix test todo

Co-authored-by: Martin Holst Swende <martin@swende.se>
2022-01-21 14:35:30 +01:00
rjl493456442
ae45c97d3d
trie: fix range prover (#24266) 2022-01-21 10:41:51 +01:00
Darioush Jalali
2c58e6b62d
trie: use keyvalue reader for non-mutating methods (#24221) 2022-01-11 22:36:48 +02:00
Péter Szilágyi
af2ca5a654
Merge pull request #24117 from holiman/db_has
trie, core, eth: use db.has over db.get where possible
2022-01-06 11:30:11 +02:00
Water
98be5f9a72
trie: fix spelling mistake (#24185)
mispelled words in comments: th enext
2022-01-04 17:23:52 +02:00
Martin Holst Swende
893502e561
trie, core, eth: use db.has over db.get where possible 2021-12-15 16:16:45 +01:00
Péter Szilágyi
58d1988349
core, eth, les, trie: remove the sync bloom, used by fast sync 2021-12-03 12:32:41 +02:00
Taeik Lim
85064ed09b
all: fix 'the the' in comments (#24036) 2021-12-02 15:42:09 +01:00
Paweł Bylica
86fe359a56
trie: simplify StackTrie implementation (#23950)
Trim the search key from head as it's being pushed deeper into the trie. Previously the search key was never modified but each node kept information how to slice and compare it in keyOffset. Now the keyOffset is not needed as this information is included in the slice of the search key. This way the keyOffset can be removed and key manipulation
simplified.
2021-11-29 11:02:40 +01:00