Commit Graph

187 Commits

Author SHA1 Message Date
Martin Holst Swende
8b6cf128af
metrics: refactor metrics (#28035)
This change includes a lot of things, listed below. 

### Split up interfaces, write vs read

The interfaces have been split up into one write-interface and one read-interface, with `Snapshot` being the gateway from write to read. This simplifies the semantics _a lot_. 

Example of splitting up an interface into one readonly 'snapshot' part, and one updatable writeonly part: 

```golang
type MeterSnapshot interface {
	Count() int64
	Rate1() float64
	Rate5() float64
	Rate15() float64
	RateMean() float64
}

// Meters count events to produce exponentially-weighted moving average rates
// at one-, five-, and fifteen-minutes and a mean rate.
type Meter interface {
	Mark(int64)
	Snapshot() MeterSnapshot
	Stop()
}
```

### A note about concurrency

This PR makes the concurrency model clearer. We have actual meters and snapshot of meters. The `meter` is the thing which can be accessed from the registry, and updates can be made to it. 

- For all `meters`, (`Gauge`, `Timer` etc), it is assumed that they are accessed by different threads, making updates. Therefore, all `meters` update-methods (`Inc`, `Add`, `Update`, `Clear` etc) need to be concurrency-safe. 
- All `meters` have a `Snapshot()` method. This method is _usually_ called from one thread, a backend-exporter. But it's fully possible to have several exporters simultaneously: therefore this method should also be concurrency-safe. 

TLDR: `meter`s are accessible via registry, all their methods must be concurrency-safe. 

For all `Snapshot`s, it is assumed that an individual exporter-thread has obtained a `meter` from the registry, and called the `Snapshot` method to obtain a readonly snapshot. This snapshot is _not_ guaranteed to be concurrency-safe. There's no need for a snapshot to be concurrency-safe, since exporters should not share snapshots. 

Note, though: that by happenstance a lot of the snapshots _are_ concurrency-safe, being unmutable minimal representations of a value. Only the more complex ones are _not_ threadsafe, those that lazily calculate things like `Variance()`, `Mean()`.

Example of how a background exporter typically works, obtaining the snapshot and sequentially accessing the non-threadsafe methods in it: 
```golang
		ms := metric.Snapshot()
                ...
		fields := map[string]interface{}{
			"count":    ms.Count(),
			"max":      ms.Max(),
			"mean":     ms.Mean(),
			"min":      ms.Min(),
			"stddev":   ms.StdDev(),
			"variance": ms.Variance(),
```

TLDR: `snapshots` are not guaranteed to be concurrency-safe (but often are).

### Sample changes

I also changed the `Sample` type: previously, it iterated the samples fully every time `Mean()`,`Sum()`, `Min()` or `Max()` was invoked. Since we now have readonly base data, we can just iterate it once, in the constructor, and set all four values at once. 

The same thing has been done for runtimehistogram. 

### ResettingTimer API

Back when ResettingTImer was implemented, as part of https://github.com/ethereum/go-ethereum/pull/15910, Anton implemented a `Percentiles` on the new type. However, the method did not conform to the other existing types which also had a `Percentiles`. 

1. The existing ones, on input, took `0.5` to mean `50%`. Anton used `50` to mean `50%`. 
2. The existing ones returned `float64` outputs, thus interpolating between values. A value-set of `0, 10`, at `50%` would return `5`, whereas Anton's would return either `0` or `10`. 

This PR removes the 'new' version, and uses only the 'legacy' percentiles, also for the ResettingTimer type. 

The resetting timer snapshot was also defined so that it would expose the internal values. This has been removed, and getters for `Max, Min, Mean` have been added instead. 

### Unexport types

A lot of types were exported, but do not need to be. This PR unexports quite a lot of them.
2023-09-13 13:13:47 -04:00
rjl493456442
a7842c9cae
core, trie: cleanup trie database (#28062) 2023-09-07 21:17:14 +08:00
rjl493456442
0acc0a1f86
core/state: simplify storage trie update and commit (#28030)
This change improves function description and simplifies logic in statedb update and commit operations.
2023-08-31 14:33:18 -04:00
Martin Holst Swende
d4e345c7d4
core/state: fix missing import (#28010) 2023-08-26 04:43:36 -04:00
rjl493456442
3ff6b3c31e
core/state: implement fast storage deletion (#27955)
This changes implements faster post-selfdestruct iteration of storage slots for deletion, by using snapshot-storage+stacktrie to recover the trienodes to be deleted. This mechanism is only implemented for path-based schema. 

For hash-based schema, the entire post-selfdestruct storage iteration is skipped, with this change, since hash-based does not actually perform deletion anyway. 

---------

Co-authored-by: Martin Holst Swende <martin@swende.se>
2023-08-26 04:13:22 -04:00
Martin Holst Swende
eb6cbe37e1
core/state: remove public method ForEachStorage (#27986)
Co-authored-by: Felix Lange <fjl@twurst.com>
2023-08-23 21:53:31 +02:00
Guillaume Ballet
67979022aa
core/state: move UpdateContractCode before the trie hash is computed (#27853)
Context: The UpdateContractCode method was introduced for the state storage commitment
schemes that include the whole code for their commitment computation. It must therefore be called
before the root hash is computed at the end of IntermediateRoot.

This should have no impact on the MPT since, in this context, the method is a no-op.
2023-08-09 18:02:45 +02:00
rjl493456442
7de748d3f6
all: implement path-based state scheme (#25963)
* all: implement path-based state scheme

* all: edits from review

* core/rawdb, trie/triedb/pathdb: review changes

* core, light, trie, eth, tests: reimplement pbss history

* core, trie/triedb/pathdb: track block number in state history

* trie/triedb/pathdb: add history documentation

* core, trie/triedb/pathdb: address comments from Peter's review

Important changes to list:

- Cache trie nodes by path in clean cache
- Remove root->id mappings when history is truncated

* trie/triedb/pathdb: fallback to disk if unexpect node in clean cache

* core/rawdb: fix tests

* trie/triedb/pathdb: rename metrics, change clean cache key

* trie/triedb: manage the clean cache inside of disk layer

* trie/triedb/pathdb: move journal function

* trie/triedb/path: fix tests

* trie/triedb/pathdb: fix journal

* trie/triedb/pathdb: fix history

* trie/triedb/pathdb: try to fix tests on windows

* core, trie: address comments

* trie/triedb/pathdb: fix test issues

---------

Co-authored-by: Felix Lange <fjl@twurst.com>
Co-authored-by: Martin Holst Swende <martin@swende.se>
2023-08-01 15:17:32 +03:00
rjl493456442
817553cc28
core, trie: track state change set with account address (#27815) 2023-07-31 15:07:51 +03:00
rjl493456442
88f3d61468
all: expose block number information to statedb (#27753)
* core/state: clean up

* all: add block number infomration to statedb

* core, trie: rename blockNumber to block
2023-07-24 13:22:09 +03:00
jwasinger
988d84aa7c
core/state, core/vm: implement EIP 6780 (#27189)
EIP-6780: SELFDESTRUCT only in same transaction

>     SELFDESTRUCT will recover all funds to the caller but not delete the account, except when called in the same transaction as creation

---------

Co-authored-by: Martin Holst Swende <martin@swende.se>
2023-07-17 13:02:18 -04:00
jwasinger
d233b6b23a
core: replace instances of 'suicide' with 'selfdestruct' to improve code consistency. (#27716)
---------

Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: lightclient <14004106+lightclient@users.noreply.github.com>
2023-07-15 10:35:30 -04:00
rjl493456442
4b06e4f25e
core/state: value diff tracking in StateDB (#27349)
This change makes the StateDB track the state key value diff of a block transition.
We already tracked current account and storage values for the purpose of updating
the state snapshot. With this PR, we now also track the original (pre-transition) values
of accounts and storage slots.
2023-07-11 15:43:23 +02:00
Guillaume Ballet
c7b099b2ea
trie, core/state: revert error removal in (*state.Trie).Commit (#27544)
* trie, core/state: revert error removal in (*state.Trie).Commit

* Gary's nitpick :)

Co-Authored-By:  rjl493456442 <garyrong0905@gmail.com>

---------

Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
2023-06-27 15:36:38 +03:00
Guillaume Ballet
699243f8ae
core/state, light, trie: add UpdateContractCode to the Trie interface (#27476)
Verkle trees store the code inside the trie. This PR changes the interface to pass the code, as well as the dirty flag to tell the trie package if the code is dirty and needs to be updated. This is a no-op for the MPT and the odr trie.
2023-06-22 08:52:52 -04:00
rjl493456442
6d2aeb43d5
cmd, core/state, eth, tests, trie: improve state reader (#27428)
The state availability is checked during the creation of a state reader.

-    In hash-based database, if the specified root node does not exist on disk disk, then
    the state reader won't be created and an error will be returned.

-    In path-based database, if the specified state layer is not available, then the
    state reader won't be created and an error will be returned.

This change also contains a stricter semantics regarding the `Commit` operation: once it has been performed, the trie is no longer usable, and certain operations will return an error.
2023-06-20 15:31:45 -04:00
rjl493456442
ceca4578ca
trie: remove parameter 'fromLevel' in Prove (#27512)
This removes the feature where top nodes of the proof can be elided.
It was intended to be used by the LES server, to save bandwidth 
when the client had already fetched parts of the state and only needed
some extra nodes to complete the proof. Alas, it never got implemented
in the client.
2023-06-19 16:28:40 +02:00
rjl493456442
0e5d2c7c53
core/state/snapshot, core/types, eth: move account definition to type (#27323)
* core/state/snapshot, core/types, eth: move account definition to type

* core, eth: revert snapshot Account API change
2023-06-06 11:17:39 +03:00
rjl493456442
380fb4e249
core/state: clear out cached state data when reset occurs (#27376)
* core/state: remove cached snap data if reset occurs

* core/state: address comment from peter

* core/state: skip revert in case data is nil
2023-06-05 16:25:57 +03:00
rjl493456442
15bd21f3c8
core/state: mark account as dirty when resetObject occurs (#27339)
This changes the journal logic to mark the state object dirty immediately when it
is reset. 

We're mostly adding this change to appease the fuzzer. Marking it dirty immediately
makes no difference in practice because accounts will always be modified by EVM
right after creation.
2023-06-01 11:09:32 +02:00
rjl493456442
5021d36d35
all: port boring changes from pbss (#27176)
* all: port boring changes from pbss

* core, trie: address comments from martin

* trie: minor fixes

* core/rawdb: update comment

* core, eth, tests, trie: address comments

* tests, trie: add extra check when update trie database

* trie/triedb/hashdb: degrade the error to warning
2023-05-09 10:11:04 +03:00
ucwong
cc8d40c65f
core/state: initialize maps with known size (#27222)
* core/state : fix map size avoid resizing

* core/state : fixed size
2023-05-08 09:59:14 +03:00
Guillaume Ballet
41f89ca944
core/state, trie: remove Try prefix in Trie accessors (#26975)
This change renames StateTrie methods to remove the Try* prefix. 

We added the Trie methods with prefix 'Try' a long time ago, working
around the problem that most existing methods of Trie did not return the
database error. This weird naming convention has persisted until now.

Co-authored-by: Gary Rong <garyrong0905@gmail.com>
2023-03-27 10:48:46 +02:00
Guillaume Ballet
8990c92aea
core/state: add account address to Trie slot accessors (#26934)
This changes the Trie interface to add the plain account address as a
parameter to all storage-related methods.

After the introduction of the TryAccount* functions, TryGet, TryUpdate and
TryDelete are now only meant to read an account's storage. In their current
form, they assume that an account storage is stored in a separate trie, and
that the hashing of the slot is independent of its account's address.

The proposed structure for a stateless storage breaks these two
assumptions: the hashing of a slot key requires the address and all slots
and accounts are stored in a single trie.

This PR therefore adds an address parameter to the interface. It is ignored
in the MPT version, so this change has no functional impact, however it
will reduce the diff size when merging verkle trees.
2023-03-23 11:52:22 +01:00
rjl493456442
bba2a1bac5
core: show db error-info in case of mismatched hash root (#26870)
When a database failure occurs, bubble it up a into statedb, and report it in suitable places, such as during a 'bad block' report.
2023-03-16 03:12:34 -04:00
rjl493456442
c8a6b7100c
core/state, trie: port changes from PBSS (#26763) 2023-03-14 04:50:53 -04:00
rjl493456442
fe01a2f63b
all: use unified emptyRootHash and emptyCodeHash (#26718)
The EmptyRootHash and EmptyCodeHash are defined everywhere in the codebase, this PR replaces all of them with unified one defined in core/types package, and also defines constants for TxRoot, WithdrawalsRoot and UncleRoot
2023-02-21 06:12:27 -05:00
rjl493456442
13ef21d467
Revert "core/trie: remove trie tracer (#26665)" (#26732)
This reverts commit 7c749c947a.
2023-02-20 09:54:52 -05:00
Martin Holst Swende
7c749c947a
core/trie: remove trie tracer (#26665)
This PR contains a small portion of the full pbss PR, namely

    Remove the tracer from trie (and comitter), and instead using an accessList.
    Related changes to the Nodeset.


---------

Co-authored-by: Gary Rong <garyrong0905@gmail.com>
2023-02-19 14:10:19 -05:00
Martin Holst Swende
22c3ad1d12
core/state, trie: remove unused error-return from trie Commit operation (#26641) 2023-02-09 08:56:59 -05:00
Martin Holst Swende
7a489623ac
core/state: remove notion of fake storage (#24916)
This PR removes the notion of fakeStorage from the state objects, and instead, for any state modifications that are needed, it simply makes the changes.
2023-01-10 08:24:30 -05:00
rjl493456442
6c149fd4ad
core, eth, trie, light: clean up trie interface (#26388)
* all: cleanup trie interface

* eth, trie: address comments
2023-01-03 15:41:40 +02:00
rjl493456442
c87f321b8f
core/state: maintain destruction flag by default (#26371)
This changes moves the tracking of "deleted in this block" out from snap-only domain, so that it happens regardless of whether the execution is snapshot-backed or trie-backed.
2022-12-28 08:53:43 -05:00
rjl493456442
01808421e2
core/state: return error when storage trie can't be opened (#26350)
This changes the StorageTrie method to return an error when the trie
is not available. It used to return an 'empty trie' in this case, but that's
not possible anymore under PBSS.
2022-12-21 10:21:21 +01:00
rjl493456442
cda051eba7
core, cmd: fill blockNumber in logs (#26345)
* core, cmd: fill blockNumber in logs

* Update core/state/statedb.go

Co-authored-by: Martin Holst Swende <martin@swende.se>

* core/types: revert

* core/state: improve comments

Co-authored-by: Martin Holst Swende <martin@swende.se>
2022-12-13 07:54:16 -05:00
Marius van der Wijden
ec2ec2d08e
core: implement EIP-3651, warm coinbase (#25819)
Implements EIP-3651, "Warm Coinbase", for Shanghai hardfork. Specification: https://eips.ethereum.org/EIPS/eip-3651.
2022-11-22 22:39:52 +01:00
Mark Tyneway
b4ea2bf7dd
all: implement EIP-1153 transient storage (#26003)
Implements TSTORE and TLOAD as specified by the following EIP:

https://eips.ethereum.org/EIPS/eip-1153
https://ethereum-magicians.org/t/eip-1153-transient-storage-opcodes/553


Co-authored-by: Sara Reynolds <snreynolds2506@gmail.com>
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
2022-11-16 10:18:52 +01:00
rjl493456442
60e30a940b
core/rawdb: refactor db inspector for extending multiple ancient store (#25896)
This PR ports a few changes from PBSS:

- Fix the snapshot generator waiter in case the generation is not even initialized
- Refactor db inspector for ancient store
2022-10-28 10:23:49 +02:00
Martin Holst Swende
5a02b2d6d0
all: fix spelling mistakes (#25961) 2022-10-11 09:37:00 +02:00
rjl493456442
bff84a99fe
cmd, core, eth, les, light: track deleted nodes (#25757)
* cmd, core, eth, les, light: track deleted nodes

* trie: add docs

* trie: address comments

* cmd, core, eth, les, light, trie: trie id

* trie: add tests

* trie, core: updates

* trie: fix imports

* trie: add utility print-method for nodeset

* trie: import err

* trie: fix go vet warnings

Co-authored-by: Martin Holst Swende <martin@swende.se>
2022-09-27 10:01:02 +02:00
Felix Lange
b628d72766
build: upgrade to go 1.19 (#25726)
This changes the CI / release builds to use the latest Go version. It also
upgrades golangci-lint to a newer version compatible with Go 1.19.

In Go 1.19, godoc has gained official support for links and lists. The
syntax for code blocks in doc comments has changed and now requires a
leading tab character. gofmt adapts comments to the new syntax
automatically, so there are a lot of comment re-formatting changes in this
PR. We need to apply the new format in order to pass the CI lint stage with
Go 1.19.

With the linter upgrade, I have decided to disable 'gosec' - it produces
too many false-positive warnings. The 'deadcode' and 'varcheck' linters
have also been removed because golangci-lint warns about them being
unmaintained. 'unused' provides similar coverage and we already have it
enabled, so we don't lose much with this change.
2022-09-10 13:25:40 +02:00
rjl493456442
5ddedd2f83
core, light, trie: remove DiskDB function from trie database (#25690) 2022-09-07 09:08:56 +02:00
Justin Traglia
2c5648d891
all: fix some typos (#25551)
* Fix some typos

* Fix some mistakes

* Revert 4byte.json

* Fix an incorrect fix

* Change files to fails
2022-08-19 09:00:21 +03:00
Guillaume Ballet
6da5c1644d
core/state, trie, light: add a TryDeleteAccount method (#25531)
* core/state, trie, light: Add a DeleteAccount method

* review feedback

* Update database.go

* pr triage feedback

Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
2022-08-17 14:14:49 +03:00
Guillaume Ballet
f67e54c92f
core: use TryGetAccount to read what TryUpdateAccount has written (#25458)
* core: use TryGetAccount to read where TryUpdateAccount has been used to write

* Gary's review feedback

* implement Gary's suggestion

* fix bug + rename NewSecure into NewStateTrie

* trie: add backwards-compatibility aliases for SecureTrie

* Update database.go

* make the linter happy

Co-authored-by: Felix Lange <fjl@twurst.com>
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
2022-08-04 16:13:18 +02:00
rjl493456442
8b53b92eb4
core, trie: rework trie committer (#25320)
* all: rework trie and trie committer

* all: get rid of internal cache in trie

* all: fixes

* trie: polish

* core, trie: address comments

* trie: fix imports

* core/state: address comments

* core/state/snapshot: polish

* trie: remove unused code

* trie: update tests

* trie: don't set db as nil

* trie: address comments

* trie: unskip test
2022-08-04 11:03:20 +03:00
rjl493456442
1657e43931
core, les, eth: port snap sync changes (#24898)
core, eth, les, trie: rework snap sync
2022-07-15 14:55:51 +03:00
rjl493456442
22defa5af7
all: introduce trie owner notion (#24750)
* cmd, core/state, light, trie, eth: add trie owner notion

* all: refactor

* tests: fix goimports

* core/state/snapshot: fix ineffasigns

Co-authored-by: Martin Holst Swende <martin@swende.se>
2022-06-06 17:14:55 +02:00
Péter Szilágyi
48605b5f61
core/statedb: always clear out access list when setting a new one 2022-03-08 12:00:29 +02:00
Martin Holst Swende
51e7968b8b
core/state: fix read-meters + simplify code (#24304) 2022-02-14 09:22:57 +01:00