Commit Graph

974 Commits

Author SHA1 Message Date
philip-morlier
7e8ed6b43d Merge commit '02eb36afc' into merge/geth-v1.13.12 2024-02-09 07:15:32 -08:00
lightclient
1f50aa7631
cmd,internal/era: implement export-history subcommand (#26621)
* all: implement era format, add history importer/export

* internal/era/e2store: refactor e2store to provide ReadAt interface

* internal/era/e2store: export HeaderSize

* internal/era: refactor era to use ReadAt interface

* internal/era: elevate anonymous func to named

* cmd/utils: don't store entire era file in-memory during import / export

* internal/era: better abstraction between era and e2store

* cmd/era: properly close era files

* cmd/era: don't let defers stack

* cmd/geth: add description for import-history

* cmd/utils: better bytes buffer

* internal/era: error if accumulator has more records than max allowed

* internal/era: better doc comment

* internal/era/e2store: rm superfluous reader, rm superfluous testcases, add fuzzer

* internal/era: avoid some repetition

* internal/era: simplify clauses

* internal/era: unexport things

* internal/era,cmd/utils,cmd/era: change to iterator interface for reading era entries

* cmd/utils: better defer handling in history test

* internal/era,cmd: add number method to era iterator to get the current block number

* internal/era/e2store: avoid double allocation during write

* internal/era,cmd/utils: fix lint issues

* internal/era: add ReaderAt func so entry value can be read lazily

Co-authored-by: lightclient <lightclient@protonmail.com>
Co-authored-by: Martin Holst Swende <martin@swende.se>

* internal/era: improve iterator interface

* internal/era: fix rlp decode of header and correctly read total difficulty

* cmd/era: fix rebase errors

* cmd/era: clearer comments

* cmd,internal: fix comment typos

---------

Co-authored-by: Martin Holst Swende <martin@swende.se>
2024-02-07 09:18:27 -07:00
philip-morlier
2e29160068 Merge remote-tracking branch 'origin/bugfix/trie-plugin-cleanup-dev' into merge/geth-v1.13.11 2024-01-24 14:39:09 -08:00
philip-morlier
48be241dbf Merge commit '8f7eb9ccd' into merge/geth-v1.13.11 2024-01-24 09:31:20 -08:00
Martin HS
6a724b94db
docs: remove reference to being official (#28858) 2024-01-23 09:26:00 +01:00
Péter Szilágyi
43ba7d65a8
cmd/geth, internal/debug: get rid of by-default log config (#28801) 2024-01-12 15:59:03 +02:00
Austin Roberts
c2fa267503 Merge tag 'v1.13.9' into merge/geth-v1.13.9 2024-01-10 10:51:25 -06:00
jwasinger
9e018ce3a5
cmd/geth: update log test data (#28780)
update logger test data
2024-01-09 15:35:49 +01:00
Martin HS
1010a79c7c
cmd/geth: make it possible to autopilot removedb (#28725)
When managing geth, it is sometimes desirable to do a partial wipe; deleting state but retaining freezer data. A partial wipe can be somewhat tricky to accomplish. 

This change implements the ability to perform partial wipe by making it possible to run geth removedb non-interactive, using command line options instead.
2024-01-09 08:56:01 +01:00
Marius Kjærstad
99eb49e601
internal/flags: update copyright year to 2024 (#28760)
Co-authored-by: Felix Lange <fjl@twurst.com>
2024-01-04 15:03:58 +01:00
philip-morlier
b4a5cf811e Merge commit 'b20b4a715' into merge/geth-v1.13.8 2023-12-22 07:37:09 -08:00
rjl493456442
cca94792a4
core, cmd, trie: fix the condition of pathdb initialization (#28718)
Original problem was caused by #28595, where we made it so that as soon as we start to sync, the root of the disk layer is deleted. That is not wrong per se, but another part of the code uses the "presence of the root" as an init-check for the pathdb. And, since the init-check now failed, the code tried to re-initialize it which failed since a sync was already ongoing.

The total impact being: after a state-sync has begun, if the node for some reason is is shut down, it will refuse to start up again, with the error message: `Fatal: Failed to register the Ethereum service: waiting for sync.`. 

This change also modifies how `geth removedb` works, so that the user is prompted for two things: `state data` and `ancient chain`. The former includes both the chaindb aswell as any state history stored in ancients. 

---------

Co-authored-by: Martin HS <martin@swende.se>
2023-12-21 20:28:32 +01:00
philip-morlier
ac499a7ff1 Merge commit 'da6cdaf63' into merge/geth-v1.13.6 2023-12-18 15:08:33 -08:00
ucwong
81fd1b3cf9
core/txpool : small cleanup refactors (#28654) 2023-12-12 17:23:36 +02:00
Austin Roberts
bc14d12fd7 Use the statedb database to get Tries
The old approach of getting state tries involved instantiating a
new state database and using it to instantiate a trie. What we didn't
realize was that state database needed to have Close() called on it,
which we didn't offer a way for plugins to do, resulting in memory
leaking.

This approach reuses the primary trie database associated with the
blockchain object, albeit a bit indirectly. This will allow access
to recent tries that are stored in memory, where previously only
tries that had been committed to disk were accessible.
2023-12-06 16:10:17 -06:00
Martin Holst Swende
b8d44ed98b
log: remove lazy, remove unused interfaces, unexport methods (#28622)
This change 

- Removes interface `log.Format`, 
- Removes method `log.FormatFunc`, 
- unexports `TerminalHandler.TerminalFormat` formatting methods (renamed to `TerminalHandler.format`)
- removes the notion of `log.Lazy` values


The lazy handler was useful in the old log package, since it
could defer the evaluation of costly attributes until later in the
log pipeline: thus, if the logging was done at 'Trace', we could
skip evaluation if logging only was set to 'Info'.

With the move to slog, this way of deferring evaluation is no longer
needed, since slog introduced 'Enabled': the caller can thus do
the evaluate-or-not decision at the callsite, which is much more
straight-forward than dealing with lazy reflect-based evaluation.

Also, lazy evaluation would not work with 'native' slog, as in, these
two statements would be evaluated differently:

```golang
  log.Info("foo", "my lazy", lazyObj)
  slog.Info("foo", "my lazy", lazyObj)
```
2023-12-05 11:54:44 +01:00
jwasinger
28e7371701
all: replace log15 with slog (#28187)
This PR replaces Geth's logger package (a fork of [log15](https://github.com/inconshreveable/log15)) with an implementation using slog, a logging library included as part of the Go standard library as of Go1.21.

Main changes are as follows:
* removes any log handlers that were unused in the Geth codebase.
* Json, logfmt, and terminal formatters are now slog handlers.
* Verbosity level constants are changed to match slog constant values.  Internal translation is done to make this opaque to the user and backwards compatible with existing `--verbosity` and `--vmodule` options.
* `--log.backtraceat` and `--log.debug` are removed.

The external-facing API is largely the same as the existing Geth logger.  Logger method signatures remain unchanged.

A small semantic difference is that a `Handler` can only be set once per `Logger` and not changed dynamically.  This just means that a new logger must be instantiated every time the handler of the root logger is changed.

----
For users of the `go-ethereum/log` module. If you were using this module for your own project, you will need to change the initialization. If you previously did 
```golang
log.Root().SetHandler(log.LvlFilterHandler(log.LvlInfo, log.StreamHandler(os.Stderr, log.TerminalFormat(true))))
```
You now instead need to do 
```golang
log.SetDefault(log.NewLogger(log.NewTerminalHandlerWithLevel(os.Stderr, log.LevelInfo, true)))
```
See more about reasoning here: https://github.com/ethereum/go-ethereum/issues/28558#issuecomment-1820606613
2023-11-29 08:33:50 +01:00
Martin Holst Swende
63979bc9cc
cmd/evm, core/state: fix post-exec dump of state (statetests, blockchaintests) (#28504)
There were several problems related to dumping state. 

- If a preimage was missing, even if we had set the `OnlyWithAddresses` to `false`, to export them anyway, the way the mapping was constructed (using `common.Address` as key) made the entries get lost anyway. Concerns both state- and blockchain tests. 
- Blockchain test execution was not configured to store preimages.

This changes makes it so that the block test executor takes a callback, just like the state test executor already does. This callback can be used to examine the post-execution state, e.g. to aid debugging of test failures.
2023-11-28 13:54:17 +01:00
Marius van der Wijden
bdf5e388ca
cmd, les, tests: remove light client code (#28586)
* cmd, les, tests: remove light client code

This commit removes the light client (LES) code.
Since the merge the light client has been broken and
it is hard to maintain it alongside the normal client.
We decided it would be best to remove it for now and
maybe rework and reintroduce it in the future.

* cmd, eth: remove some more mentions of light mode

* cmd: re-add flags and mark as deprecated

* cmd: warn the user about deprecated flags

* eth: better error message
2023-11-23 16:28:26 +02:00
Felix Lange
63127f5443
cmd/geth: fix build error (#28585) 2023-11-22 16:32:43 +01:00
Guillaume Ballet
d468c333a7
cmd/{geth,utils}: add cmd to export preimages in snap enumeration order (#28256)
Adds a subcommand: `geth snapshot export-preimages`, to export preimages of every hash found during a snapshot enumeration: that is, it exports _only the active state_, and not _all_ preimages that have been used but are no longer part of the state. 

This tool is needed for the verkle transition, in order to distribute the preimages needed for the conversion. Since only the 'active' preimages are exported, the output is shrunk from ~70GB to ~4GB.

The order of the output is the order used by the snapshot enumeration, which avoids database thrashing. However, it also means that storage-slot preimages are not deduplicated.
2023-11-22 14:48:25 +01:00
jwasinger
3cfcd252db
cmd/geth: add support for --dev flag in dumpgenesis (#28463)
Co-authored-by: Felix Lange <fjl@twurst.com>
Co-authored-by: lightclient <lightclient@protonmail.com>
2023-11-22 12:08:39 +01:00
Håvard Anda Estensen
460cc1673e
cmd: run tests in parallel (#28546) 2023-11-20 10:52:14 +01:00
jwasinger
984f82629c
cmd/geth: more special cases logging tests (#28527)
adds logging tests for errors and custom fmt.Stringer-types which output strings that needs to be quoted/escaped.
2023-11-15 09:54:35 +01:00
philip-morlier
406ad4d1fb Merge commit '916d6a441' into merge/geth-v1.13.5 2023-11-14 08:39:46 -08:00
Guillaume Ballet
fa8d39807d
cmd, core, trie: verkle-capable geth init (#28270)
This change allows the creation of a genesis block for verkle testnets. This makes for a chunk of code that is easier to review and still touches many discussion points.
2023-11-14 13:09:40 +01:00
Delweng
f265cc24b4
cmd/geth: remove some whitespace in code and comments (#28148)
This changes just removes some whitespace
2023-11-10 13:26:13 +01:00
Martin Holst Swende
b77a9b127c
cmd/geth: more testcases for logging (#28501)
* cmd/geth: more testcases for logging

This adds more edgecases around logging, particularly around handling of different types of nil-values
as concrete types and within interfaces.

Also adds tests with 'reserved' values which breaks json/logfmt formats. The json output is checked in,
but not actively used by any testcase at the moment.

* cmd/geth/testdata: remove timestamps
2023-11-09 10:46:48 +01:00
Martin Holst Swende
233db64cc1
all: make vendored copy of reexec (#28382)
a little copying is better than a little dependency

-- go proverb

We have this dependency on docker, a.k.a moby: a gigantic library, and we only need ~70 LOC,
so here I tried moving it inline instead.

Co-authored-by: Felix Lange <fjl@twurst.com>
2023-10-28 00:14:43 +02:00
Guillaume Ballet
4cbca5178a
core, cmd/geth: add --override.* flags to geth init (#28407)
* core, cmd/geth: add --override.* flags to geth init

* also apply overrides before genesis commit with new block

* review feedback
2023-10-27 10:15:14 +08:00
Martin Holst Swende
58ae1df684
cmd/geth: test for logging-output (#28373)
This PR is a bit in preparation for the slog work in #28187 .

Our current test re logging mostly test the internals, but we have no real end-to-end test of the logging output. This PR introduces a simple reexec-based log tester. This also relies upon a special mode in geth, which can be made to eject a set of predefined log messages (only available if the build-tag `integrationtests` is used

e.g. go run --tags=integrationtests ./cmd/geth --log.format terminal logtest

While working on this, I also noticed a quirk in the setup: when geth was configured to use a file output, then two separate handlers were used (one handler for the file, one handler for the console). Using two separate handlers means that two formatters are used, thus the formatting of any/all records happened twice. This PR changes the mechanism to use two separate io.Writers instead, which is both more optimal and fixes a bug which occurs due to a global statefulness in the formatter.
2023-10-25 17:57:12 +02:00
philip-morlier
00ce5674f7 Adjusted utility function, ParseStateScheme(), in plugethCaptureTrieConfig().
The function was moved from /cmd/utils/ to /core/rawdb/ and the areguments adjusted.

Test Plugin passing without error.
2023-10-12 07:59:36 -07:00
philip-morlier
f912a2192d Merge commit '0d45d72d7' into merge/geth-v1.13.3 2023-10-12 07:16:53 -07:00
Martin Holst Swende
d2c0bed9d5
cmd/geth: fix failing test (#28322) 2023-10-12 09:54:00 +02:00
rjl493456442
eeb5dc3ccf
cmd, core: resolve scheme from a read-write database (#28313)
* cmd, core: resolve scheme from a read-write database

* cmd, core, eth: move the scheme check in the ethereum constructor

* cmd/geth: dump should in ro mode

* cmd: reverts
2023-10-11 11:27:44 +03:00
Péter Szilágyi
7b6ff527d5
cmd, eth: switch the dev synctarget to hash from block (#28209)
* cmd, eth: switch the dev synctarget to hash from block

* cmd/utils, eth/catalyst: terminate node wyen synctarget reached
2023-09-29 22:11:15 +03:00
philip-morlier
18be67b3a3 Merge commit 'dc34fe829' into merge/geth-v1.13.2 2023-09-28 08:41:54 -07:00
Delweng
3d297fc2d7
cmd/geth: ensure db is closed before exit (#28150) 2023-09-25 18:28:20 +02:00
Péter Szilágyi
82ec555d70
cmd: add state.scheme to the database flag group for local flag handling (#28107) 2023-09-22 14:56:48 +03:00
Péter Szilágyi
d135bafdcb
cmd/geth: print progress logs when iterating large contracts too (#28179) 2023-09-22 11:07:20 +03:00
philip-morlier
5d773a540d Merge commit '3f40e65c4' into merge/geth-v1.13.1 2023-09-18 16:13:24 -07:00
Péter Szilágyi
16cd1a7561
cmd/geth, internal/flags: print envvar config source and bad names (#28119) 2023-09-15 15:52:53 +03:00
philip-morlier
54b4a7d9c7 Modifications to pluGethCaptureTrieConfig 2023-09-14 16:34:35 -07:00
Péter Szilágyi
d9fbb71d63
cmd/geth, internal/flags, go.mod: colorize cli help, support env vars (#28103)
* cmd/geth, internal/flags, go.mod: colorize cli help, support env vars

* internal/flags: use stdout, not stderr for terminal detection
2023-09-14 10:33:59 +03:00
philip-morlier
51c041850b Manual touches to plugeth code to accommodate geth v1.13.0 2023-09-13 12:57:48 -07:00
Péter Szilágyi
eb7438997b
cmd/geth: rename the protocols field in the metrics gague (#28102) 2023-09-13 13:17:55 -04:00
philip-morlier
bdc42a9128 Merge commit '7371b3817' into merge/geth-v1.13.0 2023-09-12 12:34:24 -07:00
philip-morlier
3ca53046eb Merge remote-tracking branch 'laconic/add-plugindir-flag' into laconic-merge 2023-09-07 18:21:23 -07:00
Martin Holst Swende
c60f7dd08d
deps: update minisign (#28066)
This updates minisign to the latest version. One new thing is that minisign (not go-minisign) has started to prehash the file, and in order to make geth pass the version-check, we need to sign the file in legacy-mode.
2023-09-07 04:18:46 -04:00
Jorge
53f3c2ae65
metrics, cmd/geth: informational metrics (prometheus, influxdb, opentsb) (#24877)
This chang creates a GaugeInfo metrics type for registering informational (textual) metrics, e.g. geth version number. It also improves the testing for backend-exporters, and uses a shared subpackage in 'internal' to provide sample datasets and ordered registry. 

Implements #21783

---------

Co-authored-by: Martin Holst Swende <martin@swende.se>
2023-08-31 13:37:17 -04:00