Commit Graph

441 Commits

Author SHA1 Message Date
Péter Szilágyi
90d25514af
core, eth: merge snap-sync chain download progress logs (#26676) 2023-02-21 12:17:34 +02:00
ucwong
e9d42499bb
eth/downloader: fix typo (#26716) 2023-02-16 19:40:16 +02:00
Martin Holst Swende
13d7de77f4
eth/downloader: fix empty-body case in queue fetchresult (#26707) 2023-02-16 12:28:01 +02:00
Martin Holst Swende
645e3e86c4
core, eth/downloader: make body validation more strict (#26704) 2023-02-16 12:10:16 +02:00
Marius van der Wijden
7fb42e6db2
eth/downloader: handle missing withdrawals if empty list is expected (#26675)
This PR relaxes the block body ingress handling a bit: if block body withdrawals are missing (but expected to be empty), the body withdrawals are set to 'empty list' before being passed to upper layers. 

This fixes an issue where a block passed from EthereumJS to geth was deemed invalid.
2023-02-15 07:23:07 -05:00
Péter Szilágyi
3086c256c9
eth/downloader: fix timeout resurrection panic (#26652)
* common/prque, eth/downloader: fix timeout resurrection panic

* common/prque: revert -1 hack for les, temporaryly!
2023-02-09 14:56:15 +02:00
Péter Szilágyi
bf1798e04e
common/prque: generic priority queue (#26290)
* common, core, eth, les, trie: make prque generic

* les/vflux/server: fixed issues in priorityPool

* common, core, eth, les, trie: make priority also generic in prque

* les/flowcontrol: add test case for priority accumulator overflow

* les/flowcontrol: avoid priority value overflow

* common/prque: use int priority in some tests

No need to convert to int64 when we can just change the type used by the
queue.

* common/prque: remove comment about int64 range

---------

Co-authored-by: Zsolt Felfoldi <zsfelfoldi@gmail.com>
Co-authored-by: Felix Lange <fjl@twurst.com>
2023-02-09 13:03:54 +02:00
Marius van der Wijden
2a2b0419fb
all: implement withdrawals (EIP-4895) (#26484)
This change implements withdrawals as specified in EIP-4895.

Co-authored-by: lightclient@protonmail.com <lightclient@protonmail.com>
Co-authored-by: marioevz <marioevz@gmail.com>
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Felix Lange <fjl@twurst.com>
2023-01-25 15:32:25 +01:00
rjl493456442
b748709a11
eth/downloader: fix cornercase when clean stale beacon headers (#26441) 2023-01-12 11:19:16 +01:00
Péter Szilágyi
452a12aa79
eth/downloader: move the pivot in beacon sync mode too (#26453)
In legacy (pre-merge) sync mode, headers were contiguously downloaded from the network and when no more headers were available, we checked every few seconds whether there are 64 new blocks to move the pivot.

In beacon (post-merge) sync mode, we don't need to check for new skeleton headers non stop, since those re delivered one by one by the engine API. The missing code snippet from the header fetcher was to actually look at the latest head and move the pivot if it was more than 2*64-8 away. This PR adds the missing movement logic.
2023-01-10 12:16:54 -05:00
Péter Szilágyi
71f7988b0f
eth/downloader: create repro testcase for beacon header loss 2023-01-09 12:12:25 +02:00
Gary Rong
686f7438d3 eth/downloader: fix unexpected skeleton header deletion 2023-01-09 15:07:08 +08:00
Ha DANG
9d38466437
eth/downloader: fix some typos (#26396) 2023-01-03 11:03:44 +02:00
rjl493456442
743e404906
core, eth, les, tests, trie: abstract node scheme (#25532)
This PR introduces a node scheme abstraction. The interface is only implemented by `hashScheme` at the moment, but will be extended by `pathScheme` very soon.

Apart from that, a few changes are also included which is worth mentioning:

-  port the changes in the stacktrie, tracking the path prefix of nodes during commit
-  use ethdb.Database for constructing trie.Database. This is not necessary right now, but it is required for path-based used to open reverse diff freezer
2022-11-28 14:31:28 +01:00
Péter Szilágyi
c4a662176e
core, eth: for types with accurate size calcs, return uint64, not float (#26046)
* core, eth: for types with accurate size calcs, return uint64, not float

* core/types: proper tx size tests

* core/types: extend tx size test with decoded sizes, fix error

* core/txpool: fix linter

Co-authored-by: Martin Holst Swende <martin@swende.se>
2022-10-26 15:23:07 +03:00
Martin Holst Swende
5a02b2d6d0
all: fix spelling mistakes (#25961) 2022-10-11 09:37:00 +02:00
Martin Holst Swende
ee301c750b
all: fix docstrings 2022-10-04 09:18:02 +02:00
Martin Holst Swende
80cc34ac3c
eth/downloader: make flakey test less flakey (#25879) 2022-09-27 15:49:52 +02:00
zhiqiangxu
b32d20324e
eth/downloader, les/downloader: fix subtle flaw in queue delivery (#25861)
* fix queue.deliver

* les/downloader: fix queue.deliver

Co-authored-by: Martin Holst Swende <martin@swende.se>
2022-09-26 12:33:21 +02:00
Felix Lange
b628d72766
build: upgrade to go 1.19 (#25726)
This changes the CI / release builds to use the latest Go version. It also
upgrades golangci-lint to a newer version compatible with Go 1.19.

In Go 1.19, godoc has gained official support for links and lists. The
syntax for code blocks in doc comments has changed and now requires a
leading tab character. gofmt adapts comments to the new syntax
automatically, so there are a lot of comment re-formatting changes in this
PR. We need to apply the new format in order to pass the CI lint stage with
Go 1.19.

With the linter upgrade, I have decided to disable 'gosec' - it produces
too many false-positive warnings. The 'deadcode' and 'varcheck' linters
have also been removed because golangci-lint warns about them being
unmaintained. 'unused' provides similar coverage and we already have it
enabled, so we don't lose much with this change.
2022-09-10 13:25:40 +02:00
rjl493456442
dea1fb3cfc
all: cleanup tests (#25641)
Follow-up to PR #25523 to cleanup all relevant tests.
2022-09-07 20:21:59 +02:00
rjl493456442
d10c280309
all: move genesis initialization to blockchain (#25523)
* all: move genesis initialization to blockchain

* core: add one more check

* core: fix tests
2022-08-30 18:22:28 +02:00
Péter Szilágyi
81bd998353
core, eth/downloader: handle spurious junk bodies from racey rollbacks (#25578)
* eth/downloader: handle junkbodies/receipts in the beacon sync

* core: check for header presence when checking for blocks
2022-08-23 14:02:51 +03:00
Justin Traglia
656dc8cc00
eth, les: unlock downloader peerSet if there's an error (#25546)
Unlock peerSet if there's an error in the downloader
2022-08-19 09:02:47 +03:00
Justin Traglia
2c5648d891
all: fix some typos (#25551)
* Fix some typos

* Fix some mistakes

* Revert 4byte.json

* Fix an incorrect fix

* Change files to fails
2022-08-19 09:00:21 +03:00
rjl493456442
a41ea8a97c
all: cleanup the APIs for initializing genesis (#25473)
* all: polish tests

* core: apply feedback from Guillaume

* core: fix comment
2022-08-09 12:44:39 +03:00
int88
e93442c6cf
eth/downloader: fix log errors of queue_test.go (#25494) 2022-08-08 16:07:54 +03:00
Péter Szilágyi
f3af3fd8df
eth: support bubbling up bad blocks from sync to the engine API (#25190)
* eth: support bubbling up bad blocks from sync to the engine API

* eth/catalyst: fix typo

Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>

* eth/catalyst: fix typo

Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>

* Update eth/catalyst/api.go

* eth/catalyst: when forgetting bad hashes, also forget descendants

* eth/catalyst: minor bad block tweaks for resilience

Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
Co-authored-by: Martin Holst Swende <martin@swende.se>
2022-07-25 16:51:04 +03:00
Delweng
b196ad1c16
all: add whitespace linter (#25312)
* golangci: typo

Signed-off-by: Delweng <delweng@gmail.com>

* golangci: add whietspace

Signed-off-by: Delweng <delweng@gmail.com>

* *: rm whitesapce using golangci-lint

Signed-off-by: Delweng <delweng@gmail.com>

* cmd/puppeth: revert accidental resurrection

Co-authored-by: Péter Szilágyi <peterke@gmail.com>
2022-07-25 13:14:03 +03:00
rjl493456442
1657e43931
core, les, eth: port snap sync changes (#24898)
core, eth, les, trie: rework snap sync
2022-07-15 14:55:51 +03:00
lightclient
10dc5dce08
all: remove concept of public/private API definitions (#25053)
* internal/ethapi: rename PublicEthereumAPI to EthereumAPI

* eth: rename PublicEthereumAPI to EthereumAPI

* internal/ethapi: rename PublicTxPoolAPI to TxPoolAPI

* internal/ethapi: rename PublicAccountAPI to EthereumAccountAPI

* internal/ethapi: rename PrivateAccountAPI to PersonalAccountAPI

* internal/ethapi: rename PublicBlockChainAPI to BlockChainAPI

* internal/ethapi: rename PublicTransactionPoolAPI to TransactionAPI

* internal/ethapi: rename PublicDebugAPI to DebugAPI

* internal/ethapi: move PrivateDebugAPI methods to DebugAPI

* internal/ethapi: rename PublicNetAPI to NetAPI

* les: rename PrivateLightServerAPI to LightServerAPI

* les: rename PrivateLightAPI to LightAPI

* les: rename PrivateDebugAPI to DebugAPI

* les: rename PublicDownloaderAPI to DownloaderAPI

* eth,les: rename PublicFilterAPI to FilterAPI

* eth: rename PublicMinerAPI to MinerAPI

* eth: rename PublicDownloaderAPI to DownloaderAPI

* eth: move PrivateMinerAPI methods to MinerAPI

* eth: rename PrivateAdminAPI to AdminAPI

* eth: rename PublicDebugAPI to DebugAPI

* eth: move PrivateDebugAPI methods to DebugAPI

* node: rename publicAdminAPI to adminAPI

* node: move privateAdminAPI methods to adminAPI

* node: rename publicWeb3API to web3API

* eth,internal/ethapi: sync comments with previous renamings
2022-06-21 12:05:43 +03:00
rjl493456442
30602163d5
eth: introduce eth67 protocol (#24093)
The new protocol version removes support for GetNodeData.
See https://eips.ethereum.org/EIPS/eip-4938 for more information.

Co-authored-by: Felix Lange <fjl@twurst.com>
Co-authored-by: Martin Holst Swende <martin@swende.se>
2022-06-15 12:56:47 +02:00
lightclient
3273ad1a58
eth: add missing period at end of sentences (#25058)
eth: add missing periods on end of comments
2022-06-15 09:29:23 +02:00
Martin Holst Swende
a907d7e81a
all: more linters (#24783)
This enables the following linters

- typecheck
- unused
- staticcheck
- bidichk
- durationcheck
- exportloopref
- gosec

WIth a few exceptions.

- We use a deprecated protobuf in trezor. I didn't want to mess with that, since I cannot meaningfully test any changes there.
- The deprecated TypeMux is used in a few places still, so the warning for it is silenced for now.
- Using string type in context.WithValue is apparently wrong, one should use a custom type, to prevent collisions between different places in the hierarchy of callers. That should be fixed at some point, but may require some attention.
- The warnings for using weak random generator are squashed, since we use a lot of random without need for cryptographic guarantees.
2022-06-13 16:24:45 +02:00
Felix Lange
9244d5cd61
all: update license headers and AUTHORS from git history (#24947) 2022-05-24 20:39:40 +02:00
s7v7nislands
7caa2d8163
all: replace strings.Replace with string.ReplaceAll (#24835) 2022-05-09 13:13:23 +03:00
Ikko Ashimine
40cfe71002
eth/downloader: fix typo in downloader.go (#24704)
synchornization -> synchronization
2022-04-20 16:17:29 +02:00
rjl493456442
f0328f241b
eth/downloader: resolve local header by hash for beacon sync (#24691)
* eth/downlaoder: resolve local header by hash for beacon sync

* eth/downloader: fix error message

* eth/downloader: cap the reverse header resolving

* eth/downloader: re-enable tests

* eth/downloader: add warning logs
2022-04-14 09:49:23 +03:00
Péter Szilágyi
86216189a5
eth/downloader: remove stale beacon headers as backfilling progresses (#24670)
* eth/downloader: remove stale beacon headers as backfilling progresses

* eth/downloader: remove leftover from a previous design

* eth/downloader: do partial beacon cleanups if chain is large

* eth/downloader: linter != heart
2022-04-13 20:31:08 +03:00
Eng Zer Jun
8d066f1f42
all: use T.TempDir to create temporary test directories (#24633)
This commit replaces ioutil.TempDir with t.TempDir in tests. The
directory created by t.TempDir is automatically removed when the test
and all its subtests complete.

Prior to this commit, temporary directory created using ioutil.TempDir
had to be removed manually by calling os.RemoveAll, which is omitted in
some tests. The error handling boilerplate e.g.

	defer func() {
		if err := os.RemoveAll(dir); err != nil {
			t.Fatal(err)
		}
	}

is also tedious, but t.TempDir handles this for us nicely.

Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2022-04-08 15:44:55 +02:00
rjl493456442
28ec26094b
eth/downloader: retrieve pivot header from local chain if necessary (#24610)
* eth/downloader: retrieve pivot header from local chain if necessary

* eth/downloader: improve readability

* eth/downloader: update fix

* eth/downloader: add beacon sync tests

* eth/downloader: remove duplicated code
2022-04-04 10:10:16 +03:00
Martin Holst Swende
f252154599
eth/downloader: fix flakey test (#24576) 2022-03-23 20:57:53 +01:00
rjl493456442
e0e8bf31c5
eth/downloader: ignore zero size header batch for importing (#24569)
* eth/downloader: ignore zero size header batch for importing

* core, light: reject empty header batch for importing
2022-03-22 18:58:05 +01:00
Péter Szilágyi
51de2bc9dc
eth/downloader: terminate beacon sync early when linked to local chain (#24550)
* eth/downloader: terminate beacon sync early when linked to local chain

* eth/downloader: fix backfiller resume on early beacon termination
2022-03-21 10:43:54 +02:00
Péter Szilágyi
6b8718c374
eth/downloader: fix off-by-one error in test causing 50% fails 2022-03-16 08:57:00 +02:00
Péter Szilágyi
8f66ea3786
eth/downloader: implement beacon sync (#23982)
* eth/downloader: implement beacon sync

* eth/downloader: fix a crash if the beacon chain is reduced in length

* eth/downloader: fix beacon sync start/stop thrashing data race

* eth/downloader: use a non-nil pivot even in degenerate sync requests

* eth/downloader: don't touch internal state on beacon Head retrieval

* eth/downloader: fix spelling mistakes

* eth/downloader: fix some typos

* eth: integrate legacy/beacon sync switchover and UX

* eth: handle UX wise being stuck on post-merge TTD

* core, eth: integrate the beacon client with the beacon sync

* eth/catalyst: make some warning messages nicer

* eth/downloader: remove Ethereum 1&2 notions in favor of merge

* core/beacon, eth: clean up engine API returns a bit

* eth/downloader: add skeleton extension tests

* eth/catalyst: keep non-kiln spec, handle mining on ttd

* eth/downloader: add beacon header retrieval tests

* eth: fixed spelling, commented failing tests out

* eth/downloader: review fixes

* eth/downloader: drop peers failing to deliver beacon headers

* core/rawdb: track beacon sync data in db inspect

* eth: fix review concerns

* internal/web3ext: nit

Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
2022-03-11 14:14:45 +02:00
Martin Holst Swende
db03faa10d
core, eth: improve delivery speed on header requests (#23105)
This PR reduces the amount of work we do when answering header queries, e.g. when a peer
is syncing from us.

For some items, e.g block bodies, when we read the rlp-data from database, we plug it
directly into the response package. We didn't do that for headers, but instead read
headers-rlp, decode to types.Header, and re-encode to rlp. This PR changes that to keep it
in RLP-form as much as possible. When a node is syncing from us, it typically requests 192
contiguous headers. On master it has the following effect:

- For headers not in ancient: 2 db lookups. One for translating hash->number (even though
  the request is by number), and another for reading by hash (this latter one is sometimes
  cached).
  
- For headers in ancient: 1 file lookup/syscall for translating hash->number (even though
  the request is by number), and another for reading the header itself. After this, it
  also performes a hashing of the header, to ensure that the hash is what it expected. In
  this PR, I instead move the logic for "give me a sequence of blocks" into the lower
  layers, where the database can determine how and what to read from leveldb and/or
  ancients.

There are basically four types of requests; three of them are improved this way. The
fourth, by hash going backwards, is more tricky to optimize. However, since we know that
the gap is 0, we can look up by the parentHash, and stlil shave off all the number->hash
lookups.

The gapped collection can be optimized similarly, as a follow-up, at least in three out of
four cases.

Co-authored-by: Felix Lange <fjl@twurst.com>
2021-12-07 17:50:58 +01:00
Péter Szilágyi
58d1988349
core, eth, les, trie: remove the sync bloom, used by fast sync 2021-12-03 12:32:41 +02:00
Péter Szilágyi
c893488349
eth: pre-process downloader responses on the peer reader thread 2021-12-01 20:18:12 +02:00
Péter Szilágyi
c10a0a62c3
eth: request id dispatcher and direct req/reply APIs (#23576)
* eth: request ID based message dispatcher

* eth: fix dispatcher cancellation, rework fetchers idleness tracker

* eth/downloader: drop peers who refuse to serve advertised chains
2021-11-26 13:26:03 +02:00