Commit Graph

406 Commits

Author SHA1 Message Date
rjl493456442
c1b69bd121
les: fix panic in ultralight client sync (#24641) 2022-04-08 15:48:52 +02:00
rjl493456442
da16d089c0
trie, les, tests, core: implement trie tracer (#24403)
Trie tracer is an auxiliary tool to capture all deleted nodes
which can't be captured by trie.Committer. The deleted nodes
can be removed from the disk later.
2022-03-31 09:28:32 +02:00
Marius van der Wijden
4f4622bc8b
core: eth: implement Kiln-v2 spec (#24506)
* core/beacon: eth/catalyst: updated engine api to new version

* core: implement exchangeTransitionConfig

* core/beacon: prevRandao instead of Random

* eth/catalyst: Fix ExchangeTransitionConfig, add test

* eth/catalyst: stop external miners on TTD reached

* node: implement --authrpc.vhosts flag

* core: allow for config override on non-mainnet networks

* eth/catalyst: fix peters comments

* eth/catalyst: make stop remote sealer more explicit

* eth/catalyst: add log output

* cmd/utils: rename authrpc.host to authrpc.addr

* eth/catalyst: disable the disabling of the miner

* eth: core: remove notion of terminal pow block

* eth: les: more of peters nitpicks
2022-03-17 17:20:03 +02:00
rjl493456442
7a80cf6543
les, tests: fix vflux fuzzer by removing unnecessary panic (#24537) 2022-03-16 11:13:10 +01:00
Péter Szilágyi
8f66ea3786
eth/downloader: implement beacon sync (#23982)
* eth/downloader: implement beacon sync

* eth/downloader: fix a crash if the beacon chain is reduced in length

* eth/downloader: fix beacon sync start/stop thrashing data race

* eth/downloader: use a non-nil pivot even in degenerate sync requests

* eth/downloader: don't touch internal state on beacon Head retrieval

* eth/downloader: fix spelling mistakes

* eth/downloader: fix some typos

* eth: integrate legacy/beacon sync switchover and UX

* eth: handle UX wise being stuck on post-merge TTD

* core, eth: integrate the beacon client with the beacon sync

* eth/catalyst: make some warning messages nicer

* eth/downloader: remove Ethereum 1&2 notions in favor of merge

* core/beacon, eth: clean up engine API returns a bit

* eth/downloader: add skeleton extension tests

* eth/catalyst: keep non-kiln spec, handle mining on ttd

* eth/downloader: add beacon header retrieval tests

* eth: fixed spelling, commented failing tests out

* eth/downloader: review fixes

* eth/downloader: drop peers failing to deliver beacon headers

* core/rawdb: track beacon sync data in db inspect

* eth: fix review concerns

* internal/web3ext: nit

Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
2022-03-11 14:14:45 +02:00
Martin Holst Swende
4860e50e05
cmd/geth, node, rpc: implement jwt tokens (#24364)
* rpc, node: refactor request validation and add jwt validation

* node, rpc: fix error message, ignore engine api in RegisterAPIs

* node: make authenticated port configurable

* eth/catalyst: enable unauthenticated version of engine api

* node: rework obtainjwtsecret (backport later)

* cmd/geth: added auth port flag

* node: happy lint, happy life

* node: refactor authenticated api

Modifies the authentication mechanism to use default values

* node: trim spaces and newline away from secret

Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
2022-03-07 09:30:27 +02:00
Felföldi Zsolt
7033724522
les/vflux/client: fix goroutine leak in testIter (#24449) 2022-03-01 10:36:42 +01:00
Felix Lange
d6f49bf764
core/types: faster RLP encoding of Header, StateAcccount, ReceiptForStorage (#24420)
This change makes use of the new code generator rlp/rlpgen to improve the
performance of RLP encoding for Header and StateAccount. It also speeds up
encoding of ReceiptForStorage using the new rlp.EncoderBuffer API.

The change is much less transparent than I wanted it to be, because Header and
StateAccount now have an EncodeRLP method defined with pointer receiver. It
used to be possible to encode non-pointer values of these types, but the new
method prevents that and attempting to encode unadressable values (even if
part of another value) will return an error. The error can be surprising and may
pop up in places that previously didn't expect any errors.

To make things work, I also needed to update all code paths (mostly in unit tests)
that lead to encoding of non-pointer values, and pass a pointer instead.

Benchmark results:

    name                             old time/op    new time/op    delta
    EncodeRLP/legacy-header-8           328ns ± 0%     237ns ± 1%   -27.63%  (p=0.000 n=8+8)
    EncodeRLP/london-header-8           353ns ± 0%     247ns ± 1%   -30.06%  (p=0.000 n=8+8)
    EncodeRLP/receipt-for-storage-8     237ns ± 0%     123ns ± 0%   -47.86%  (p=0.000 n=8+7)
    EncodeRLP/receipt-full-8            297ns ± 0%     301ns ± 1%    +1.39%  (p=0.000 n=8+8)

    name                             old speed      new speed      delta
    EncodeRLP/legacy-header-8        1.66GB/s ± 0%  2.29GB/s ± 1%   +38.19%  (p=0.000 n=8+8)
    EncodeRLP/london-header-8        1.55GB/s ± 0%  2.22GB/s ± 1%   +42.99%  (p=0.000 n=8+8)
    EncodeRLP/receipt-for-storage-8  38.0MB/s ± 0%  64.8MB/s ± 0%   +70.48%  (p=0.000 n=8+7)
    EncodeRLP/receipt-full-8          910MB/s ± 0%   897MB/s ± 1%    -1.37%  (p=0.000 n=8+8)

    name                             old alloc/op   new alloc/op   delta
    EncodeRLP/legacy-header-8           0.00B          0.00B           ~     (all equal)
    EncodeRLP/london-header-8           0.00B          0.00B           ~     (all equal)
    EncodeRLP/receipt-for-storage-8     64.0B ± 0%      0.0B       -100.00%  (p=0.000 n=8+8)
    EncodeRLP/receipt-full-8             320B ± 0%      320B ± 0%      ~     (all equal)
2022-02-18 08:10:26 +01:00
rjl493456442
9da25c5db7
all: separate catalyst package (#24280)
* all: seperate catalyst package

* eth/catalyst: moved some methods, added docs

* eth/catalyst, les/catalyst: add method docs

* core, eth, les, miner: move common function to beacon package

* eth/catalyst: goimported

* cmd/utils, miner/stress/beacon: naming nitpicks

Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
2022-01-31 14:22:35 +02:00
Martin Holst Swende
335914a63a
les: fix serverHandler crash after setHead (#24200) 2022-01-05 16:40:45 +01:00
Sina Mahmoodi
ada9c774e9
eth, les: update unclean shutdown markers regularly (#24077)
Fixes #22580

Co-authored-by: Felix Lange <fjl@twurst.com>
2021-12-17 15:18:51 +01:00
Felföldi Zsolt
fc01a7ce8e
les/vflux/client, p2p/nodestate: fix data races (#24058)
Fixes #23848
2021-12-14 11:34:50 +01:00
Péter Szilágyi
58d1988349
core, eth, les, trie: remove the sync bloom, used by fast sync 2021-12-03 12:32:41 +02:00
Taeik Lim
85064ed09b
all: fix 'the the' in comments (#24036) 2021-12-02 15:42:09 +01:00
Péter Szilágyi
c10a0a62c3
eth: request id dispatcher and direct req/reply APIs (#23576)
* eth: request ID based message dispatcher

* eth: fix dispatcher cancellation, rework fetchers idleness tracker

* eth/downloader: drop peers who refuse to serve advertised chains
2021-11-26 13:26:03 +02:00
Marius van der Wijden
3038e480f5
all: core rework for the merge transition (#23761)
* all: work for eth1/2 transtition

* consensus/beacon, eth: change beacon difficulty to 0

* eth: updates

* all: add terminalBlockDifficulty config, fix rebasing issues

* eth: implemented merge interop spec

* internal/ethapi: update to v1.0.0.alpha.2

                                                                 This commit updates the code to the new spec, moving payloadId into
                                                                 it's own object. It also fixes an issue with finalizing an empty blockhash.
                                                                 It also properly sets the basefee

* all: sync polishes, other fixes + refactors

* core, eth: correct semantics for LeavePoW, EnterPoS

* core: fixed rebasing artifacts

* core: light: performance improvements

* core: use keyed field (f)

* core: eth: fix compilation issues + tests

* eth/catalyst: dbetter error codes

* all: move Merger to consensus/, remove reliance on it in bc

* all: renamed EnterPoS and LeavePoW to ReachTDD and FinalizePoS

* core: make mergelogs a function

* core: use InsertChain instead of InsertBlock

* les: drop merger from lightchain object

* consensus: add merger

* core: recoverAncestors in catalyst mode

* core: fix nitpick

* all: removed merger from beacon, use TTD, nitpicks

* consensus: eth: add docstring, removed unnecessary code duplication

* consensus/beacon: better comment

* all: easy to fix nitpicks by karalabe

* consensus/beacon: verify known headers to be sure

* core: comments

* core: eth: don't drop peers who advertise blocks, nitpicks

* core: never add beacon blocks to the future queue

* core: fixed nitpicks

* consensus/beacon: simplify IsTTDReached check

* consensus/beacon: correct IsTTDReached check

Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
2021-11-26 13:23:02 +02:00
Felföldi Zsolt
8be8ba450e
les/vflux: fixed panic and data races (#23865)
* les/vflux/server: fix BalanceOperation

* les/vflux/client: fixed data races
2021-11-08 10:29:59 +01:00
Martin Holst Swende
3bbeb94c1c
eth: make traceChain avoid OOM on long-running tracing (#23736)
This PR changes long-running chain tracing, so that it at some points releases the memory trie db, and switch over to a fresh disk-backed trie.
2021-11-04 18:54:00 +01:00
Martin Holst Swende
32150f8aa9
cmd/geth, cmd/evm, params: implement Arrow Glacier (EIP 4345) (#23810)
This PR adds support for ArrowGlacier, as defined by

    https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/arrow-glacier.md
    https://eips.ethereum.org/EIPS/eip-4345

> Starting with FORK_BLOCK_NUMBER the client will calculate the difficulty based on a fake block number suggesting to the client that the difficulty bomb is adjusting 10,700,000 blocks later than the actual block number.

This also adds support for evm t8n to return the calculated difficulty, so it can be used to construct test.
2021-10-28 22:18:14 +02:00
meowsbits
c72b16c340
core: use block difficulty for genesis (#23793)
* core: write test showing that TD is not stored properly at genesis

The ToBlock method applies a default value for an empty
difficulty value. This default is not carried over through the Commit
method because the TotalDifficulty database write writes the
original difficulty value (nil) instead of the defaulty value
present on the genesis Block.

Date: 2021-10-22 08:25:32-07:00
Signed-off-by: meows <b5c6@protonmail.com>

* core: write TD value from Block, not original genesis value

This an issue where a default TD value was not written to
the database, resulting in a 0 value TD at genesis.

A test for this issue was provided at 90e3ffd393

Date: 2021-10-22 08:28:00-07:00
Signed-off-by: meows <b5c6@protonmail.com>

* core: fix tests by adding GenesisDifficulty to expected result

See prior two commits.

Date: 2021-10-22 09:16:01-07:00
Signed-off-by: meows <b5c6@protonmail.com>

* les: fix test with genesis change

Co-authored-by: Martin Holst Swende <martin@swende.se>
2021-10-26 08:44:43 +02:00
noam-alchemy
633e7ef478
eth,rpc: allow for flag configured timeouts for eth_call (#23645)
* eth,rpc: allow for flag configured timeouts for eth_call

* lint: account for package-local import order

* cr: rename `rpc.calltimeout` to `rpc.evmtimeout`
2021-10-12 10:46:04 +02:00
rjl493456442
088bc34194
les/vflux/server: fix metrics (#22946)
* les/vflux/server: fix metrics

* les/vflux/server: fix metrics
2021-10-11 18:49:26 +02:00
Guillaume Ballet
443afc975c
core/state: move state account to core/types + abstracted "write account to trie" (#23567)
* core/state: abstracted "write account to trie" method

* fix appveyor build

* Apply suggestions from code review

Co-authored-by: Martin Holst Swende <martin@swende.se>

* review feedback

* core/state/accounts: move Account to core/types

* core/types: rename Account -> StateAccount

* core/state: restore EncodeRLP for stateObject

* core/types: add the missing file

* more review feedback

* more review feedback

Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Felix Lange <fjl@twurst.com>
2021-09-28 10:48:07 +02:00
Péter Szilágyi
9e17648d8c
les: duplicate downloader and fetcher to allow progressive refactoring 2021-09-10 11:11:59 +03:00
gary rong
83ad92c421
les: fix data races in tests (#23457) 2021-08-25 11:56:25 +02:00
Péter Szilágyi
0db0b27754
Revert "Revert "eth: drop eth/65, the last non-reqid protocol version" (#23426)" (#23456)
This reverts commit c368f728c1.
2021-08-24 21:52:58 +03:00
chuwt
79bb9300c1
trie, les: add missing calls to Ticker.Stop (#23415) 2021-08-24 10:51:28 +02:00
Marius van der Wijden
d02c605367
core: only check sendernoeoa in non fake mode (#23424) 2021-08-23 12:49:39 +03:00
Péter Szilágyi
c368f728c1
Revert "eth: drop eth/65, the last non-reqid protocol version" (#23426) 2021-08-20 15:14:21 +03:00
Péter Szilágyi
fd604becbb
Merge pull request #23120 from karalabe/drop-eth-65
eth: drop eth/65, the last non-reqid protocol version
2021-08-13 11:52:47 +03:00
lightclient
bbfa6488ac
Use hexutil.Uint for blockCount parameter in feeHistory method (#23239)
* internal/ethapi/api: use hexutil.uint for blockCount parameter instead of int for feeHistory

* return hex value for oldestBlock instead of number

* return uint64 from oracle.resolveBlockRange

* eth/gasprice: fixed test

Co-authored-by: Zsolt Felfoldi <zsfelfoldi@gmail.com>
2021-07-27 05:27:28 +02:00
gary rong
f05419f0fb
les: fix eth_sendTransaction API (#23215) 2021-07-16 01:52:40 +02:00
Mark
c49e065fea
internal: get pending and queued transaction by address (#22992)
* core, eth, internal, les, light: get pending and queued transaction by address

* core: tiny nitpick fixes

* light: tiny nitpick

Co-authored-by: mark <mark@amis.com>
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
2021-07-13 13:40:58 +03:00
Martin Holst Swende
6ed812db13
les: avoid shutdown hang (#23139) 2021-07-01 14:01:19 +02:00
Marius van der Wijden
686b2884ee
all: removed blockhash from statedb (#23126)
This PR removes the blockhash from the statedb
2021-06-30 15:17:01 +02:00
Péter Szilágyi
d3f018fde8
eth: drop eth/65, the last non-reqid protocol version 2021-06-29 12:31:30 +03:00
Felföldi Zsolt
35dbf7a8a3
eth/gasprice: implement feeHistory API (#23033)
* eth/gasprice: implement feeHistory API

* eth/gasprice: factored out resolveBlockRange

* eth/gasprice: add sanity check for missing block

* eth/gasprice: fetch actual gas used from receipts

* miner, eth/gasprice: add PendingBlockAndReceipts

* internal/ethapi: use hexutil.Big

* eth/gasprice: return error when requesting beyond head block

* eth/gasprice: fixed tests and return errors correctly

* eth/gasprice: rename receiver name

* eth/gasprice: return directly if blockCount == 0

Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
2021-06-28 16:16:32 +02:00
Péter Szilágyi
7a7abe3de8
accounts/abi/bind: fix bounded contracts and sim backend for 1559 (#23038)
* accounts/abi/bind: fix bounded contracts and sim backend for 1559

* accounts/abi/bind, ethclient: don't rely on chain config for gas prices

* all: enable London for all internal tests

* les: get receipt type info in les tests

* les: fix weird test

Co-authored-by: Martin Holst Swende <martin@swende.se>
2021-06-15 13:56:14 +03:00
Martin Holst Swende
5cff9754d7
core, eth, internal, les: RPC methods and fields for EIP 1559 (#22964)
* internal/ethapi: add baseFee to RPCMarshalHeader

* internal/ethapi: add FeeCap, Tip and correct GasPrice to EIP-1559 RPCTransaction results

* core,eth,les,internal: add support for tip estimation in gas price oracle

* internal/ethapi,eth/gasprice: don't suggest tip larger than fee cap

* core/types,internal: use correct eip1559 terminology for json marshalling

* eth, internal/ethapi: fix rebase problems

* internal/ethapi: fix rpc name of basefee

* internal/ethapi: address review concerns

* core, eth, internal, les: simplify gasprice oracle (#25)

* core, eth, internal, les: simplify gasprice oracle

* eth/gasprice: fix typo

* internal/ethapi: minor tweak in tx args

* internal/ethapi: calculate basefee for pending block

* internal/ethapi: fix panic

* internal/ethapi, eth/tracers: simplify txargs ToMessage

* internal/ethapi: remove unused param

* core, eth, internal: fix regressions wrt effective gas price in the evm

* eth/gasprice: drop weird debug println

* internal/jsre/deps: hack in 1559 gas conversions into embedded web3

* internal/jsre/deps: hack basFee to decimal conversion

* internal/ethapi: init feecap and tipcap for legacy txs too

* eth, graphql, internal, les: fix gas price suggestion on all combos

* internal/jsre/deps: handle decimal tipcap and feecap

* eth, internal: minor review fixes

* graphql, internal: export max fee cap RPC endpoint

* internal/ethapi: fix crash in transaction_args

* internal/ethapi: minor refactor to make the code safer

Co-authored-by: Ryan Schneider <ryanleeschneider@gmail.com>
Co-authored-by: lightclient@protonmail.com <lightclient@protonmail.com>
Co-authored-by: gary rong <garyrong0905@gmail.com>
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
2021-06-02 16:13:10 +03:00
Péter Szilágyi
835fe06f1d
les: generate random nums directly, not via strange conversions 2021-05-21 12:36:04 +03:00
Martin Holst Swende
94451c2788
all: implement EIP-1559 (#22837)
This is the initial implementation of EIP-1559 in packages core/types and core.
Mining, RPC, etc. will be added in subsequent commits.

Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
Co-authored-by: lightclient@protonmail.com <lightclient@protonmail.com>
Co-authored-by: Felix Lange <fjl@twurst.com>
2021-05-17 15:13:22 +02:00
Felix Lange
c0e201b690
eth/protocols/eth, les: avoid Raw() when decoding HashOrNumber (#22841)
Getting the raw value is not necessary to decode this type, and
decoding it directly from the stream is faster.
2021-05-10 12:38:54 +02:00
Martin Holst Swende
cc606be74c
all: define London+baikal, undefine yolov3, add london override flag (#22822)
* all: define London+baikal, undefine yolov3, add london override flag

* cmd, core, params: add baikal genesis definition
2021-05-06 12:07:42 +03:00
gary rong
6d7c9566df
les, tests: fix les clientpool (#22756)
* les, tests: fix les clientpool

* tests: disable debug mode

* les: polish code
2021-04-28 14:18:25 +02:00
gary rong
854f068ed6
les: polish code (#22625)
* les: polish code

* les/vflus/server: fixes

* les: fix lint
2021-04-27 09:44:59 +02:00
Martin Holst Swende
3e68d627b1
les: fix goroutine leaks in tests (#22707) 2021-04-21 10:19:28 +02:00
Felföldi Zsolt
65689e7fce
les/vflux/server: fix priority cornercase causing fuzzer timeout (#22650)
* les/vflux/server: fix estimatePriority corner case

* les/vflux/server: simplify inactiveAllowance == 0 case
2021-04-16 09:52:33 +02:00
Péter Szilágyi
fe1586b094
eth, les: drop support for eth/64, fix eth/66 tests 2021-04-09 10:39:45 +03:00
Marius van der Wijden
9d10856e84
core, eth, internal/ethapi: create access list RPC API (#22550)
* core/vm: implement AccessListTracer

* eth: implement debug.createAccessList

* core/vm: fixed nil panics in accessListTracer

* eth: better error messages for createAccessList

* eth: some fixes on CreateAccessList

* eth: allow for provided accesslists

* eth: pass accesslist by value

* eth: remove created acocunt from accesslist

* core/vm: simplify access list tracer

* core/vm: unexport accessListTracer

* eth: return best guess if al iteration times out

* eth: return best guess if al iteration times out

* core: docstring, unexport methods

* eth: typo

* internal/ethapi: move createAccessList to eth package

* internal/ethapi: remove reexec from createAccessList

* internal/ethapi: break if al is equal to last run, not if gas is equal

* internal/web3ext: fixed arguments

* core/types: fixed equality check for accesslist

* core/types: no hardcoded vals

* core, internal: simplify access list generation, make it precise

* core/vm: fix typo

Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
2021-04-07 17:54:31 +03:00
gary rong
a600dab7e5
eth, les: fix tracers (#22473)
* eth, les: fix tracer

* eth: isolate live trie database in tracer

* eth: fix nil

* eth: fix

* eth, les: add checkLive param

* eth/tracer: fix
2021-04-07 10:30:26 +03:00
Felföldi Zsolt
2d89fe0883
les: move client pool to les/vflux/server (#22495)
* les: move client pool to les/vflux/server

* les/vflux/server: un-expose NodeBalance, remove unused fn, fix bugs

* tests/fuzzers/vflux: add ClientPool fuzzer

* les/vflux/server: fixed balance tests

* les: rebase fix

* les/vflux/server: fixed more bugs

* les/vflux/server: unexported NodeStateMachine fields and flags

* les/vflux/server: unexport all internal components and functions

* les/vflux/server: fixed priorityPool test

* les/vflux/server: polish balance

* les/vflux/server: fixed mutex locking error

* les/vflux/server: priorityPool bug fixed

* common/prque: make Prque wrap-around priority handling optional

* les/vflux/server: rename funcs, small optimizations

* les/vflux/server: fixed timeUntil

* les/vflux/server: separated balance.posValue and negValue

* les/vflux/server: polish setup

* les/vflux/server: enforce capacity curve monotonicity

* les/vflux/server: simplified requestCapacity

* les/vflux/server: requestCapacity with target range, no iterations in SetCapacity

* les/vflux/server: minor changes

* les/vflux/server: moved default factors to balanceTracker

* les/vflux/server: set inactiveFlag in priorityPool

* les/vflux/server: moved related metrics to vfs package

* les/vflux/client: make priorityPool temp state logic cleaner

* les/vflux/server: changed log.Crit to log.Error

* add vflux fuzzer to oss-fuzz

Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
2021-04-06 20:42:50 +02:00
Felix Lange
5129cdc4f0
cmd/devp2p: skip ENR field tails properly in nodeset filter (#22565)
In Geth v1.10, we changed the structure of the "les" ENR entry. As a result, the DHT crawler that creates the DNS lists
no longer recognizes the les nodes, which is fixed in this commit. 

* cmd/devp2p: skip ENR field tails properly in nodeset filter
* cmd/devp2p: fix tail decoder for snap as well
* les: fix tail decoding in "eth" ENR entry
2021-03-24 12:32:39 +01:00
gary rong
0c70b83e00
all: add read-only option to database (#22407)
* all: add read-only option to database

* all: fixes tests

* cmd/geth: migrate flags

* cmd/geth: fix the compact

* cmd/geth: fix the format

* cmd/geth: fix log

* cmd: add chain-readonly

* core: add readonly notion to freezer

* core/rawdb: add log

* core/rawdb: fix freezer close

* cmd: fix

* cmd, core: construct db

* core: update tests
2021-03-22 19:06:30 +01:00
Felföldi Zsolt
91726e8aad
les: allow either full enode strings or raw hex ids in the API (#22423) 2021-03-16 12:55:43 +01:00
Felföldi Zsolt
62d8022b51
les: fix UDP connection query (#22451)
This PR fixes multiple issues with the UDP connection pre-negotiation feature:

- the enable condition was wrong (it checked the existence of the DiscV5 struct where it wasn't initialized yet, disabling the feature even if discv5 was enabled)
- the server pool queried already connected nodes when the discovery iterators returned them again
- servers responded positively before they were synced and really willing to accept connections

Metrics are also added on the server side that count the positive and negative replies to served connection queries.
2021-03-16 12:53:54 +01:00
Martin Holst Swende
bc47993692
tests/fuzzers: fix goroutine leak in les fuzzer (#22455)
The oss-fuzz fuzzer has been reporting some failing testcases for les. They're all spurious, and cannot reliably be reproduced. However, running them showed that there was a goroutine leak: the tests created a lot of new clients, which started an exec queue that was never torn down.

This PR fixes the goroutine leak, and also a log message which was erroneously formatted.
2021-03-16 09:43:33 +01:00
Martin Holst Swende
c6d45009f1
eth, les: properly init statedb accesslist during tracing (#22480)
* eth/state, les/state: properly init statedb accesslist when tracing, fixes #22475

* eth: review comments

* eth/tracers: fix compilation err

* eth/tracers: apply @karalabe's suggested fix
2021-03-14 17:13:25 +02:00
Martin Holst Swende
3d299b7468
les: fix errors in metric namespace (#22459)
* les: add trailing slash to metric namespace

* les: omit '.' in metric namespace
2021-03-09 09:04:03 +01:00
Felföldi Zsolt
b24804d88c
les: fix nodiscover option on the client side (#22422) 2021-03-03 15:05:24 +01:00
Felföldi Zsolt
d96870428f
les: UDP pre-negotiation of available server capacity (#22183)
This PR implements the first one of the "lespay" UDP queries which
is already useful in itself: the capacity query. The server pool is making
use of this query by doing a cheap UDP query to determine whether it is
worth starting the more expensive TCP connection process.
2021-03-01 10:24:20 +01:00
Péter Szilágyi
0928562670
all: define Berlin hard fork spec 2021-02-26 14:24:07 +02:00
Felföldi Zsolt
dc109cce26
les: move server pool to les/vflux/client (#22377)
* les: move serverPool to les/vflux/client

* les: add metrics

* les: moved ValueTracker inside ServerPool

* les: protect against node registration before server pool is started

* les/vflux/client: fixed tests

* les: make peer registration safe
2021-02-25 21:08:34 +01:00
lightclient
bbfb1e4008
all: add support for EIP-2718, EIP-2930 transactions (#21502)
This adds support for EIP-2718 typed transactions as well as EIP-2930
access list transactions (tx type 1). These EIPs are scheduled for the
Berlin fork.

There very few changes to existing APIs in core/types, and several new APIs
to deal with access list transactions. In particular, there are two new
constructor functions for transactions: types.NewTx and types.SignNewTx.
Since the canonical encoding of typed transactions is not RLP-compatible,
Transaction now has new methods for encoding and decoding: MarshalBinary
and UnmarshalBinary.

The existing EIP-155 signer does not support the new transaction types.
All code dealing with transaction signatures should be updated to use the
newer EIP-2930 signer. To make this easier for future updates, we have
added new constructor functions for types.Signer: types.LatestSigner and
types.LatestSignerForChainID. 

This change also adds support for the YoloV3 testnet.

Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Felix Lange <fjl@twurst.com>
Co-authored-by: Ryan Schneider <ryanleeschneider@gmail.com>
2021-02-25 15:26:57 +01:00
gary rong
7a3c890009
les, light: improve txstatus retrieval (#22349)
Transaction unindexing will be enabled by default as of 1.10, which causes tx status retrieval will be broken without this PR. 

This PR introduces a retry mechanism in TxStatus retrieval.
2021-02-25 14:24:04 +01:00
gary rong
378e961d85
cmd, eth, les: enable serving light clients when non-synced (#22250)
This PR adds a more CLI flag, so that the les-server can serve light clients even the local node is not synced yet.

This functionality is needed in some testing environments(e.g. hive). After launching the les server, no more blocks will be imported so the node is always marked as "non-synced".
2021-02-25 13:55:07 +01:00
Marius van der Wijden
142fbcfd6f
internal/ethapi: reject non-replay-protected txs over RPC (#22339)
This PR prevents users from submitting transactions without EIP-155 enabled. This behaviour can be overridden by specifying the flag --rpc.allow-unprotected-txs=true.
2021-02-23 13:09:19 +01:00
gary rong
3ecfdccd9a
les: clean up server handler (#22357) 2021-02-22 14:33:11 +01:00
Felföldi Zsolt
8f03e3b107
tests/fuzzers/les: add fuzzer for les server handler (#22282)
* les: refactored server handler

* tests/fuzzers/les: add fuzzer for les server handler

* tests, les: update les fuzzer

tests: update les fuzzer

tests/fuzzer/les: release resources

tests/fuzzer/les: pre-initialize all resources

* les: refactored server handler and fuzzer

Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
2021-02-20 10:40:38 +01:00
Felföldi Zsolt
8647233a8e
les: fix balance expiration (#22343)
* les/lespay/server: fix balance expiration and add test

* les: move client balances to a new db

* les: rename lespayDb to lesDb
2021-02-19 15:53:12 +01:00
Felföldi Zsolt
c027507e03
les: renamed lespay to vflux (#22347) 2021-02-19 14:44:16 +01:00
Felföldi Zsolt
7d1b711c7d
les: enable les/4 and add tests (#22321) 2021-02-12 20:48:18 +01:00
gary rong
098a2b6e26
eth: move eth.Config to a common package (#22205)
This moves the eth config definition into a separate package, eth/ethconfig. 
Packages eth and les can now import this common package instead of
importing eth from les, reducing dependencies.

Co-authored-by: Felix Lange <fjl@twurst.com>
2021-02-05 13:51:15 +01:00
ucwong
83e4c49e2b
trie : use trie.NewStackTrie instead of new(trie.Trie) (#22246)
The PR makes use of the stacktrie, which is is more lenient on resource consumption, than the regular trie, in cases where we only need it for DeriveSha
2021-02-02 13:09:23 +01:00
Felföldi Zsolt
7a800f98f6
les/utils: UDP rate limiter (#21930)
* les/utils: Limiter

* les/utils: dropped prior weight vs variable cost logic, using fixed weights

* les/utils: always create node selector in addressGroup

* les/utils: renamed request weight to request cost

* les/utils: simplified and improved the DoS penalty mechanism

* les/utils: minor fixes

* les/utils: made selection weight calculation nicer

* les/utils: fixed linter warning

* les/utils: more precise and reliable probabilistic test

* les/utils: fixed linter warning
2021-01-28 22:47:15 +01:00
Felföldi Zsolt
a72fa88a0d
les: switch to new discv5 (#21940)
This PR enables running the new discv5 protocol in both LES client
and server mode. In client mode it mixes discv5 and dnsdisc iterators
(if both are enabled) and filters incoming ENRs for "les" tag and fork ID.
The old p2p/discv5 package and all references to it are removed.

Co-authored-by: Felix Lange <fjl@twurst.com>
2021-01-26 21:41:35 +01:00
gary rong
adf130def8
eth/tracers: move tracing APIs into eth/tracers (#22161)
This moves the tracing RPC API implementation to package eth/tracers.
By doing so, package eth no longer depends on tracing and the duktape JS engine.

The change also enables tracing using the light client. All tracing methods work with the
light client, but it's a lot slower compared to using a full node.
2021-01-25 14:36:39 +01:00
Sina Mahmoodi
c4307a9339
eth/filters: fix potential deadlock in filter timeout loop (#22178)
This fixes #22131 and adds a test reproducing the issue.
2021-01-21 12:17:10 +01:00
gary rong
45cb1a580a
eth, les: add new config field SyncFromCheckpoint (#22123)
This PR introduces a new config field SyncFromCheckpoint for light client.

In some special scenarios, it's required to start synchronization from some
arbitrary checkpoint or even from the scratch. So this PR offers this
flexibility to users so that the synchronization start point can be configured.

There are two relevant configs: SyncFromCheckpoint and Checkpoint.

- If the SyncFromCheckpoint is true, the light client will try to sync from the
  specified checkpoint.

- If the Checkpoint is not configured, then the light client will sync from the
  scratch(from the latest header if the database is not empty)

Additional notes: these two configs are not visible in the CLI flags but only
accessable in the config file.

Example Usage:

[Eth]
SyncFromCheckpoint = true

[Eth.Checkpoint]
SectionIndex = 100
SectionHead = "0xabc"
CHTRoot = "0xabc"
BloomRoot = "0xabc"

PS. Historical checkpoint can be retrieved from the synced full node or light
client via les_getCheckpoint API.
2021-01-19 10:52:45 +01:00
gary rong
034ecc3210
les: remove useless protocol defines (#22115)
This PR has two changes in the les protocol:

- the auxRoot is not supported. See ethereum/devp2p#171 for more information
- the empty response will be returned in GetHelperTrieProofsMsg request if the merkle
   proving is failed. note, for backward compatibility, the empty merkle proof as well as
   the request auxiliary data will still be returned in  les2/3 protocol no matter the proving
   is successful or not. the proving failure can happen e.g. request the proving for a
   non-included entry in helper trie (unstable header).
2021-01-16 19:06:18 +01:00
gary rong
8d62ee65b2
les: don't drop sentTo for normal cases (#22048) 2021-01-15 23:04:38 +01:00
gary rong
165f53fc6e
les: remove transaction propagation limits (#22125) 2021-01-07 23:39:35 +01:00
Marius van der Wijden
158f72cc0c
internal/ethapi: restore net_version RPC method (#22061)
During the snap and eth refactor, the net_version rpc call was falsely deprecated.
This restores the net_version RPC handler as most eth2 nodes and other software
depend on it.
2020-12-23 13:43:22 +01:00
Felföldi Zsolt
c7f2536735
les: les/4 minimalistic version (#21909)
* les: allow tx unindexing in les/4 light server mode

* les: minor fixes

* les: more small fixes

* les: add meaningful constants for recentTxIndex handshake field
2020-12-15 20:12:14 +01:00
Péter Szilágyi
017831dd5b
core, eth: split eth package, implement snap protocol (#21482)
This commit splits the eth package, separating the handling of eth and snap protocols. It also includes the capability to run snap sync (https://github.com/ethereum/devp2p/blob/master/caps/snap.md) , but does not enable it by default. 

Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
Co-authored-by: Martin Holst Swende <martin@swende.se>
2020-12-14 10:27:15 +01:00
Martin Holst Swende
4d48980e74
core, eth, les: implement unclean-shutdown marker (#21893)
This PR implements unclean shutdown marker. Every time geth boots, it adds a timestamp to a list of timestamps in the database. This list is capped at 10. At a clean shutdown, the timestamp is removed again. 
Thus, when geth exits unclean, the marker remains, and at boot up we show the most recent unclean shutdowns to the user, which makes it easier to diagnose root-causes to certain problems. 

Co-authored-by: Nagy Salem <me@muhnagy.com>
2020-12-11 15:56:00 +01:00
Guillaume Ballet
1a715d7db5
les: rework float conversion on arm64 and other architectures (#21994)
The previous fix #21960 converted the float to an intermediate signed int, before attempting the uint conversion. Although this works, this doesn't guarantee that other architectures will work the same.
2020-12-11 10:28:01 +01:00
gary rong
004541098d
les: introduce forkID (#21974)
* les: introduce forkID

* les: address comment
2020-12-10 17:20:55 +01:00
gary rong
9f6bb492bb
les, light: remove untrusted header retrieval in ODR (#21907)
* les, light: remove untrusted header retrieval in ODR

* les: polish

* light: check the hash equality in odr
2020-12-10 14:33:52 +01:00
Marius van der Wijden
ed0670cb17
accounts/abi/bind: allow specifying signer on transactOpts (#21356)
This commit enables users to specify which signer they want to use while creating their transactOpts.
Previously all contract interactions used the homestead signer. Now a user can specify whether they
want to sign with homestead or EIP155 and specify the chainID which adds another layer of security.

Closes #16484
2020-12-08 14:44:56 +01:00
Guillaume Ballet
581c028d18
les: cosmetic rewrite of the arm64 float bug workaround (#21960)
* les: revert arm float bug workaround to check go 1.15

* add traces to reproduce outside travis

* simpler workaround
2020-12-07 14:04:27 +01:00
Felföldi Zsolt
a2795c8055
les: fix nodiscover option (#21906) 2020-12-01 10:03:41 +01:00
Alex Prut
c92faee66e
all: simplify nested complexity and if blocks ending with a return statement (#21854)
Changes:

    Simplify nested complexity
    If an if blocks ends with a return statement then remove the else nesting.

Most of the changes has also been reported in golint https://goreportcard.com/report/github.com/ethereum/go-ethereum#golint
2020-11-25 09:24:50 +01:00
Felföldi Zsolt
bddd103a9f
les: fix GetProofsV2 bug (#21896) 2020-11-24 10:55:17 +01:00
Felföldi Zsolt
bddf5aaa2f
les/utils: protect against WeightedRandomSelect overflow (#21839)
Also fixes a bug in les/flowcontrol that caused the overflow.
2020-11-23 10:18:33 +01:00
Marius van der Wijden
2045a2bba3
core, all: split vm.Context into BlockContext and TxContext (#21672)
* all: core: split vm.Config into BlockConfig and TxConfig

* core: core/vm: reset EVM between tx in block instead of creating new

* core/vm: added docs
2020-11-13 13:42:19 +01:00
gary rong
b63bffe820
les, p2p/simulations/adapters: fix issues found while simulating les (#21761)
This adds a few tiny fixes for les and the p2p simulation framework:

LES Parts

- Keep the LES-SERVER connection even it's non-synced

  We had this idea to reject the connections in LES protocol if the les-server itself is
  not synced. However, in LES protocol we will also receive the connection from another
  les-server. In this case even the local node is not synced yet, we should keep the tcp
  connection for other protocols(e.g. eth protocol).

- Don't count "invalid message" for non-existing GetBlockHeadersMsg request

  In the eth syncing mechanism (full sync, fast sync, light sync), it will try to fetch
  some non-existent blocks or headers(to ensure we indeed download all the missing chain).
  In this case, it's possible that the les-server will receive the request for
  non-existent headers. So don't count it as the "invalid message" for scheduling
  dropping.

- Copy the announce object in the closure

  Before the les-server pushes the latest headers to all connected clients, it will create
  a closure and queue it in the underlying request scheduler. In some scenarios it's
  problematic. E.g, in private networks, the block can be mined very fast. So before the
  first closure is executed, we may already update the latest_announce object. So actually
  the "announce" object we want to send is replaced.

  The downsize is the client will receive two announces with the same td and then drop the
  server.

P2P Simulation Framework

- Don't double register the protocol services in p2p-simulation "Start".

  The protocols upon the devp2p are registered in the "New node stage". So don't reigster
  them again when starting a node in the p2p simulation framework

- Add one more new config field "ExternalSigner", in order to use clef service in the
  framework.
2020-10-30 18:04:38 +01:00
Felföldi Zsolt
85d81b2cdd
les: remove clientPeerSet and serverSet (#21566)
* les: move NodeStateMachine from clientPool to LesServer

* les: new header broadcaster

* les: peerCommons.headInfo always contains last announced head

* les: remove clientPeerSet and serverSet

* les: fixed panic

* les: fixed --nodiscover option

* les: disconnect all peers at ns.Stop()

* les: added comments and fixed signed broadcasts

* les: removed unused parameter, fixed tests
2020-10-21 10:56:33 +02:00
mr_franklin
c37e68e7c1
all: replace RWMutex with Mutex in places where RLock is not used (#21622) 2020-10-13 10:58:41 +02:00
gary rong
93f047023f
les/lespay/server: bump database version (#21571) 2020-09-16 11:51:16 +02:00
Felföldi Zsolt
4996fce25a
les, les/lespay/server: refactor client pool (#21236)
* les, les/lespay/server: refactor client pool

* les: use ns.Operation and sub calls where needed

* les: fixed tests

* les: removed active/inactive logic from peerSet

* les: removed active/inactive peer logic

* les: fixed linter warnings

* les: fixed more linter errors and added missing metrics

* les: addressed comments

* cmd/geth: fixed TestPriorityClient

* les: simplified clientPool state machine

* les/lespay/server: do not use goroutine for balance callbacks

* internal/web3ext: fix addBalance required parameters

* les: removed freeCapacity, always connect at minCapacity initially

* les: only allow capacity change with priority status

Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
2020-09-14 22:44:20 +02:00