* eth/fetcher: introduce some lag in tx fetching
* eth/fetcher: change conditions a bit
* eth/fetcher: use per-batch quota check
* eth/fetcher: fix some comments
* eth/fetcher: address review concerns
* eth/fetcher: fix panic + add warn log
* eth/fetcher: fix log
* eth/fetcher: fix log
* cmd/devp2p/internal/ethtest: fix ignorign tx announcements from prev. tests
* cmd/devp2p/internal/ethtest: fix TestLargeTxRequest
This increases the number of tx relay messages the test waits for. Since
go-ethereum now processes incoming txs in smaller batches, the
announcement messages it sends are also smaller.
Co-authored-by: Felix Lange <fjl@twurst.com>
This change updates our urfave/cli dependency to the v2 branch of the library.
There are some Go API changes in cli v2:
- Flag values can now be accessed using the methods ctx.Bool,
ctx.Int, ctx.String, ... regardless of whether the flag is 'local' or
'global'.
- v2 has built-in support for flag categories. Our home-grown category
system is removed and the categories of flags are assigned as part of
the flag definition.
For users, there is only one observable difference with cli v2: flags must now
strictly appear before regular arguments. For example, the following command is
now invalid:
geth account import mykey.json --password file.txt
Instead, the command must be invoked as follows:
geth account import --password file.txt mykey.json
This enables the following linters
- typecheck
- unused
- staticcheck
- bidichk
- durationcheck
- exportloopref
- gosec
WIth a few exceptions.
- We use a deprecated protobuf in trezor. I didn't want to mess with that, since I cannot meaningfully test any changes there.
- The deprecated TypeMux is used in a few places still, so the warning for it is silenced for now.
- Using string type in context.WithValue is apparently wrong, one should use a custom type, to prevent collisions between different places in the hierarchy of callers. That should be fixed at some point, but may require some attention.
- The warnings for using weak random generator are squashed, since we use a lot of random without need for cryptographic guarantees.
This PR groups all built-in network flags together and list them in the command as a whole.
And all database path flags(datadir, ancient) are also grouped, since usually these two are
used together.
This PR fixes two problems in devp2p tests (and through them, hive).
- Make the output more detailed about what is returned (always print packet kind).
- Allow Ping response to unsolicited findnode.
Without this PR, nethermind fails a hive protocol test, and I misinterpreted the result (NethermindEth/nethermind#3617). Ergo, the output was not fool-proof.
This was apparently recently changed by Cloudflare, and
began returning an error: 'TTL must be between 60 and 86400
seconds, or 1 for Automatic'
Date: 2021-11-10 15:25:20-08:00
Signed-off-by: meows <b5c6@protonmail.com>
Debugging recent geth failures in hive, it took a while to realize that it's because
geth doesn't support eth/65 any longer. This PR makes such failures a bit more
easy to figure out.
* cmd/devp2p/internal/ethtest: only use eth66 if eth66 is negotiated
* cmd/devp2p/internal/ethtest: switch on concrete type not pointer
* cmd/devp2p/internal/ethtest: switch on concrete type not pointer
This PR modifies the post-PING-send expectations to both be laxer and stricter: it doesn't care what order the packets arrive, but also verifies that exactly one PING and one PONG is returned.
This PR fixes a false positive PONG 'to' endpoint mismatch seen in hive tests:
got {IP:172.17.0.7 UDP:44025 TCP:44025}, want {IP:172.17.0.7 UDP:44025 TCP:0}
Co-authored-by: Felix Lange <fjl@twurst.com>
This PR refactors the eth test suite to make it more readable and
easier to use. Some notable differences:
- A new file helpers.go stores all of the methods used between
both eth66 and eth65 and below tests, as well as methods shared
among many test functions.
- suite.go now contains all of the test functions for both eth65
tests and eth66 tests.
- The utesting.T object doesn't get passed through to other helper methods,
but is instead only used within the scope of the test function,
whereas helper methods return errors, so only the test function
itself can fatal out in the case of an error.
- The full test suite now only takes 13.5 seconds to run.
This changes the SimultaneousRequests test to send the requests from the same
connection, as it doesn't really make sense to test whether a node can respond
to two requests with different request IDs from separate connections.
The new -limit option makes the filter operate on top N nodes by score.
This also adds ENR attribute stats in the nodeset info command.
Node set commands are now documented in README.
This upgrades the cloudflare client dependency to v0.14.0. The new
version changes the API because all methods now require a context
parameter. This change also reduces the log level of the 'Skipping...'
message to debug, following a similar change in the AWS deployer.
* cmd/devp2p: fix comparison of TXT record value
The AWS API returns quoted DNS strings, so we must encode the new value
before comparing it against the existing record content.
* cmd/devp2p: add test
* cmd/devp2p: fix typo and rename val -> newValue
In Geth v1.10, we changed the structure of the "les" ENR entry. As a result, the DHT crawler that creates the DNS lists
no longer recognizes the les nodes, which is fixed in this commit.
* cmd/devp2p: skip ENR field tails properly in nodeset filter
* cmd/devp2p: fix tail decoder for snap as well
* les: fix tail decoding in "eth" ENR entry
This PR fixes a regression introduced in #22360, when we updated to the v2 of the AWS sdk, which causes current crawler to just get the same first 100 results over and over, and get stuck in a loop.
This updates the DNS deployer to use AWS SDK v2. Migration is relatively
seamless, although there were two locations that required a slightly
different approach to achieve the same results. In particular, waiting for
DNS change propagation is very different with SDK v2.
This change also optimizes DNS updates by publishing all changes before
waiting for propagation.
This PR fixes an issue with the eth66 test suite where, during a readAndServe when
the test is manually responding to GetBlockHeader requests, it now responds
with a BlockHeaders eth66 packet that includes the inbound request ID.
This PR replaces the old test genesis.json and chain.rlp files in the testdata
directory for the eth protocol test suite, and also adds documentation for
running the eth test suite locally.
It also improves the test output text and adds more timeouts.
Co-authored-by: Felix Lange <fjl@twurst.com>
TAP is a text format for test results. Parsers for it are available in many languages,
making it easy to consume. I want TAP output from our protocol tests because the
Hive wrapper around them needs to know about the test names and their individual
results and logs. It would also be possible to just write this info as JSON, but I don't
want to invent a new format.
This also improves the normal console output for tests (when running without --tap).
It now prints -- RUN lines before any output from the test, and indents the log output
by one space.
The old one was wrong in two ways: the first block in chain.rlp was the
genesis block, and the genesis difficulty was below minimum difficulty.
This also contains some other fixes to the test.
This change moves the RLPx protocol implementation into a separate package,
p2p/rlpx. The new package can be used to establish RLPx connections for
protocol testing purposes.
Co-authored-by: Felix Lange <fjl@twurst.com>
This adds a test suite for discovery v4. The test suite is a port of the Hive suite for
discovery, and will replace the current suite on Hive soon-ish. The tests can be
run locally with this command:
devp2p discv4 test -remote enode//...
Co-authored-by: Felix Lange <fjl@twurst.com>
* cmd/clef, signer/core: use better terminal input for passwords, make it possible to avoid boot-up warning
* all: move commonly used prompter to isolated (small) package
* cmd/clef: Add new --acceptWarn to clef README
* cmd/clef: rename flag 'acceptWarn' to 'suppress-bootwarn'
Co-authored-by: ligi <ligi@ligi.de>
* cmd/utils: make goerli the default testnet
* cmd/geth: explicitly rename testnet to ropsten
* core: explicitly rename testnet to ropsten
* params: explicitly rename testnet to ropsten
* cmd: explicitly rename testnet to ropsten
* miner: explicitly rename testnet to ropsten
* mobile: allow for returning the goerli spec
* tests: explicitly rename testnet to ropsten
* docs: update readme to reflect changes to the default testnet
* mobile: allow for configuring goerli and rinkeby nodes
* cmd/geth: revert --testnet back to ropsten and mark as legacy
* cmd/util: mark --testnet flag as deprecated
* docs: update readme to properly reflect the 3 testnets
* cmd/utils: add an explicit deprecation warning on startup
* cmd/utils: swap goerli and ropsten in usage
* cmd/geth: swap goerli and ropsten in usage
* cmd/geth: if running a known preset, log it for convenience
* docs: improve readme on usage of ropsten's testnet datadir
* cmd/utils: check if legacy `testnet` datadir exists for ropsten
* cmd/geth: check for legacy testnet path in console command
* cmd/geth: use switch statement for complex conditions in main
* cmd/geth: move known preset log statement to the very top
* cmd/utils: create new ropsten configurations in the ropsten datadir
* cmd/utils: makedatadir should check for existing testnet dir
* cmd/geth: add legacy testnet flag to the copy db command
* cmd/geth: add legacy testnet flag to the inspect command
This adds an implementation of the current discovery v5 spec.
There is full integration with cmd/devp2p and enode.Iterator in this
version. In theory we could enable the new protocol as a replacement of
discovery v4 at any time. In practice, there will likely be a few more
changes to the spec and implementation before this can happen.
Turns out the way RDATA limits work is documented after all,
I just didn't search right. The trick to make it work is to
count UPSERTs twice.
This also adds an additional check to ensure TTL changes are
applied on existing records.
For longer records and subtree entries, the deployer created two
separate TXT records. This doesn't work as intended because the client
will receive the two records in arbitrary order. The fix is to encode
longer values as "string1""string2" instead of "string1", "string2".
This encoding creates a single record on AWS Route53.
* log: delete RotatingFileHandler
We added this for the dashboard, which is gone now. The
handler never really worked well and had data race and file
handling issues.
* internal/debug: remove unused RotatingFileHandler setup code
This change works around the 32k RDATA character limit per change
request and fixes several issues in the deployer which prevented it from
working for our production trees.
* p2p/dnsdisc: add support for enode.Iterator
This changes the dnsdisc.Client API to support the enode.Iterator
interface.
* p2p/dnsdisc: rate-limit DNS requests
* p2p/dnsdisc: preserve linked trees across root updates
This improves the way links are handled when the link root changes.
Previously, sync would simply remove all links from the current tree and
garbage-collect all unreachable trees before syncing the new list of
links.
This behavior isn't great in certain cases: Consider a structure where
trees A, B, and C reference each other and D links to A. If D's link
root changed, the sync code would first remove trees A, B and C, only to
re-sync them later when the link to A was found again.
The fix for this problem is to track the current set of links in each
clientTree and removing old links only AFTER all links are synced.
* p2p/dnsdisc: deflake iterator test
* cmd/devp2p: adapt dnsClient to new p2p/dnsdisc API
* p2p/dnsdisc: tiny comment fix
This adds an implementation of node discovery via DNS TXT records to the
go-ethereum library. The implementation doesn't match EIP-1459 exactly,
the main difference being that this implementation uses separate merkle
trees for tree links and ENRs. The EIP will be updated to match p2p/dnsdisc.
To maintain DNS trees, cmd/devp2p provides a frontend for the p2p/dnsdisc
library. The new 'dns' subcommands can be used to create, sign and deploy DNS
discovery trees.
* p2p/discover: export Ping and RequestENR
These two are useful for checking the status of a node.
* cmd/devp2p: add devp2p debug tool
This is a new tool for debugging p2p issues. It supports a few
basic tasks for now, but many more things can and will be added
in the near future.
devp2p enrdump -- prints ENRs readably
devp2p discv4 ping -- checks if a node is up
devp2p discv4 requestenr -- gets a node's record
devp2p discv4 resolve -- finds a node through the DHT
