Commit Graph

601 Commits

Author SHA1 Message Date
Péter Szilágyi
6ef3a16869
p2p/enode: use unix timestamp as base ENR sequence number (#19903)
This PR ensures that wiping all data associated with a node (apart from its nodekey)
will not generate already used sequence number for the ENRs, since all remote nodes
would reject them until they out-number the previously published largest one.

The big complication with this scheme is that every local update to the ENR can
potentially bump the sequence number by one. In order to ensure that local updates
do not outrun the clock, the sequence number is a millisecond-precision timestamp,
and updates are throttled to occur at most once per millisecond.

Co-authored-by: Felix Lange <fjl@twurst.com>
2021-09-07 12:36:48 +02:00
Felix Lange
8a134014b4
all: add go:build lines (#23468)
Generated by go1.17 fmt ./...
2021-08-25 18:46:29 +02:00
Marius van der Wijden
8dbf261fd9
p2p, p2p/enode: fix data races (#23434)
In p2p/dial.go, conn.flags was accessed without using sync/atomic.
This race is fixed by removing the access.

In p2p/enode/iter_test.go, a similar race is resolved by writing the field atomically.

Co-authored-by: Felix Lange <fjl@twurst.com>
2021-08-24 12:22:56 +02:00
baptiste-b-pegasys
860184d542
p2p: remove term "whitelist" (#23295)
Co-authored-by: Felix Lange <fjl@twurst.com>
2021-07-29 17:50:18 +02:00
Evolution404
8f11d279d2
p2p/simulations: fix unlikely crash in probabilistic connect (#23200)
When the nodeCount is less than 10, it will panic with the out of bound error.
How about we just skip this round, when rand1 and rand2 are equal?
2021-07-29 16:03:50 +02:00
Evolution404
5afc82de6e
p2p: fix array out of bounds issue (#23165) 2021-07-06 09:33:51 +02:00
Ahyun
4fcc93d922
p2p/server: fix method name in comment (#23123) 2021-06-29 12:14:47 +03:00
ForLina
dde6f1e92d
p2p/enode: fix method doc (#23115)
This is an obvious spelling error

Co-authored-by: liuyaxiong <liuyaxiong@inspur.com>
2021-06-28 10:48:17 +03:00
Felix Lange
427175153c
p2p/msgrate: return capacity as integer, clamp to max uint32 (#22943)
* p2p/msgrate: return capacity as integer

* eth/protocols/snap: remove conversions

* p2p/msgrate: add overflow test

* p2p/msgrate: make the capacity overflow test actually overflow

* p2p/msgrate: clamp capacity to max int32

* p2p/msgrate: fix min/max confusion
2021-05-27 19:43:55 +03:00
Felix Lange
7194c847b6
p2p/rlpx: reduce allocation and syscalls (#22899)
This change significantly improves the performance of RLPx message reads
and writes. In the previous implementation, reading and writing of
message frames performed multiple reads and writes on the underlying
network connection, and allocated a new []byte buffer for every read.

In the new implementation, reads and writes re-use buffers, and perform
much fewer system calls on the underlying connection. This doubles the
theoretically achievable throughput on a single connection, as shown by
the benchmark result:

    name             old speed      new speed       delta
    Throughput-8     70.3MB/s ± 0%  155.4MB/s ± 0%  +121.11%  (p=0.000 n=9+8)

The change also removes support for the legacy, pre-EIP-8 handshake encoding.
As of May 2021, no actively maintained client sends this format.
2021-05-27 10:19:13 +02:00
meowsbits
750115ff39
p2p/nat: skip TestUPNP in non-CI environments if discover fails (#22877)
Fixes #21476
2021-05-25 22:37:30 +02:00
Felix Lange
836c647bdd
eth: unregister peer only when handler exits (#22908)
This removes the error log message that says 

    Ethereum peer removal failed ... err="peer not registered"

The error happened because removePeer was called multiple
times: once to disconnect the peer, and another time when the
handler exited. With this change, removePeer now has the sole
purpose of disconnecting the peer. Unregistering happens exactly
once, when the handler exits.
2021-05-25 22:20:36 +02:00
Felix Lange
16bc57438b
p2p/dnsdisc: fix crash when iterator closed before first call to Next (#22906) 2021-05-20 09:24:41 +02:00
Péter Szilágyi
3e795881ea
eth, p2p/msgrate: move peer QoS tracking to its own package and use it for snap (#22876)
This change extracts the peer QoS tracking logic from eth/downloader, moving
it into the new package p2p/msgrate. The job of msgrate.Tracker is determining
suitable timeout values and request sizes per peer.

The snap sync scheduler now uses msgrate.Tracker instead of the hard-coded 15s
timeout. This should make the sync work better on network links with high latency.
2021-05-19 14:09:03 +02:00
Felix Lange
3e6f46caec
p2p/discover/v4wire: use optional RLP field for EIP-868 seq (#22842)
This changes the definitions of Ping and Pong, adding an optional field
for the sequence number. This field was previously encoded/decoded using
the "tail" struct tag, but using "optional" is much nicer.
2021-05-18 11:48:41 +02:00
Felix Lange
b8040a430e
cmd/utils: use eth DNS tree for snap discovery (#22808)
This removes auto-configuration of the snap.*.ethdisco.net DNS discovery tree.
Since measurements have shown that > 75% of nodes in all.*.ethdisco.net support
snap, we have decided to retire the dedicated index for snap and just use the eth
tree instead.

The dial iterators of eth and snap now use the same DNS tree in the default configuration,
so both iterators should use the same DNS discovery client instance. This ensures that
the record cache and rate limit are shared. Records will not be requested multiple times.

While testing the change, I noticed that duplicate DNS requests do happen even
when the client instance is shared. This is because the two iterators request the tree
root, link tree root, and first levels of the tree in lockstep. To avoid this problem, the
change also adds a singleflight.Group instance in the client. When one iterator
attempts to resolve an entry which is already being resolved, the singleflight object
waits for the existing resolve call to finish and returns the entry to both places.
2021-05-04 11:29:32 +02:00
Péter Szilágyi
ff3535e8e0
p2p/tracker: only reschedule wake if previous didn't run 2021-04-27 21:47:59 +03:00
Martin Holst Swende
45fca44c24
p2p/tracker: properly clean up fulfilled requests 2021-04-27 18:09:34 +02:00
Nishant Das
34f3c9539b
p2p/discover: improve discv5 handling of IPv4-in-IPv6 addresses (#22703)
When receiving PING from an IPv4 address over IPv6, the implementation sent
back a IPv4-in-IPv6 address. This change makes it reflect the IPv4 address.
2021-04-23 18:18:10 +02:00
Péter Szilágyi
1fb9a6dd32
eth/protocols, prp/tracker: add support for req/rep rtt tracking (#22608)
* eth/protocols, prp/tracker: add support for req/rep rtt tracking

* p2p/tracker: sanity cap the number of pending requests

* pap/tracker: linter <3

* p2p/tracker: disable entire tracker if no metrics are enabled
2021-04-22 11:42:46 +03:00
Felföldi Zsolt
2d89fe0883
les: move client pool to les/vflux/server (#22495)
* les: move client pool to les/vflux/server

* les/vflux/server: un-expose NodeBalance, remove unused fn, fix bugs

* tests/fuzzers/vflux: add ClientPool fuzzer

* les/vflux/server: fixed balance tests

* les: rebase fix

* les/vflux/server: fixed more bugs

* les/vflux/server: unexported NodeStateMachine fields and flags

* les/vflux/server: unexport all internal components and functions

* les/vflux/server: fixed priorityPool test

* les/vflux/server: polish balance

* les/vflux/server: fixed mutex locking error

* les/vflux/server: priorityPool bug fixed

* common/prque: make Prque wrap-around priority handling optional

* les/vflux/server: rename funcs, small optimizations

* les/vflux/server: fixed timeUntil

* les/vflux/server: separated balance.posValue and negValue

* les/vflux/server: polish setup

* les/vflux/server: enforce capacity curve monotonicity

* les/vflux/server: simplified requestCapacity

* les/vflux/server: requestCapacity with target range, no iterations in SetCapacity

* les/vflux/server: minor changes

* les/vflux/server: moved default factors to balanceTracker

* les/vflux/server: set inactiveFlag in priorityPool

* les/vflux/server: moved related metrics to vfs package

* les/vflux/client: make priorityPool temp state logic cleaner

* les/vflux/server: changed log.Crit to log.Error

* add vflux fuzzer to oss-fuzz

Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
2021-04-06 20:42:50 +02:00
Péter Szilágyi
6d7ff6acea
eth/protocols, metrics, p2p: add handler performance metrics 2021-03-26 14:00:06 +02:00
Felix Lange
ab8fd4d005
p2p/dnsdisc: rate limit resolving before checking cache (#22566)
This makes the rate limit apply regardless of whether the node is
already cached.
2021-03-24 14:37:20 +02:00
Chen Quan
15e6c27f8b
p2p: fix minor typo and remove fd parameter in checkInboundConn (#22547) 2021-03-24 13:18:29 +01:00
Martin Holst Swende
5bc0343ed3
p2p/dnsdisc: fix flaw in dns size calculation (#22533)
This fixes the calculation of the tree branch factor. With the new
formula, we now creat at most 13 children instead of 30, ensuring
the TXT record size will be below 370 bytes.
2021-03-19 13:20:27 +01:00
Felix Lange
aae7660410
p2p/enr: fix decoding of incomplete lists (#22484)
Given a list of less than two elements DecodeRLP returned rlp.EOL,
leading to issues in outer decoders.
2021-03-11 15:09:25 +01:00
Felföldi Zsolt
d96870428f
les: UDP pre-negotiation of available server capacity (#22183)
This PR implements the first one of the "lespay" UDP queries which
is already useful in itself: the capacity query. The server pool is making
use of this query by doing a cheap UDP query to determine whether it is
worth starting the more expensive TCP connection process.
2021-03-01 10:24:20 +01:00
Felix Lange
d36276d85e
p2p/dnsdisc: fix hot-spin when all trees are empty (#22313)
In the random sync algorithm used by the DNS node iterator, we first pick a random
tree and then perform one sync action on that tree. This happens in a loop until any
node is found. If no trees contain any nodes, the iterator will enter a hot loop spinning
at 100% CPU.

The fix is complicated. The iterator now checks if a meaningful sync action can
be performed on any tree. If there is nothing to do, it waits for the next root record
recheck time to arrive and then tries again.

Fixes #22306
2021-02-19 09:54:46 +01:00
Martin Holst Swende
6ec1561044
eth: implement eth66 (#22241)
* eth/protocols/eth: split up the eth protocol handlers

* eth/protocols/eth: define eth-66 protocol messages

* eth/protocols/eth: poc implement getblockheaders on eth/66

* eth/protocols/eth: implement remaining eth-66 handlers

* eth/protocols: define handler map for eth 66

* eth/downloader: use protocol constants from eth package

* eth/protocols/eth: add ETH66 capability

* eth/downloader: tests for eth66

* eth/downloader: fix error in tests

* eth/protocols/eth: use eth66 for outgoing requests

* eth/protocols/eth: remove unused error type

* eth/protocols/eth: define protocol length

* eth/protocols/eth: fix pooled tx over eth66

* protocols/eth/handlers: revert behavioural change which caused tests to fail

* eth/downloader: fix failing test

* eth/protocols/eth: add testcases + fix flaw with header requests

* eth/protocols: change comments

* eth/protocols/eth: review fixes + fixed flaw in RequestOneHeader

* eth/protocols: documentation

* eth/protocols/eth: review concerns about types
2021-02-18 18:54:29 +02:00
Martin Holst Swende
7ed860d4f1
eth: don't wait for snap registration if we're not running snap (#22272)
Prevents a situation where we (not running snap) connects with a peer running snap, and get stalled waiting for snap registration to succeed (which will never happen), which cause a waitgroup wait to halt shutdown
2021-02-05 14:15:22 +01:00
Péter Szilágyi
e3430ac7df
eth: check snap satelliteness, delegate drop to eth (#22235)
* eth: check snap satelliteness, delegate drop to eth

* eth: better handle eth/snap satellite relation, merge reg/unreg paths
2021-02-02 10:44:36 +02:00
Felföldi Zsolt
a72fa88a0d
les: switch to new discv5 (#21940)
This PR enables running the new discv5 protocol in both LES client
and server mode. In client mode it mixes discv5 and dnsdisc iterators
(if both are enabled) and filters incoming ENRs for "les" tag and fork ID.
The old p2p/discv5 package and all references to it are removed.

Co-authored-by: Felix Lange <fjl@twurst.com>
2021-01-26 21:41:35 +01:00
Martin Holst Swende
d2779ed7ac
eth, p2p: reserve half peer slots for snap peers during snap sync (#22171)
* eth, p2p: reserve half peer slots for snap peers during snap sync

* eth: less logging

* eth: rework the eth/snap peer reservation logic

* eth: rework the eth/snap peer reservation logic (again)
2021-01-25 20:06:52 +02:00
Guillaume Ballet
c7a6be163f
cmd/utils: don't enumerate USB unless --usb is set (#22130)
USB enumeration still occured. Make sure it will only occur if --usb is set.
This also deprecates the 'NoUSB' config file option in favor of a new option 'USB'.
2021-01-13 11:14:36 +01:00
Felix Lange
817a3fb562
p2p/enode: avoid crashing for invalid IP (#21981)
The database panicked for invalid IPs. This is usually no problem
because all code paths leading to node DB access verify the IP, but it's
dangerous because improper validation can turn this panic into a DoS
vulnerability. The quick fix here is to just turn database accesses
using invalid IP into a noop. This isn't great, but I'm planning to
remove the node DB for discv5 long-term, so it should be fine to have
this quick fix for half a year.

Fixes #21849
2020-12-09 20:21:31 +01:00
Martin Holst Swende
e7db1dbc96
p2p/nodestate: fix deadlock during shutdown of les server (#21927)
This PR fixes a deadlock reported here: #21925

The cause is that many operations may be pending, but if the close happens, only one of them gets awoken and exits, the others remain waiting for a signal that never comes.
2020-11-30 18:58:47 +01:00
Nishant Das
429e7141f2
p2p/discover: fix deadlock in discv5 message dispatch (#21858)
This fixes a deadlock that could occur when a response packet arrived
after a call had already received enough responses and was about to
signal completion to the dispatch loop.

Co-authored-by: Felix Lange <fjl@twurst.com>
2020-11-25 22:16:36 +01:00
Alex Prut
810f9e057d
all: remove redundant conversions and import names (#21903) 2020-11-25 21:00:23 +01:00
Alex Prut
c92faee66e
all: simplify nested complexity and if blocks ending with a return statement (#21854)
Changes:

    Simplify nested complexity
    If an if blocks ends with a return statement then remove the else nesting.

Most of the changes has also been reported in golint https://goreportcard.com/report/github.com/ethereum/go-ethereum#golint
2020-11-25 09:24:50 +01:00
Martin Holst Swende
3ef52775c4
p2p: avoid spinning loop on out-of-handles (#21878)
* p2p: avoid busy-loop on temporary errors

* p2p: address review concerns
2020-11-20 15:14:25 +01:00
gary rong
c52dfd55fb
p2p/simulations/adapters/exec: fix some issues (#21801)
- Remove the ws:// prefix from the status endpoint since
  the ws:// is already included in the stack.WSEndpoint().
- Don't register the services again in the node start.
  Registration is already done in the initialization stage.
- Expose admin namespace via websocket.
  This namespace is necessary for connecting the peers via websocket.
- Offer logging relevant options for exec adapter.
  It's really painful to mix all log output in the single console. So
  this PR offers two additional options for exec adapter in this case
  testers can config the log output(e.g. file output) and log level
  for each p2p node.
2020-11-10 14:19:44 +01:00
gary rong
b63bffe820
les, p2p/simulations/adapters: fix issues found while simulating les (#21761)
This adds a few tiny fixes for les and the p2p simulation framework:

LES Parts

- Keep the LES-SERVER connection even it's non-synced

  We had this idea to reject the connections in LES protocol if the les-server itself is
  not synced. However, in LES protocol we will also receive the connection from another
  les-server. In this case even the local node is not synced yet, we should keep the tcp
  connection for other protocols(e.g. eth protocol).

- Don't count "invalid message" for non-existing GetBlockHeadersMsg request

  In the eth syncing mechanism (full sync, fast sync, light sync), it will try to fetch
  some non-existent blocks or headers(to ensure we indeed download all the missing chain).
  In this case, it's possible that the les-server will receive the request for
  non-existent headers. So don't count it as the "invalid message" for scheduling
  dropping.

- Copy the announce object in the closure

  Before the les-server pushes the latest headers to all connected clients, it will create
  a closure and queue it in the underlying request scheduler. In some scenarios it's
  problematic. E.g, in private networks, the block can be mined very fast. So before the
  first closure is executed, we may already update the latest_announce object. So actually
  the "announce" object we want to send is replaced.

  The downsize is the client will receive two announces with the same td and then drop the
  server.

P2P Simulation Framework

- Don't double register the protocol services in p2p-simulation "Start".

  The protocols upon the devp2p are registered in the "New node stage". So don't reigster
  them again when starting a node in the p2p simulation framework

- Add one more new config field "ExternalSigner", in order to use clef service in the
  framework.
2020-10-30 18:04:38 +01:00
Felix Lange
524aaf5ec6
p2p/discover: implement v5.1 wire protocol (#21647)
This change implements the Discovery v5.1 wire protocol and
also adds an interactive test suite for this protocol.
2020-10-14 12:28:17 +02:00
Martin Holst Swende
7a5a822905
eth, p2p: use truncated names (#21698)
* peer: return localAddr instead of name to prevent spam

We currently use the name (which can be freely set by the peer) in several log messages.
This enables malicious actors to write spam into your geth log.
This commit returns the localAddr instead of the freely settable name.

* p2p: reduce usage of peer.Name in warn messages

* eth, p2p: use truncated names

* Update peer.go

Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
Co-authored-by: Felix Lange <fjl@twurst.com>
2020-10-13 13:28:24 +02:00
Felix Lange
5e86e4ed29
p2p/discover: remove use of shared hash instance for key derivation (#21673)
For some reason, using the shared hash causes a cryptographic incompatibility
when using Go 1.15. I noticed this during the development of Discovery v5.1
when I added test vector verification.

The go library commit that broke this is golang/go@97240d5, but the
way we used HKDF is slightly dodgy anyway and it's not a regression.
2020-10-08 11:19:54 +02:00
mr_franklin
666092936c
p2p/enode: remove unused code (#21612) 2020-09-28 20:10:11 +02:00
rene
129cf075e9
p2p: move rlpx into separate package (#21464)
This change moves the RLPx protocol implementation into a separate package,
p2p/rlpx. The new package can be used to establish RLPx connections for
protocol testing purposes.

Co-authored-by: Felix Lange <fjl@twurst.com>
2020-09-22 10:17:39 +02:00
Felföldi Zsolt
4eb9296910
p2p/nodestate: ensure correct callback order (#21436)
This PR adds an extra guarantee to NodeStateMachine: it ensures that all
immediate effects of a certain change are processed before any subsequent
effects of any of the immediate effects on the same node. In the original
version, if a cascaded change caused a subscription callback to be called
multiple times for the same node then these calls might have happened in a
wrong chronological order.

For example:

- a subscription to flag0 changes flag1 and flag2
- a subscription to flag1 changes flag3
- a subscription to flag1, flag2 and flag3 was called in the following order:

   [flag1] -> [flag1, flag3]
   [] -> [flag1]
   [flag1, flag3] -> [flag1, flag2, flag3]

This happened because the tree of changes was traversed in a "depth-first
order". Now it is traversed in a "breadth-first order"; each node has a
FIFO queue for pending callbacks and each triggered subscription callback
is added to the end of the list. The already existing guarantees are
retained; no SetState or SetField returns until the callback queue of the
node is empty again. Just like before, it is the responsibility of the
state machine design to ensure that infinite state loops are not possible.
Multiple changes affecting the same node can still happen simultaneously;
in this case the changes can be interleaved in the FIFO of the node but the
correct order is still guaranteed.

A new unit test is also added to verify callback order in the above scenario.
2020-09-14 14:01:18 +02:00
Guillaume Ballet
4e2641319b
p2p/discover: fix typo in comments (#21554) 2020-09-11 20:35:38 +02:00
timcooijmans
7b5107b73f
p2p/discover: avoid dropping unverified nodes when table is almost empty (#21396)
This change improves discovery behavior in small networks. Very small
networks would often fail to bootstrap because all member nodes were
dropping table content due to findnode failure. The check is now changed
to avoid dropping nodes on findnode failure when their bucket is almost
empty. It also relaxes the liveness check requirement for FINDNODE/v4
response nodes, returning unverified nodes as results when there aren't
any verified nodes yet.

The "findnode failed" log now reports whether the node was dropped
instead of the number of results. The value of the "results" was
always zero by definition.

Co-authored-by: Felix Lange <fjl@twurst.com>
2020-08-24 14:42:39 +02:00
Felix Lange
1d25039ff5
p2p/nat: limit UPNP request concurrency (#21390)
This adds a lock around requests because some routers can't handle
concurrent requests. Requests are also rate-limited.
 
The Map function request a new mapping exactly when the map timeout
occurs instead of 5 minutes earlier. This should prevent duplicate mappings.
2020-08-05 09:51:37 +02:00
rene
c0c01612e9
node: refactor package node (#21105)
This PR significantly changes the APIs for instantiating Ethereum nodes in
a Go program. The new APIs are not backwards-compatible, but we feel that
this is made up for by the much simpler way of registering services on
node.Node. You can find more information and rationale in the design
document: https://gist.github.com/renaynay/5bec2de19fde66f4d04c535fd24f0775.

There is also a new feature in Node's Go API: it is now possible to
register arbitrary handlers on the user-facing HTTP server. In geth, this
facility is used to enable GraphQL.

There is a single minor change relevant for geth users in this PR: The
GraphQL API is no longer available separately from the JSON-RPC HTTP
server. If you want GraphQL, you need to enable it using the
./geth --http --graphql flag combination.

The --graphql.port and --graphql.addr flags are no longer available.
2020-08-03 19:40:46 +02:00
Felix Lange
6ef4495a8f
p2p/discover: require table nodes to have an IP (#21330)
This fixes a corner case in discv5. The issue cannot happen in discv4
because it performs IP checks on all incoming node information.
2020-07-13 22:25:45 +02:00
libotony
2e08dad9e6
p2p/discv5: unset pingEcho on pong timeout (#21324) 2020-07-13 11:20:47 +02:00
Péter Szilágyi
42ccb2fdbd
p2p: measure packet throughput too, not just bandwidth 2020-06-24 09:36:20 +03:00
Felix Lange
723bd8c17f
p2p/discover: move discv4 encoding to new 'v4wire' package (#21147)
This moves all v4 protocol definitions to a new package, p2p/discover/v4wire.
The new package will be used for low-level protocol tests.
2020-06-02 13:20:19 +02:00
Martin Michlmayr
4f2784b38f
all: fix typos in comments (#21118) 2020-05-25 10:21:28 +02:00
Felföldi Zsolt
b4a2681120
les, les/lespay: implement new server pool (#20758)
This PR reimplements the light client server pool. It is also a first step
to move certain logic into a new lespay package. This package will contain
the implementation of the lespay token sale functions, the token buying and
selling logic and other components related to peer selection/prioritization
and service quality evaluation. Over the long term this package will be
reusable for incentivizing future protocols.

Since the LES peer logic is now based on enode.Iterator, it can now use
DNS-based fallback discovery to find servers.

This document describes the function of the new components:
https://gist.github.com/zsfelfoldi/3c7ace895234b7b345ab4f71dab102d4
2020-05-22 13:46:34 +02:00
Felix Lange
6f54ae24cd
p2p: add 0 port check in dialer (#21008)
* p2p: add low port check in dialer

We already have a check like this for UDP ports, add a similar one in
the dialer. This prevents dials to port zero and it's also an extra
layer of protection against spamming HTTP servers.

* p2p/discover: use errLowPort in v4 code

* p2p: change port check

* p2p: add comment

* p2p/simulations/adapters: ensure assigned port is in all node records
2020-05-11 18:11:17 +03:00
tgyKomgo
1aa83290f5
p2p/enode: update code comment (#20972)
It is possible to specify enode URLs using domain name since
commit b90cdbaa79, but the code comment still said that only
IP addresses are allowed.

Co-authored-by: admin@komgo.io <KomgoRocks2018!>
2020-04-24 16:50:03 +02:00
ucwong
6c458f32f8
p2p: defer wait group done in protocol start (#20951) 2020-04-21 16:42:18 +02:00
Nishant Das
7a63faf734
p2p/discover: add helper methods to UDPv5 (#20918)
This adds two new methods to UDPv5, AllNodes and LocalNode.

AllNodes returns all the nodes stored in the local table; this is
useful for the purposes of metrics collection and also debugging any
potential issues with other discovery v5 implementations.

LocalNode returns the local node object. The reason for exposing this
is so that users can modify and set/delete new key-value entries in
the local record.
2020-04-16 15:58:37 +02:00
Felföldi Zsolt
0851646e48
les, les/lespay/client: add service value statistics and API (#20837)
This PR adds service value measurement statistics to the light client. It
also adds a private API that makes these statistics accessible. A follow-up
PR will add the new server pool which uses these statistics to select
servers with good performance.

This document describes the function of the new components:
https://gist.github.com/zsfelfoldi/3c7ace895234b7b345ab4f71dab102d4

Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
2020-04-09 11:55:32 +02:00
Felix Lange
c8e9a91672
build: upgrade to golangci-lint 1.24.0 (#20901)
* accounts/scwallet: remove unnecessary uses of fmt.Sprintf

* cmd/puppeth: remove unnecessary uses of fmt.Sprintf

* p2p/discv5: remove unnecessary use of fmt.Sprintf

* whisper/mailserver: remove unnecessary uses of fmt.Sprintf

* core: goimports -w tx_pool_test.go

* eth/downloader: goimports -w downloader_test.go

* build: upgrade to golangci-lint 1.24.0
2020-04-08 11:07:29 +03:00
Felix Lange
b7394d7942
p2p/discover: add initial discovery v5 implementation (#20750)
This adds an implementation of the current discovery v5 spec.

There is full integration with cmd/devp2p and enode.Iterator in this
version. In theory we could enable the new protocol as a replacement of
discovery v4 at any time. In practice, there will likely be a few more
changes to the spec and implementation before this can happen.
2020-04-08 09:57:23 +02:00
ucwong
0c359e4b9a
p2p/discv5, p2p/testing: add missing Timer.Stop calls in tests (#20869) 2020-04-02 16:03:40 +02:00
ucwong
c87cdd3053
p2p/discv5: add missing Timer.Stop calls (#20853) 2020-04-02 10:11:16 +02:00
ucwong
bf35e27ea7
p2p/server: add UDP port mapping goroutine to wait group (#20846) 2020-04-01 18:00:33 +02:00
Felix Lange
1e1b18637e
p2p/discv5: fix test on go 1.14 (#20724) 2020-02-27 14:10:28 +02:00
Boqin Qin
1b9c5b393b
all: fix goroutine leaks in unit tests by adding 1-elem channel buffer (#20666)
This fixes a bunch of cases where a timeout in the test would leak
a goroutine.
2020-02-17 17:33:11 +01:00
Felix Lange
57d4898e29
p2p/dnsdisc: re-check tree root when leaf resolution fails (#20682)
This adds additional logic to re-resolve the root name of a tree when a
couple of leaf requests have failed. We need this change to avoid
getting into a failure state where leaf requests keep failing for half
an hour when the tree has been updated.
2020-02-17 15:23:25 +01:00
Felix Lange
ac72787768
p2p: remove MeteredPeerEvent (#20679)
This event was added for the dashboard, but we don't need it anymore
since the dashboard is gone.
2020-02-17 13:22:14 +02:00
Felix Lange
90caa2cabb
p2p: new dial scheduler (#20592)
* p2p: new dial scheduler

This change replaces the peer-to-peer dial scheduler with a new and
improved implementation. The new code is better than the previous
implementation in two key aspects:

- The time between discovery of a node and dialing that node is
  significantly lower in the new version. The old dialState kept
  a buffer of nodes and launched a task to refill it whenever the buffer
  became empty. This worked well with the discovery interface we used to
  have, but doesn't really work with the new iterator-based discovery
  API.

- Selection of static dial candidates (created by Server.AddPeer or
  through static-nodes.json) performs much better for large amounts of
  static peers. Connections to static nodes are now limited like dynanic
  dials and can no longer overstep MaxPeers or the dial ratio.

* p2p/simulations/adapters: adapt to new NodeDialer interface

* p2p: re-add check for self in checkDial

* p2p: remove peersetCh

* p2p: allow static dials when discovery is disabled

* p2p: add test for dialScheduler.removeStatic

* p2p: remove blank line

* p2p: fix documentation of maxDialPeers

* p2p: change "ok" to "added" in static node log

* p2p: improve dialTask docs

Also increase log level for "Can't resolve node"

* p2p: ensure dial resolver is truly nil without discovery

* p2p: add "looking for peers" log message

* p2p: clean up Server.run comments

* p2p: fix maxDialedConns for maxpeers < dialRatio

Always allocate at least one dial slot unless dialing is disabled using
NoDial or MaxPeers == 0. Most importantly, this fixes MaxPeers == 1 to
dedicate the sole slot to dialing instead of listening.

* p2p: fix RemovePeer to disconnect the peer again

Also make RemovePeer synchronous and add a test.

* p2p: remove "Connection set up" log message

* p2p: clean up connection logging

We previously logged outgoing connection failures up to three times.

- in SetupConn() as "Setting up connection failed addr=..."
- in setupConn() with an error-specific message and "id=... addr=..."
- in dial() as "Dial error task=..."

This commit ensures a single log message is emitted per failure and adds
"id=... addr=... conn=..." everywhere (id= omitted when the ID isn't
known yet).

Also avoid printing a log message when a static dial fails but can't be
resolved because discv4 is disabled. The light client hit this case all
the time, increasing the message count to four lines per failed
connection.

* p2p: document that RemovePeer blocks
2020-02-13 11:10:03 +01:00
Boqin Qin
a9614c3c91
event, p2p/simulations/adapters: fix rare goroutine leaks (#20657)
Co-authored-by: Felix Lange <fjl@twurst.com>
2020-02-12 15:19:47 +01:00
Felix Lange
d5acc5ed9e p2p: ensure Server.loop is ticking even if discovery hangs (#20573)
This is a temporary fix for a problem which started happening when the
dialer was changed to read nodes from an enode.Iterator. Before the
iterator change, discovery queries would always return within a couple
seconds even if there was no Internet access. Since the iterator won't
return unless a node is actually found, discoverTask can take much
longer. This means that the 'emergency connect' logic might not execute
in time, leading to a stuck node.
2020-01-17 12:29:16 +02:00
Felix Lange
fcafa0baa5 p2p: wait for listener goroutines on shutdown (#20569)
* p2p: wait for goroutine exit, fixes #20558

* p2p: wait for all slots on exit

Co-authored-by: Martin Holst Swende <martin@swende.se>
2020-01-16 14:10:15 +02:00
Felix Lange
191364c350 p2p/dnsdisc: add enode.Iterator API (#20437)
* p2p/dnsdisc: add support for enode.Iterator

This changes the dnsdisc.Client API to support the enode.Iterator
interface.

* p2p/dnsdisc: rate-limit DNS requests

* p2p/dnsdisc: preserve linked trees across root updates

This improves the way links are handled when the link root changes.
Previously, sync would simply remove all links from the current tree and
garbage-collect all unreachable trees before syncing the new list of
links.

This behavior isn't great in certain cases: Consider a structure where
trees A, B, and C reference each other and D links to A. If D's link
root changed, the sync code would first remove trees A, B and C, only to
re-sync them later when the link to A was found again.

The fix for this problem is to track the current set of links in each
clientTree and removing old links only AFTER all links are synced.

* p2p/dnsdisc: deflake iterator test

* cmd/devp2p: adapt dnsClient to new p2p/dnsdisc API

* p2p/dnsdisc: tiny comment fix
2019-12-12 11:15:36 +02:00
Marius van der Wijden
c9dce0bfd7 p2p/enode: remove data race in sliceIter (#20421) 2019-12-05 22:16:35 +01:00
Felix Lange
2e98706a99 p2p/discover: slow down lookups on empty table (#20389)
* p2p/discover: slow down lookups on empty table

* p2p/discover: wake from slowdown sleep when table is closed
2019-11-26 12:14:43 +02:00
Felix Lange
fdff182f11 p2p/discv5: add deprecation warning and remove unused code (#20367)
* p2p/discv5: add deprecation warning and remove unused code

* p2p/discv5: remove unused variables
2019-11-22 18:02:13 +02:00
Felix Lange
df206d2513
p2p/simulations: fix staticcheck warnings (#20322) 2019-11-19 17:16:42 +01:00
Felix Lange
9e8cc00b73
p2p: remove unused code (#20325) 2019-11-19 17:16:08 +01:00
Felix Lange
5fefe39ba5 p2p/netutil: fix staticcheck warning (#20315) 2019-11-19 11:17:41 +02:00
Felix Lange
689486449d build: use golangci-lint (#20295)
* build: use golangci-lint

This changes build/ci.go to download and run golangci-lint instead
of gometalinter.

* core/state: fix unnecessary conversion

* p2p/simulations: fix lock copying (found by go vet)

* signer/core: fix unnecessary conversions

* crypto/ecies: remove unused function cmpPublic

* core/rawdb: remove unused function print

* core/state: remove unused function xTestFuzzCutter

* core/vm: disable TestWriteExpectedValues in a different way

* core/forkid: remove unused function checksum

* les: remove unused type proofsData

* cmd/utils: remove unused functions prefixedNames, prefixFor

* crypto/bn256: run goimports

* p2p/nat: fix goimports lint issue

* cmd/clef: avoid using unkeyed struct fields

* les: cancel context in testRequest

* rlp: delete unreachable code

* core: gofmt

* internal/build: simplify DownloadFile for Go 1.11 compatibility

* build: remove go test --short flag

* .travis.yml: disable build cache

* whisper/whisperv6: fix ineffectual assignment in TestWhisperIdentityManagement

* .golangci.yml: enable goconst and ineffassign linters

* build: print message when there are no lint issues

* internal/build: refactor download a bit
2019-11-18 10:49:17 +02:00
Felix Lange
7c4a4eb58a rpc, p2p/simulations: use github.com/gorilla/websocket (#20289)
* rpc: improve codec abstraction

rpc.ServerCodec is an opaque interface. There was only one way to get a
codec using existing APIs: rpc.NewJSONCodec. This change exports
newCodec (as NewFuncCodec) and NewJSONCodec (as NewCodec). It also makes
all codec methods non-public to avoid showing internals in godoc.

While here, remove codec options in tests because they are not
supported anymore.

* p2p/simulations: use github.com/gorilla/websocket

This package was the last remaining user of golang.org/x/net/websocket.
Migrating to the new library wasn't straightforward because it is no
longer possible to treat WebSocket connections as a net.Conn.

* vendor: delete golang.org/x/net/websocket

* rpc: fix godoc comments and run gofmt
2019-11-18 10:40:59 +02:00
Kurkó Mihály
4ea9b62b5c dashboard: send current block to the dashboard client (#19762)
This adds all dashboard changes from the last couple months.
We're about to remove the dashboard, but decided that we should
get all the recent work in first in case anyone wants to pick up this
project later on.

* cmd, dashboard, eth, p2p: send peer info to the dashboard
* dashboard: update npm packages, improve UI, rebase
* dashboard, p2p: remove println, change doc
* cmd, dashboard, eth, p2p: cleanup after review
* dashboard: send current block to the dashboard client
2019-11-13 12:13:13 +01:00
Rick
6f1a600f6c p2p: fix bug in TestPeerDisconnect (#20277) 2019-11-13 12:01:52 +01:00
Felix Lange
adf007dadc
p2p/enode: mock DNS resolver in URL parsing test (#20252) 2019-11-07 16:40:37 +01:00
Felix Lange
2c37142d2f cmd/devp2p, p2p: dial using node iterator, discovery crawler (#20132)
* p2p/enode: add Iterator and associated utilities

* p2p/discover: add RandomNodes iterator

* p2p: dial using iterator

* cmd/devp2p: add discv4 crawler

* cmd/devp2p: WIP nodeset filter

* cmd/devp2p: fixup lesFilter

* core/forkid: add NewStaticFilter

* cmd/devp2p: make -eth-network filter actually work

* cmd/devp2p: improve crawl timestamp handling

* cmd/devp2p: fix typo

* p2p/enode: fix comment typos

* p2p/discover: fix comment typos

* p2p/discover: rename lookup.next to 'advance'

* p2p: lower discovery mixer timeout

* p2p/enode: implement dynamic FairMix timeouts

* cmd/devp2p: add ropsten support in -eth-network filter

* cmd/devp2p: tweak crawler log message
2019-10-29 17:08:57 +02:00
Ross
d5b79e752e p2p/simulations: add node properties support and utility functions (#20060) 2019-10-17 10:07:09 +02:00
Felix Lange
7300365956 p2p/dnsdisc: update to latest EIP-1459 spec (#20168)
This updates the DNS TXT record format to the latest
changes in ethereum/EIPs#2313.
2019-10-16 14:35:24 +03:00
Péter Szilágyi
a2a60869c8
p2p: measure subprotocol bandwidth usage 2019-09-27 18:00:25 +03:00
Felix Lange
0568e81701
p2p/dnsdisc: add implementation of EIP-1459 (#20094)
This adds an implementation of node discovery via DNS TXT records to the
go-ethereum library. The implementation doesn't match EIP-1459 exactly,
the main difference being that this implementation uses separate merkle
trees for tree links and ENRs. The EIP will be updated to match p2p/dnsdisc.

To maintain DNS trees, cmd/devp2p provides a frontend for the p2p/dnsdisc
library. The new 'dns' subcommands can be used to create, sign and deploy DNS
discovery trees.
2019-09-25 11:38:13 +02:00
Felix Lange
39b0b1a1a6
all: make unit tests work with Go 1.13 (#20053)
Most of these changes are related to the Go 1.13 changes to test binary
flag handling. 

* cmd/geth: make attach tests more reliable

This makes the test wait for the endpoint to come up by polling
it instead of waiting for two seconds.

* tests: fix test binary flags for Go 1.13

Calling flag.Parse during package initialization is prohibited
as of Go 1.13 and causes test failures. Call it in TestMain instead.

* crypto/ecies: remove useless -dump flag in tests

* p2p/simulations: fix test binary flags for Go 1.13

Calling flag.Parse during package initialization is prohibited
as of Go 1.13 and causes test failures. Call it in TestMain instead.

* build: remove workaround for ./... vendor matching

This workaround was necessary for Go 1.8. The Go 1.9 release changed
the expansion rules to exclude vendored packages.

* Makefile: use relative path for GOBIN

This makes the "Run ./build/bin/..." line look nicer.

* les: fix test binary flags for Go 1.13

Calling flag.Parse during package initialization is prohibited
as of Go 1.13 and causes test failures. Call it in TestMain instead.
2019-09-11 14:41:22 +02:00
Péter Szilágyi
72d5a27a39
core, metrics, p2p: switch some invalid counters to gauges 2019-09-10 14:39:07 +03:00
Felix Lange
54b271a86d
crypto: add SignatureLength constant and use it everywhere (#19996)
Original change by @jpeletier
2019-08-22 15:14:06 +02:00
alexwang
b90cdbaa79 p2p/enode: allow DNS names in enode URLs (#18524) 2019-08-22 14:23:40 +02:00
Felix Lange
26f538b0e5
p2p/enode, p2p/discv5: fix URL parsing test for go 1.12.8 (#19963) 2019-08-15 10:36:36 +02:00
Christian Muehlhaeuser
a32a2b933a cmd, contracts, eth, p2p, signer, whisper: fixed ineffectual assignments (#19869)
Fixed assigning values to variables we don't end up using.
2019-07-22 13:34:41 +03:00
Péter Szilágyi
1a83114c74
all: update author list and licenses 2019-07-22 12:17:27 +03:00