Commit Graph

133 Commits

Author SHA1 Message Date
Felix Lange
fa538ee7ed p2p/discover: improve randomness of ReadRandomNodes (#19799)
Make it select from all live nodes instead of selecting the heads of
random buckets.
2019-07-08 18:58:03 +03:00
Felix Lange
e83c3ccc47
p2p/enode: improve IPv6 support, add ENR text representation (#19663)
* p2p/enr: add entries for for IPv4/IPv6 separation

This adds entry types for "ip6", "udp6", "tcp6" keys. The IP type stays
around because removing it would break a lot of code and force everyone
to care about the distinction.

* p2p/enode: track IPv4 and IPv6 address separately

LocalNode predicts the local node's UDP endpoint and updates the record.
This change makes it predict IPv4 and IPv6 endpoints separately since
they can now be in the record at the same time.

* p2p/enode: implement base64 text format
* all: switch to enode.Parse(...)

This allows passing base64-encoded node records to all the places that
previously accepted enode:// URLs. The URL format is still supported.

* cmd/bootnode, p2p: log node URL instead of ENR

...and return the base64 record in NodeInfo.
2019-06-07 15:31:00 +02:00
Felix Lange
896322bf88
cmd/devp2p: add devp2p debug tool (#19657)
* p2p/discover: export Ping and RequestENR

These two are useful for checking the status of a node.

* cmd/devp2p: add devp2p debug tool

This is a new tool for debugging p2p issues. It supports a few
basic tasks for now, but many more things can and will be added
in the near future.

   devp2p enrdump            -- prints ENRs readably
   devp2p discv4 ping        -- checks if a node is up
   devp2p discv4 requestenr  -- gets a node's record
   devp2p discv4 resolve     -- finds a node through the DHT
2019-06-07 15:29:16 +02:00
Martin Holst Swende
fec3b56f7f accounts, p2p, rpc: make CGO_ENABLED=0 build again (#19593)
* p2p: remove direct import of cgo-library

* accounts, rpc: more nocgo alternatives

* rpc: move unix path constant into separate file

* accounts/scwallet: address review concerns, remove copy-pasta
2019-05-26 01:07:10 +03:00
Felix Lange
b548b5aeb0
p2p/discover: fix crash in Resolve (#19579) 2019-05-15 11:11:17 -04:00
Felix Lange
350a87dd3c
p2p/discover: add support for EIP-868 (v4 ENR extension) (#19540)
This change implements EIP-868. The UDPv4 transport announces support
for the extension in ping/pong and handles enrRequest messages.

There are two uses of the extension: If a remote node announces support
for EIP-868 in their pong, node revalidation pulls the node's record.
The Resolve method requests the record unconditionally.
2019-05-15 06:47:45 +02:00
Martin Holst Swende
95263914fc
p2p/discover: fix a race where table loop would self-lookup before returning from constructor 2019-05-13 11:30:31 +02:00
Felix Lange
dba1750eda p2p/discover: split out discv4 code
This change restructures the internals of p2p/discover to make room for
the discv5 code which will soon be added to this package.

- packet type names now have a "V4" suffix.
- ListenUDP returns *UDPv4 instead of *Table. This technically breaks
  the API but the only caller in go-ethereum is package p2p, which uses
  a compatible interface and doesn't need changes.
- The internal transport interface is changed to make Table reusable for v5.
- The 'lookup' code moves from table to transport. This required
  updating the lookup unit test to use udpTest instead of a custom transport.
2019-04-30 13:13:22 +02:00
Felix Lange
ed97517ff4 p2p/discover: bump failure counter only if no nodes were provided (#19362)
This resolves a minor issue where neighbors responses containing less
than 16 nodes would bump the failure counter, removing the node. One
situation where this can happen is a private deployment where the total
number of extant nodes is less than 16.

Issue found by @jsying.
2019-04-08 14:35:11 +03:00
Péter Szilágyi
dac7cbcf21
p2p/enode: use localItemKey for local sequence number (#19131)
* p2p/discover: remove unused function

* p2p/enode: use localItemKey for local sequence number

I added localItemKey for this purpose in #18963, but then
forgot to actually use it. This changes the database layout
yet again and requires bumping the version number.
2019-02-28 13:14:45 +02:00
Felix Lange
cf147c71d5 p2p/discover: remove unused function 2019-02-19 13:29:19 +01:00
Matthew Halpern
f1537b774c p2p/discover: make maximum packet size a constant (#19061) 2019-02-19 12:27:29 +01:00
Felix Lange
a89170cfb2
p2p/discover: improve table addition code (#18974)
This change clears up confusion around the two ways in which nodes
can be added to the table.

When a neighbors packet is received as a reply to findnode, the nodes
contained in the reply are added as 'seen' entries if sufficient space
is available.

When a ping is received and the endpoint verification has taken place,
the remote node is added as a 'verified' entry or moved to the front of
the bucket if present. This also updates the node's IP address and port
if they have changed.
2019-01-31 11:48:54 +01:00
Felix Lange
f0c6f92140
p2p/discover, p2p/enode: rework endpoint proof handling, packet logging (#18963)
This change resolves multiple issues around handling of endpoint proofs.
The proof is now done separately for each IP and completing the proof
requires a matching ping hash.

Also remove waitping because it's equivalent to sleep. waitping was
slightly more efficient, but that may cause issues with findnode if
packets are reordered and the remote end sees findnode before pong.

Logging of received packets was hitherto done after handling the packet,
which meant that sent replies were logged before the packet that
generated them. This change splits up packet handling into 'preverify'
and 'handle'. The error from 'preverify' is logged, but 'handle' happens
after the message is logged. This fixes the order. Packet logs now
contain the node ID.
2019-01-29 17:39:20 +01:00
Martin Holst Swende
493903eede
core: better side-chain importing 2018-11-20 12:28:43 +02:00
Felix Lange
6f607de5d5
p2p, p2p/discover: add signed ENR generation (#17753)
This PR adds enode.LocalNode and integrates it into the p2p
subsystem. This new object is the keeper of the local node
record. For now, a new version of the record is produced every
time the client restarts. We'll make it smarter to avoid that in
the future.

There are a couple of other changes in this commit: discovery now
waits for all of its goroutines at shutdown and the p2p server
now closes the node database after discovery has shut down. This
fixes a leveldb crash in tests. p2p server startup is faster
because it doesn't need to wait for the external IP query
anymore.
2018-10-12 11:47:24 +02:00
Felix Lange
30cd5c1854
all: new p2p node representation (#17643)
Package p2p/enode provides a generalized representation of p2p nodes
which can contain arbitrary information in key/value pairs. It is also
the new home for the node database. The "v4" identity scheme is also
moved here from p2p/enr to remove the dependency on Ethereum crypto from
that package.

Record signature handling is changed significantly. The identity scheme
registry is removed and acceptable schemes must be passed to any method
that needs identity. This means records must now be validated explicitly
after decoding.

The enode API is designed to make signature handling easy and safe: most
APIs around the codebase work with enode.Node, which is a wrapper around
a valid record. Going from enr.Record to enode.Node requires a valid
signature.

* p2p/discover: port to p2p/enode

This ports the discovery code to the new node representation in
p2p/enode. The wire protocol is unchanged, this can be considered a
refactoring change. The Kademlia table can now deal with nodes using an
arbitrary identity scheme. This requires a few incompatible API changes:

  - Table.Lookup is not available anymore. It used to take a public key
    as argument because v4 protocol requires one. Its replacement is
    LookupRandom.
  - Table.Resolve takes *enode.Node instead of NodeID. This is also for
    v4 protocol compatibility because nodes cannot be looked up by ID
    alone.
  - Types Node and NodeID are gone. Further commits in the series will be
    fixes all over the the codebase to deal with those removals.

* p2p: port to p2p/enode and discovery changes

This adapts package p2p to the changes in p2p/discover. All uses of
discover.Node and discover.NodeID are replaced by their equivalents from
p2p/enode.

New API is added to retrieve the enode.Node instance of a peer. The
behavior of Server.Self with discovery disabled is improved. It now
tries much harder to report a working IP address, falling back to
127.0.0.1 if no suitable address can be determined through other means.
These changes were needed for tests of other packages later in the
series.

* p2p/simulations, p2p/testing: port to p2p/enode

No surprises here, mostly replacements of discover.Node, discover.NodeID
with their new equivalents. The 'interesting' API changes are:

 - testing.ProtocolSession tracks complete nodes, not just their IDs.
 - adapters.NodeConfig has a new method to create a complete node.

These changes were needed to make swarm tests work.

Note that the NodeID change makes the code incompatible with old
simulation snapshots.

* whisper/whisperv5, whisper/whisperv6: port to p2p/enode

This port was easy because whisper uses []byte for node IDs and
URL strings in the API.

* eth: port to p2p/enode

Again, easy to port because eth uses strings for node IDs and doesn't
care about node information in any way.

* les: port to p2p/enode

Apart from replacing discover.NodeID with enode.ID, most changes are in
the server pool code. It now deals with complete nodes instead
of (Pubkey, IP, Port) triples. The database format is unchanged for now,
but we should probably change it to use the node database later.

* node: port to p2p/enode

This change simply replaces discover.Node and discover.NodeID with their
new equivalents.

* swarm/network: port to p2p/enode

Swarm has its own node address representation, BzzAddr, containing both
an overlay address (the hash of a secp256k1 public key) and an underlay
address (enode:// URL).

There are no changes to the BzzAddr format in this commit, but certain
operations such as creating a BzzAddr from a node ID are now impossible
because node IDs aren't public keys anymore.

Most swarm-related changes in the series remove uses of
NewAddrFromNodeID, replacing it with NewAddr which takes a complete node
as argument. ToOverlayAddr is removed because we can just use the node
ID directly.
2018-09-25 00:59:00 +02:00
Emil
86a03f97d3 all: simplify s[:] to s where s is a slice (#17673) 2018-09-14 22:07:13 +02:00
Oleg Kovalov
cf05ef9106 p2p, swarm, trie: avoid copying slices in loops (#17265) 2018-08-07 13:56:40 +03:00
Felix Lange
c73b654fd1 p2p/discover: move bond logic from table to transport (#17048)
* p2p/discover: move bond logic from table to transport

This commit moves node endpoint verification (bonding) from the table to
the UDP transport implementation. Previously, adding a node to the table
entailed pinging the node if needed. With this change, the ping-back
logic is embedded in the packet handler at a lower level.

It is easy to verify that the basic protocol is unchanged: we still
require a valid pong reply from the node before findnode is accepted.

The node database tracked the time of last ping sent to the node and
time of last valid pong received from the node. Node endpoints are
considered verified when a valid pong is received and the time of last
pong was called 'bond time'. The time of last ping sent was unused. In
this commit, the last ping database entry is repurposed to mean last
ping _received_. This entry is now used to track whether the node needs
to be pinged back.

The other big change is how nodes are added to the table. We used to add
nodes in Table.bond, which ran when a remote node pinged us or when we
encountered the node in a neighbors reply. The transport now adds to the
table directly after the endpoint is verified through ping. To ensure
that the Table can't be filled just by pinging the node repeatedly, we
retain the isInitDone check. During init, only nodes from neighbors
replies are added.

* p2p/discover: reduce findnode failure counter on success

* p2p/discover: remove unused parameter of loadSeedNodes

* p2p/discover: improve ping-back check and comments

* p2p/discover: add neighbors reply nodes always, not just during init
2018-07-03 16:24:12 +03:00
Elad
1836366ac1 all: library changes for swarm-network-rewrite (#16898)
This commit adds all changes needed for the merge of swarm-network-rewrite.
The changes:

- build: increase linter timeout
- contracts/ens: export ensNode
- log: add Output method and enable fractional seconds in format
- metrics: relax test timeout
- p2p: reduced some log levels, updates to simulation packages
- rpc: increased maxClientSubscriptionBuffer to 20000
2018-06-14 11:21:17 +02:00
kiel barry
864e80a48f p2p: fix some golint warnings (#16577) 2018-05-08 13:08:43 +02:00
thomasmodeneis
ba1030b6b8 build: enable goimports and varcheck linters (#16446) 2018-04-18 00:53:50 +02:00
Felix Lange
aeedec4078 p2p/discover: s/lastPong/bondTime/, update TestUDP_findnode
I forgot to change the check in udp.go when I changed Table.bond to be
based on lastPong instead of node presence in db. Rename lastPong to
bondTime and add hasBond so it's clearer what this DB key is used for
now.
2018-02-16 21:29:20 +01:00
Péter Szilágyi
32301a4d6b
p2p/discover: validate bond against lastpong, not db presence 2018-02-16 17:05:08 +02:00
Péter Szilágyi
20797348ca
p2p/discover: fix out-of-bounds issue 2018-02-13 20:59:43 +02:00
Felix Lange
9123eceb0f p2p, p2p/discover: misc connectivity improvements (#16069)
* p2p: add DialRatio for configuration of inbound vs. dialed connections

* p2p: add connection flags to PeerInfo

* p2p/netutil: add SameNet, DistinctNetSet

* p2p/discover: improve revalidation and seeding

This changes node revalidation to be periodic instead of on-demand. This
should prevent issues where dead nodes get stuck in closer buckets
because no other node will ever come along to replace them.

Every 5 seconds (on average), the last node in a random bucket is
checked and moved to the front of the bucket if it is still responding.
If revalidation fails, the last node is replaced by an entry of the
'replacement list' containing recently-seen nodes.

Most close buckets are removed because it's very unlikely we'll ever
encounter a node that would fall into any of those buckets.

Table seeding is also improved: we now require a few minutes of table
membership before considering a node as a potential seed node. This
should make it less likely to store short-lived nodes as potential
seeds.

* p2p/discover: fix nits in UDP transport

We would skip sending neighbors replies if there were fewer than
maxNeighbors results and CheckRelayIP returned an error for the last
one. While here, also resolve a TODO about pong reply tokens.
2018-02-12 14:36:09 +02:00
Felföldi Zsolt
92580d69d3 p2p, p2p/discover, p2p/discv5: implement UDP port sharing (#15200)
This commit affects p2p/discv5 "topic discovery" by running it on
the same UDP port where the old discovery works. This is realized
by giving an "unhandled" packet channel to the old v4 discovery
packet handler where all invalid packets are sent. These packets
are then processed by v5. v5 packets are always invalid when
interpreted by v4 and vice versa. This is ensured by adding one
to the first byte of the packet hash in v5 packets.

DiscoveryV5Bootnodes is also changed to point to new bootnodes
that are implementing the changed packet format with modified
hash. Existing and new v5 bootnodes are both running on different
ports ATM.
2018-01-22 13:38:34 +01:00
ferhat elmas
afa3c72c40 p2p/discover: fix leaked goroutine in data expiration 2017-12-18 09:16:54 +01:00
ferhat elmas
86f6568f66 build: enable unconvert linter (#15456)
* build: enable unconvert linter

 - fixes #15453
 - update code base for failing cases

* cmd/puppeth: replace syscall.Stdin with os.Stdin.Fd() for unconvert linter
2017-11-10 19:06:45 +02:00
Lewis Marshall
9feec51e2d p2p: add network simulation framework (#14982)
This commit introduces a network simulation framework which
can be used to run simulated networks of devp2p nodes. The
intention is to use this for testing protocols, performing
benchmarks and visualising emergent network behaviour.
2017-09-25 10:08:07 +02:00
Ali Hajimirza
33b158e0ed discover: Changed Logging from Debug to Info (#14485) 2017-05-20 13:10:59 +02:00
Felix Lange
30d706c35e cmd/geth: add --config file flag (#13875)
* p2p/discover, p2p/discv5: add marshaling methods to Node

* p2p/netutil: make Netlist decodable from TOML

* common/math: encode nil HexOrDecimal256 as 0x0

* cmd/geth: add --config file flag

* cmd/geth: add missing license header

* eth: prettify Config again, fix tests

* eth: use gasprice.Config instead of duplicating its fields

* eth/gasprice: hide nil default from dumpconfig output

* cmd/geth: hide genesis block in dumpconfig output

* node: make tests compile

* console: fix tests

* cmd/geth: make TOML keys look exactly like Go struct fields

* p2p: use discovery by default

This makes the zero Config slightly more useful. It also fixes package
node tests because Node detects reuse of the datadir through the
NodeDatabase.

* cmd/geth: make ethstats URL settable through config file

* cmd/faucet: fix configuration

* cmd/geth: dedup attach tests

* eth: add comment for DefaultConfig

* eth: pass downloader.SyncMode in Config

This removes the FastSync, LightSync flags in favour of a more
general SyncMode flag.

* cmd/utils: remove jitvm flags

* cmd/utils: make mutually exclusive flag error prettier

It now reads:

   Fatal: flags --dev, --testnet can't be used at the same time

* p2p: fix typo

* node: add DefaultConfig, use it for geth

* mobile: add missing NoDiscovery option

* cmd/utils: drop MakeNode

This exposed a couple of places that needed to be updated to use
node.DefaultConfig.

* node: fix typo

* eth: make fast sync the default mode

* cmd/utils: remove IPCApiFlag (unused)

* node: remove default IPC path

Set it in the frontends instead.

* cmd/geth: add --syncmode

* cmd/utils: make --ipcdisable and --ipcpath mutually exclusive

* cmd/utils: don't enable WS, HTTP when setting addr

* cmd/utils: fix --identity
2017-04-12 17:27:23 +03:00
Felix Lange
96ae35e2ac p2p, p2p/discover, p2p/nat: rework logging using context keys 2017-02-28 10:20:29 +01:00
Péter Szilágyi
d4fd06c3dc
all: blidly swap out glog to our log15, logs need rework 2017-02-23 12:16:44 +02:00
Péter Szilágyi
189dee26c6
p2p: remove trailing newlines from log messages 2017-02-23 12:00:04 +02:00
Felix Lange
9b0af51386 crypto: add btcec fallback for sign/recover without cgo (#3680)
* vendor: add github.com/btcsuite/btcd/btcec

* crypto: add btcec fallback for sign/recover without cgo

This commit adds a non-cgo fallback implementation of secp256k1
operations.

* crypto, core/vm: remove wrappers for sha256, ripemd160
2017-02-18 09:24:12 +01:00
Péter Szilágyi
18c77744ff
all: fix spelling errors 2017-01-06 19:44:35 +02:00
Felix Lange
f2da6581ba all: fix issues reported by honnef.co/go/simple/cmd/gosimple 2017-01-06 18:18:07 +01:00
Felix Lange
35a7dcb162 all: gofmt -w -s 2017-01-06 15:52:03 +01:00
Péter Szilágyi
2573094df2
p2p/discover, p2p/discv5: use flexible comparison for IPs 2016-12-14 18:40:49 +02:00
Felix Lange
a47341cf96 p2p, p2p/discover, p2p/discv5: add IP network restriction feature
The p2p packages can now be configured to restrict all communication to
a certain subset of IP networks. This feature is meant to be used for
private networks.
2016-11-22 22:21:18 +01:00
Felix Lange
a98d1d67d6 p2p/discover, p2p/discv5: prevent relay of invalid IPs and low ports
The discovery DHT contains a number of hosts with LAN and loopback IPs.
These get relayed because some implementations do not perform any checks
on the IP.

go-ethereum already prevented relay in most cases because it verifies
that the host actually exists before adding it to the local table. But
this verification causes other issues. We have received several reports
where people's VPSs got shut down by hosting providers because sending
packets to random LAN hosts is indistinguishable from a slow port scan.

The new check prevents sending random packets to LAN by discarding LAN
IPs sent by Internet hosts (and loopback IPs from LAN and Internet
hosts). The new check also blacklists almost all currently registered
special-purpose networks assigned by IANA to avoid inciting random
responses from services in the LAN.

As another precaution against abuse of the DHT, ports below 1024 are now
considered invalid.
2016-11-22 22:21:18 +01:00
Felix Lange
ba2884f343 p2p/discover, p2p/discv5: use netutil.IsTemporaryError 2016-11-22 22:21:15 +01:00
Péter Szilágyi
178da7c6a9
mobile: initial wrappers for mobile support 2016-11-14 17:56:58 +02:00
villesundell
c7442ef0d2 ethdb, p2p/discover: replace "alloted" with "allotted" (#2785) 2016-07-12 11:42:47 +02:00
Felix Lange
8110671960 p2p/discover: prevent bonding self 2016-05-02 18:50:16 +02:00
Felix Lange
6fdd0893c3 all: fix go vet warnings 2016-04-15 11:17:27 +02:00
Jeffrey Wilcke
483feb0d3f Merge pull request #2242 from jimenezrick/upstream-crypto
Closes #2241: Use Keccak-256 from golang.org/x/crypto/sha3 and mention explicitly
2016-02-24 12:57:57 +01:00
Péter Szilágyi
ac954f48bd p2p/discover: emphasize warning, add 10 min cooldown 2016-02-24 12:16:28 +02:00