Commit Graph

3690 Commits

Author SHA1 Message Date
divma
2acf75785c More sync updates (#1791)
## Issue Addressed
#1614 and a couple of sync-stalling problems, the most important is a cyclic dependency between the sync manager and the peer manager
2020-10-20 22:34:18 +00:00
Michael Sproul
703c33bdc7 Fix head tracker concurrency bugs (#1771)
## Issue Addressed

Closes #1557

## Proposed Changes

Modify the pruning algorithm so that it mutates the head-tracker _before_ committing the database transaction to disk, and _only if_ all the heads to be removed are still present in the head-tracker (i.e. no concurrent mutations).

In the process of writing and testing this I also had to make a few other changes:

* Use internal mutability for all `BeaconChainHarness` functions (namely the RNG and the graffiti), in order to enable parallel calls (see testing section below).
* Disable logging in harness tests unless the `test_logger` feature is turned on

And chose to make some clean-ups:

* Delete the `NullMigrator`
* Remove type-based configuration for the migrator in favour of runtime config (simpler, less duplicated code)
* Use the non-blocking migrator unless the blocking migrator is required. In the store tests we need the blocking migrator because some tests make asserts about the state of the DB after the migration has run.
* Rename `validators_keypairs` -> `validator_keypairs` in the `BeaconChainHarness`

## Testing

To confirm that the fix worked, I wrote a test using [Hiatus](https://crates.io/crates/hiatus), which can be found here:

https://github.com/michaelsproul/lighthouse/tree/hiatus-issue-1557

That test can't be merged because it inserts random breakpoints everywhere, but if you check out that branch you can run the test with:

```
$ cd beacon_node/beacon_chain
$ cargo test --release --test parallel_tests --features test_logger
```

It should pass, and the log output should show:

```
WARN Pruning deferred because of a concurrent mutation, message: this is expected only very rarely!
```

## Additional Info

This is a backwards-compatible change with no impact on consensus.
2020-10-19 05:58:39 +00:00
blacktemplar
6ba997b88e add direction information to PeerInfo (#1768)
## Issue Addressed

NA

## Proposed Changes

Adds a direction field to `PeerConnectionStatus` that can be accessed by calling `is_outgoing` which will return `true` iff the peer is connected and the first connection was an outgoing one.
2020-10-16 05:24:21 +00:00
Herman Junge
d7b9d0dd9f Implement matches! macro (#1777)
Fix #1775
2020-10-15 21:42:43 +00:00
Pawan Dhananjay
97be2ca295 Simulator and attestation service fixes (#1747)
## Issue Addressed

#1729 #1730 

Which issue # does this PR address?

## Proposed Changes

1. Fixes a bug in the simulator where nodes can't find each other due to 0 udp ports in their enr.
2. Fixes bugs in attestation service where we are unsubscribing from a subnet prematurely.

More testing is needed for attestation service fixes.
2020-10-15 07:11:31 +00:00
Pawan Dhananjay
aadbab47cc Doc fixes (#1762)
## Issue Addressed

N/A

## Proposed Changes

Minor doc fixes. Adds a section on custom data directories.



Co-authored-by: Michael Sproul <micsproul@gmail.com>
2020-10-15 00:37:00 +00:00
blacktemplar
a0634cc64f Gossipsub topic filters (#1767)
## Proposed Changes

Adds a gossipsub topic filter that only allows subscribing and incoming subscriptions from valid ETH2 topics.

## Additional Info

Currently the preparation of the valid topic hashes uses only the current fork id but in the future it must also use all possible future fork ids for planned forks. This has to get added when hard coded forks get implemented.

DO NOT MERGE: We first need to merge the libp2p changes (see https://github.com/sigp/rust-libp2p/pull/70) so that we can refer from here to a commit hash inside the lighthouse branch.
2020-10-14 10:12:57 +00:00
blacktemplar
8248afa793 Updates the message-id according to the Networking Spec (#1752)
## Proposed Changes

Implement the new message id function (see https://github.com/ethereum/eth2.0-specs/pull/2089) using an additional fast message id function for better performance + caching decompressed data.
2020-10-14 06:51:58 +00:00
Michael Sproul
467de4c8d0 Add docs for slashing protection (#1760)
## Proposed Changes

* Add documentation about slashing protection, including how to troubleshoot issues and move between clients.
* Add an error message if the validator client is started with 0 validators. Previously it would hit an error relating to the slashing protection database not existing, which wrongly pushed people towards using the unsafe `--init-slashing-protection` flag.
2020-10-13 22:10:07 +00:00
realbigsean
95c96ac567 Small doc fix (#1761)
## Issue Addressed

N/A

## Proposed Changes

Looks like there was some text left over from a merge.

## Additional Info
2020-10-12 23:56:49 +00:00
ethDreamer
e9d5bade36 Fixed cross-compiling by replacing wget with curl (#1759)
It looks like the default docker image used by cross doesn't have
wget installed. This causes builds to fail. This can be fixed by
switching to curl.

## Issue Addressed
cross-compiling was broken (at least for build-aarch64)

## Proposed Changes
swap wget for curl
2020-10-11 23:58:13 +00:00
realbigsean
83ae12a1b4 Fix epoch, slot, and effective balance quoting (#1756)
## Issue Addressed

Resolves #1717

## Proposed Changes

Add quoting for epochs, slots, and `effective_balance`

## Additional Info
2020-10-11 23:58:12 +00:00
Pawan Dhananjay
99a02fd2ab Limit snappy input stream (#1738)
## Issue Addressed

N/A

## Proposed Changes

This PR limits the length of the stream received by the snappy decoder to be the maximum allowed size for the received rpc message type. Also adds further checks to ensure that the length specified in the rpc [encoding-dependent header](https://github.com/ethereum/eth2.0-specs/blob/dev/specs/phase0/p2p-interface.md#encoding-strategies) is within the bounds for the rpc message type being decoded.
2020-10-11 22:45:33 +00:00
Paul Hauner
b185d7bbd8
Merge pull request #1671 from sigp/v0.3.0-staging
Staging: v0.3.0
2020-10-09 17:08:32 +11:00
Paul Hauner
0e4cc50262
Remove unused deps 2020-10-09 15:58:20 +11:00
Paul Hauner
db3e0578e9
Merge branch 'v0.3.0-staging' into v3-master 2020-10-09 15:27:08 +11:00
Michael Sproul
b0833033b7 Strict slashing protection by default (#1750)
## Proposed Changes

Replace `--strict-slashing-protection` by `--init-slashing-protection` and remove mentions of `--auto-register`
2020-10-09 02:05:32 +00:00
Paul Hauner
72cc5e35af Bump version to v0.3.0 (#1743)
## Issue Addressed

NA

## Proposed Changes

- Bump version to v0.3.0
- Run `cargo update`

## Additional Info

NA
2020-10-09 02:05:30 +00:00
Paul Hauner
414138f137 Update docs for v0.3.0 (#1742)
## Issue Addressed

NA

## Proposed Changes

- Remove Metamask deposits from the docs.
    - Restructure docs to be launchpad-centric.
- Remove references to sigp/lighthouse-docker.
- Add section about binaries.


## Additional Info

Please provide any additional information. For example, future considerations
or information useful for reviewers.
2020-10-09 00:43:49 +00:00
realbigsean
b69c63d486 Validator dir creation (#1746)
## Issue Addressed

Resolves #1744

## Proposed Changes

- Add `directory::ensure_dir_exists` to the `ValidatorDefinition::open_or_create` method 
- As @pawanjay176 suggested, making the `--validator-dir` non-global so users are forced to include the flag after the `validator` subcommand. Current behavior seems to be ignoring the flag if it comes after something like `validator import`

## Additional Info
N/A
2020-10-08 21:01:32 +00:00
Paul Hauner
a67fa5f4a4 Add zinken testnet (#1741)
## Issue Addressed

- Resolves #1722

## Proposed Changes

This extends @danielschonfeld's work in #1739 with:

- Use an empty boot node list
- Remove the genesis state

## Additional Info

NA


Co-authored-by: Daniel Schonfeld <daniel@schonfeld.org>
2020-10-07 10:10:35 +00:00
Herman Junge
a886afd3ca Improve command help (#1740)
A little help for the future generations.
2020-10-07 00:31:19 +00:00
Michael Sproul
56ffe91f90 Update Cargo.lock (#1735)
## Issue Addressed

Fix the lockfile after it was broken by the manual merge of https://github.com/sigp/lighthouse/pull/1654
2020-10-06 00:01:17 +00:00
blacktemplar
59adc5ba00 Implement key cache to reduce keystore loading times for validator_client (#1695)
## Issue Addressed

#1618 

## Proposed Changes

Adds an encrypted key cache that is loaded on validator_client startup. It stores the keypairs for all enabled keystores and uses as password the concatenation the passwords of all enabled keystores. This reduces the number of time intensive key derivitions for `N` validators from `N` to `1`. On changes the cache gets updated asynchronously to avoid blocking the main thread.

## Additional Info

If the cache contains the keypair of a keystore that is not in the validator_definitions.yml file during loading the cache cannot get decrypted. In this case all the keystores get decrypted and then the cache gets overwritten. To avoid that one can disable keystores in validator_definitions.yml and restart the client which will remove them from the cache, after that one can entirely remove the keystore (from the validator_definitions.yml and from the disk). 

Other solutions to the above "problem" might be:
* Add a CLI and/or API function for removing keystores which will update the cache (asynchronously).
* Add a CLI and/or API function that just updates the cache (asynchronously) after a modification of the `validator_definitions.yml` file.

Note that the cache file has a lock file which gets removed immediatly after the cache was used or updated.
2020-10-05 10:50:43 +00:00
Paul Hauner
da44821e39
Clean up obsolete TODOs (#1734)
Squashed commit of the following:

commit f99373cbaec9adb2bdbae3f7e903284327962083
Author: Age Manning <Age@AgeManning.com>
Date:   Mon Oct 5 18:44:09 2020 +1100

    Clean up obsolute TODOs
2020-10-05 21:08:14 +11:00
Paul Hauner
ee7c8a0b7e Update external deps (#1711)
## Issue Addressed

- Resolves #1706 

## Proposed Changes

Updates dependencies across the workspace. Any crate that was not able to be brought to the latest version is listed in #1712.

## Additional Info

NA
2020-10-05 08:22:19 +00:00
Age Manning
240181e840
Upgrade discovery and restructure task execution (#1693)
* Initial rebase

* Remove old code

* Correct release tests

* Rebase commit

* Remove eth2-testnet dep on eth2libp2p

* Remove crates lost in rebase

* Remove unused dep
2020-10-05 18:45:54 +11:00
Age Manning
bcb629564a
Improve error handling in network processing (#1654)
* Improve error handling in network processing

* Cargo fmt

* Cargo fmt

* Improve error handling for prior genesis

* Remove dep
2020-10-05 17:34:56 +11:00
divma
113758a4f5
From panic to crit (#1726)
## Issue Addressed
Downgrade inconsistent chain segment states from `panic` to `crit`. I don't love this solution but since range can always bounce back from any of those, we don't panic.

Co-authored-by: Age Manning <Age@AgeManning.com>
2020-10-05 17:34:49 +11:00
Justin
cf74e0baed
Document need for port 9000 to be open (fix #730) (#731)
Co-authored-by: Age Manning <Age@AgeManning.com>

Edited by Paul H when cherry-picking from master to v0.3.0-staging
2020-10-05 17:34:24 +11:00
Age Manning
a8c5af8874
Increase content-id length (#1725)
## Issue Addressed

N/A

## Proposed Changes

Increase gossipsub's content-id length to the full 32 byte hash. 

## Additional Info

N/A
2020-10-05 17:33:42 +11:00
divma
6997776494
Sync fixes (#1716)
## Issue Addressed

chain state inconsistencies

## Proposed Changes
- a batch can be fake-failed by Range if it needs to move a peer to another chain. The peer will still send blocks/ errors / produce timeouts for those  requests, so check when we get a response from the RPC that the request id matches, instead of only the peer, since a re-request can be directed to the same peer.
- if an optimistic batch succeeds, store the attempt to avoid trying it again when quickly switching chains. Also, use it only if ahead of our current target, instead of the segment's start epoch
2020-10-05 17:33:36 +11:00
Paul Hauner
e7eb99cb5e
Use Drop impl to send worker idle message (#1718)
## Issue Addressed

NA

## Proposed Changes

Uses a `Drop` implementation to help ensure that `BeaconProcessor` workers are freed. This will help prevent against regression, if someone happens to add an early return and it will also help in the case of a panic.

## Additional Info

NA
2020-10-05 17:33:25 +11:00
Age Manning
fe07a3c21c
Improve error handling in network processing (#1654)
* Improve error handling in network processing

* Cargo fmt

* Cargo fmt

* Improve error handling for prior genesis

* Remove dep
2020-10-05 17:30:43 +11:00
Age Manning
47c921f326 Update libp2p (#1728)
## Issue Addressed

N/A

## Proposed Changes

Updates the libp2p dependency to the latest version

## Additional Info

N/A
2020-10-05 05:16:27 +00:00
divma
b1c121b880 From panic to crit (#1726)
## Issue Addressed
Downgrade inconsistent chain segment states from `panic` to `crit`. I don't love this solution but since range can always bounce back from any of those, we don't panic.

Co-authored-by: Age Manning <Age@AgeManning.com>
2020-10-05 04:02:09 +00:00
Justin
39bd511838 Document need for port 9000 to be open (fix #730) (#731)
Co-authored-by: Age Manning <Age@AgeManning.com>
2020-10-05 03:20:53 +00:00
Paul Hauner
cee3e6483a Tidy some TODOs (#1721)
## Issue Addressed

- Resolves #1705

## Proposed Changes

Cleans up some of my TODOs in the code base.

- Adds link to issue in this repo for BLST `unsafe` block.
- Confirms that the `nextaccount` field *is* required on an EIP-2386 wallet.
    - Reference: https://github.com/mcdee/EIPs/blob/master/EIPS/eip-2386.md#json-schema
- Removes TODO about Zeroize on bip39 that was resolved in #1701 
- Removes a TODO about an early randao reveal since we use the slot clock to generate the reveal: c4bd9c86e6/validator_client/src/block_service.rs (L212-L220)

## Additional Info

NA
2020-10-05 00:39:30 +00:00
Age Manning
6b68c628df Increase content-id length (#1725)
## Issue Addressed

N/A

## Proposed Changes

Increase gossipsub's content-id length to the full 32 byte hash. 

## Additional Info

N/A
2020-10-04 23:49:16 +00:00
divma
86a18e72c4 Sync fixes (#1716)
## Issue Addressed

chain state inconsistencies

## Proposed Changes
- a batch can be fake-failed by Range if it needs to move a peer to another chain. The peer will still send blocks/ errors / produce timeouts for those  requests, so check when we get a response from the RPC that the request id matches, instead of only the peer, since a re-request can be directed to the same peer.
- if an optimistic batch succeeds, store the attempt to avoid trying it again when quickly switching chains. Also, use it only if ahead of our current target, instead of the segment's start epoch
2020-10-04 23:49:14 +00:00
divma
e3c7b58657 Address a couple of TODOs (#1724)
## Issue Addressed
couple of TODOs
2020-10-04 22:50:44 +00:00
Paul Hauner
d72c026d32 Use Drop impl to send worker idle message (#1718)
## Issue Addressed

NA

## Proposed Changes

Uses a `Drop` implementation to help ensure that `BeaconProcessor` workers are freed. This will help prevent against regression, if someone happens to add an early return and it will also help in the case of a panic.

## Additional Info

NA
2020-10-04 21:59:20 +00:00
Paul Hauner
c4bd9c86e6
Add check for head/target consistency (#1702)
## Issue Addressed

NA

## Proposed Changes

Addresses an interesting DoS vector raised by @protolambda by verifying that the head and target are consistent when processing aggregate attestations. This check prevents us from loading very old target blocks and doing lots of work to skip them to the current slot.

## Additional Info

NA
2020-10-03 10:08:06 +10:00
Sean
6af3bc9ce2
Add UPnP support for Lighthouse (#1587)
This commit was modified by Paul H whilst rebasing master onto
v0.3.0-staging

Adding UPnP support will help grow the DHT by allowing NAT traversal for peers with UPnP supported routers.

Using IGD library: https://docs.rs/igd/0.10.0/igd/

Adding the  the libp2p tcp port and discovery udp port. If this fails it simply logs the attempt and moves on

Co-authored-by: Age Manning <Age@AgeManning.com>
2020-10-03 10:07:47 +10:00
Geoffry Song
8fde9a4016
Wallet creation: Make mnemonic length configurable, default to 24 words. (#1697)
## Issue Addressed

Fixes #1665.

## Proposed Changes

`lighthouse account_manager wallet create` now generates a 24-word
mnemonic. The user can override this by passing `--mnemonic-length 12`
(or another legal bip39 length).

## Additional Info

CLI `--help`:
```
        --mnemonic-length <MNEMONIC_LENGTH>       The number of words to use for the mnemonic phrase. [default: 24]
```

In case of an invalid argument:
```
% lighthouse account_manager wallet create --mnemonic-length 25
error: Invalid value for '--mnemonic-length <MNEMONIC_LENGTH>': Mnemonic length must be one of 12, 15, 18, 21, 24
```
2020-10-03 10:01:06 +10:00
realbigsean
17c5da478e
Update tiny-bip39 dependency to one implementing zeroize (#1701)
## Issue Addressed

Resolves #1130

## Proposed Changes

Use the sigp fork of tiny-bip39, which includes `Zeroize` for `Mnemonic` and `Seed`

## Additional Info
N/A
2020-10-03 10:00:58 +10:00
realbigsean
255cc25623
Weak subjectivity start from genesis (#1675)
This commit was edited by Paul H when rebasing from master to
v0.3.0-staging.

Solution 2 proposed here: https://github.com/sigp/lighthouse/issues/1435#issuecomment-692317639

- Adds an optional `--wss-checkpoint` flag that takes a string `root:epoch`
- Verify that the given checkpoint exists in the chain, or that the the chain syncs through this checkpoint. If not, shutdown and prompt the user to purge state before restarting.

Co-authored-by: Paul Hauner <paul@paulhauner.com>
2020-10-03 10:00:28 +10:00
Paul Hauner
32338bcafa Add check for head/target consistency (#1702)
## Issue Addressed

NA

## Proposed Changes

Addresses an interesting DoS vector raised by @protolambda by verifying that the head and target are consistent when processing aggregate attestations. This check prevents us from loading very old target blocks and doing lots of work to skip them to the current slot.

## Additional Info

NA
2020-10-02 10:46:37 +00:00
Paul Hauner
6ea3bc5e52 Implement VC API (#1657)
## Issue Addressed

NA

## Proposed Changes

- Implements a HTTP API for the validator client.
- Creates EIP-2335 keystores with an empty `description` field, instead of a missing `description` field. Adds option to set name.
- Be more graceful with setups without any validators (yet)
    - Remove an error log when there are no validators.
    - Create the `validator` dir if it doesn't exist.
- Allow building a `ValidatorDir` without a withdrawal keystore (required for the API method where we only post a voting keystore).
- Add optional `description` field to `validator_definitions.yml`

## TODO

- [x] Signature header, as per https://github.com/sigp/lighthouse/issues/1269#issuecomment-649879855
- [x] Return validator descriptions
- [x] Return deposit data
- [x] Respect the mnemonic offset
- [x] Check that mnemonic can derive returned keys
- [x] Be strict about non-localhost
- [x] Allow graceful start without any validators (+ create validator dir)
- [x] Docs final pass
- [x] Swap to EIP-2335 description field. 
- [x] Fix Zerioze TODO in VC api types.
- [x] Zeroize secp256k1 key

## Endpoints

- [x] `GET /lighthouse/version`
- [x] `GET /lighthouse/health`
- [x] `GET /lighthouse/validators` 
- [x] `POST /lighthouse/validators/hd`
- [x] `POST /lighthouse/validators/keystore`
- [x] `PATCH /lighthouse/validators/:validator_pubkey`
- [ ] ~~`POST /lighthouse/validators/:validator_pubkey/exit/:epoch`~~ Future works


## Additional Info

TBC
2020-10-02 09:42:19 +00:00
Sean
94b17ce02b Add UPnP support for Lighthouse (#1587)
Adding UPnP support will help grow the DHT by allowing NAT traversal for peers with UPnP supported routers.

## Issue Addressed

#927 

## Proposed Changes

Using IGD library: https://docs.rs/igd/0.10.0/igd/

Adding the  the libp2p tcp port and discovery udp port. If this fails it simply logs the attempt and moves on

## Additional Info



Co-authored-by: Age Manning <Age@AgeManning.com>
2020-10-02 08:47:00 +00:00