Commit Graph

3756 Commits

Author SHA1 Message Date
Age Manning
c00e6c2c6f Small network adjustments (#1884)
## Issue Addressed

- Asymmetric pings - Currently with symmetric ping intervals, lighthouse nodes race each other to ping often ending in simultaneous ping connections. This shifts the ping interval to be asymmetric based on inbound/outbound connections
- Correct inbound/outbound peer-db registering - It appears we were accounting inbound as outbound and vice versa in the peerdb, this has been corrected
- Improved logging

There is likely more to come - I'll leave this open as we investigate further testnets
2020-11-13 06:06:33 +00:00
Paul Hauner
8772c02fa0 Reduce temp allocations in network metrics (#1895)
## Issue Addressed

Using `heaptrack` I could see that ~75% of Lighthouse temporary allocations are caused by temporary string allocations here.

## Proposed Changes

Reduces temporary `String` allocations when updating metrics in the `network` crate. The solution isn't perfect since we rebuild our caches with each call, but it's a significant improvement.

## Additional Info

NA
2020-11-13 04:19:38 +00:00
blacktemplar
c7ac967d5a handle peer state transitions on gossipsub score changes + refactoring (#1892)
## Issue Addressed

NA

## Proposed Changes

Correctly handles peer state transitions on gossipsub changes + refactors handling of peer state transitions into one function used for lighthouse score changes and gossipsub score changes.


Co-authored-by: Age Manning <Age@AgeManning.com>
2020-11-13 03:15:03 +00:00
realbigsean
cb26c15eb6 Peer endpoint updates (#1893)
## Issue Addressed

N/A

## Proposed Changes

- rename `address` -> `last_seen_p2p_address`
- state and direction filters for `peers` endpoint
- metadata count addition to `peers` endpoint
- add `peer_count` endpoint


Co-authored-by: realbigsean <seananderson33@gmail.com>
2020-11-13 02:02:41 +00:00
blacktemplar
fcb4893f72 do subnet discoveries until we have MESH_N_LOW many peers (#1886)
## Issue Addressed

NA

## Proposed Changes

Increases the target peers for a subnet, so that subnet queries are executed until we have at least the minimum required peers for a mesh (`MESH_N_LOW`). We keep the limit of `6` target peers for aggregated subnet discovery queries, therefore the size (and the time needed) for a query doesn't change.
2020-11-13 00:56:05 +00:00
Michael Sproul
11076912d9 Update EF tests to 1.0.0 (#1875)
## Proposed Changes

Bump the EF tests from `1.0.0-rc.0` to `1.0.0`

## Additional Info

Builds on #1862
2020-11-12 23:52:38 +00:00
blacktemplar
7404f1ce54 Gossipsub scoring (#1668)
## Issue Addressed

#1606 

## Proposed Changes

Uses dynamic gossipsub scoring parameters depending on the number of active validators as specified in https://gist.github.com/blacktemplar/5c1862cb3f0e32a1a7fb0b25e79e6e2c.

## Additional Info

Although the parameters got tested on Medalla, extensive testing using simulations on larger networks is still to be done and we expect that we need to change the parameters, although this might only affect constants within the dynamic parameter framework.
2020-11-12 01:48:28 +00:00
realbigsean
f0c9339153 Update tiny-bip39 dependency (#1887)
## Issue Addressed

Resolves #1704

## Proposed Changes

Update tiny-bip39 from using the sigp fork to the newly released v0.8.0 in the upstream.



Co-authored-by: realbigsean <seananderson33@gmail.com>
2020-11-12 00:46:33 +00:00
Paul Hauner
9ee71d6fec Add toledo support (#1874)
## Issue Addressed

NA

## Proposed Changes

Adds support for the [Toledo](https://github.com/protolambda/toledo) dev-net.

```bash
lighthouse --testnet toledo bn --http
```

This is for development only, we do not recommend users to join this testnet.

## Additional Info

- ~~Blocked on #1862~~
2020-11-10 22:42:59 +00:00
Age Manning
5ed4c1daca Update vc testnet script (#1859)
Updates the local testnet VC scripts to match current functionality.
2020-11-10 02:36:14 +00:00
realbigsean
f8da151b0b Standard beacon api updates (#1831)
## Issue Addressed

Resolves #1809 
Resolves #1824
Resolves #1818
Resolves #1828 (hopefully)

## Proposed Changes

- add `validator_index` to the proposer duties endpoint
- add the ability to query for historical proposer duties
- `StateId` deserialization now fails with a 400 warp rejection
- add the `validator_balances` endpoint
- update the `aggregate_and_proofs` endpoint to accept an array
- updates the attester duties endpoint from a `GET` to a `POST`
- reduces the number of times we query for proposer duties from once per slot per validator to only once per slot 


Co-authored-by: realbigsean <seananderson33@gmail.com>
Co-authored-by: Paul Hauner <paul@paulhauner.com>
2020-11-09 23:13:56 +00:00
Michael Sproul
556190ff46 Compact database on finalization (#1871)
## Issue Addressed

Closes #1866

## Proposed Changes

* Compact the database on finalization. This removes the deleted states from disk completely. Because it happens in the background migrator, it doesn't block other database operations while it runs. On my Medalla node it took about 1 minute and shrank the database from 90GB to 9GB.
* Fix an inefficiency in the pruning algorithm where it would always use the genesis checkpoint as the `old_finalized_checkpoint` when running for the first time after start-up. This would result in loading lots of states one-at-a-time back to genesis, and storing a lot of block roots in memory. The new code stores the old finalized checkpoint on disk and only uses genesis if no checkpoint is already stored. This makes it both backwards compatible _and_ forwards compatible -- no schema change required!
* Introduce two new `INFO` logs to indicate when pruning has started and completed. Users seem to want to know this information without enabling debug logs!
2020-11-09 07:02:21 +00:00
blacktemplar
b711cfe2bb Improve validator key cache lock handling (#1837)
## Issue Addressed

NA

## Proposed Changes

Improves the deletion of the validator key cache lock file in case of program interrupts.

## Additional Info

This should reduce cases where a lock file doesn't get removed on shutdown and reduce complaints on Discord. This will be superseded by issue #1823.
2020-11-09 06:03:35 +00:00
Paul Hauner
2f9999752e Add --testnet mainnet and start HTTP server before genesis (#1862)
## Issue Addressed

NA

## Proposed Changes

- Adds support for `--testnet mainnet`
- Start HTTP server prior to genesis

## Additional Info

**Note: This is an incomplete work-in-progress. Use Lighthouse for mainnet at your own risk.**

With this PR, you can check the deposits:

```bash
lighthouse --testnet mainnet bn --http
```
```bash
curl localhost:5052/lighthouse/eth1/deposit_cache | jq
```

```json
{
  "data": [
    {
      "deposit_data": {
        "pubkey": "0x854980aa9bf2e84723e1fa6ef682e3537257984cc9cb1daea2ce6b268084b414f0bb43206e9fa6fd7a202357d6eb2b0d",
        "withdrawal_credentials": "0x00cacf703c658b802d55baa2a5c1777500ef5051fc084330d2761bcb6ab6182b",
        "amount": "32000000000",
        "signature": "0xace226cdfd9da6b1d827c3a6ab93f91f53e8e090eb6ca5ee7c7c5fe3acc75558240ca9291684a2a7af5cac67f0558d1109cc95309f5cdf8c125185ec9dcd22635f900d791316924aed7c40cff2ffccdac0d44cf496853db678c8c53745b3545b"
      },
      "block_number": 3492981,
      "index": 0,
      "signature_is_valid": true
    },
    {
      "deposit_data": {
        "pubkey": "0x93da03a71bc4ed163c2f91c8a54ea3ba2461383dd615388fd494670f8ce571b46e698fc8d04b49e4a8ffe653f581806b",
        "withdrawal_credentials": "0x006ebfbb7c8269a78018c8b810492979561d0404d74ba9c234650baa7524dcc4",
        "amount": "32000000000",
        "signature": "0x8d1f4a1683f798a76effcc6e2cdb8c3eed5a79123d201c5ecd4ab91f768a03c30885455b8a952aeec3c02110457f97ae0a60724187b6d4129d7c352f2e1ac19b4210daacd892fe4629ad3260ce2911dceae3890b04ed28267b2d8cb831f6a92d"
      },
      "block_number": 3493427,
      "index": 1,
      "signature_is_valid": true
    },
```
2020-11-09 05:04:03 +00:00
Michael Sproul
b3fc48e887 Update slashing protection interchange to v5 (#1816)
## Proposed Changes

Update the slashing protection interchange format to v5 in preparation for finalisation as part of an EIP.

Also, add some more tests and update the commit hash for https://github.com/eth2-clients/slashing-protection-interchange-tests to include the new generated tests.
2020-11-09 05:04:01 +00:00
divma
b0e9e3dcef Seen addresses store port (#1841)
## Issue Addressed
#1764
2020-11-09 04:01:03 +00:00
Geoffry Song
63fe5542e7 Remove mention of OpenSSL from documentation (#1844)
If I'm not mistaken, openssl is no longer a dependency of lighthouse, so it can no longer cause build issues.
2020-11-09 02:31:31 +00:00
Marius Kjærstad
3574bad6cd Changed http:// to https:// on some links (#1869)
Changed http:// to https:// on some links in README.md
2020-11-09 01:28:39 +00:00
Herman Junge
78744cd07a Update remote signer README (#1870)
Forgot to update the executable. Also fix to the roadmap.
2020-11-07 03:06:17 +00:00
Herman Junge
492ce07ed3 Update README.md (#1868)
Just one line of doc
2020-11-06 17:23:43 +00:00
Herman Junge
e004b98eab [Remote signer] Fold signer into Lighthouse repository (#1852)
The remote signer relies on the `types` and `crypto/bls` crates from Lighthouse. Moreover, a number of tests of the remote signer consumption of LH leverages this very signer, making any important update a potential dependency nightmare.

Co-authored-by: Paul Hauner <paul@paulhauner.com>
2020-11-06 06:17:11 +00:00
Age Manning
e2ae5010a6 Update libp2p (#1865)
Updates libp2p to the latest version. 

This adds tokio 0.3 support and brings back yamux support. 

This also updates some discv5 configuration parameters for leaner discovery queries
2020-11-06 04:14:14 +00:00
Herman Junge
4c4dad9fb5
Fix fn documentation 2020-11-05 17:53:35 +00:00
Paul Hauner
157e31027a Add warnings for deposits (#1858)
## Issue Addressed

NA

## Proposed Changes

Add some warnings to discourage users to user Lighthouse for mainnet.

## Additional Info

NA
2020-11-04 19:46:42 +00:00
blacktemplar
7e7fad5734 Ignore RPC messages of disconnected peers and remove old peers based on disconnection time (#1854)
## Issue Addressed

NA

## Proposed Changes

Lets the networking behavior ignore messages of peers that are not connected. Furthermore, old peers are not removed from the peerdb based on score anymore but based on the disconnection time.
2020-11-03 23:43:10 +00:00
Age Manning
0a0f4daf9d Prevent errors for stream termination race (#1853)
Prevents an error being propagated on a race condition for RPC stream termination
2020-11-03 10:37:00 +00:00
Paul Hauner
0cde4e285c Bump version to v0.3.3 (#1850)
## Issue Addressed

NA

## Proposed Changes

- Update versions
- Run `cargo update`

## Additional Info

- Blocked on #1846
2020-11-02 23:55:15 +00:00
Michael Sproul
2ff5828310 Downgrade ADX check to a warning (#1846)
## Issue Addressed

Closes #1842

## Proposed Changes

Due to the lies told to us by VPS providers about what CPU features they support, we are forced to check for the availability of CPU features like ADX by just _running code and seeing if it crashes_. The prominent warning should hopefully help users who have truly incompatible CPUs work out what is going on, while not burdening users of cheap VPSs.
2020-11-02 22:35:37 +00:00
Pawan Dhananjay
863ee7c9f2 Update to discv5 bootnodes (#1849)
## Issue Addressed

We seem to have roll backed to old discv5 bootnodes with #1799 because of which fresh nodes with no cached peers cannot find any peers.

## Proposed Changes

Updates `boot_enr.yaml` to discv5.1 bootnodes.
2020-11-02 21:29:43 +00:00
Paul Hauner
7afbaa807e Return eth1-related data via the API (#1797)
## Issue Addressed

- Related to #1691

## Proposed Changes

Adds the following API endpoints:

- `GET lighthouse/eth1/syncing`: status about how synced we are with Eth1.
- `GET lighthouse/eth1/block_cache`: all locally cached eth1 blocks.
- `GET lighthouse/eth1/deposit_cache`: all locally cached eth1 deposits.

Additionally:

- Moves some types from the `beacon_node/eth1` to the `common/eth2` crate, so they can be used in the API without duplication.
- Allow `update_deposit_cache` and `update_block_cache` to take an optional head block number to avoid duplicate requests.

## Additional Info

TBC
2020-11-02 00:37:30 +00:00
divma
6c0c050fbb Tweak head syncing (#1845)
## Issue Addressed

Fixes head syncing

## Proposed Changes

- Get back to statusing peers after removing chain segments and making the peer manager deal with status according to the Sync status, preventing an old known deadlock
- Also a bug where a chain would get removed if the optimistic batch succeeds being empty

## Additional Info

Tested on Medalla and looking good
2020-11-01 23:37:39 +00:00
realbigsean
304793a6ab add quoted serialization util for FixedVector and VariableList (#1794)
## Issue Addressed

This comment: https://github.com/sigp/lighthouse/issues/1776#issuecomment-712349841

## Proposed Changes

- Add quoted serde utils for `FixedVector` and `VariableList`
- Had to remove the dependency that `ssz_types` has on `serde_utils` to avoid a circular dependency.

## Additional Info


Co-authored-by: realbigsean <seananderson33@gmail.com>
2020-10-29 23:25:21 +00:00
Pawan Dhananjay
56f9394141 Add cli option for voluntary exits (#1781)
## Issue Addressed

Resolve #1652 

## Proposed Changes

Adds a cli option for voluntary exits. The flow is similar to prysm's where after entering the password for the validator keystore (or load password from `secrets` if present) the user is given multiple warnings about the operation being irreversible, then redirected to the docs webpage(not added yet) which explains what a voluntary exit is and the consequences of exiting and then prompted to enter a phrase from the docs webpage as a final confirmation. 

Example usage
```
$ lighthouse --testnet zinken account validator exit --validator <validator-pubkey> --beacon-node http://localhost:5052

Running account manager for zinken testnet                                                                                                          
validator-dir path: "..."

Enter the keystore password:  for validator in ...

Password is correct

Publishing a voluntary exit for validator: ...              
WARNING: This is an irreversible operation                                                                                                                    
WARNING: Withdrawing staked eth will not be possible until Eth1/Eth2 merge Please visit [website] to make sure you understand the implications of a voluntary exit.            
                                                                                                                                             
Enter the phrase from the above URL to confirm the voluntary exit:
Exit my validator
Published voluntary exit for validator ...
```

## Additional info

Not sure if we should have batch exits (`--validator all`) option for exiting all the validators in the `validators` directory. I'm slightly leaning towards having only single exits but don't have a strong preference.
2020-10-29 23:25:19 +00:00
Paul Hauner
f64f8246db Only run http_api tests in release (#1827)
## Issue Addressed

NA

## Proposed Changes

As raised by @hermanjunge in a DM, the `http_api` tests have been observed taking 100+ minutes on debug. This PR:

- Moves the `http_api` tests to only run in release.
- Groups some `http_api` tests to reduce test-setup overhead.

## Additional Info

NA
2020-10-29 22:25:20 +00:00
realbigsean
ae0f025375 Beacon state validator id filter (#1803)
## Issue Addressed

Michael's comment here: https://github.com/sigp/lighthouse/issues/1434#issuecomment-708834079
Resolves #1808

## Proposed Changes

- Add query param `id` and `status` to the `validators` endpoint
- Add string serialization and deserialization for `ValidatorStatus`
- Drop `Epoch` from `ValidatorStatus` variants

## Additional Info

Please provide any additional information. For example, future considerations
or information useful for reviewers.
2020-10-29 05:13:04 +00:00
divma
9f45ac2f5e More sync edge cases + prettify range (#1834)
## Issue Addressed
Sync edge case when we get an empty optimistic batch that passes validation and is inside the download buffer. Eventually the chain would reach the batch and treat it as an ugly state. 

## Proposed Changes
- Handle the edge case advancing the chain's target + code clarification
- Some largey changes for readability + ergonomics since rust has try ops
- Better handling of bad batch and chain states
2020-10-29 02:29:24 +00:00
blacktemplar
2bd5b9182f fix unbanning of peers (#1838)
## Issue Addressed

NA

## Proposed Changes

Currently a banned peer will remain banned indefinitely as long as update is called on the score struct regularly. This fixes this bug and the score decay starts after `BANNED_BEFORE_DECAY` seconds after banning.
2020-10-29 01:25:02 +00:00
Michael Sproul
36bd4d87f0 Update to spec v1.0.0-rc.0 and BLSv4 (#1765)
## Issue Addressed

Closes #1504 
Closes #1505
Replaces #1703
Closes #1707

## Proposed Changes

* Update BLST and Milagro to versions compatible with BLSv4 spec
* Update Lighthouse to spec v1.0.0-rc.0, and update EF test vectors
* Use the v1.0.0 constants for `MainnetEthSpec`.
* Rename `InteropEthSpec` -> `V012LegacyEthSpec`
    * Change all constants to suit the mainnet `v0.12.3` specification (i.e., Medalla).
* Deprecate the `--spec` flag for the `lighthouse` binary
    * This value is now obtained from the `config_name` field of the `YamlConfig`.
        * Built in testnet YAML files have been updated.
    * Ignore the `--spec` value, if supplied, log a warning that it will be deprecated
    * `lcli` still has the spec flag, that's fine because it's dev tooling.
* Remove the `E: EthSpec` from `YamlConfig`
    * This means we need to deser the genesis `BeaconState` on-demand, but this is fine.
* Swap the old "minimal", "mainnet" strings over to the new `EthSpecId` enum.
* Always require a `CONFIG_NAME` field in `YamlConfig` (it used to have a default).

## Additional Info

Lots of breaking changes, do not merge! ~~We will likely need a Lighthouse v0.4.0 branch, and possibly a long-term v0.3.0 branch to keep Medalla alive~~.

Co-authored-by: Kirk Baird <baird.k@outlook.com>
Co-authored-by: Paul Hauner <paul@paulhauner.com>
2020-10-28 22:19:38 +00:00
divma
ad846ad280 Inform peers of requests that exceed the maximum rate limit + log downgrade (#1830)
## Issue Addressed

#1825 

## Proposed Changes

Since we penalize more blocks by range requests that have large steps, it is possible to get requests that will never be processed. We were not informing peers about this requests and also logging CRIT that is no longer relevant. Later we should check if more sophisticated handling for those requests is needed
2020-10-27 11:46:38 +00:00
Paul Hauner
92c8eba8ca Ensure eth1 deposit/chain IDs are used from YamlConfig (#1829)
## Issue Addressed

 NA

## Proposed Changes

Fixes a bug which causes the node to reject valid eth1 nodes.

- Fix core bug: failure to apply `YamlConfig` values to `ChainSpec`.
- Add a test to prevent regression in this specific case.
- Fix an invalid log message

## Additional Info

NA
2020-10-26 03:34:14 +00:00
Paul Hauner
f157d61cc7 Address clippy lints, panic in ssz_derive on overflow (#1714)
## Issue Addressed

NA

## Proposed Changes

- Panic or return error if we overflow `usize` in SSZ decoding/encoding derive macros.
  - I claim that the panics can only be triggered by a faulty type definition in lighthouse, they cannot be triggered externally on a validly defined struct.
- Use `Ordering` instead of some `if` statements, as demanded by clippy.
- Remove some old clippy `allow` that seem to no longer be required.
- Add comments to interesting clippy statements that we're going to continue to ignore.
- Create #1713

## Additional Info

NA
2020-10-25 23:27:39 +00:00
Paul Hauner
eba51f0973 Update testnet configs, change on-disk format (#1799)
## Issue Addressed

- Related to #1691

## Proposed Changes

- Add `DEPOSIT_CHAIN_ID` and `DEPOSIT_NETWORK_ID` to `config.yaml`.
    - Pass the `DEPOSIT_NETWORK_ID` to the `eth1::Service`.
- Remove the unused `MAX_EPOCHS_PER_CROSSLINK` from the `altona` and `medalla` configs (see [spec commit](2befe90032 (diff-efb845ac2ebd4aafbc23df40f47ce25699255064e99d36d0406d0a14ca7953ec))).
- Change from compressing the whole testnet directory, to only compressing the genesis state file. This is the only file we need to compress and *not* compressing the others makes them work nicely with git.
    - We can modify the boot nodes, configs, etc. without incurring an eternal binary-blob cost on our git history.
    - This change is backwards compatible (i.e., non-breaking).

## Additional Info

NA
2020-10-25 22:15:46 +00:00
Age Manning
7453f39d68 Prevent unbanning of disconnected peers (#1822)
## Issue Addressed

Further testing revealed another edge case where we attempt to unban a peer that can be in a disconnected start. Although this causes no real issue, it does log an error to the user. 

This PR adds a check to prevent this edge case and prevents the error being logged to the user.
2020-10-24 05:24:20 +00:00
Age Manning
a3cc1a1e0f Call unban only when necessary (#1821)
This PR prevents a user-facing error. 

It prevents optimistically unbanning a peer and instead checks the state of the peer before requesting the peers state to be unbanned.
2020-10-24 03:24:19 +00:00
blacktemplar
1644289a08 Updates the libp2p to the second newest commit => Allow only one topic per message (#1819)
As @AgeManning mentioned the newest libp2p version had some problems and got downgraded again on lighthouse master. This is an intermediate version that makes no problems and only adds a small change of allowing only one topic per message.
2020-10-24 01:05:37 +00:00
Age Manning
7870b81ade Downgrade libp2p (#1817)
## Description

This downgrades the recent libp2p upgrade. 

There were issues with the RPC which prevented syncing of the chain and this upgrade needs to be further investigated.
2020-10-23 09:33:59 +00:00
Paul Hauner
fa2daa7d6c Update readme, add banner (#1814)
## Issue Addressed

NA

## Proposed Changes

- Update progress timeline
- Remove the qualification that the eth2 spec is "emerging".
- Remove the terminal animation, replace with new banner.

## Additional Info

NA
2020-10-23 04:16:38 +00:00
Age Manning
55eee18ebb Version bump to 0.3.1 (#1813)
## Description

Bumps Lighthouse to version 0.3.1.
2020-10-23 04:16:36 +00:00
Age Manning
64c5899d25 Adds colour help to bn and vc subcommands (#1811)
Adds coloured help to the bn and vc subcommands
2020-10-23 04:16:34 +00:00
Age Manning
2c7f362908 Discovery v5.1 (#1786)
## Overview 

This updates lighthouse to discovery v5.1

Note: This makes lighthouse's discovery not compatible with any previous version. Lighthouse cannot discover peers or send/receive ENR's from any previous version. This is a breaking change. 

This resolves #1605
2020-10-23 04:16:33 +00:00