Commit Graph

3988 Commits

Author SHA1 Message Date
Pawan Dhananjay
7933596c89 Add a purge-eth1-cache cli option (#2039)
## Issue

Some eth1 clients are missing deposit logs on mainnet for multiple reasons (not fully synced, eth1 client issues) because of which we are getting `FailedToInsertDeposit` errors.
Ideally, LH should pick up where it left off after pointing it to a nice eth1 client endpoint (which has all deposits). 

However, I have seen instances where LH keeps getting `FailedToInsertDeposit` even after switching to a good endpoint. Only deleting the beacon directory (which also wipes the eth1 cache) and resyncing the eth1 caches seems to be the solution. This wouldn't be great for mainnet if you have to sync your beacon node again as well.

## Proposed Changes

Add a `--purge-eth1-db` option which just wipes the eth1 cache and doesn't touch the rest of the beacon db. 
Still need to investigate if and why LH isn't picking up where it left off for the deposit logs sync, but I think it would be good to have an option to just delete eth1 caches regardless.
2020-12-04 05:03:28 +00:00
realbigsean
fdfb81a74a Server sent events (#1920)
## Issue Addressed

Resolves #1434 (this is the last major feature in the standard spec. There are only a couple of places we may be off-spec due to recent spec changes or ongoing discussion)
Partly addresses #1669
 
## Proposed Changes

- remove the websocket server
- remove the `TeeEventHandler` and `NullEventHandler` 
- add server sent events according to the eth2 API spec

## Additional Info

This is according to the currently unmerged PR here: https://github.com/ethereum/eth2.0-APIs/pull/117


Co-authored-by: realbigsean <seananderson33@gmail.com>
2020-12-04 00:18:58 +00:00
realbigsean
2b5c0df9e5 Validators endpoint status code (#2040)
## Issue Addressed

Resolves #2035 

## Proposed Changes

Update 405's to 400's for failures when we are parsing path params.

## Additional Info

Haven't updated the same for non-standard endpoints

Co-authored-by: realbigsean <seananderson33@gmail.com>
2020-12-03 23:10:08 +00:00
Michael Sproul
e06d040b5d Update blst to 0.3.2 (#2034)
## Issue Addressed

Should resolve `blst` build issues that previously required `cargo clean` 🤞

## Proposed Changes

BLST cleaned up some of their validation logic: https://github.com/supranational/blst/compare/v0.3.1...v0.3.2

And included my build system PR: https://github.com/supranational/blst/pull/45
2020-12-03 22:07:16 +00:00
Age Manning
2682f46025 Fingerprint new client identify agent string (#2027)
Nimbus have modified their identify agent string. 

This PR adds their new agent string to identify new nimbus peers.
2020-12-03 22:07:14 +00:00
Michael Sproul
686b605112 Pretty-print EIP-3076 tests (#1977)
## Proposed Changes

* Pretty-print the EIP-3076 tests to match https://github.com/eth2-clients/slashing-protection-interchange-tests/pull/4
* Move the `curl` invocation that downloads the tests to the test executor, removing the build script (closes #1982)
2020-12-03 22:07:12 +00:00
Pawan Dhananjay
e1353088e0 Normalize keystore passwords (#1972)
## Issue Addressed

Resolves #1879 

## Proposed Changes

Do NFKD normalization for keystore passwords.
2020-12-03 22:07:09 +00:00
Pawan Dhananjay
482695142a Minor fixes (#2038)
Fixes a couple of low hanging fruits.

- Fixes #2037 
- `validators-dir` and `secrets-dir` flags don't really need to depend upon each other
- Fixes #2006 and Fixes #1995
2020-12-03 01:10:28 +00:00
blacktemplar
d8cda2d86e Fix new clippy lints (#2036)
## Issue Addressed

NA

## Proposed Changes

Fixes new clippy lints in the whole project (mainly [manual_strip](https://rust-lang.github.io/rust-clippy/master/index.html#manual_strip) and [unnecessary_lazy_evaluations](https://rust-lang.github.io/rust-clippy/master/index.html#unnecessary_lazy_evaluations)). Furthermore, removes `to_string()` calls on literals when used with the `?`-operator.
2020-12-03 01:10:26 +00:00
realbigsean
d3f0a21436 delete validator-dir path printing in subcommands (#2025)
## Issue Addressed

Resolves #2004

## Proposed Changes

Only print validator dir path once

## Additional Info

N/A

Co-authored-by: realbigsean <seananderson33@gmail.com>
2020-12-01 00:04:48 +00:00
Paul Hauner
b8bd80d2fb Add Content-Type to metrics server (#2019)
## Issue Addressed

- Resolves #2013

## Proposed Changes

Adds the `Content-Type text/plain` header as per #2013

## Additional Info

NA
2020-12-01 00:04:46 +00:00
Conor Svensson
075eecdcb1 Fix broken custom data directories link (#2000)
## Issue Addressed

No issue - its a broken link in the docs.

## Proposed Changes

Fix the broken link.

## Additional Info

N/A in this instance.
2020-12-01 00:04:44 +00:00
Paul Hauner
65dcdc361b Bump version to v1.0.3 (#2024)
## Issue Addressed

NA

## Proposed Changes

- Set version to `v1.0.3`
- Run cargo update

## Additional Info

- ~~Blocked on #2008~~
2020-11-30 22:55:10 +00:00
Age Manning
c718e81eaf Add privacy option (#2016)
Adds a `--privacy` CLI flag to the beacon node that users may opt into. 

This does two things:
- Removes client identifying information from the identify libp2p protocol
- Changes the default graffiti to "" if no graffiti is set.
2020-11-30 22:55:08 +00:00
Paul Hauner
77f3539654 Improve eth1 block sync (#2008)
## Issue Addressed

NA

## Proposed Changes

- Log about eth1 whilst waiting for genesis.
- For the block and deposit caches, update them after each download instead of when *all* downloads are complete.
  - This prevents the case where a single timeout error can cause us to drop *all* previously download blocks/deposits.
- Set `max_log_requests_per_update` to avoid timeouts due to very large log counts in a response.
- Set `max_blocks_per_update` to prevent a single update of the block cache to download an unreasonable number of blocks.
  - This shouldn't have any affect in normal use, it's just a safe-guard against bugs.
- Increase the timeout for eth1 calls from 15s to 60s, as per @pawanjay176's experience with Infura.

## Additional Info

NA
2020-11-30 20:29:17 +00:00
divma
8fcd22992c No string in slog (#2017)
## Issue Addressed

Following slog's documentation, this should help a bit with string allocations. I left it run for two days and mem usage is lower. This is of course anecdotal, but shouldn't harm anyway 

## Proposed Changes

remove `String` creation in logs when possible
2020-11-30 10:33:00 +00:00
Mehdi Zerouali
3f036fd193 Update PGP key in README (#1986)
## Proposed Changes

Update Sigma Prime's PGP key.
2020-11-30 09:28:54 +00:00
Paul Hauner
85e69249e6 Drop discovery log to trace (#2007)
## Issue Addressed

NA

## Proposed Changes

This was causing:

```
Nov 28 21:56:08.154 ERRO slog-async: logger dropped messages due to channel overflow, count: 44, service: libp2p
```

## Additional Info

NA
2020-11-29 03:02:23 +00:00
Age Manning
f7183098ee Bump to version v1.0.2 (#2001)
Update lighthouse to version `v1.0.2`. 

There are two major updates in this version:
- Updates to the task executor to tokio 0.3 and all sub-dependencies relying on core execution, including libp2p
- Update BLST
2020-11-28 13:22:37 +00:00
Justin
cadcc9a76b Fix possible typo in build from source instructions (#1990) 2020-11-28 06:41:34 +00:00
Sean Gulley
9a37f356a9 Update blst to official crate and incorporate subgroup changes (#1979)
## Issue Addressed

Move to latest official version of blst (v0.3.1).  Incorporate all the subgroup check API changes.

## Proposed Changes

Update Cargo.toml to use official blst crate 0.3.1
Modifications to blst.rs wrapper for subgroup check API changes

## Additional Info

The overall subgroup check methodology is public keys should be check for validity using key_validate() at time of first seeing them.  This will check for infinity and in group.  Those keys can then be cached for future usage.  All calls into blst set the pk_validate boolean to false to indicate there is no need for on the fly checking of public keys in the library.  Additionally the public keys are supposed to be validated for proof of possession outside of blst.

For signatures the subgroup check can be done at time of deserialization, prior to being used in aggregation or verification, or in the blst aggregation or verification functions themselves.  In the interface wrapper the call to subgroup_check has been left for one instance, although that could be moved into the 
verify_multiple_aggregate_signatures() call if wanted.  Checking beforehand does save some compute resources in the scenario a bad signature is received.  Elsewhere the subgroup check is being done inside the higher level operations.  See comments in the code.

All checks on signature are done for subgroup only.  There are no checks for infinity.  The rationale is an aggregate signature could technically equal infinity.  If any individual signature was infinity (invalid) then it would fail at time of verification.  A loss of compute resources, although safety would be preserved.
2020-11-28 06:41:32 +00:00
Age Manning
a567f788bd Upgrade to tokio 0.3 (#1839)
## Description

This PR updates Lighthouse to tokio 0.3. It includes a number of dependency updates and some structural changes as to how we create and spawn tasks.

This also brings with it a number of various improvements:

- Discv5 update
- Libp2p update
- Fix for recompilation issues
- Improved UPnP port mapping handling
- Futures dependency update
- Log downgrade to traces for rejecting peers when we've reached our max



Co-authored-by: blacktemplar <blacktemplar@a1.net>
2020-11-28 05:30:57 +00:00
Paul Hauner
5a3b94cbb4
Update to v1.0.1, run cargo update 2020-11-27 21:16:59 +11:00
blacktemplar
38b15deccb Fallback nodes for eth1 access (#1918)
## Issue Addressed

part of  #1883

## Proposed Changes

Adds a new cli argument `--eth1-endpoints` that can be used instead of `--eth1-endpoint` to specify a comma-separated list of endpoints. If the first endpoint returns an error for some request the other endpoints are tried in the given order.

## Additional Info

Currently if the first endpoint fails the fallbacks are used silently (except for `try_fallback_test_endpoint` that is used in `do_update` which logs a `WARN` for each endpoint that is not reachable). A question is if we should add more logs so that the user gets warned if his main endpoint is for example just slow and sometimes hits timeouts.
2020-11-27 08:37:44 +00:00
Michael Sproul
1312844f29 Disable snappy in LevelDB to fix build issues (#1983)
## Proposed Changes

A user on Discord reported build issues when trying to compile Lighthouse checked out to a path with spaces in it. I've fixed the issue upstream in `leveldb-sys` (https://github.com/skade/leveldb-sys/pull/22), but rather than waiting for a new release of the `leveldb` crate, we can also work around the issue by disabling Snappy in LevelDB, which we weren't using anyway.

This may also have the side-effect of slightly improving compilation times, as LevelDB+Snappy was found to be a substantial contributor to build time (although I'm not sure how much was LevelDB and how much was Snappy).
2020-11-27 03:01:57 +00:00
Pawan Dhananjay
0589a14afe Log better error message (#1981)
## Issue Addressed

Fixes #1965 

## Proposed Changes

Log an error and don't update eth1 caches if `chain_id = 0`
2020-11-26 23:13:46 +00:00
Michael Sproul
3486d6a809 Use OS file locks in validator client (#1958)
## Issue Addressed

Closes #1823

## Proposed Changes

* Use OS-level file locking for validator keystores, eliminating problems with lockfiles lingering after ungraceful shutdowns (`SIGKILL`, power outage). I'm using the `fs2` crate because it's cross-platform (unlike `file-lock`), and it seems to have the most downloads on crates.io.
* Deprecate + disable `--delete-lockfiles` CLI param, it's no longer necessary
* Delete the `validator_dir::Manager`, as it was mostly dead code and was only used in the `validator list` command, which has been rewritten to read the validator definitions YAML instead.

## Additional Info

Tested on:

- [x] Linux
- [x] macOS
- [x] Docker Linux
- [x] Docker macOS
- [ ] Windows
2020-11-26 11:25:46 +00:00
divma
fc07cc3fdf Sync metrics (#1975)
## Issue Addressed
- Add metrics to keep track of peer counts by sync type
- Add metric to keep track of the number of syncing chains in range

## Proposed Changes
Plugin to the network metrics update interval and update too the counts for peers wrt to their sync status with us

## Additional Info
For the peer counts
- By the way it is implemented the numbers won't always match to the total peer count in the `libp2p` metric.
- Updating the gauge with every change is messy because it requires to be updated on connection (in the `eth2_libp2p` crate, while metrics are defined in the `network` crate) on Goodbye sent (for an `IrrelevantPeer`) either in the `beacon_processor` or the `peer_manager`, and on disconnection. Since this is not a critical metric I think counting once every second is enough. If you think more accuracy is needed we can do it too, but it would be harder to maintain)

ATM those look like this
![image](https://user-images.githubusercontent.com/26765164/100275387-22137b00-2f60-11eb-93b9-94b0f265240c.png)
2020-11-26 05:23:17 +00:00
Paul Hauner
26741944b1 Add metrics to VC (#1954)
## Issue Addressed

NA

## Proposed Changes

- Adds a HTTP server to the VC which provides Prometheus metrics.
- Moves the health metrics into the `lighthouse_metrics` crate so it can be shared between BN/VC.
- Sprinkle some metrics around the VC.
- Update the book to indicate that we now have VC metrics.
- Shifts the "waiting for genesis" logic later in the `ProductionValidatorClient::new_from_cli`
  - This is worth attention during the review.

## Additional Info

- ~~`clippy` has some new lints that are failing. I'll deal with that in another PR.~~
2020-11-26 01:10:51 +00:00
SjonHortensius
50558e61f7 Fix #1964: remove mainnet warnings which no longer apply (#1970)
## Issue Addressed

#1964

## Proposed Changes

* remove two mainnet warnings
* reword `testnet` in logmessage
* update test
2020-11-25 23:56:21 +00:00
Age Manning
198c4a873d Update ENR construction and mainnet bootnodes (#1968)
## Issue Addressed

Boot nodes were being successfully created and publishing valid ENRs however the `eth2` field was not being saved to disk leading to a discrepancy between published ENR and disk ENR. 

If the `eth2` field is known, it is now constructed in the initial ENR and saved to disk. 

Previous mainnet bootnodes did not contain the `eth2` field and these have also been updated.
2020-11-25 22:48:07 +00:00
realbigsean
7b6a97e73c FAQ/Doc updates (#1966)
## Issue Addressed

N/A

## Proposed Changes

Adding a few FAQ's, updating some formatting


Co-authored-by: realbigsean <seananderson33@gmail.com>
2020-11-25 05:51:10 +00:00
Paul Hauner
7020f5df40 Update docs whenever unstable changes (#1969)
## Issue Addressed

NA

## Proposed Changes

Presently `master` is stable (and will be sunsetted) which means our docs only update after a release. This PR sets the docs to build on the `unstable` branch, which is equivalent to what what we've always had. 

## Additional Info

This does raise the question of whether or not docs should target `stable` or `unstable`, but I'd prefer to maintain current functionality and merge #1966 for now. I think having two versions might be handy, one for stable and one for unstable; I don't imagine this very difficult to achieve.
2020-11-25 03:20:23 +00:00
divma
3b4afc27bf Status race condition (#1967)
## Issue Addressed

Sync stalls due to race conditions between dc notifications and status processing
2020-11-25 02:15:38 +00:00
Paul Hauner
c6baa0eed1
Bump to v1.0.0, run cargo update 2020-11-25 02:02:19 +11:00
Age Manning
a96893744c
Update bootnodes and boot_node cli (#1961) 2020-11-25 02:01:37 +11:00
Paul Hauner
11c4968ea0
DO spec check before waiting for genesis (#1962) 2020-11-25 02:00:11 +11:00
Age Manning
b6eff50ffa
Add lighthouse boot nodes (#1960) 2020-11-25 00:05:53 +11:00
Paul Hauner
61277e3a72
Add mainnet genesis state (#1959)
* Add mainnet genesis state

* Add compressed, remove uncompressed
2020-11-24 23:21:00 +11:00
Mehdi Zerouali
ead6be074e Remove experimental software warning (#1957)
## Proposed Changes

Remove warning message on startup.
2020-11-24 10:29:41 +00:00
Mehdi Zerouali
011cea93b3 Update security details in README (#1956)
## Proposed Changes

Introduces a few minor changes to the README, mainly updating mentions about security.
2020-11-24 10:29:39 +00:00
Michael Sproul
20339ade01 Refine and test slashing protection semantics (#1885)
## Issue Addressed

Closes #1873

## Proposed Changes

Fixes the bug in slashing protection import (#1873) by pruning the database upon import.

Also expands the test generator to cover this case and a few others which are under discussion here:

https://ethereum-magicians.org/t/eip-3076-validator-client-interchange-format-slashing-protection/4883

## Additional Info

Depending on the outcome of the discussion on Eth Magicians, we can either wait for consensus before merging, or merge our preferred solution and patch things later.
2020-11-24 07:21:14 +00:00
Paul Hauner
84b3387d09 Add Prysm and Teku boot nodes (#1953)
## Issue Addressed

NA

## Proposed Changes

- Adds Prysm and Teku's boot nodes.

The boot ENR were collected from [this Prysm PR](https://github.com/prysmaticlabs/prysm/pull/7925/files#diff-c20494db2dc1354ad056bcacaa192681386854bf036fdeef375dfe57336f27a7R42).

## Additional Info

NA
2020-11-24 06:02:28 +00:00
Paul Hauner
e504645767 Update validator guide for mainnet (#1951)
## Issue Addressed

NA

## Proposed Changes

Updates the validator guide to provide instructions for mainnet users.

## Additional Info

- ~~Blocked on #1751~~
2020-11-24 04:42:17 +00:00
realbigsean
a171fb8843 check if the slashing protection database is locked before creating keys (#1949)
## Issue Addressed

Closes #1790

## Proposed Changes

Make a new method that creates an empty transaction with `TransactionBehavior::Exclusive` to check whether the slashing protection is locked. Call this method before attempting to create or import new validator keystores.  

## Additional Info

N/A


Co-authored-by: realbigsean <seananderson33@gmail.com>
2020-11-24 03:25:40 +00:00
divma
6f890c398e Sync Bug fixes (#1950)
## Issue Addressed

Two issues related to empty batches
- Chain target's was not being advanced when the batch was successful, empty and the chain didn't have an optimistic batch
- Not switching finalized chains. We now switch finalized chains requiring a minimum work first
2020-11-24 02:11:31 +00:00
Paul Hauner
21617aa87f Change --testnet flag to --network (#1751)
## Issue Addressed

- Resolves #1689

## Proposed Changes

TBC

## Additional Info

NA
2020-11-23 23:54:03 +00:00
Michael Sproul
7d644103c6 Tweak slasher DB schema and pruning (#1948)
## Issue Addressed

Resolves #1890

## Proposed Changes

Change the slasher database schema to key indexed attestations by `(target_epoch, indexed_attestation_root)` instead of just `indexed_attestation_root`. This allows more straight-forward pruning (linear scan), that is also "re-entrant". By re-entrant, we mean that a pruning pass that gets stuck because of a `MapFull` error can attempt to commit midway, and be resumed later without issue. The previous pruning strategy for indexed attestations did not have this property. There was also a flaw in the previous pruning that could leave "zombie" indexed attestations in the database (ones not referenced by any attester record), which could build up and contribute to bloat (although in practice I think they occur quite infrequently).

## Additional Info

During testing I noticed that a `MapFull` error can still occur during the commit of the transaction itself, which is irritating, but not unbearable. This PR should at least reduce the frequency with which users need to manually resize their DB, and if the `MapFull` on commit rears its ugly head too often we could use a dynamic strategy (temporarily increase the size of the map until the transaction commits).

The extra bytes for the epoch make the database a bit heavier, so the size estimate docs have been updated to reflect this. This is also a breaking schema change, so anyone using a v0 database from a few hours ago will need to drop it and update 😅
2020-11-23 21:33:51 +00:00
Michael Sproul
5828ff1204 Implement slasher (#1567)
This is an implementation of a slasher that lives inside the BN and can be enabled via `lighthouse bn --slasher`.

Features included in this PR:

- [x] Detection of attester slashing conditions (double votes, surrounds existing, surrounded by existing)
- [x] Integration into Lighthouse's attestation verification flow
- [x] Detection of proposer slashing conditions
- [x] Extraction of attestations from blocks as they are verified
- [x] Compression of chunks
- [x] Configurable history length
- [x] Pruning of old attestations and blocks
- [x] More tests

Future work:

* Focus on a slice of history separate from the most recent N epochs (e.g. epochs `current - K` to `current - M`)
* Run out-of-process
* Ingest attestations from the chain without a resync

Design notes are here https://hackmd.io/@sproul/HJSEklmPL
2020-11-23 03:43:22 +00:00
Paul Hauner
59b2247ab8 Improve UX whilst VC is waiting for genesis (#1915)
## Issue Addressed

- Resolves #1424

## Proposed Changes

Add a `GET lighthouse/staking` that returns 200 if the node is ready to stake (i.e., `--eth1` flag is present) or a 404 otherwise.

Whilst the VC is waiting for the genesis time to start (i.e., when the genesis state is known), check the `lighthouse/staking` endpoint and log an error if the node isn't configured for staking.

## Additional Info

NA
2020-11-23 01:00:22 +00:00