Commit Graph

438 Commits

Author SHA1 Message Date
Age Manning
31707ccf45 Shift author to sigma prime on some crates (#1440)
Shifts the author to sigma prime on some crates
2020-08-04 02:31:41 +00:00
Paul Hauner
142e033c34 Add Medalla genesis state, more boot enr (#1438)
## Issue Addressed

NA

## Proposed Changes

- Include the Medalla `genesis.ssz` file.
- Adds the boot nodes from here: 27e7e7cc87/medalla/bootnodes.txt

## Additional Info

NA
2020-08-03 07:49:24 +00:00
Paul Hauner
36d3d37cb4 Add support for multiple testnet flags (#1396)
## Issue Addressed

NA

## Proposed Changes

Allows for multiple "hardcoded" testnets.

## Additional Info

This PR is incomplete.

## TODO

- [x] Add flag to CLI, integrate with rest of Lighthouse.


Co-authored-by: Pawan Dhananjay <pawandhananjay@gmail.com>
Co-authored-by: Michael Sproul <michael@sigmaprime.io>
2020-07-29 06:39:29 +00:00
Michael Sproul
f53dedb27d Improve account manager CLI (#1404)
## Proposed Changes

Fixes some sharp edges on the new `lighthouse account validator list` command, and the account manager CLI.

* Validator names/keys are always printed in the same order due to the use of a sorted `BTreeMap`
* The `validator list` subcommand now respects the `--validator-dir` flag, instead of always looking in `~/.lighthouse/validators`
* The `--help` now shows a description for the `wallet` subcommand instead of just `TODO`
2020-07-29 04:32:52 +00:00
Paul Hauner
eaa9f9744f Add EF launchpad import (#1381)
## Issue Addressed

NA

## Proposed Changes

Adds an integration for keys generated via https://github.com/ethereum/eth2.0-deposit (In reality keys are *actually* generated here: https://github.com/ethereum/eth2.0-deposit-cli).

## Additional Info

NA

## TODO

- [x] Docs
- [x] Tests

Co-authored-by: Michael Sproul <michael@sigmaprime.io>
2020-07-29 04:32:50 +00:00
Paul Hauner
5680355b31 Fix naming of validators in CLI (#1332)
## Issue Addressed

NA

## Proposed Changes

- Adds a `lighthouse account validator list` command, to list all known validators.
- Fixes the validator names; previously they were using a full path (e.g., `"/home/paul/.lighthouse/validators/0x8ce25415d078bdc83133758604578ba51707a55965eeca8982f44695db7432d6ff1c23529020a971faa68ab60baf3118"` but now we only use the final directory name (e.g., `0x8ce25415d078bdc83133758604578ba51707a55965eeca8982f44695db7432d6ff1c23529020a971faa68ab60baf3118`).
2020-07-27 01:25:20 +00:00
Paul Hauner
b73c497be2 Support multiple BLS implementations (#1335)
## Issue Addressed

NA

## Proposed Changes

- Refactor the `bls` crate to support multiple BLS "backends" (e.g., milagro, blst, etc).
- Removes some duplicate, unused code in `common/rest_types/src/validator.rs`.
- Removes the old "upgrade legacy keypairs" functionality (these were unencrypted keys that haven't been supported for a few testnets, no one should be using them anymore).

## Additional Info

Most of the files changed are just inconsequential changes to function names.

## TODO

- [x] Optimization levels
- [x] Infinity point: https://github.com/supranational/blst/issues/11
- [x] Ensure milagro *and* blst are tested via CI
- [x] What to do with unsafe code?
- [x] Test infinity point in signature sets
2020-07-25 02:03:18 +00:00
blacktemplar
23a8f31f83 Fix clippy warnings (#1385)
## Issue Addressed

NA

## Proposed Changes

Fixes most clippy warnings and ignores the rest of them, see issue #1388.
2020-07-23 14:18:00 +00:00
Paul Hauner
e26da35cbf Introduce validator definition file for VC (#1357)
## Issue Addressed

NA

## Proposed Changes

- Introduces the `valdiator_definitions.yml` file which serves as an explicit list of validators that should be run by the validator client.
  - Removes `--strict` flag, split into `--strict-lockfiles` and `--disable-auto-discover`  
  - Adds a "Validator Management" page to the book.
- Adds the `common/account_utils` crate which contains some logic that was starting to duplicate across the codebase.

The new docs for this feature are the best description of it (apart from the code, I guess): 9cb87e93ce/book/src/validator-management.md

## API Changes

This change should be transparent for *most* existing users. If the `valdiator_definitions.yml` doesn't exist then it will be automatically generated using a method that will detect all the validators in their `validators_dir`.

Users will have issues if they are:

1. Using `--strict`.
1. Have keystores in their `~/.lighthouse/validators` directory that weren't being detected by the current keystore discovery method.

For users with (1), the VC will refuse to start because the `--strict` flag has been removed. They will be forced to review `--help` and choose an equivalent flag.

For users with (2), this seems fairly unlikely and since we're only in testnets there's no *real* value on the line here. I'm happy to take the risk, it would be a different case for mainnet.

## Additional Info

This PR adds functionality we will need for #1347.

## TODO

- [x] Reconsider flags
- [x] Move doc into a more reasonable chapter.
- [x] Check for compile warnings.
2020-07-22 09:34:55 +00:00
Michael Sproul
20a48df80a
Fix race condition in VC block proposal service (#1282)
Closes #918
Closes #923
2020-07-07 14:03:21 +10:00
Age Manning
5bc8fea2e0
Activate peer scoring (#1284)
* Initial score structure

* Peer manager update

* Updates to dialing

* Correct tests

* Correct typos and remove unused function

* Integrate scoring into the network crate

* Clean warnings

* Formatting

* Shift core functionality into the behaviour

* Temp commit

* Shift disconnections into the behaviour

* Temp commit

* Update libp2p and gossipsub

* Remove gossipsub lru cache

* Correct merge conflicts

* Modify handler and correct tests

* Update enr network globals on socket update

* Apply clippy lints

* Add new prysm fingerprint

* More clippy fixes
2020-07-07 10:13:16 +10:00
Paul Hauner
2856f5122d
Add checksum verification for deposit contract (#1330) 2020-07-06 11:56:44 +10:00
Paul Hauner
25cd91ce26
Update deps (#1322)
* Run cargo update

* Upgrade prometheus

* Update hex

* Upgrade parking-lot

* Upgrade num-bigint

* Upgrade sha2

* Update dockerfile Rust version

* Run cargo update
2020-07-06 11:55:56 +10:00
Paul Hauner
916a133043
Loose VC lockfile and slashing protection registers (#1314) 2020-06-29 21:04:07 +10:00
Paul Hauner
163fda2c26
Add more Altona boot ENR (#1310)
* Add other client boot enr

* Include lighthouse boot nodes
2020-06-29 16:51:26 +10:00
Paul Hauner
3953204727
Update to latest testnet dir (#1303) 2020-06-28 10:46:49 +10:00
Paul Hauner
38b9bf98ac
Update testnet-dir (#1301) 2020-06-26 15:05:49 +10:00
Michael Sproul
7688b5f1dd
Merge remote-tracking branch 'origin/master' into spec-v0.12 2020-06-26 12:57:56 +10:00
Paul Hauner
69e15af0b2
Bake in Altona (without genesis state) (#1295)
* Bake in Altona testnet (without genesis state)

* Update config.yml

* Fix eth2_testnet_config test

* Update min genesis time
2020-06-26 10:54:57 +10:00
pscott
02174e21d8
Fix clippy's performance lints (#1286)
* Fix clippy perf lints

* Cargo fmt

* Add  and  to lint rule in Makefile

* Fix some leftover clippy lints
2020-06-26 00:04:08 +10:00
Paul Hauner
decea48c78
Merge branch 'master' into spec-v0.12 2020-06-21 10:33:02 +10:00
Age Manning
f3380c00b8
Discovery metrics (#1276)
* Silky smooth squash

* Add discovery metrics

* Fix gauge metric, increase discv5 sessions

* Formatting
2020-06-19 15:36:03 +10:00
Michael Sproul
305724770d
Bump all spec tags to v0.12.1 (#1275) 2020-06-19 11:18:27 +10:00
Pawan Dhananjay
3199b1a6f2
Use all attestation subnets (#1257)
* Update `milagro_bls` to new release (#1183)

* Update milagro_bls to new release

Signed-off-by: Kirk Baird <baird.k@outlook.com>

* Tidy up fake cryptos

Signed-off-by: Kirk Baird <baird.k@outlook.com>

* move SecretHash to bls and put plaintext back

Signed-off-by: Kirk Baird <baird.k@outlook.com>

* Update v0.12.0 to v0.12.1

* Add compute_subnet_for_attestation

* Replace CommitteeIndex topic with Attestation

* Fix warnings

* Fix attestation service tests

* fmt

* Appease clippy

* return error from validator_subscriptions

* move state out of loop

* Fix early break on error

* Get state from slot clock

* Fix beacon state in attestation tests

* Add failing test for lookahead > 1

* Minor change

* Address some review comments

* Add subnet verification to beacon chain

* Move subnet verification to processor

* Pass committee_count_at_slot to ValidatorDuty and ValidatorSubscription

* Pass subnet id for publishing attestations

* Fix attestation service tests

* Fix more tests

* Fix fork choice test

* Remove unused code

* Remove more unused and expensive code

Co-authored-by: Kirk Baird <baird.k@outlook.com>
Co-authored-by: Michael Sproul <michael@sigmaprime.io>
Co-authored-by: Age Manning <Age@AgeManning.com>
Co-authored-by: Paul Hauner <paul@paulhauner.com>
2020-06-18 19:11:03 +10:00
Michael Sproul
e6f97bf466
Merge remote-tracking branch 'origin/master' into spec-v0.12 2020-06-17 12:34:11 +10:00
Paul Hauner
764cb2d32a
v0.12 fork choice update (#1229)
* Incomplete scraps

* Add progress on new fork choice impl

* Further progress

* First complete compiling version

* Remove chain reference

* Add new lmd_ghost crate

* Start integrating into beacon chain

* Update `milagro_bls` to new release (#1183)

* Update milagro_bls to new release

Signed-off-by: Kirk Baird <baird.k@outlook.com>

* Tidy up fake cryptos

Signed-off-by: Kirk Baird <baird.k@outlook.com>

* move SecretHash to bls and put plaintext back

Signed-off-by: Kirk Baird <baird.k@outlook.com>

* Update state processing for v0.12

* Fix EF test runners for v0.12

* Fix some tests

* Fix broken attestation verification test

* More test fixes

* Rough beacon chain impl working

* Remove fork_choice_2

* Remove checkpoint manager

* Half finished ssz impl

* Add missed file

* Add persistence

* Tidy, fix some compile errors

* Remove RwLock from ProtoArrayForkChoice

* Fix store-based compile errors

* Add comments, tidy

* Move function out of ForkChoice struct

* Start testing

* More testing

* Fix compile error

* Tidy beacon_chain::fork_choice

* Queue attestations from the current slot

* Allow fork choice to handle prior-to-genesis start

* Improve error granularity

* Test attestation dequeuing

* Process attestations during block

* Store target root in fork choice

* Move fork choice verification into new crate

* Update tests

* Consensus updates for v0.12 (#1228)

* Update state processing for v0.12

* Fix EF test runners for v0.12

* Fix some tests

* Fix broken attestation verification test

* More test fixes

* Fix typo found in review

* Add `Block` struct to ProtoArray

* Start fixing get_ancestor

* Add rough progress on testing

* Get fork choice tests working

* Progress with testing

* Fix partialeq impl

* Move slot clock from fc_store

* Improve testing

* Add testing for best justified

* Add clone back to SystemTimeSlotClock

* Add balances test

* Start adding balances cache again

* Wire-in balances cache

* Improve tests

* Remove commented-out tests

* Remove beacon_chain::ForkChoice

* Rename crates

* Update wider codebase to new fork_choice layout

* Move advance_slot in test harness

* Tidy ForkChoice::update_time

* Fix verification tests

* Fix compile error with iter::once

* Fix fork choice tests

* Ensure block attestations are processed

* Fix failing beacon_chain tests

* Add first invalid block check

* Add finalized block check

* Progress with testing, new store builder

* Add fixes to get_ancestor

* Fix old genesis justification test

* Fix remaining fork choice tests

* Change root iteration method

* Move on_verified_block

* Remove unused method

* Start adding attestation verification tests

* Add invalid ffg target test

* Add target epoch test

* Add queued attestation test

* Remove old fork choice verification tests

* Tidy, add test

* Move fork choice lock drop

* Rename BeaconForkChoiceStore

* Add comments, tidy BeaconForkChoiceStore

* Update metrics, rename fork_choice_store.rs

* Remove genesis_block_root from ForkChoice

* Tidy

* Update fork_choice comments

* Tidy, add comments

* Tidy, simplify ForkChoice, fix compile issue

* Tidy, removed dead file

* Increase http request timeout

* Fix failing rest_api test

* Set HTTP timeout back to 5s

* Apply fix to get_ancestor

* Address Michael's comments

* Fix typo

* Revert "Fix broken attestation verification test"

This reverts commit 722cdc903b12611de27916a57eeecfa3224f2279.

Co-authored-by: Kirk Baird <baird.k@outlook.com>
Co-authored-by: Michael Sproul <michael@sigmaprime.io>
2020-06-17 11:10:22 +10:00
Michael Sproul
7818447fd2
Check for unused deps in CI (#1262)
* Check for unused deps in CI

* Bump slashing protection parking_lot version
2020-06-14 10:59:50 +10:00
Michael Sproul
7ce9a252a4
Merge remote-tracking branch 'origin/master' into spec-v0.12 2020-06-09 18:34:44 +10:00
Pawan Dhananjay
042e80570c
Improve tokio task execution (#1181)
* Add logging on shutdown

* Replace tokio::spawn with handle.spawn

* Upgrade tokio

* Add a task executor

* Beacon chain tasks use task executor

* Validator client tasks use task executor

* Rename runtime_handle to executor

* Add duration histograms; minor fixes

* Cleanup

* Fix logs

* Fix tests

* Remove random file

* Get enr dependency instead of libp2p

* Address some review comments

* Libp2p takes a TaskExecutor

* Ugly fix libp2p tests

* Move TaskExecutor to own file

* Upgrade Dockerfile rust version

* Minor fixes

* Revert "Ugly fix libp2p tests"

This reverts commit 58d4bb690f52de28d893943b7504d2d0c6621429.

* Pretty fix libp2p tests

* Add spawn_without_exit; change Counter to Gauge

* Tidy

* Move log from RuntimeContext to TaskExecutor

* Fix errors

* Replace histogram with int_gauge for async tasks

* Fix todo

* Fix memory leak in test by exiting all spawned tasks at the end
2020-06-04 21:48:05 +10:00
Michael Sproul
fe03ff0f21
Consensus updates for v0.12 (#1228)
* Update state processing for v0.12

* Fix EF test runners for v0.12

* Fix some tests

* Fix broken attestation verification test

* More test fixes

* Fix typo found in review
2020-06-03 14:56:54 +10:00
Kirk Baird
197adeff0b
Update milagro_bls to new release (#1183)
* Update milagro_bls to new release

Signed-off-by: Kirk Baird <baird.k@outlook.com>

* Tidy up fake cryptos

Signed-off-by: Kirk Baird <baird.k@outlook.com>

* move SecretHash to bls and put plaintext back

Signed-off-by: Kirk Baird <baird.k@outlook.com>
2020-06-03 14:56:53 +10:00
Paul Hauner
a214032e1f
Update docs to suit sigp/docker-compose updates (#1235) 2020-06-01 21:18:48 +10:00
Paul Hauner
723c7cbd27
Ensure new health endpoint builds on MacOS (#1215)
* Add mac build to CI

* Always return an error for Health when not linux

* Change macos workflow

* Rename macos tests

* Disable health API test on Mac

Co-authored-by: Michael Sproul <michael@sigmaprime.io>
2020-06-01 21:18:31 +10:00
Paul Hauner
ea4a52984c
Add /node/health endpoint (#1197)
* Start adding health endpoint

* Use psutil more

* Add get_health test

* Expose health to Prom

* Update comments

* Add /node/health to docs

* Update Prom naming
2020-05-28 11:24:24 +10:00
Paul Hauner
8bc82c573d
Update local testnet scripts, fix eth1 sim (#1184)
* Update local testnet scripts

* Add logs when decrypting validators

* Update comment

* Update account manager

* Make random key generation explicit

* Remove unnecessary clap constraint

* Only decrypt voting keypair for eth1 deposit

* Use insecure kdf for insecure keypairs

* Simplify local testnet keygen

* Update local testnet

* Fix eth1 sim

* Add eth1 sim to CI again

* Remove old local testnet docs

* Tidy

* Remove checks for existing validators

* Tidy

* Fix typos
2020-05-26 18:30:44 +10:00
Paul Hauner
ad4e5adabc
Integrate Witti testnet (#1193)
* Update for latest master

* Shift delay inside loop

* Clean up genesis service

* Tidy

* Tidy logs

* Address Michael's comments

* Add pre-genesis logging

* Remove est time till genesis

* Fix time formatting

* Tidy

* Update docs for Witti

* Update JS for Witti

* Ensure deposit data is 0x-prefixed hex

* Hard code witti testnet dir

* Add --auto-register warning

* Integrate local sigp testnet source

* Reword warning
2020-05-26 15:31:41 +10:00
Paul Hauner
c93f9c351b
Improve bls::SecretKey privacy (#1164)
* Improve bls::SecretKey privacy

* Add missed file

* Remove more methods from bls::SecretKey

* Add as_bytes() to SecretKey, remove as_raw

* Remove as_raw

* Add back as_raw

* Address review comments
2020-05-19 11:23:08 +10:00
Paul Hauner
4331834003
Directory Restructure (#1163)
* Move tests -> testing

* Directory restructure

* Update Cargo.toml during restructure

* Update Makefile during restructure

* Fix arbitrary path
2020-05-18 21:24:23 +10:00