Commit Graph

101 Commits

Author SHA1 Message Date
Paul Hauner
1ab495738d Fix short sha in github actions (#2210)
## Issue Addressed

NA

## Proposed Changes

Swap to a full sha as per: https://github.com/sigp/lighthouse/runs/1922773402
2021-02-18 06:18:47 +00:00
Paul Hauner
3ffee365e2 Update rustup in release action (#2206)
## Issue Addressed

NA

## Proposed Changes

Ensure the arm64 binary is build with the latest version of Rust.
2021-02-17 23:04:09 +00:00
Michael Sproul
1a748699d9 Rust 1.50 fixes (#2199)
## Issue Addressed

https://rustsec.org/advisories/RUSTSEC-2021-0021.html

## Proposed Changes

* Bump `nb-connect` to avoid the above advisory (and migrate from the yanked crate)
* Add a `rustup update stable` to the Docker build to fix this issue: https://github.com/sigp/lighthouse/actions/runs/567075245
2021-02-15 06:09:54 +00:00
realbigsean
ff1813e618 Automated release Github upload fix (#2192)
## Issue Addressed

Our v1.1.0 automated release failed to upload to Github. The `hub` command failed with a `403`, which seems like this issue: https://github.com/github/hub/issues/2149
 
## Proposed Changes

The suggested fix in that issue is to set the `$GITHUB_USER` environment variable. I can't really test this because this hasn't been failing on my fork, but seems low risk



Co-authored-by: realbigsean <seananderson33@gmail.com>
Co-authored-by: Paul Hauner <paul@paulhauner.com>
2021-02-15 06:09:53 +00:00
Akihito Nakano
fee24539ac Add a GitHub Action to cancel previous runs (#2180)
## Issue Addressed

It takes over 20 minutes to run the GitHub Workflow for lighthouse. It would be time-saving to cancel previous runs. 

## Proposed Changes

Added [styfle/cancel-workflow-action](https://github.com/styfle/cancel-workflow-action) to our workflow. I've configured the action according to [Advanced settings](https://github.com/styfle/cancel-workflow-action#advanced).
2021-02-15 06:09:51 +00:00
Paul Hauner
8e5c20b6d1 Update for clippy 1.50 (#2193)
## Issue Addressed

NA

## Proposed Changes

Rust 1.50 has landed 🎉

The shiny new `clippy` peers down upon us mere mortals with disgust. Brutish peasants wrapping our `usize`s in superfluous `Option`s... tsk tsk.

I've performed the goat sacrifice and corrected our evil ways in this PR. Tonight we shall pray that Github Actions bestows the almighty green tick upon us.

## Additional Info

NA


Co-authored-by: realbigsean <seananderson33@gmail.com>
Co-authored-by: Michael Sproul <michael@sigmaprime.io>
2021-02-15 00:09:12 +00:00
Michael Sproul
e2ff9c66a1 Remove links to old master branch (#2190)
## Proposed Changes

In preparation for deleting the `master` branch, remove all links to it from the book/README.
2021-02-11 06:06:54 +00:00
Michael Sproul
6f4da9a5d2 Check that pull requests target unstable (#2187)
Attempt to prevent accidental merges to `stable` due to GitHub's default behaviour of opening PRs against it.

I've intentionally opened this PR against `stable` to test the functionality ;)
2021-02-09 02:00:53 +00:00
realbigsean
51f7724c76 Automate docker version tag (#2150)
## Issue Addressed

N/A

## Proposed Changes

On any tag formatted `v*`, a full multi-arch docker build will be kicked off and automatically pushed to docker hub with the version tag.

This is a bit repetitive, because the image built will usually be the same as the image built on pushes to `stable`, but it seems like the simplest way to go about it and this will also work if we incorporate a workflow with `vX.X.X-rc` tags. 

## Additional Info

This may also need to wait for env variable updates: https://github.com/sigp/lighthouse/pull/2135#issuecomment-754977433

Co-authored-by: realbigsean <seananderson33@gmail.com>
2021-01-19 03:50:10 +00:00
realbigsean
908c8eadf3 remove protected environment (#2135)
## Issue Addressed

N/A

## Proposed Changes

Remove Github Action environments

## Additional Info

N/A


Co-authored-by: realbigsean <seananderson33@gmail.com>
2021-01-19 01:29:06 +00:00
realbigsean
14df5d5c32 Use cross in linux x86 64 release flow (#2136)
## Issue Addressed

Resolves  #2120

## Proposed Changes

This updates github actions to use `cross` when compiling linux x86_64 binaries.

## Additional Info

I think we could alternatively be explicit with the version of macOS or ubuntu we are running actions on and that could solve #2120. I'm not sure which method is preferred here though. Github actions supports Ubuntu 16.04

Co-authored-by: realbigsean <seananderson33@gmail.com>
2021-01-12 06:38:22 +00:00
realbigsean
b5e81eb6b2 add automated release workflow (#2077)
## Issue Addressed

Resolves #1674 

## Proposed Changes

- Whenever a tag is pushed with the prefix `v` this workflow is triggered
- creates portable and non-portable binaries for linux x86_64, linux aarch64, macOS
  - an attempt at using github actions caching
- signs each binary using GPG
- auto-generates full changelog based on commit messages since the last release
- creates a **draft** release
- hot new formatting (preview [here](https://github.com/realbigsean/lighthouse/releases/tag/v0.9.23))
- has been taking around 35 minutes

## Additional Info

TODOs:
- Figure out how we should automate dockerhub's version tag. 
  - It'd be quickest just to tag `latest`, but we'd need to make sure the docker workflow completes before this starts
- we do the same cross-compile in the `docker` workflow, we could try to use the same binary
- integrate a similar flow for unstable binaries (`-rc` tag?)
- improve caching, potentially use sccache
- if we start using a self-hosted runner this'll require some re-working

Need to add the following secrets to Github: 

- `GPG_PASSPHRASE`
- ~~`GPG_PUBLIC_KEY`~~ hard-coded this, because it was tough manage as a secret
- `GPG_SIGNING_KEY` 


Co-authored-by: realbigsean <seananderson33@gmail.com>
2020-12-23 07:53:34 +00:00
realbigsean
ca08fc7831 Revert "add caching to test suite (#2089)" (#2098)
## Issue Addressed

N/A

## Proposed Changes

I didn't realize the `PORTABLE` env variable is only picked up by `install` in the `Makefile` so we are still getting `SIGILL`s:

https://github.com/sigp/lighthouse/runs/1565004525?check_suite_focus=true

## Additional Info



Co-authored-by: realbigsean <seananderson33@gmail.com>
2020-12-16 23:29:07 +00:00
blacktemplar
3fcc517993 Fix Syncing Simulator (#2049)
## Issue Addressed

NA

## Proposed Changes

Fixes problems with slot times below 1 second which got revealed by running the syncing simulator with the default speedup time.
2020-12-16 05:37:38 +00:00
realbigsean
80f47fcfff add caching to test suite (#2089)
## Issue Addressed

N/A

## Proposed Changes

Add some caching to the test suite and to the aarch64 cross-compile in the docker build. 

## Additional Info

Cache hits only occur if the Cargo.lock file is unchanged, Github Actions runner OS matches, and the cache is "in scope". Some documentation on github actions cache scoping is here:

https://docs.github.com/en/free-pro-team@latest/actions/guides/caching-dependencies-to-speed-up-workflows#matching-a-cache-key

I'm not sure how frequently we'll get cache hits, I imagine only on smaller PR's or updates to the same PR.  And there is a cache size limit that we may end up reaching quickly.  But Github actions handles evictions if we go over that limit. 

Not sure how much of an impact this will end up having but I don't really see a downside to trying it out.

Co-authored-by: realbigsean <seananderson33@gmail.com>
2020-12-16 03:44:03 +00:00
realbigsean
c1e27f4c89 Improve docker auto builds (#2078)
## Issue Addressed

N/A

## Proposed Changes

- hardcode `ubuntu-18.04` -- I don't think this was causing us issues, but github actions is in the process of migrating `ubuntu-latest` from Ubuntu 18 -> 20.. so just in case
- different source of emulation dependencies -> https://github.com/tonistiigi/binfmt 
  - this one is explicitly referenced in the `buildx` github docs
- install emulation dependencies and run `docker buildx` in the same `run` command
- enable `buildx` with  `DOCKER_CLI_EXPERIMENTAL: enabled` rather than re-building it

## Additional Info

N/A


Co-authored-by: realbigsean <seananderson33@gmail.com>
2020-12-11 00:19:35 +00:00
realbigsean
adbd49ddc6 Multiarch docker GitHub actions (#2065)
## Issue Addressed

Resolves #1512

## Proposed Changes

- Adds a new docker Github Actions workflow  
- Removes the Dockerhub hook
- Adds a new Dockerfile for use with pre-existing cross-compiled binaries 
- on pushes to `unstable` 
  - builds an ARM64 image and tags it `latest-arm64-unstable`
  - builds an AMD64 image and tags it `latest-amd64-unstable`
  - builds an multiarch image by creating a manifest list referencing the prior two images and tags it `latest-unstable`
- on pushes to `stable` 
  - builds an ARM64 image and tags it `latest-arm64`
  - builds an AMD64 image and tags it `latest-amd64`
  - builds an multiarch image by creating a manifest list referencing the prior two images and tags it `latest`

## Additional Info
- for ARM64, first `cross` is used to cross compile the `lighthouse` and  `lcli` binaries, then `docker buildx` is installed to actually build the docker image for the correct target platform. The image build pretty much just copies the binaries from local into the docker image (thanks @michaelsproul :) )
- The AMD64 and ARM64 builds run in parallel, in total it's been taking around 45mins on a local runner
- This PR does **not** cover version tags on docker images at the moment

Co-authored-by: realbigsean <seananderson33@gmail.com>
2020-12-09 06:06:37 +00:00
Paul Hauner
7020f5df40 Update docs whenever unstable changes (#1969)
## Issue Addressed

NA

## Proposed Changes

Presently `master` is stable (and will be sunsetted) which means our docs only update after a release. This PR sets the docs to build on the `unstable` branch, which is equivalent to what what we've always had. 

## Additional Info

This does raise the question of whether or not docs should target `stable` or `unstable`, but I'd prefer to maintain current functionality and merge #1966 for now. I think having two versions might be handy, one for stable and one for unstable; I don't imagine this very difficult to achieve.
2020-11-25 03:20:23 +00:00
Age Manning
a1a6b01acb Remove macos tests (#1687)
## Issue Addressed

N/A

## Proposed Changes

Remove the MacOs tests. They routinely fail, causing bors to retry and slowing down the whole merge process.

## Additional Info

N/A


Co-authored-by: Michael Sproul <michael@sigmaprime.io>
2020-09-30 01:27:36 +00:00
Michael Sproul
3412a3ec54 Remove saturating arith from state_processing (#1644)
## Issue Addressed

Resolves #1100

## Proposed Changes

* Implement the `SafeArith` trait for `Slot` and `Epoch`, so that methods like `safe_add` become available.
* Tweak the `SafeArith` trait to allow a different `Rhs` type (analagous to `std::ops::Add`, etc).
* Add a `legacy-arith` feature to `types` and `state_processing` that conditionally enables implementations of
  the `std` ops with saturating semantics.
* Check compilation of `types` and `state_processing` _without_ `legacy-arith` on CI,
  thus guaranteeing that they only use the `SafeArith` primitives 🎉

## Additional Info

The `legacy-arith` feature gets turned on by all higher-level crates that depend on `state_processing` or `types`, thus allowing the beacon chain, networking, and other components to continue to rely on the availability of ops like `+`, `-`, `*`, etc.

**This is a consensus-breaking change**, but brings us in line with the spec, and our incompatibilities shouldn't have been reachable with any valid configuration of Eth2 parameters.
2020-09-25 05:18:21 +00:00
Michael Sproul
211109bbc0 Revert "add a github action for build multi-arch docker images (#1574)" (#1591)
This reverts commit 2627463366.

## Issue Addressed

This is a temporary fix for #1589, by reverting #1574. The Docker image needs to be built with `--build-arg PORTABLE=true`, and we could probably integrate that into the multi-arch build, but in the interests of expediting a fix, this PR opts for a revert.
2020-09-06 04:46:25 +00:00
realbigsean
2627463366 add a github action for build multi-arch docker images (#1574)
## Issue Addressed

#1512

## Proposed Changes

Use Github Actions to automate the Docker image build, so that we can make a multi-arch image.  

## Additional Info

This change will require adding the DOCKER_USERNAME and DOCKER_PASSWORD secrets in Github. It will also require disabling the Docker Hub automated build.
2020-09-04 02:43:32 +00:00
Michael Sproul
30bb7aecfb Check Cargo.lock freshness on CI (#1565)
Check that `Cargo.lock` is up-to-date on CI so we're not having to push messy lockfile fix ups after releases.
2020-08-26 00:01:08 +00:00
Paul Hauner
b73c497be2 Support multiple BLS implementations (#1335)
## Issue Addressed

NA

## Proposed Changes

- Refactor the `bls` crate to support multiple BLS "backends" (e.g., milagro, blst, etc).
- Removes some duplicate, unused code in `common/rest_types/src/validator.rs`.
- Removes the old "upgrade legacy keypairs" functionality (these were unencrypted keys that haven't been supported for a few testnets, no one should be using them anymore).

## Additional Info

Most of the files changed are just inconsequential changes to function names.

## TODO

- [x] Optimization levels
- [x] Infinity point: https://github.com/supranational/blst/issues/11
- [x] Ensure milagro *and* blst are tested via CI
- [x] What to do with unsafe code?
- [x] Test infinity point in signature sets
2020-07-25 02:03:18 +00:00
Michael Sproul
393782f632 Prefer viable nightly Rust builds for CI (#1379)
## Proposed Changes

CI is failing on PRs because of a regression in nightly Rust. This change forces `rustup` to install a version of the nightly compiler that can at least build `rustfmt`, which should reduce the frequency of this happening.

## Additional Info

Example failing run: https://github.com/sigp/lighthouse/runs/896694659

Alternatively, we could allow CI to succeed even when `cargo-udeps` fails, but I think we should avoid doing that unless we really have to.
2020-07-22 07:34:30 +00:00
Michael Sproul
2870172e0d Deny warnings on CI (#1372)
## Issue Addressed

Prevent CI from succeeding when there are warnings. Code can still be built and tested with warnings locally, but CI may fail during a Rust update (which is fine IMO).

## Proposed Changes

* Deny warnings for all stable compiler jobs on CI (excludes `cargo udeps`, which runs under nightly)
* Fix the warnings currently on `master` related to unnecessary `mem::replace`
2020-07-21 05:51:33 +00:00
Adam Szkoda
e8d5d37bc1 Tune CI (#1344)
`pr/*` branches now trigger CI before creating a PR to avoid submitting PR with failing tests.

Details in the commit messages.
2020-07-09 09:23:28 +00:00
Adam Szkoda
559b7c8faa Fix bors timeouts (#1341)
... by moving "license/cla" from `status` to `pr_status` in `bors.toml`.
2020-07-07 13:44:35 +00:00
Adam Szkoda
51fbaefe41 Add bors configuration (#1340)
Melded together stuff from https://github.com/sigp/lighthouse/pull/1326 and https://github.com/sigp/lighthouse/pull/1337.

Note that I've reduced the timeout to 2 hours to save some time in case the build gets stuck.

Co-authored-by: Michael Sproul <michael@sigmaprime.io>
2020-07-07 06:31:20 +00:00
Michael Sproul
7818447fd2
Check for unused deps in CI (#1262)
* Check for unused deps in CI

* Bump slashing protection parking_lot version
2020-06-14 10:59:50 +10:00
Paul Hauner
723c7cbd27
Ensure new health endpoint builds on MacOS (#1215)
* Add mac build to CI

* Always return an error for Health when not linux

* Change macos workflow

* Rename macos tests

* Disable health API test on Mac

Co-authored-by: Michael Sproul <michael@sigmaprime.io>
2020-06-01 21:18:31 +10:00
Paul Hauner
8bc82c573d
Update local testnet scripts, fix eth1 sim (#1184)
* Update local testnet scripts

* Add logs when decrypting validators

* Update comment

* Update account manager

* Make random key generation explicit

* Remove unnecessary clap constraint

* Only decrypt voting keypair for eth1 deposit

* Use insecure kdf for insecure keypairs

* Simplify local testnet keygen

* Update local testnet

* Fix eth1 sim

* Add eth1 sim to CI again

* Remove old local testnet docs

* Tidy

* Remove checks for existing validators

* Tidy

* Fix typos
2020-05-26 18:30:44 +10:00
Mehdi Zerouali
d41a9f7aa6
Add cargo audit to CI (#1192) 2020-05-26 17:04:39 +10:00
Age Manning
309cd95b2c
Update work-flow for pull_request (#1156)
* Update work-flow for pull_request

* Update to push for master
2020-05-19 13:17:49 +10:00
Paul Hauner
c571afb8d8
Wallet-based, encrypted key management (#1138)
* Update hashmap hashset to stable futures

* Adds panic test to hashset delay

* Port remote_beacon_node to stable futures

* Fix lcli merge conflicts

* Non rpc stuff compiles

* Remove padding

* Add error enum, zeroize more things

* Fix comment

* protocol.rs compiles

* Port websockets, timer and notifier to stable futures (#1035)

* Fix lcli

* Port timer to stable futures

* Fix timer

* Port websocket_server to stable futures

* Port notifier to stable futures

* Add TODOS

* Port remote_beacon_node to stable futures

* Partial eth2-libp2p stable future upgrade

* Finished first round of fighting RPC types

* Further progress towards porting eth2-libp2p adds caching to discovery

* Update behaviour

* Add keystore builder

* Remove keystore stuff from val client

* Add more tests, comments

* RPC handler to stable futures

* Update RPC to master libp2p

* Add more comments, test vectors

* Network service additions

* Progress on improving JSON validation

* More JSON verification

* Start moving JSON into own mod

* Remove old code

* Add more tests, reader/writers

* Tidy

* Move keystore into own file

* Move more logic into keystore file

* Tidy

* Tidy

* Fix the fallback transport construction (#1102)

* Allow for odd-character hex

* Correct warning

* Remove hashmap delay

* Compiling version of eth2-libp2p

* Update all crates versions

* Fix conversion function and add tests (#1113)

* Add more json missing field checks

* Use scrypt by default

* Tidy, address comments

* Test path and uuid in vectors

* Fix comment

* Add checks for kdf params

* Enforce empty kdf message

* Port validator_client to stable futures (#1114)

* Add PH & MS slot clock changes

* Account for genesis time

* Add progress on duties refactor

* Add simple is_aggregator bool to val subscription

* Start work on attestation_verification.rs

* Add progress on ObservedAttestations

* Progress with ObservedAttestations

* Fix tests

* Add observed attestations to the beacon chain

* Add attestation observation to processing code

* Add progress on attestation verification

* Add first draft of ObservedAttesters

* Add more tests

* Add observed attesters to beacon chain

* Add observers to attestation processing

* Add more attestation verification

* Create ObservedAggregators map

* Remove commented-out code

* Add observed aggregators into chain

* Add progress

* Finish adding features to attestation verification

* Ensure beacon chain compiles

* Link attn verification into chain

* Integrate new attn verification in chain

* Remove old attestation processing code

* Start trying to fix beacon_chain tests

* Split adding into pools into two functions

* Add aggregation to harness

* Get test harness working again

* Adjust the number of aggregators for test harness

* Fix edge-case in harness

* Integrate new attn processing in network

* Fix compile bug in validator_client

* Update validator API endpoints

* Fix aggreagation in test harness

* Fix enum thing

* Fix attestation observation bug:

* Patch failing API tests

* Start adding comments to attestation verification

* Remove unused attestation field

* Unify "is block known" logic

* Update comments

* Supress fork choice errors for network processing

* Add todos

* Tidy

* Add gossip attn tests

* Disallow test harness to produce old attns

* Comment out in-progress tests

* Partially address pruning tests

* Fix failing store test

* Add aggregate tests

* Add comments about which spec conditions we check

* Dont re-aggregate

* Split apart test harness attn production

* Fix compile error in network

* Make progress on commented-out test

* Fix skipping attestation test

* Add fork choice verification tests

* Tidy attn tests, remove dead code

* Remove some accidentally added code

* Fix clippy lint

* Rename test file

* Add block tests, add cheap block proposer check

* Rename block testing file

* Add observed_block_producers

* Tidy

* Switch around block signature verification

* Finish block testing

* Remove gossip from signature tests

* First pass of self review

* Fix deviation in spec

* Update test spec tags

* Start moving over to hashset

* Finish moving observed attesters to hashmap

* Move aggregation pool over to hashmap

* Make fc attn borrow again

* Fix rest_api compile error

* Fix missing comments

* Fix monster test

* Uncomment increasing slots test

* Address remaining comments

* Remove unsafe, use cfg test

* Remove cfg test flag

* Fix dodgy comment

* Revert "Update hashmap hashset to stable futures"

This reverts commit d432378a3cc5cd67fc29c0b15b96b886c1323554.

* Revert "Adds panic test to hashset delay"

This reverts commit 281502396fc5b90d9c421a309c2c056982c9525b.

* Ported attestation_service

* Ported duties_service

* Ported fork_service

* More ports

* Port block_service

* Minor fixes

* VC compiles

* Update TODOS

* Borrow self where possible

* Ignore aggregates that are already known.

* Unify aggregator modulo logic

* Fix typo in logs

* Refactor validator subscription logic

* Avoid reproducing selection proof

* Skip HTTP call if no subscriptions

* Rename DutyAndState -> DutyAndProof

* Tidy logs

* Print root as dbg

* Fix compile errors in tests

* Fix compile error in test

* Re-Fix attestation and duties service

* Minor fixes

Co-authored-by: Paul Hauner <paul@paulhauner.com>

* Expose json_keystore mod

* First commits on path derivation

* Progress with implementation

* More progress

* Passing intermediate test vectors

* Tidy, add comments

* Add DerivedKey structs

* Move key derivation into own crate

* Add zeroize structs

* Return error for empty seed

* Add tests

* Tidy

* First commits on path derivation

* Progress with implementation

* Move key derivation into own crate

* Start defining JSON wallet

* Add progress

* Split out encrypt/decrypt

* First commits on path derivation

* Progress with implementation

* More progress

* Passing intermediate test vectors

* Tidy, add comments

* Add DerivedKey structs

* Move key derivation into own crate

* Add zeroize structs

* Return error for empty seed

* Add tests

* Tidy

* Add progress

* Replace some password usage with slice

* First commits on path derivation

* Progress with implementation

* More progress

* Passing intermediate test vectors

* Tidy, add comments

* Add DerivedKey structs

* Move key derivation into own crate

* Add zeroize structs

* Return error for empty seed

* Add tests

* Tidy

* Add progress

* Expose PlainText struct

* First commits on path derivation

* Progress with implementation

* More progress

* Passing intermediate test vectors

* Tidy, add comments

* Add DerivedKey structs

* Move key derivation into own crate

* Add zeroize structs

* Return error for empty seed

* Add tests

* Tidy

* Add builder

* Expose consts, remove Password

* Minor progress

* Expose SALT_SIZE

* First compiling version

* Add test vectors

* Network crate update to stable futures

* Move dbg assert statement

* Port account_manager to stable futures (#1121)

* Port account_manager to stable futures

* Run async fns in tokio environment

* Port rest_api crate to stable futures (#1118)

* Port rest_api lib to stable futures

* Reduce tokio features

* Update notifier to stable futures

* Builder update

* Further updates

* Add mnemonic, tidy

* Convert self referential async functions

* Tidy

* Add testing

* Add first attempt at validator_dir

* Present pubkey field

* stable futures fixes (#1124)

* Fix eth1 update functions

* Fix genesis and client

* Fix beacon node lib

* Return appropriate runtimes from environment

* Fix test rig

* Refactor eth1 service update

* Upgrade simulator to stable futures

* Lighthouse compiles on stable futures

* Add first pass of wallet manager

* Progress with CLI

* Remove println debugging statement

* Tidy output

* Tidy 600 perms

* Update libp2p service, start rpc test upgrade

* Add validator creation flow

* Update network crate for new libp2p

* Start tidying, adding comments

* Update tokio::codec to futures_codec (#1128)

* Further work towards RPC corrections

* Correct http timeout and network service select

* Add wallet mgr testing

* Shift LockedWallet into own file

* Add comments to fs

* Start integration into VC

* Use tokio runtime for libp2p

* Revert "Update tokio::codec to futures_codec (#1128)"

This reverts commit e57aea924acf5cbabdcea18895ac07e38a425ed7.

* Upgrade RPC libp2p tests

* Upgrade secio fallback test

* Add lcli keypair upgrade command

* Upgrade gossipsub examples

* Clean up RPC protocol

* Test fixes (#1133)

* Correct websocket timeout and run on os thread

* Fix network test

* Add --secrets-dir to VC

* Remove --legacy-keys from VC

* Clean up PR

* Correct tokio tcp move attestation service tests

* Upgrade attestation service tests

* Fix sim

* Correct network test

* Correct genesis test

* Start docs

* Add progress for validator generation

* Tidy error messages

* Test corrections

* Log info when block is received

* Modify logs and update attester service events

* Stable futures: fixes to vc, eth1 and account manager (#1142)

* Add local testnet scripts

* Remove whiteblock script

* Rename local testnet script

* Move spawns onto handle

* Fix VC panic

* Initial fix to block production issue

* Tidy block producer fix

* Tidy further

* Add local testnet clean script

* Run cargo fmt

* Tidy duties service

* Tidy fork service

* Tidy ForkService

* Tidy AttestationService

* Tidy notifier

* Ensure await is not suppressed in eth1

* Ensure await is not suppressed in account_manager

* Use .ok() instead of .unwrap_or(())

* RPC decoding test for proto

* Update discv5 and eth2-libp2p deps

* Run cargo fmt

* Pre-build keystores for sim

* Fix lcli double runtime issue (#1144)

* Handle stream termination and dialing peer errors

* Correct peer_info variant types

* Add progress on new deposit flow

* Remove unnecessary warnings

* Handle subnet unsubscription removal and improve logigng

* Add logs around ping

* Upgrade discv5 and improve logging

* Handle peer connection status for multiple connections

* Improve network service logging

* Add more incomplete progress

* Improve logging around peer manager

* Upgrade swarm poll centralise peer management

* Identify clients on error

* Fix `remove_peer` in sync (#1150)

* remove_peer removes from all chains

* Remove logs

* Fix early return from loop

* Improved logging, fix panic

* Partially correct tests

* Add deposit command

* Remove old validator directory

* Start adding AM tests

* Stable futures: Vc sync (#1149)

* Improve syncing heuristic

* Add comments

* Use safer method for tolerance

* Fix tests

* Binary testing progress

* Progress with CLI tests

* Use constants for flags

* More account manager testing

* Improve CLI tests

* Move upgrade-legacy-keypairs into account man

* Use rayon for VC key generation

* Add comments to `validator_dir`

* Add testing to validator_dir

* Add fix to eth1-sim

* Check errors in eth1-sim

* Fix mutability issue

* Ensure password file ends in .pass

* Add more tests to wallet manager

* Tidy deposit

* Tidy account manager

* Tidy account manager

* Remove panic

* Generate keypairs earlier in sim

* Tidy eth1-sime

* Try to fix eth1 sim

* Address review comments

* Fix typo in CLI command

* Update docs

* Disable eth1 sim

* Remove eth1 sim completely

Co-authored-by: Age Manning <Age@AgeManning.com>
Co-authored-by: pawanjay176 <pawandhananjay@gmail.com>
2020-05-18 19:01:45 +10:00
Michael Sproul
338cb2fba7
Fix parallelism bug in exit processing (#1110)
* Fix parallelism bug in exit processing

Also:

* Remove parallelism for all other operations except deposit merkle proofs
* Improve exit tests
* Fix broken attestation test

Closes #1090

* Allow for generating block/pre/post states from some unit tests (#1123)

* Add post-state checks, comments

* Add state_transition_vectors crate

* Integrate new testing crate with CI

* Add readme

* Add additional valid tests

* Remove ExitTests (they were moved to new crate)

* Small test fixes

* Delete incorrect saturating_sub in slash_validator

And clean-up the balance increase/decrease functions to look more like the spec.

Co-authored-by: Paul Hauner <paul@paulhauner.com>
2020-05-09 09:37:21 +10:00
Kirk Baird
611a0c7d19
Arbitrary trait for eth2/types (#1040)
* Add the arbitrary type to eth2/types and their deps

Signed-off-by: Kirk Baird <baird.k@outlook.com>

* Wrap arbitrary in a feature flag

Signed-off-by: Kirk Baird <baird.k@outlook.com>

* Fix feature for types

Signed-off-by: Kirk Baird <baird.k@outlook.com>

* Fix comment

Signed-off-by: Kirk Baird <baird.k@outlook.com>

* Patch versioning

Signed-off-by: Kirk Baird <baird.k@outlook.com>

* Allow expanded crate reference for arbitrary 0.4.3

Signed-off-by: Kirk Baird <baird.k@outlook.com>

* Add arbitrary to remaining types

Signed-off-by: Kirk Baird <baird.k@outlook.com>

* use cmp::min

Signed-off-by: Kirk Baird <baird.k@outlook.com>

* Derive Arbitrary trait for ValidatorStatus, TotalBalances and InclusionInfo

* Add CI check for state processing arbitrary faetures

Signed-off-by: Kirk Baird <baird.k@outlook.com>

* Fix indentation

Signed-off-by: Kirk Baird <baird.k@outlook.com>

Co-authored-by: Mehdi Zerouali <mehdi@sigmaprime.io>
2020-05-06 09:12:28 +10:00
Age Manning
ca538e887e
Cleanup v0.2.0 2020-04-22 01:31:07 +10:00
Age Manning
9e416a9bcd
Merge latest master 2020-04-22 01:05:46 +10:00
Michael Sproul
32074f0d09
Use checked arithmetic in types and state proc (#1009) 2020-04-20 12:35:11 +10:00
Paul Hauner
7b86c9a08f
Update testnet tooling (#1001)
* Add progress on new deposits

* Add deposited command to account manager

* Remove old lcli::helpers mod

* Clean clap_utils

* Refactor lcli deposit contract commands to use IPC

* Make testnet optional for environment

* Use dbg formatting for deploy address

* Add command to generate bootnode enr

* Ensure lcli returns with 1 on error

* Ensure account manager returns 1 on error

* Disallow deposits to the zero address

* Update web3 in eth1 crate

* Ensure correct lighthouse dir is created

* Reduce deposit gas requirement

* Update cargo.lock

* Add progress on new deposits

* Add deposited command to account manager

* Remove old lcli::helpers mod

* Clean clap_utils

* Refactor lcli deposit contract commands to use IPC

* Add command to generate bootnode enr

* Ensure lcli returns with 1 on error

* Ensure account manager returns 1 on error

* Update web3 in eth1 crate

* Update Cargo.lock

* Move lcli out of main install script

* Change --limit to --at-least

* Change --datadir to --validator-dir

* Remove duplication in docs
2020-04-19 02:20:43 +00:00
Age Manning
1779aa6a8a
Merge latest master in v0.2.0 2020-04-08 16:46:37 +10:00
Michael Sproul
869b0621d6
Update benchmarks, add to CI (#988) 2020-04-06 20:16:08 +10:00
Paul Hauner
7305e9e5d9
Upgrade sim (#972)
* Add progress on duties refactor

* Add simple is_aggregator bool to val subscription

* Add the no-eth1-sim, refactor sim
2020-04-01 17:54:49 +11:00
/raw PONG _GHMoaCXLT
a5fbaef469
Github: Add version to issue template (#937) 2020-03-20 12:18:31 +11:00
Paul Hauner
a87e8c55fc
Add a GitHub action to publish The Book (#877) 2020-03-06 10:54:07 +11:00
Pawan Dhananjay
89f05e4a4f Adds simulator for syncing (#758)
* Add CLI for beacon_chain_sim

* Rename beacon-chain-sim to simulator

* Fix simulator workflow

* Push Cargo.lock

* WIP syncing simulator

* Add cli args

* Remove eth1 stuff and deposits

* Add syncing strategy simulations

* Successful one node sync

* Clean up

* Rename to avoid confusion

* add command line args

* fix cargo fmt issues

* Add additional syncing strategies

* Run all syncing strategies one after other; add comments

* Improve cli argument parsing

* Change `end_after_checks` default to true

* Small modifications to syncing-sim

* Add `strategy` cli argument

* Documented defaults in cli help

Co-authored-by: mkinney <mike.kinney@gmail.com>
Co-authored-by: Age Manning <Age@AgeManning.com>
2020-01-23 18:07:39 +11:00
Paul Hauner
5e4cf2f3da
Add simulator to CI (#723)
* Add simulator to CI

* Install ganache before running sim
2019-12-17 08:48:20 +11:00
Age Manning
97aa8b75b8 Testnet compatible network upgrade (#587)
* Create libp2p instance

* Change logger to stdlog

* test_connection initial commit

* Add gossipsub test

* Delete tests in network crate

* Add test module

* Clean tests

* Remove dependency on discovery

* Working publish between 2 nodes
TODO: Publish should be called just once

* Working 2 peer gossipsub test with additional events

* Cleanup test

* Add rpc test

* Star topology discovery WIP

* build_nodes builds and connects n nodes. Increase nodes in gossipsub test

* Add unsubscribe method and expose reference to gossipsub object for gossipsub tests

* Add gossipsub message forwarding test

* Fix gossipsub forward test

* Test improvements
* Remove discovery tests
* Simplify gossipsub forward test topology
* Add helper functions for topology building

* Clean up tests

* Update naming to new network spec

* Correct ssz encoding of protocol names

* Further additions to network upgrade

* Initial network spec update WIP

* Temp commit

* Builds one side of the streamed RPC responses

* Temporary commit

* Propagates streaming changes up into message handler

* Intermediate network update

* Partial update in upgrading to the new network spec

* Update dependencies, remove redundant deps

* Correct sync manager for block stream handling

* Re-write of RPC handler, improves efficiency and corrects bugs

* Stream termination update

* Completed refactor of rpc handler

* Remove crates

* Correct compile issues associated with test merge

* Build basic tests and testing structure for eth2-libp2p

* Enhance RPC tests and add logging

* Complete RPC testing framework and STATUS test

* Decoding bug fixes, log improvements, stream test

* Clean up RPC handler logging

* Decoder bug fix, empty block stream test

* Add BlocksByRoot RPC test

* Add Goodbye RPC test

* Syncing and stream handling bug fixes and performance improvements

* Applies discv5 bug fixes

* Adds DHT IP filtering for lighthouse - currently disabled

* Adds randomized network propagation as a CLI arg

* Add disconnect functionality

* Adds attestation handling and parent lookup

* Adds RPC error handling for the sync manager

* Allow parent's blocks to be already processed

* Update github workflow

* Adds reviewer suggestions
2019-11-27 12:47:46 +11:00
Paul Hauner
78d82d9193
Validator client refactor (#618)
* Update to spec v0.9.0

* Update to v0.9.1

* Bump spec tags for v0.9.1

* Formatting, fix CI failures

* Resolve accidental KeyPair merge conflict

* Document new BeaconState functions

* Add `validator` changes from `validator-to-rest`

* Add initial (failing) REST api tests

* Fix signature parsing

* Add more tests

* Refactor http router

* Add working tests for publish beacon block

* Add validator duties tests

* Move account_manager under `lighthouse` binary

* Unify logfile handling in `environment` crate.

* Fix incorrect cache drops in `advance_caches`

* Update fork choice for v0.9.1

* Add `deposit_contract` crate

* Add progress on validator onboarding

* Add unfinished attesation code

* Update account manager CLI

* Write eth1 data file as hex string

* Integrate ValidatorDirectory with validator_client

* Move ValidatorDirectory into validator_client

* Clean up some FIXMEs

* Add beacon_chain_sim

* Fix a few docs/logs

* Expand `beacon_chain_sim`

* Fix spec for `beacon_chain_sim

* More testing for api

* Start work on attestation endpoint

* Reject empty attestations

* Allow attestations to genesis block

* Add working tests for `rest_api` validator endpoint

* Remove grpc from beacon_node

* Start heavy refactor of validator client

- Block production is working

* Prune old validator client files

* Start works on attestation service

* Add attestation service to validator client

* Use full pubkey for validator directories

* Add validator duties post endpoint

* Use par_iter for keypair generation

* Use bulk duties request in validator client

* Add version http endpoint tests

* Add interop keys and startup wait

* Ensure a prompt exit

* Add duties pruning

* Fix compile error in beacon node tests

* Add github workflow

* Modify rust.yaml

* Modify gitlab actions

* Add to CI file

* Add sudo to CI npm install

* Move cargo fmt to own job in tests

* Fix cargo fmt in CI

* Add rustup update before cargo fmt

* Change name of CI job

* Make other CI jobs require cargo fmt

* Add CI badge

* Remove gitlab and travis files

* Add different http timeout for debug

* Update docker file, use makefile in CI

* Use make in the dockerfile, skip the test

* Use the makefile for debug GI test

* Update book

* Tidy grpc and misc things

* Apply discv5 fixes

* Address other minor issues

* Fix warnings

* Attempt fix for addr parsing

* Tidy validator config, CLIs

* Tidy comments

* Tidy signing, reduce ForkService duplication

* Fail if skipping too many slots

* Set default recent genesis time to 0

* Add custom http timeout to validator

* Fix compile bug in node_test_rig

* Remove old bootstrap flag from val CLI

* Update docs

* Tidy val client

* Change val client log levels

* Add comments, more validity checks

* Fix compile error, add comments

* Undo changes to eth2-libp2p/src

* Reduce duplication of keypair generation

* Add more logging for validator duties

* Fix beacon_chain_sim, nitpicks

* Fix compile error, minor nits

* Address Michael's comments
2019-11-25 15:48:24 +11:00
Paul Hauner
c327ef970b
Add PR and issue templates 2018-10-14 14:51:59 +11:00