Prepare for public testnet (#628)

* Update to spec v0.9.0

* Update to v0.9.1

* Bump spec tags for v0.9.1

* Formatting, fix CI failures

* Resolve accidental KeyPair merge conflict

* Document new BeaconState functions

* Add `validator` changes from `validator-to-rest`

* Add initial (failing) REST api tests

* Fix signature parsing

* Add more tests

* Refactor http router

* Add working tests for publish beacon block

* Add validator duties tests

* Move account_manager under `lighthouse` binary

* Unify logfile handling in `environment` crate.

* Fix incorrect cache drops in `advance_caches`

* Update fork choice for v0.9.1

* Add `deposit_contract` crate

* Add progress on validator onboarding

* Add unfinished attesation code

* Update account manager CLI

* Write eth1 data file as hex string

* Integrate ValidatorDirectory with validator_client

* Move ValidatorDirectory into validator_client

* Clean up some FIXMEs

* Add beacon_chain_sim

* Fix a few docs/logs

* Expand `beacon_chain_sim`

* Fix spec for `beacon_chain_sim

* More testing for api

* Start work on attestation endpoint

* Reject empty attestations

* Allow attestations to genesis block

* Add working tests for `rest_api` validator endpoint

* Remove grpc from beacon_node

* Start heavy refactor of validator client

- Block production is working

* Prune old validator client files

* Start works on attestation service

* Add attestation service to validator client

* Use full pubkey for validator directories

* Add validator duties post endpoint

* Use par_iter for keypair generation

* Use bulk duties request in validator client

* Add version http endpoint tests

* Add interop keys and startup wait

* Ensure a prompt exit

* Add duties pruning

* Fix compile error in beacon node tests

* Add github workflow

* Modify rust.yaml

* Modify gitlab actions

* Add to CI file

* Add sudo to CI npm install

* Move cargo fmt to own job in tests

* Fix cargo fmt in CI

* Add rustup update before cargo fmt

* Change name of CI job

* Make other CI jobs require cargo fmt

* Add CI badge

* Remove gitlab and travis files

* Add different http timeout for debug

* Update docker file, use makefile in CI

* Use make in the dockerfile, skip the test

* Use the makefile for debug GI test

* Update book

* Tidy grpc and misc things

* Apply discv5 fixes

* Address other minor issues

* Fix warnings

* Attempt fix for addr parsing

* Tidy validator config, CLIs

* Tidy comments

* Tidy signing, reduce ForkService duplication

* Fail if skipping too many slots

* Set default recent genesis time to 0

* Add custom http timeout to validator

* Fix compile bug in node_test_rig

* Remove old bootstrap flag from val CLI

* Update docs

* Tidy val client

* Change val client log levels

* Add comments, more validity checks

* Fix compile error, add comments

* Undo changes to eth2-libp2p/src

* Reduce duplication of keypair generation

* Add more logging for validator duties

* Fix beacon_chain_sim, nitpicks

* Fix compile error, minor nits

* Update to use v0.9.2 version of deposit contract

* Add efforts to automate eth1 testnet deployment

* Fix lcli testnet deployer

* Modify bn CLI to parse eth2_testnet_dir

* Progress with account_manager deposit tools

* Make account manager submit deposits

* Add password option for submitting deposits

* Allow custom deposit amount

* Add long names to lcli clap

* Add password option to lcli deploy command

* Fix minor bugs whilst testing

* Address Michael's comments

* Add refund-deposit-contract to lcli

* Use time instead of skip count for denying long skips

* Improve logging for eth1

* Fix bug with validator services exiting on error

* Drop the block cache after genesis

* Modify eth1 testnet config

* Improve eth1 logging

* Make validator wait until genesis time

* Fix bug in eth1 voting

* Add more logging to eth1 voting

* Handle errors in eth1 http module

* Set SECONDS_PER_DAY to sensible minimum

* Shorten delay before testnet start

* Ensure eth1 block is produced without any votes

* Improve eth1 logging

* Fix broken tests in eth1

* Tidy code in rest_api

* Fix failing test in deposit_contract

* Make CLI args more consistent

* Change validator/duties endpoint

* Add time-based skip slot limiting

* Add new error type missed in previous commit

* Add log when waiting for genesis

* Refactor beacon node CLI

* Remove unused dep

* Add lcli eth1-genesis command

* Fix bug in master merge

* Apply clippy lints to beacon node

* Add support for YamlConfig in Eth2TestnetDir

* Upgrade tesnet deposit contract version

* Remove unnecessary logging and correct formatting

* Add a hardcoded eth2 testnet config

* Ensure http server flag works. Overwrite configs with flags.

* Ensure boot nodes are loaded from testnet dir

* Fix account manager CLI bugs

* Fix bugs with beacon node cli

* Allow testnet dir without boot nodes

* Write genesis state as SSZ

* Remove ---/n from the start of testnet_dir files

* Set default libp2p address

* Tidy account manager CLI, add logging

* Add check to see if testnet dir exists

* Apply reviewers suggestions

* Add HeadTracker struct

* Add fork choice persistence

* Shorten slot time for simulator

* Add the /beacon/heads API endpoint

* Update hardcoded testnet

* Add tests for BeaconChain persistence + fix bugs

* Extend BeaconChain persistence testing

* Ensure chain is finalized b4 persistence tests

* Ensure boot_enr.yaml is include in binary

* Refactor beacon_chain_sim

* Move files about in beacon sim

* Update beacon_chain_sim

* Fix bug with deposit inclusion

* Increase log in genesis service, fix todo

* Tidy sim, fix broken rest_api tests

* Fix more broken tests

* Update testnet

* Fix broken rest api test

* Tidy account manager CLI

* Use tempdir for account manager

* Stop hardcoded testnet dir from creating dir

* Rename Eth2TestnetDir to Eth2TestnetConfig

* Change hardcoded -> hard_coded

* Tidy account manager

* Add log to account manager

* Tidy, ensure head tracker is loaded from disk

* Tidy beacon chain builder

* Tidy eth1_chain

* Adds log support for simulator

* Revert "Adds log support for simulator"

This reverts commit ec77c66a052350f551db145cf20f213823428dd3.

* Adds log support for simulator

* Tidy after self-review

* Change default log level

* Address Michael's delicious PR comments

* Fix off-by-one in tests

beacon_node/rest_api/Cargo.toml

@@ -22,7 +22,6 @@ eth2_ssz = { path = "../../eth2/utils/ssz" }
 eth2_ssz_derive = { path = "../../eth2/utils/ssz_derive" }
 state_processing = { path = "../../eth2/state_processing" }
 types = { path = "../../eth2/types" }
-clap = "2.33"
 http = "0.1"
 hyper = "0.12"
 exit-future = "0.1.4"

beacon_node/rest_api/src/beacon.rs

@@ -56,6 +56,29 @@ pub fn get_head<T: BeaconChainTypes>(
     ResponseBuilder::new(&req)?.body(&head)
 }
 
+#[derive(Serialize, Deserialize, Encode)]
+pub struct HeadBeaconBlock {
+    beacon_block_root: Hash256,
+    beacon_block_slot: Slot,
+}
+
+/// HTTP handler to return a list of head BeaconBlocks.
+pub fn get_heads<T: BeaconChainTypes>(
+    req: Request<Body>,
+    beacon_chain: Arc<BeaconChain<T>>,
+) -> ApiResult {
+    let heads = beacon_chain
+        .heads()
+        .into_iter()
+        .map(|(beacon_block_root, beacon_block_slot)| HeadBeaconBlock {
+            beacon_block_root,
+            beacon_block_slot,
+        })
+        .collect::<Vec<_>>();
+
+    ResponseBuilder::new(&req)?.body(&heads)
+}
+
 #[derive(Serialize, Encode)]
 #[serde(bound = "T: EthSpec")]
 pub struct BlockResponse<T: EthSpec> {

beacon_node/rest_api/src/config.rs

@@ -1,4 +1,3 @@
-use clap::ArgMatches;
 use serde::{Deserialize, Serialize};
 use std::net::Ipv4Addr;
 
@@ -50,25 +49,3 @@ impl Default for Config {
         }
     }
 }
-
-impl Config {
-    pub fn apply_cli_args(&mut self, args: &ArgMatches) -> Result<(), &'static str> {
-        if args.is_present("no-api") {
-            self.enabled = false;
-        }
-
-        if let Some(rpc_address) = args.value_of("api-address") {
-            self.listen_address = rpc_address
-                .parse::<Ipv4Addr>()
-                .map_err(|_| "api-address is not a valid IPv4 address.")?;
-        }
-
-        if let Some(rpc_port) = args.value_of("api-port") {
-            self.port = rpc_port
-                .parse::<u16>()
-                .map_err(|_| "api-port is not a valid u16.")?;
-        }
-
-        Ok(())
-    }
-}

beacon_node/rest_api/src/router.rs

@@ -64,6 +64,9 @@ pub fn route<T: BeaconChainTypes>(
 
             // Methods for Beacon Node
             (&Method::GET, "/beacon/head") => into_boxfut(beacon::get_head::<T>(req, beacon_chain)),
+            (&Method::GET, "/beacon/heads") => {
+                into_boxfut(beacon::get_heads::<T>(req, beacon_chain))
+            }
             (&Method::GET, "/beacon/block") => {
                 into_boxfut(beacon::get_block::<T>(req, beacon_chain))
             }
@@ -154,7 +157,7 @@ pub fn route<T: BeaconChainTypes>(
     // (e.g., a response with a 404 or 500 status).
     request_result.then(move |result| match result {
         Ok(response) => {
-            debug!(local_log, "Request successful: {:?}", path);
+            debug!(local_log, "HTTP API request successful"; "path" => path);
             metrics::inc_counter(&metrics::SUCCESS_COUNT);
             metrics::stop_timer(timer);
 
@@ -163,7 +166,7 @@ pub fn route<T: BeaconChainTypes>(
         Err(e) => {
             let error_response = e.into();
 
-            debug!(local_log, "Request failure: {:?}", path);
+            debug!(local_log, "HTTP API request failure"; "path" => path);
             metrics::stop_timer(timer);
 
             Ok(error_response)

beacon_node/rest_api/tests/test.rs

@@ -3,7 +3,7 @@
 use beacon_chain::{BeaconChain, BeaconChainTypes};
 use node_test_rig::{
     environment::{Environment, EnvironmentBuilder},
-    testing_client_config, ClientGenesis, LocalBeaconNode,
+    testing_client_config, ClientConfig, ClientGenesis, LocalBeaconNode,
 };
 use remote_beacon_node::{PublishStatus, ValidatorDuty};
 use std::sync::Arc;
@@ -26,6 +26,13 @@ fn build_env() -> Environment<E> {
         .expect("environment should build")
 }
 
+fn build_node<E: EthSpec>(env: &mut Environment<E>, config: ClientConfig) -> LocalBeaconNode<E> {
+    let context = env.core_context();
+    env.runtime()
+        .block_on(LocalBeaconNode::production(context, config))
+        .expect("should block until node created")
+}
+
 /// Returns the randao reveal for the given slot (assuming the given `beacon_chain` uses
 /// deterministic keypairs).
 fn get_randao_reveal<T: BeaconChainTypes>(
@@ -64,7 +71,7 @@ fn validator_produce_attestation() {
 
     let spec = &E::default_spec();
 
-    let node = LocalBeaconNode::production(env.core_context(), testing_client_config());
+    let node = build_node(&mut env, testing_client_config());
     let remote_node = node.remote_node().expect("should produce remote node");
 
     let beacon_chain = node
@@ -160,7 +167,7 @@ fn validator_duties_bulk() {
 
     let spec = &E::default_spec();
 
-    let node = LocalBeaconNode::production(env.core_context(), testing_client_config());
+    let node = build_node(&mut env, testing_client_config());
     let remote_node = node.remote_node().expect("should produce remote node");
 
     let beacon_chain = node
@@ -197,7 +204,7 @@ fn validator_duties() {
 
     let spec = &E::default_spec();
 
-    let node = LocalBeaconNode::production(env.core_context(), testing_client_config());
+    let node = build_node(&mut env, testing_client_config());
     let remote_node = node.remote_node().expect("should produce remote node");
 
     let beacon_chain = node
@@ -321,7 +328,7 @@ fn validator_block_post() {
         genesis_time: 13_371_337,
     };
 
-    let node = LocalBeaconNode::production(env.core_context(), config);
+    let node = build_node(&mut env, config);
     let remote_node = node.remote_node().expect("should produce remote node");
 
     let beacon_chain = node
@@ -387,7 +394,7 @@ fn validator_block_get() {
 
     let spec = &E::default_spec();
 
-    let node = LocalBeaconNode::production(env.core_context(), testing_client_config());
+    let node = build_node(&mut env, testing_client_config());
     let remote_node = node.remote_node().expect("should produce remote node");
 
     let beacon_chain = node
@@ -425,7 +432,7 @@ fn validator_block_get() {
 fn beacon_state() {
     let mut env = build_env();
 
-    let node = LocalBeaconNode::production(env.core_context(), testing_client_config());
+    let node = build_node(&mut env, testing_client_config());
     let remote_node = node.remote_node().expect("should produce remote node");
 
     let (state_by_slot, root) = env
@@ -469,7 +476,7 @@ fn beacon_state() {
 fn beacon_block() {
     let mut env = build_env();
 
-    let node = LocalBeaconNode::production(env.core_context(), testing_client_config());
+    let node = build_node(&mut env, testing_client_config());
     let remote_node = node.remote_node().expect("should produce remote node");
 
     let (block_by_slot, root) = env
@@ -513,7 +520,7 @@ fn beacon_block() {
 fn genesis_time() {
     let mut env = build_env();
 
-    let node = LocalBeaconNode::production(env.core_context(), testing_client_config());
+    let node = build_node(&mut env, testing_client_config());
     let remote_node = node.remote_node().expect("should produce remote node");
 
     let genesis_time = env
@@ -537,7 +544,7 @@ fn genesis_time() {
 fn fork() {
     let mut env = build_env();
 
-    let node = LocalBeaconNode::production(env.core_context(), testing_client_config());
+    let node = build_node(&mut env, testing_client_config());
     let remote_node = node.remote_node().expect("should produce remote node");
 
     let fork = env
@@ -561,7 +568,7 @@ fn fork() {
 fn eth2_config() {
     let mut env = build_env();
 
-    let node = LocalBeaconNode::production(env.core_context(), testing_client_config());
+    let node = build_node(&mut env, testing_client_config());
     let remote_node = node.remote_node().expect("should produce remote node");
 
     let eth2_config = env
@@ -585,7 +592,7 @@ fn eth2_config() {
 fn get_version() {
     let mut env = build_env();
 
-    let node = LocalBeaconNode::production(env.core_context(), testing_client_config());
+    let node = build_node(&mut env, testing_client_config());
     let remote_node = node.remote_node().expect("should produce remote node");
 
     let version = env