Prepare for public testnet (#628)
* Update to spec v0.9.0 * Update to v0.9.1 * Bump spec tags for v0.9.1 * Formatting, fix CI failures * Resolve accidental KeyPair merge conflict * Document new BeaconState functions * Add `validator` changes from `validator-to-rest` * Add initial (failing) REST api tests * Fix signature parsing * Add more tests * Refactor http router * Add working tests for publish beacon block * Add validator duties tests * Move account_manager under `lighthouse` binary * Unify logfile handling in `environment` crate. * Fix incorrect cache drops in `advance_caches` * Update fork choice for v0.9.1 * Add `deposit_contract` crate * Add progress on validator onboarding * Add unfinished attesation code * Update account manager CLI * Write eth1 data file as hex string * Integrate ValidatorDirectory with validator_client * Move ValidatorDirectory into validator_client * Clean up some FIXMEs * Add beacon_chain_sim * Fix a few docs/logs * Expand `beacon_chain_sim` * Fix spec for `beacon_chain_sim * More testing for api * Start work on attestation endpoint * Reject empty attestations * Allow attestations to genesis block * Add working tests for `rest_api` validator endpoint * Remove grpc from beacon_node * Start heavy refactor of validator client - Block production is working * Prune old validator client files * Start works on attestation service * Add attestation service to validator client * Use full pubkey for validator directories * Add validator duties post endpoint * Use par_iter for keypair generation * Use bulk duties request in validator client * Add version http endpoint tests * Add interop keys and startup wait * Ensure a prompt exit * Add duties pruning * Fix compile error in beacon node tests * Add github workflow * Modify rust.yaml * Modify gitlab actions * Add to CI file * Add sudo to CI npm install * Move cargo fmt to own job in tests * Fix cargo fmt in CI * Add rustup update before cargo fmt * Change name of CI job * Make other CI jobs require cargo fmt * Add CI badge * Remove gitlab and travis files * Add different http timeout for debug * Update docker file, use makefile in CI * Use make in the dockerfile, skip the test * Use the makefile for debug GI test * Update book * Tidy grpc and misc things * Apply discv5 fixes * Address other minor issues * Fix warnings * Attempt fix for addr parsing * Tidy validator config, CLIs * Tidy comments * Tidy signing, reduce ForkService duplication * Fail if skipping too many slots * Set default recent genesis time to 0 * Add custom http timeout to validator * Fix compile bug in node_test_rig * Remove old bootstrap flag from val CLI * Update docs * Tidy val client * Change val client log levels * Add comments, more validity checks * Fix compile error, add comments * Undo changes to eth2-libp2p/src * Reduce duplication of keypair generation * Add more logging for validator duties * Fix beacon_chain_sim, nitpicks * Fix compile error, minor nits * Update to use v0.9.2 version of deposit contract * Add efforts to automate eth1 testnet deployment * Fix lcli testnet deployer * Modify bn CLI to parse eth2_testnet_dir * Progress with account_manager deposit tools * Make account manager submit deposits * Add password option for submitting deposits * Allow custom deposit amount * Add long names to lcli clap * Add password option to lcli deploy command * Fix minor bugs whilst testing * Address Michael's comments * Add refund-deposit-contract to lcli * Use time instead of skip count for denying long skips * Improve logging for eth1 * Fix bug with validator services exiting on error * Drop the block cache after genesis * Modify eth1 testnet config * Improve eth1 logging * Make validator wait until genesis time * Fix bug in eth1 voting * Add more logging to eth1 voting * Handle errors in eth1 http module * Set SECONDS_PER_DAY to sensible minimum * Shorten delay before testnet start * Ensure eth1 block is produced without any votes * Improve eth1 logging * Fix broken tests in eth1 * Tidy code in rest_api * Fix failing test in deposit_contract * Make CLI args more consistent * Change validator/duties endpoint * Add time-based skip slot limiting * Add new error type missed in previous commit * Add log when waiting for genesis * Refactor beacon node CLI * Remove unused dep * Add lcli eth1-genesis command * Fix bug in master merge * Apply clippy lints to beacon node * Add support for YamlConfig in Eth2TestnetDir * Upgrade tesnet deposit contract version * Remove unnecessary logging and correct formatting * Add a hardcoded eth2 testnet config * Ensure http server flag works. Overwrite configs with flags. * Ensure boot nodes are loaded from testnet dir * Fix account manager CLI bugs * Fix bugs with beacon node cli * Allow testnet dir without boot nodes * Write genesis state as SSZ * Remove ---/n from the start of testnet_dir files * Set default libp2p address * Tidy account manager CLI, add logging * Add check to see if testnet dir exists * Apply reviewers suggestions * Add HeadTracker struct * Add fork choice persistence * Shorten slot time for simulator * Add the /beacon/heads API endpoint * Update hardcoded testnet * Add tests for BeaconChain persistence + fix bugs * Extend BeaconChain persistence testing * Ensure chain is finalized b4 persistence tests * Ensure boot_enr.yaml is include in binary * Refactor beacon_chain_sim * Move files about in beacon sim * Update beacon_chain_sim * Fix bug with deposit inclusion * Increase log in genesis service, fix todo * Tidy sim, fix broken rest_api tests * Fix more broken tests * Update testnet * Fix broken rest api test * Tidy account manager CLI * Use tempdir for account manager * Stop hardcoded testnet dir from creating dir * Rename Eth2TestnetDir to Eth2TestnetConfig * Change hardcoded -> hard_coded * Tidy account manager * Add log to account manager * Tidy, ensure head tracker is loaded from disk * Tidy beacon chain builder * Tidy eth1_chain * Adds log support for simulator * Revert "Adds log support for simulator" This reverts commit ec77c66a052350f551db145cf20f213823428dd3. * Adds log support for simulator * Tidy after self-review * Change default log level * Address Michael's delicious PR comments * Fix off-by-one in tests
This commit is contained in:
Paul Hauner
@@ -28,7 +28,6 @@ serde_yaml = "0.8.11"
|
||||
slog = { version = "2.5.2", features = ["max_level_trace"] }
|
||||
slog-async = "2.3.0"
|
||||
tokio = "0.1.22"
|
||||
clap = "2.33.0"
|
||||
dirs = "2.0.2"
|
||||
exit-future = "0.1.4"
|
||||
futures = "0.1.29"
|
||||
@@ -39,3 +38,4 @@ eth1 = { path = "../eth1" }
|
||||
genesis = { path = "../genesis" }
|
||||
environment = { path = "../../lighthouse/environment" }
|
||||
lighthouse_bootstrap = { path = "../../eth2/utils/lighthouse_bootstrap" }
|
||||
eth2_ssz = { path = "../../eth2/utils/ssz" }
|
||||
|
||||
@@ -23,13 +23,14 @@ use lighthouse_bootstrap::Bootstrapper;
|
||||
use lmd_ghost::LmdGhost;
|
||||
use network::{NetworkConfig, NetworkMessage, Service as NetworkService};
|
||||
use slog::{debug, error, info, warn};
|
||||
use ssz::Decode;
|
||||
use std::net::SocketAddr;
|
||||
use std::path::Path;
|
||||
use std::sync::Arc;
|
||||
use std::time::{Duration, Instant};
|
||||
use tokio::sync::mpsc::UnboundedSender;
|
||||
use tokio::timer::Interval;
|
||||
use types::{ChainSpec, EthSpec};
|
||||
use types::{BeaconState, ChainSpec, EthSpec};
|
||||
use websocket_server::{Config as WebSocketConfig, WebSocketSender};
|
||||
|
||||
/// The interval between notifier events.
|
||||
@@ -37,7 +38,7 @@ pub const NOTIFIER_INTERVAL_SECONDS: u64 = 15;
|
||||
/// Create a warning log whenever the peer count is at or below this value.
|
||||
pub const WARN_PEER_COUNT: usize = 1;
|
||||
/// Interval between polling the eth1 node for genesis information.
|
||||
pub const ETH1_GENESIS_UPDATE_INTERVAL_MILLIS: u64 = 500;
|
||||
pub const ETH1_GENESIS_UPDATE_INTERVAL_MILLIS: u64 = 7_000;
|
||||
|
||||
/// Builds a `Client` instance.
|
||||
///
|
||||
@@ -148,7 +149,7 @@ where
|
||||
})?;
|
||||
let context = runtime_context
|
||||
.ok_or_else(|| "beacon_chain_start_method requires a log".to_string())?
|
||||
.service_context("beacon");
|
||||
.service_context("beacon".into());
|
||||
let spec = chain_spec
|
||||
.ok_or_else(|| "beacon_chain_start_method requires a chain spec".to_string())?;
|
||||
|
||||
@@ -187,40 +188,34 @@ where
|
||||
|
||||
Box::new(future)
|
||||
}
|
||||
ClientGenesis::DepositContract => {
|
||||
let genesis_service = Eth1GenesisService::new(
|
||||
// Some of the configuration options for `Eth1Config` are
|
||||
// hard-coded when listening for genesis from the deposit contract.
|
||||
//
|
||||
// The idea is that the `Eth1Config` supplied to this function
|
||||
// (`config`) is intended for block production duties (i.e.,
|
||||
// listening for deposit events and voting on eth1 data) and that
|
||||
// we can make listening for genesis more efficient if we modify
|
||||
// some params.
|
||||
Eth1Config {
|
||||
// Truncating the block cache makes searching for genesis more
|
||||
// complicated.
|
||||
block_cache_truncation: None,
|
||||
// Scan large ranges of blocks when awaiting genesis.
|
||||
blocks_per_log_query: 1_000,
|
||||
// Only perform a single log request each time the eth1 node is
|
||||
// polled.
|
||||
//
|
||||
// For small testnets this makes finding genesis much faster,
|
||||
// as it usually happens within 1,000 blocks.
|
||||
max_log_requests_per_update: Some(1),
|
||||
// Only perform a single block request each time the eth1 node
|
||||
// is polled.
|
||||
//
|
||||
// For small testnets, this is much faster as they do not have
|
||||
// a `MIN_GENESIS_SECONDS`, so after `MIN_GENESIS_VALIDATOR_COUNT`
|
||||
// has been reached only a single block needs to be read.
|
||||
max_blocks_per_update: Some(1),
|
||||
..config
|
||||
},
|
||||
context.log.clone(),
|
||||
ClientGenesis::SszBytes {
|
||||
genesis_state_bytes,
|
||||
} => {
|
||||
info!(
|
||||
context.log,
|
||||
"Starting from known genesis state";
|
||||
);
|
||||
|
||||
let result = BeaconState::from_ssz_bytes(&genesis_state_bytes)
|
||||
.map_err(|e| format!("Unable to parse genesis state SSZ: {:?}", e));
|
||||
|
||||
let future = result
|
||||
.and_then(move |genesis_state| builder.genesis_state(genesis_state))
|
||||
.into_future()
|
||||
.map(|v| (v, None));
|
||||
|
||||
Box::new(future)
|
||||
}
|
||||
ClientGenesis::DepositContract => {
|
||||
info!(
|
||||
context.log,
|
||||
"Waiting for eth2 genesis from eth1";
|
||||
"eth1_node" => &config.endpoint
|
||||
);
|
||||
|
||||
let genesis_service =
|
||||
Eth1GenesisService::new(config, context.log.clone());
|
||||
|
||||
let future = genesis_service
|
||||
.wait_for_genesis_state(
|
||||
Duration::from_millis(ETH1_GENESIS_UPDATE_INTERVAL_MILLIS),
|
||||
@@ -275,7 +270,7 @@ where
|
||||
.runtime_context
|
||||
.as_ref()
|
||||
.ok_or_else(|| "libp2p_network requires a runtime_context")?
|
||||
.service_context("network");
|
||||
.service_context("network".into());
|
||||
|
||||
let (network, network_send) =
|
||||
NetworkService::new(beacon_chain, config, &context.executor, context.log)
|
||||
@@ -301,7 +296,7 @@ where
|
||||
.runtime_context
|
||||
.as_ref()
|
||||
.ok_or_else(|| "http_server requires a runtime_context")?
|
||||
.service_context("http");
|
||||
.service_context("http".into());
|
||||
let network = self
|
||||
.libp2p_network
|
||||
.clone()
|
||||
@@ -341,7 +336,7 @@ where
|
||||
.runtime_context
|
||||
.as_ref()
|
||||
.ok_or_else(|| "peer_count_notifier requires a runtime_context")?
|
||||
.service_context("peer_notifier");
|
||||
.service_context("peer_notifier".into());
|
||||
let log = context.log.clone();
|
||||
let log_2 = context.log.clone();
|
||||
let network = self
|
||||
@@ -384,7 +379,7 @@ where
|
||||
.runtime_context
|
||||
.as_ref()
|
||||
.ok_or_else(|| "slot_notifier requires a runtime_context")?
|
||||
.service_context("slot_notifier");
|
||||
.service_context("slot_notifier".into());
|
||||
let log = context.log.clone();
|
||||
let log_2 = log.clone();
|
||||
let beacon_chain = self
|
||||
@@ -498,7 +493,7 @@ where
|
||||
.clone()
|
||||
.ok_or_else(|| "beacon_chain requires a slot clock")?,
|
||||
)
|
||||
.empty_reduced_tree_fork_choice()
|
||||
.reduced_tree_fork_choice()
|
||||
.map_err(|e| format!("Failed to init fork choice: {}", e))?
|
||||
.build()
|
||||
.map_err(|e| format!("Failed to build beacon chain: {}", e))?;
|
||||
@@ -537,7 +532,7 @@ where
|
||||
.runtime_context
|
||||
.as_ref()
|
||||
.ok_or_else(|| "websocket_event_handler requires a runtime_context")?
|
||||
.service_context("ws");
|
||||
.service_context("ws".into());
|
||||
|
||||
let (sender, exit_signal, listening_addr): (
|
||||
WebSocketSender<TEthSpec>,
|
||||
@@ -587,7 +582,7 @@ where
|
||||
.runtime_context
|
||||
.as_ref()
|
||||
.ok_or_else(|| "disk_store requires a log".to_string())?
|
||||
.service_context("freezer_db");
|
||||
.service_context("freezer_db".into());
|
||||
let spec = self
|
||||
.chain_spec
|
||||
.clone()
|
||||
@@ -715,7 +710,7 @@ where
|
||||
.runtime_context
|
||||
.as_ref()
|
||||
.ok_or_else(|| "caching_eth1_backend requires a runtime_context")?
|
||||
.service_context("eth1_rpc");
|
||||
.service_context("eth1_rpc".into());
|
||||
let beacon_chain_builder = self
|
||||
.beacon_chain_builder
|
||||
.ok_or_else(|| "caching_eth1_backend requires a beacon_chain_builder")?;
|
||||
@@ -726,6 +721,17 @@ where
|
||||
|
||||
let backend = if let Some(eth1_service_from_genesis) = self.eth1_service {
|
||||
eth1_service_from_genesis.update_config(config.clone())?;
|
||||
|
||||
// This cache is not useful because it's first (earliest) block likely the block that
|
||||
// triggered genesis.
|
||||
//
|
||||
// In order to vote we need to be able to go back at least 2 * `ETH1_FOLLOW_DISTANCE`
|
||||
// from the genesis-triggering block. Presently the block cache does not support
|
||||
// importing blocks with decreasing block numbers, it only accepts them in increasing
|
||||
// order. If this turns out to be a bottleneck we can update the block cache to allow
|
||||
// adding earlier blocks too.
|
||||
eth1_service_from_genesis.drop_block_cache();
|
||||
|
||||
CachingEth1Backend::from_service(eth1_service_from_genesis, store)
|
||||
} else {
|
||||
CachingEth1Backend::new(config, context.log, store)
|
||||
|
||||
@@ -1,4 +1,3 @@
|
||||
use clap::ArgMatches;
|
||||
use network::NetworkConfig;
|
||||
use serde_derive::{Deserialize, Serialize};
|
||||
use std::fs;
|
||||
@@ -25,6 +24,11 @@ pub enum ClientGenesis {
|
||||
DepositContract,
|
||||
/// Loads the genesis state from a SSZ-encoded `BeaconState` file.
|
||||
SszFile { path: PathBuf },
|
||||
/// Loads the genesis state from SSZ-encoded `BeaconState` bytes.
|
||||
///
|
||||
/// We include the bytes instead of the `BeaconState<E>` because the `EthSpec` type
|
||||
/// parameter would be very annoying.
|
||||
SszBytes { genesis_state_bytes: Vec<u8> },
|
||||
/// Connects to another Lighthouse instance and reads the genesis state and other data via the
|
||||
/// HTTP API.
|
||||
RemoteNode { server: String, port: Option<u16> },
|
||||
@@ -40,9 +44,10 @@ impl Default for ClientGenesis {
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct Config {
|
||||
pub data_dir: PathBuf,
|
||||
pub testnet_dir: Option<PathBuf>,
|
||||
pub db_type: String,
|
||||
db_name: String,
|
||||
freezer_db_path: Option<PathBuf>,
|
||||
pub db_name: String,
|
||||
pub freezer_db_path: Option<PathBuf>,
|
||||
pub log_file: PathBuf,
|
||||
pub spec_constants: String,
|
||||
/// If true, the node will use co-ordinated junk for eth1 values.
|
||||
@@ -64,12 +69,13 @@ impl Default for Config {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
data_dir: PathBuf::from(".lighthouse"),
|
||||
testnet_dir: None,
|
||||
log_file: PathBuf::from(""),
|
||||
db_type: "disk".to_string(),
|
||||
db_name: "chain_db".to_string(),
|
||||
freezer_db_path: None,
|
||||
genesis: <_>::default(),
|
||||
network: NetworkConfig::new(),
|
||||
network: NetworkConfig::default(),
|
||||
rest_api: <_>::default(),
|
||||
websocket_server: <_>::default(),
|
||||
spec_constants: TESTNET_SPEC_CONSTANTS.into(),
|
||||
@@ -135,26 +141,6 @@ impl Config {
|
||||
.ok_or_else(|| "Unable to locate user home directory".to_string())?;
|
||||
ensure_dir_exists(path)
|
||||
}
|
||||
|
||||
/// Apply the following arguments to `self`, replacing values if they are specified in `args`.
|
||||
///
|
||||
/// Returns an error if arguments are obviously invalid. May succeed even if some values are
|
||||
/// invalid.
|
||||
pub fn apply_cli_args(&mut self, args: &ArgMatches, _log: &slog::Logger) -> Result<(), String> {
|
||||
if let Some(dir) = args.value_of("datadir") {
|
||||
self.data_dir = PathBuf::from(dir);
|
||||
};
|
||||
|
||||
if let Some(freezer_dir) = args.value_of("freezer-dir") {
|
||||
self.freezer_db_path = Some(PathBuf::from(freezer_dir));
|
||||
}
|
||||
|
||||
self.network.apply_cli_args(args)?;
|
||||
self.rest_api.apply_cli_args(args)?;
|
||||
self.websocket_server.apply_cli_args(args)?;
|
||||
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
/// Ensure that the directory at `path` exists, by creating it and all parents if necessary.
|
||||
|
||||
Reference in New Issue
Block a user