lighthouse/validator_client/src/config.rs

371 lines
14 KiB
Rust
Raw Normal View History

use crate::graffiti_file::GraffitiFile;
use crate::{http_api, http_metrics};
use clap::ArgMatches;
use clap_utils::{flags::DISABLE_MALLOC_TUNING_FLAG, parse_optional, parse_required};
use directory::{
get_network_dir, DEFAULT_HARDCODED_NETWORK, DEFAULT_ROOT_DIR, DEFAULT_SECRET_DIR,
DEFAULT_VALIDATOR_DIR,
};
Implement standard eth2.0 API (#1569) - Resolves #1550 - Resolves #824 - Resolves #825 - Resolves #1131 - Resolves #1411 - Resolves #1256 - Resolve #1177 - Includes the `ShufflingId` struct initially defined in #1492. That PR is now closed and the changes are included here, with significant bug fixes. - Implement the https://github.com/ethereum/eth2.0-APIs in a new `http_api` crate using `warp`. This replaces the `rest_api` crate. - Add a new `common/eth2` crate which provides a wrapper around `reqwest`, providing the HTTP client that is used by the validator client and for testing. This replaces the `common/remote_beacon_node` crate. - Create a `http_metrics` crate which is a dedicated server for Prometheus metrics (they are no longer served on the same port as the REST API). We now have flags for `--metrics`, `--metrics-address`, etc. - Allow the `subnet_id` to be an optional parameter for `VerifiedUnaggregatedAttestation::verify`. This means it does not need to be provided unnecessarily by the validator client. - Move `fn map_attestation_committee` in `mod beacon_chain::attestation_verification` to a new `fn with_committee_cache` on the `BeaconChain` so the same cache can be used for obtaining validator duties. - Add some other helpers to `BeaconChain` to assist with common API duties (e.g., `block_root_at_slot`, `head_beacon_block_root`). - Change the `NaiveAggregationPool` so it can index attestations by `hash_tree_root(attestation.data)`. This is a requirement of the API. - Add functions to `BeaconChainHarness` to allow it to create slashings and exits. - Allow for `eth1::Eth1NetworkId` to go to/from a `String`. - Add functions to the `OperationPool` to allow getting all objects in the pool. - Add function to `BeaconState` to check if a committee cache is initialized. - Fix bug where `seconds_per_eth1_block` was not transferring over from `YamlConfig` to `ChainSpec`. - Add the `deposit_contract_address` to `YamlConfig` and `ChainSpec`. We needed to be able to return it in an API response. - Change some uses of serde `serialize_with` and `deserialize_with` to a single use of `with` (code quality). - Impl `Display` and `FromStr` for several BLS fields. - Check for clock discrepancy when VC polls BN for sync state (with +/- 1 slot tolerance). This is not intended to be comprehensive, it was just easy to do. - See #1434 for a per-endpoint overview. - Seeking clarity here: https://github.com/ethereum/eth2.0-APIs/issues/75 - [x] Add docs for prom port to close #1256 - [x] Follow up on this #1177 - [x] ~~Follow up with #1424~~ Will fix in future PR. - [x] Follow up with #1411 - [x] ~~Follow up with #1260~~ Will fix in future PR. - [x] Add quotes to all integers. - [x] Remove `rest_types` - [x] Address missing beacon block error. (#1629) - [x] ~~Add tests for lighthouse/peers endpoints~~ Wontfix - [x] ~~Follow up with validator status proposal~~ Tracked in #1434 - [x] Unify graffiti structs - [x] ~~Start server when waiting for genesis?~~ Will fix in future PR. - [x] TODO in http_api tests - [x] Move lighthouse endpoints off /eth/v1 - [x] Update docs to link to standard - ~~Blocked on #1586~~ Co-authored-by: Michael Sproul <michael@sigmaprime.io>
2020-09-29 03:46:54 +00:00
use eth2::types::Graffiti;
use sensitive_url::SensitiveUrl;
use serde_derive::{Deserialize, Serialize};
use slog::{info, warn, Logger};
use std::fs;
use std::net::IpAddr;
use std::path::PathBuf;
Enable proposer boost re-orging (#2860) ## Proposed Changes With proposer boosting implemented (#2822) we have an opportunity to re-org out late blocks. This PR adds three flags to the BN to control this behaviour: * `--disable-proposer-reorgs`: turn aggressive re-orging off (it's on by default). * `--proposer-reorg-threshold N`: attempt to orphan blocks with less than N% of the committee vote. If this parameter isn't set then N defaults to 20% when the feature is enabled. * `--proposer-reorg-epochs-since-finalization N`: only attempt to re-org late blocks when the number of epochs since finalization is less than or equal to N. The default is 2 epochs, meaning re-orgs will only be attempted when the chain is finalizing optimally. For safety Lighthouse will only attempt a re-org under very specific conditions: 1. The block being proposed is 1 slot after the canonical head, and the canonical head is 1 slot after its parent. i.e. at slot `n + 1` rather than building on the block from slot `n` we build on the block from slot `n - 1`. 2. The current canonical head received less than N% of the committee vote. N should be set depending on the proposer boost fraction itself, the fraction of the network that is believed to be applying it, and the size of the largest entity that could be hoarding votes. 3. The current canonical head arrived after the attestation deadline from our perspective. This condition was only added to support suppression of forkchoiceUpdated messages, but makes intuitive sense. 4. The block is being proposed in the first 2 seconds of the slot. This gives it time to propagate and receive the proposer boost. ## Additional Info For the initial idea and background, see: https://github.com/ethereum/consensus-specs/pull/2353#issuecomment-950238004 There is also a specification for this feature here: https://github.com/ethereum/consensus-specs/pull/3034 Co-authored-by: Michael Sproul <micsproul@gmail.com> Co-authored-by: pawan <pawandhananjay@gmail.com>
2022-12-13 09:57:26 +00:00
use std::time::Duration;
use types::{Address, GRAFFITI_BYTES_LEN};
pub const DEFAULT_BEACON_NODE: &str = "http://localhost:5052/";
2019-09-01 09:33:43 +00:00
/// Stores the core configuration for this validator instance.
#[derive(Clone, Serialize, Deserialize)]
pub struct Config {
/// The data directory, which stores all validator databases
pub validator_dir: PathBuf,
Wallet-based, encrypted key management (#1138) * Update hashmap hashset to stable futures * Adds panic test to hashset delay * Port remote_beacon_node to stable futures * Fix lcli merge conflicts * Non rpc stuff compiles * Remove padding * Add error enum, zeroize more things * Fix comment * protocol.rs compiles * Port websockets, timer and notifier to stable futures (#1035) * Fix lcli * Port timer to stable futures * Fix timer * Port websocket_server to stable futures * Port notifier to stable futures * Add TODOS * Port remote_beacon_node to stable futures * Partial eth2-libp2p stable future upgrade * Finished first round of fighting RPC types * Further progress towards porting eth2-libp2p adds caching to discovery * Update behaviour * Add keystore builder * Remove keystore stuff from val client * Add more tests, comments * RPC handler to stable futures * Update RPC to master libp2p * Add more comments, test vectors * Network service additions * Progress on improving JSON validation * More JSON verification * Start moving JSON into own mod * Remove old code * Add more tests, reader/writers * Tidy * Move keystore into own file * Move more logic into keystore file * Tidy * Tidy * Fix the fallback transport construction (#1102) * Allow for odd-character hex * Correct warning * Remove hashmap delay * Compiling version of eth2-libp2p * Update all crates versions * Fix conversion function and add tests (#1113) * Add more json missing field checks * Use scrypt by default * Tidy, address comments * Test path and uuid in vectors * Fix comment * Add checks for kdf params * Enforce empty kdf message * Port validator_client to stable futures (#1114) * Add PH & MS slot clock changes * Account for genesis time * Add progress on duties refactor * Add simple is_aggregator bool to val subscription * Start work on attestation_verification.rs * Add progress on ObservedAttestations * Progress with ObservedAttestations * Fix tests * Add observed attestations to the beacon chain * Add attestation observation to processing code * Add progress on attestation verification * Add first draft of ObservedAttesters * Add more tests * Add observed attesters to beacon chain * Add observers to attestation processing * Add more attestation verification * Create ObservedAggregators map * Remove commented-out code * Add observed aggregators into chain * Add progress * Finish adding features to attestation verification * Ensure beacon chain compiles * Link attn verification into chain * Integrate new attn verification in chain * Remove old attestation processing code * Start trying to fix beacon_chain tests * Split adding into pools into two functions * Add aggregation to harness * Get test harness working again * Adjust the number of aggregators for test harness * Fix edge-case in harness * Integrate new attn processing in network * Fix compile bug in validator_client * Update validator API endpoints * Fix aggreagation in test harness * Fix enum thing * Fix attestation observation bug: * Patch failing API tests * Start adding comments to attestation verification * Remove unused attestation field * Unify "is block known" logic * Update comments * Supress fork choice errors for network processing * Add todos * Tidy * Add gossip attn tests * Disallow test harness to produce old attns * Comment out in-progress tests * Partially address pruning tests * Fix failing store test * Add aggregate tests * Add comments about which spec conditions we check * Dont re-aggregate * Split apart test harness attn production * Fix compile error in network * Make progress on commented-out test * Fix skipping attestation test * Add fork choice verification tests * Tidy attn tests, remove dead code * Remove some accidentally added code * Fix clippy lint * Rename test file * Add block tests, add cheap block proposer check * Rename block testing file * Add observed_block_producers * Tidy * Switch around block signature verification * Finish block testing * Remove gossip from signature tests * First pass of self review * Fix deviation in spec * Update test spec tags * Start moving over to hashset * Finish moving observed attesters to hashmap * Move aggregation pool over to hashmap * Make fc attn borrow again * Fix rest_api compile error * Fix missing comments * Fix monster test * Uncomment increasing slots test * Address remaining comments * Remove unsafe, use cfg test * Remove cfg test flag * Fix dodgy comment * Revert "Update hashmap hashset to stable futures" This reverts commit d432378a3cc5cd67fc29c0b15b96b886c1323554. * Revert "Adds panic test to hashset delay" This reverts commit 281502396fc5b90d9c421a309c2c056982c9525b. * Ported attestation_service * Ported duties_service * Ported fork_service * More ports * Port block_service * Minor fixes * VC compiles * Update TODOS * Borrow self where possible * Ignore aggregates that are already known. * Unify aggregator modulo logic * Fix typo in logs * Refactor validator subscription logic * Avoid reproducing selection proof * Skip HTTP call if no subscriptions * Rename DutyAndState -> DutyAndProof * Tidy logs * Print root as dbg * Fix compile errors in tests * Fix compile error in test * Re-Fix attestation and duties service * Minor fixes Co-authored-by: Paul Hauner <paul@paulhauner.com> * Expose json_keystore mod * First commits on path derivation * Progress with implementation * More progress * Passing intermediate test vectors * Tidy, add comments * Add DerivedKey structs * Move key derivation into own crate * Add zeroize structs * Return error for empty seed * Add tests * Tidy * First commits on path derivation * Progress with implementation * Move key derivation into own crate * Start defining JSON wallet * Add progress * Split out encrypt/decrypt * First commits on path derivation * Progress with implementation * More progress * Passing intermediate test vectors * Tidy, add comments * Add DerivedKey structs * Move key derivation into own crate * Add zeroize structs * Return error for empty seed * Add tests * Tidy * Add progress * Replace some password usage with slice * First commits on path derivation * Progress with implementation * More progress * Passing intermediate test vectors * Tidy, add comments * Add DerivedKey structs * Move key derivation into own crate * Add zeroize structs * Return error for empty seed * Add tests * Tidy * Add progress * Expose PlainText struct * First commits on path derivation * Progress with implementation * More progress * Passing intermediate test vectors * Tidy, add comments * Add DerivedKey structs * Move key derivation into own crate * Add zeroize structs * Return error for empty seed * Add tests * Tidy * Add builder * Expose consts, remove Password * Minor progress * Expose SALT_SIZE * First compiling version * Add test vectors * Network crate update to stable futures * Move dbg assert statement * Port account_manager to stable futures (#1121) * Port account_manager to stable futures * Run async fns in tokio environment * Port rest_api crate to stable futures (#1118) * Port rest_api lib to stable futures * Reduce tokio features * Update notifier to stable futures * Builder update * Further updates * Add mnemonic, tidy * Convert self referential async functions * Tidy * Add testing * Add first attempt at validator_dir * Present pubkey field * stable futures fixes (#1124) * Fix eth1 update functions * Fix genesis and client * Fix beacon node lib * Return appropriate runtimes from environment * Fix test rig * Refactor eth1 service update * Upgrade simulator to stable futures * Lighthouse compiles on stable futures * Add first pass of wallet manager * Progress with CLI * Remove println debugging statement * Tidy output * Tidy 600 perms * Update libp2p service, start rpc test upgrade * Add validator creation flow * Update network crate for new libp2p * Start tidying, adding comments * Update tokio::codec to futures_codec (#1128) * Further work towards RPC corrections * Correct http timeout and network service select * Add wallet mgr testing * Shift LockedWallet into own file * Add comments to fs * Start integration into VC * Use tokio runtime for libp2p * Revert "Update tokio::codec to futures_codec (#1128)" This reverts commit e57aea924acf5cbabdcea18895ac07e38a425ed7. * Upgrade RPC libp2p tests * Upgrade secio fallback test * Add lcli keypair upgrade command * Upgrade gossipsub examples * Clean up RPC protocol * Test fixes (#1133) * Correct websocket timeout and run on os thread * Fix network test * Add --secrets-dir to VC * Remove --legacy-keys from VC * Clean up PR * Correct tokio tcp move attestation service tests * Upgrade attestation service tests * Fix sim * Correct network test * Correct genesis test * Start docs * Add progress for validator generation * Tidy error messages * Test corrections * Log info when block is received * Modify logs and update attester service events * Stable futures: fixes to vc, eth1 and account manager (#1142) * Add local testnet scripts * Remove whiteblock script * Rename local testnet script * Move spawns onto handle * Fix VC panic * Initial fix to block production issue * Tidy block producer fix * Tidy further * Add local testnet clean script * Run cargo fmt * Tidy duties service * Tidy fork service * Tidy ForkService * Tidy AttestationService * Tidy notifier * Ensure await is not suppressed in eth1 * Ensure await is not suppressed in account_manager * Use .ok() instead of .unwrap_or(()) * RPC decoding test for proto * Update discv5 and eth2-libp2p deps * Run cargo fmt * Pre-build keystores for sim * Fix lcli double runtime issue (#1144) * Handle stream termination and dialing peer errors * Correct peer_info variant types * Add progress on new deposit flow * Remove unnecessary warnings * Handle subnet unsubscription removal and improve logigng * Add logs around ping * Upgrade discv5 and improve logging * Handle peer connection status for multiple connections * Improve network service logging * Add more incomplete progress * Improve logging around peer manager * Upgrade swarm poll centralise peer management * Identify clients on error * Fix `remove_peer` in sync (#1150) * remove_peer removes from all chains * Remove logs * Fix early return from loop * Improved logging, fix panic * Partially correct tests * Add deposit command * Remove old validator directory * Start adding AM tests * Stable futures: Vc sync (#1149) * Improve syncing heuristic * Add comments * Use safer method for tolerance * Fix tests * Binary testing progress * Progress with CLI tests * Use constants for flags * More account manager testing * Improve CLI tests * Move upgrade-legacy-keypairs into account man * Use rayon for VC key generation * Add comments to `validator_dir` * Add testing to validator_dir * Add fix to eth1-sim * Check errors in eth1-sim * Fix mutability issue * Ensure password file ends in .pass * Add more tests to wallet manager * Tidy deposit * Tidy account manager * Tidy account manager * Remove panic * Generate keypairs earlier in sim * Tidy eth1-sime * Try to fix eth1 sim * Address review comments * Fix typo in CLI command * Update docs * Disable eth1 sim * Remove eth1 sim completely Co-authored-by: Age Manning <Age@AgeManning.com> Co-authored-by: pawanjay176 <pawandhananjay@gmail.com>
2020-05-18 09:01:45 +00:00
/// The directory containing the passwords to unlock validator keystores.
pub secrets_dir: PathBuf,
/// The http endpoints of the beacon node APIs.
Validator client refactor (#618) * Update to spec v0.9.0 * Update to v0.9.1 * Bump spec tags for v0.9.1 * Formatting, fix CI failures * Resolve accidental KeyPair merge conflict * Document new BeaconState functions * Add `validator` changes from `validator-to-rest` * Add initial (failing) REST api tests * Fix signature parsing * Add more tests * Refactor http router * Add working tests for publish beacon block * Add validator duties tests * Move account_manager under `lighthouse` binary * Unify logfile handling in `environment` crate. * Fix incorrect cache drops in `advance_caches` * Update fork choice for v0.9.1 * Add `deposit_contract` crate * Add progress on validator onboarding * Add unfinished attesation code * Update account manager CLI * Write eth1 data file as hex string * Integrate ValidatorDirectory with validator_client * Move ValidatorDirectory into validator_client * Clean up some FIXMEs * Add beacon_chain_sim * Fix a few docs/logs * Expand `beacon_chain_sim` * Fix spec for `beacon_chain_sim * More testing for api * Start work on attestation endpoint * Reject empty attestations * Allow attestations to genesis block * Add working tests for `rest_api` validator endpoint * Remove grpc from beacon_node * Start heavy refactor of validator client - Block production is working * Prune old validator client files * Start works on attestation service * Add attestation service to validator client * Use full pubkey for validator directories * Add validator duties post endpoint * Use par_iter for keypair generation * Use bulk duties request in validator client * Add version http endpoint tests * Add interop keys and startup wait * Ensure a prompt exit * Add duties pruning * Fix compile error in beacon node tests * Add github workflow * Modify rust.yaml * Modify gitlab actions * Add to CI file * Add sudo to CI npm install * Move cargo fmt to own job in tests * Fix cargo fmt in CI * Add rustup update before cargo fmt * Change name of CI job * Make other CI jobs require cargo fmt * Add CI badge * Remove gitlab and travis files * Add different http timeout for debug * Update docker file, use makefile in CI * Use make in the dockerfile, skip the test * Use the makefile for debug GI test * Update book * Tidy grpc and misc things * Apply discv5 fixes * Address other minor issues * Fix warnings * Attempt fix for addr parsing * Tidy validator config, CLIs * Tidy comments * Tidy signing, reduce ForkService duplication * Fail if skipping too many slots * Set default recent genesis time to 0 * Add custom http timeout to validator * Fix compile bug in node_test_rig * Remove old bootstrap flag from val CLI * Update docs * Tidy val client * Change val client log levels * Add comments, more validity checks * Fix compile error, add comments * Undo changes to eth2-libp2p/src * Reduce duplication of keypair generation * Add more logging for validator duties * Fix beacon_chain_sim, nitpicks * Fix compile error, minor nits * Address Michael's comments
2019-11-25 04:48:24 +00:00
///
/// Should be similar to `["http://localhost:8080"]`
pub beacon_nodes: Vec<SensitiveUrl>,
/// If true, the validator client will still poll for duties and produce blocks even if the
/// beacon node is not synced at startup.
pub allow_unsynced_beacon_node: bool,
Introduce validator definition file for VC (#1357) ## Issue Addressed NA ## Proposed Changes - Introduces the `valdiator_definitions.yml` file which serves as an explicit list of validators that should be run by the validator client. - Removes `--strict` flag, split into `--strict-lockfiles` and `--disable-auto-discover` - Adds a "Validator Management" page to the book. - Adds the `common/account_utils` crate which contains some logic that was starting to duplicate across the codebase. The new docs for this feature are the best description of it (apart from the code, I guess): https://github.com/sigp/lighthouse/blob/9cb87e93ce239bbe06a9d01e92126f978356dd13/book/src/validator-management.md ## API Changes This change should be transparent for *most* existing users. If the `valdiator_definitions.yml` doesn't exist then it will be automatically generated using a method that will detect all the validators in their `validators_dir`. Users will have issues if they are: 1. Using `--strict`. 1. Have keystores in their `~/.lighthouse/validators` directory that weren't being detected by the current keystore discovery method. For users with (1), the VC will refuse to start because the `--strict` flag has been removed. They will be forced to review `--help` and choose an equivalent flag. For users with (2), this seems fairly unlikely and since we're only in testnets there's no *real* value on the line here. I'm happy to take the risk, it would be a different case for mainnet. ## Additional Info This PR adds functionality we will need for #1347. ## TODO - [x] Reconsider flags - [x] Move doc into a more reasonable chapter. - [x] Check for compile warnings.
2020-07-22 09:34:55 +00:00
/// If true, don't scan the validators dir for new keystores.
pub disable_auto_discover: bool,
/// If true, re-register existing validators in definitions.yml for slashing protection.
pub init_slashing_protection: bool,
/// If true, use longer timeouts for requests made to the beacon node.
pub use_long_timeouts: bool,
/// Graffiti to be inserted everytime we create a block.
pub graffiti: Option<Graffiti>,
/// Graffiti file to load per validator graffitis.
pub graffiti_file: Option<GraffitiFile>,
/// Fallback fallback address.
pub fee_recipient: Option<Address>,
/// Configuration for the HTTP REST API.
pub http_api: http_api::Config,
/// Configuration for the HTTP REST API.
pub http_metrics: http_metrics::Config,
/// Configuration for sending metrics to a remote explorer endpoint.
pub monitoring_api: Option<monitoring_api::Config>,
/// If true, enable functionality that monitors the network for attestations or proposals from
/// any of the validators managed by this client before starting up.
pub enable_doppelganger_protection: bool,
/// Enable use of the blinded block endpoints during proposals.
pub builder_proposals: bool,
/// Overrides the timestamp field in builder api ValidatorRegistrationV1
pub builder_registration_timestamp_override: Option<u64>,
/// Fallback gas limit.
pub gas_limit: Option<u64>,
/// A list of custom certificates that the validator client will additionally use when
/// connecting to a beacon node over SSL/TLS.
pub beacon_nodes_tls_certs: Option<Vec<PathBuf>>,
Enable proposer boost re-orging (#2860) ## Proposed Changes With proposer boosting implemented (#2822) we have an opportunity to re-org out late blocks. This PR adds three flags to the BN to control this behaviour: * `--disable-proposer-reorgs`: turn aggressive re-orging off (it's on by default). * `--proposer-reorg-threshold N`: attempt to orphan blocks with less than N% of the committee vote. If this parameter isn't set then N defaults to 20% when the feature is enabled. * `--proposer-reorg-epochs-since-finalization N`: only attempt to re-org late blocks when the number of epochs since finalization is less than or equal to N. The default is 2 epochs, meaning re-orgs will only be attempted when the chain is finalizing optimally. For safety Lighthouse will only attempt a re-org under very specific conditions: 1. The block being proposed is 1 slot after the canonical head, and the canonical head is 1 slot after its parent. i.e. at slot `n + 1` rather than building on the block from slot `n` we build on the block from slot `n - 1`. 2. The current canonical head received less than N% of the committee vote. N should be set depending on the proposer boost fraction itself, the fraction of the network that is believed to be applying it, and the size of the largest entity that could be hoarding votes. 3. The current canonical head arrived after the attestation deadline from our perspective. This condition was only added to support suppression of forkchoiceUpdated messages, but makes intuitive sense. 4. The block is being proposed in the first 2 seconds of the slot. This gives it time to propagate and receive the proposer boost. ## Additional Info For the initial idea and background, see: https://github.com/ethereum/consensus-specs/pull/2353#issuecomment-950238004 There is also a specification for this feature here: https://github.com/ethereum/consensus-specs/pull/3034 Co-authored-by: Michael Sproul <micsproul@gmail.com> Co-authored-by: pawan <pawandhananjay@gmail.com>
2022-12-13 09:57:26 +00:00
/// Delay from the start of the slot to wait before publishing a block.
///
/// This is *not* recommended in prod and should only be used for testing.
pub block_delay: Option<Duration>,
/// Disables publishing http api requests to all beacon nodes for select api calls.
pub disable_run_on_all: bool,
}
impl Default for Config {
/// Build a new configuration from defaults.
fn default() -> Self {
// WARNING: these directory defaults should be always overwritten with parameters from cli
// for specific networks.
let base_dir = dirs::home_dir()
.unwrap_or_else(|| PathBuf::from("."))
.join(DEFAULT_ROOT_DIR)
.join(DEFAULT_HARDCODED_NETWORK);
let validator_dir = base_dir.join(DEFAULT_VALIDATOR_DIR);
let secrets_dir = base_dir.join(DEFAULT_SECRET_DIR);
let beacon_nodes = vec![SensitiveUrl::parse(DEFAULT_BEACON_NODE)
.expect("beacon_nodes must always be a valid url.")];
Self {
validator_dir,
Wallet-based, encrypted key management (#1138) * Update hashmap hashset to stable futures * Adds panic test to hashset delay * Port remote_beacon_node to stable futures * Fix lcli merge conflicts * Non rpc stuff compiles * Remove padding * Add error enum, zeroize more things * Fix comment * protocol.rs compiles * Port websockets, timer and notifier to stable futures (#1035) * Fix lcli * Port timer to stable futures * Fix timer * Port websocket_server to stable futures * Port notifier to stable futures * Add TODOS * Port remote_beacon_node to stable futures * Partial eth2-libp2p stable future upgrade * Finished first round of fighting RPC types * Further progress towards porting eth2-libp2p adds caching to discovery * Update behaviour * Add keystore builder * Remove keystore stuff from val client * Add more tests, comments * RPC handler to stable futures * Update RPC to master libp2p * Add more comments, test vectors * Network service additions * Progress on improving JSON validation * More JSON verification * Start moving JSON into own mod * Remove old code * Add more tests, reader/writers * Tidy * Move keystore into own file * Move more logic into keystore file * Tidy * Tidy * Fix the fallback transport construction (#1102) * Allow for odd-character hex * Correct warning * Remove hashmap delay * Compiling version of eth2-libp2p * Update all crates versions * Fix conversion function and add tests (#1113) * Add more json missing field checks * Use scrypt by default * Tidy, address comments * Test path and uuid in vectors * Fix comment * Add checks for kdf params * Enforce empty kdf message * Port validator_client to stable futures (#1114) * Add PH & MS slot clock changes * Account for genesis time * Add progress on duties refactor * Add simple is_aggregator bool to val subscription * Start work on attestation_verification.rs * Add progress on ObservedAttestations * Progress with ObservedAttestations * Fix tests * Add observed attestations to the beacon chain * Add attestation observation to processing code * Add progress on attestation verification * Add first draft of ObservedAttesters * Add more tests * Add observed attesters to beacon chain * Add observers to attestation processing * Add more attestation verification * Create ObservedAggregators map * Remove commented-out code * Add observed aggregators into chain * Add progress * Finish adding features to attestation verification * Ensure beacon chain compiles * Link attn verification into chain * Integrate new attn verification in chain * Remove old attestation processing code * Start trying to fix beacon_chain tests * Split adding into pools into two functions * Add aggregation to harness * Get test harness working again * Adjust the number of aggregators for test harness * Fix edge-case in harness * Integrate new attn processing in network * Fix compile bug in validator_client * Update validator API endpoints * Fix aggreagation in test harness * Fix enum thing * Fix attestation observation bug: * Patch failing API tests * Start adding comments to attestation verification * Remove unused attestation field * Unify "is block known" logic * Update comments * Supress fork choice errors for network processing * Add todos * Tidy * Add gossip attn tests * Disallow test harness to produce old attns * Comment out in-progress tests * Partially address pruning tests * Fix failing store test * Add aggregate tests * Add comments about which spec conditions we check * Dont re-aggregate * Split apart test harness attn production * Fix compile error in network * Make progress on commented-out test * Fix skipping attestation test * Add fork choice verification tests * Tidy attn tests, remove dead code * Remove some accidentally added code * Fix clippy lint * Rename test file * Add block tests, add cheap block proposer check * Rename block testing file * Add observed_block_producers * Tidy * Switch around block signature verification * Finish block testing * Remove gossip from signature tests * First pass of self review * Fix deviation in spec * Update test spec tags * Start moving over to hashset * Finish moving observed attesters to hashmap * Move aggregation pool over to hashmap * Make fc attn borrow again * Fix rest_api compile error * Fix missing comments * Fix monster test * Uncomment increasing slots test * Address remaining comments * Remove unsafe, use cfg test * Remove cfg test flag * Fix dodgy comment * Revert "Update hashmap hashset to stable futures" This reverts commit d432378a3cc5cd67fc29c0b15b96b886c1323554. * Revert "Adds panic test to hashset delay" This reverts commit 281502396fc5b90d9c421a309c2c056982c9525b. * Ported attestation_service * Ported duties_service * Ported fork_service * More ports * Port block_service * Minor fixes * VC compiles * Update TODOS * Borrow self where possible * Ignore aggregates that are already known. * Unify aggregator modulo logic * Fix typo in logs * Refactor validator subscription logic * Avoid reproducing selection proof * Skip HTTP call if no subscriptions * Rename DutyAndState -> DutyAndProof * Tidy logs * Print root as dbg * Fix compile errors in tests * Fix compile error in test * Re-Fix attestation and duties service * Minor fixes Co-authored-by: Paul Hauner <paul@paulhauner.com> * Expose json_keystore mod * First commits on path derivation * Progress with implementation * More progress * Passing intermediate test vectors * Tidy, add comments * Add DerivedKey structs * Move key derivation into own crate * Add zeroize structs * Return error for empty seed * Add tests * Tidy * First commits on path derivation * Progress with implementation * Move key derivation into own crate * Start defining JSON wallet * Add progress * Split out encrypt/decrypt * First commits on path derivation * Progress with implementation * More progress * Passing intermediate test vectors * Tidy, add comments * Add DerivedKey structs * Move key derivation into own crate * Add zeroize structs * Return error for empty seed * Add tests * Tidy * Add progress * Replace some password usage with slice * First commits on path derivation * Progress with implementation * More progress * Passing intermediate test vectors * Tidy, add comments * Add DerivedKey structs * Move key derivation into own crate * Add zeroize structs * Return error for empty seed * Add tests * Tidy * Add progress * Expose PlainText struct * First commits on path derivation * Progress with implementation * More progress * Passing intermediate test vectors * Tidy, add comments * Add DerivedKey structs * Move key derivation into own crate * Add zeroize structs * Return error for empty seed * Add tests * Tidy * Add builder * Expose consts, remove Password * Minor progress * Expose SALT_SIZE * First compiling version * Add test vectors * Network crate update to stable futures * Move dbg assert statement * Port account_manager to stable futures (#1121) * Port account_manager to stable futures * Run async fns in tokio environment * Port rest_api crate to stable futures (#1118) * Port rest_api lib to stable futures * Reduce tokio features * Update notifier to stable futures * Builder update * Further updates * Add mnemonic, tidy * Convert self referential async functions * Tidy * Add testing * Add first attempt at validator_dir * Present pubkey field * stable futures fixes (#1124) * Fix eth1 update functions * Fix genesis and client * Fix beacon node lib * Return appropriate runtimes from environment * Fix test rig * Refactor eth1 service update * Upgrade simulator to stable futures * Lighthouse compiles on stable futures * Add first pass of wallet manager * Progress with CLI * Remove println debugging statement * Tidy output * Tidy 600 perms * Update libp2p service, start rpc test upgrade * Add validator creation flow * Update network crate for new libp2p * Start tidying, adding comments * Update tokio::codec to futures_codec (#1128) * Further work towards RPC corrections * Correct http timeout and network service select * Add wallet mgr testing * Shift LockedWallet into own file * Add comments to fs * Start integration into VC * Use tokio runtime for libp2p * Revert "Update tokio::codec to futures_codec (#1128)" This reverts commit e57aea924acf5cbabdcea18895ac07e38a425ed7. * Upgrade RPC libp2p tests * Upgrade secio fallback test * Add lcli keypair upgrade command * Upgrade gossipsub examples * Clean up RPC protocol * Test fixes (#1133) * Correct websocket timeout and run on os thread * Fix network test * Add --secrets-dir to VC * Remove --legacy-keys from VC * Clean up PR * Correct tokio tcp move attestation service tests * Upgrade attestation service tests * Fix sim * Correct network test * Correct genesis test * Start docs * Add progress for validator generation * Tidy error messages * Test corrections * Log info when block is received * Modify logs and update attester service events * Stable futures: fixes to vc, eth1 and account manager (#1142) * Add local testnet scripts * Remove whiteblock script * Rename local testnet script * Move spawns onto handle * Fix VC panic * Initial fix to block production issue * Tidy block producer fix * Tidy further * Add local testnet clean script * Run cargo fmt * Tidy duties service * Tidy fork service * Tidy ForkService * Tidy AttestationService * Tidy notifier * Ensure await is not suppressed in eth1 * Ensure await is not suppressed in account_manager * Use .ok() instead of .unwrap_or(()) * RPC decoding test for proto * Update discv5 and eth2-libp2p deps * Run cargo fmt * Pre-build keystores for sim * Fix lcli double runtime issue (#1144) * Handle stream termination and dialing peer errors * Correct peer_info variant types * Add progress on new deposit flow * Remove unnecessary warnings * Handle subnet unsubscription removal and improve logigng * Add logs around ping * Upgrade discv5 and improve logging * Handle peer connection status for multiple connections * Improve network service logging * Add more incomplete progress * Improve logging around peer manager * Upgrade swarm poll centralise peer management * Identify clients on error * Fix `remove_peer` in sync (#1150) * remove_peer removes from all chains * Remove logs * Fix early return from loop * Improved logging, fix panic * Partially correct tests * Add deposit command * Remove old validator directory * Start adding AM tests * Stable futures: Vc sync (#1149) * Improve syncing heuristic * Add comments * Use safer method for tolerance * Fix tests * Binary testing progress * Progress with CLI tests * Use constants for flags * More account manager testing * Improve CLI tests * Move upgrade-legacy-keypairs into account man * Use rayon for VC key generation * Add comments to `validator_dir` * Add testing to validator_dir * Add fix to eth1-sim * Check errors in eth1-sim * Fix mutability issue * Ensure password file ends in .pass * Add more tests to wallet manager * Tidy deposit * Tidy account manager * Tidy account manager * Remove panic * Generate keypairs earlier in sim * Tidy eth1-sime * Try to fix eth1 sim * Address review comments * Fix typo in CLI command * Update docs * Disable eth1 sim * Remove eth1 sim completely Co-authored-by: Age Manning <Age@AgeManning.com> Co-authored-by: pawanjay176 <pawandhananjay@gmail.com>
2020-05-18 09:01:45 +00:00
secrets_dir,
beacon_nodes,
allow_unsynced_beacon_node: false,
Introduce validator definition file for VC (#1357) ## Issue Addressed NA ## Proposed Changes - Introduces the `valdiator_definitions.yml` file which serves as an explicit list of validators that should be run by the validator client. - Removes `--strict` flag, split into `--strict-lockfiles` and `--disable-auto-discover` - Adds a "Validator Management" page to the book. - Adds the `common/account_utils` crate which contains some logic that was starting to duplicate across the codebase. The new docs for this feature are the best description of it (apart from the code, I guess): https://github.com/sigp/lighthouse/blob/9cb87e93ce239bbe06a9d01e92126f978356dd13/book/src/validator-management.md ## API Changes This change should be transparent for *most* existing users. If the `valdiator_definitions.yml` doesn't exist then it will be automatically generated using a method that will detect all the validators in their `validators_dir`. Users will have issues if they are: 1. Using `--strict`. 1. Have keystores in their `~/.lighthouse/validators` directory that weren't being detected by the current keystore discovery method. For users with (1), the VC will refuse to start because the `--strict` flag has been removed. They will be forced to review `--help` and choose an equivalent flag. For users with (2), this seems fairly unlikely and since we're only in testnets there's no *real* value on the line here. I'm happy to take the risk, it would be a different case for mainnet. ## Additional Info This PR adds functionality we will need for #1347. ## TODO - [x] Reconsider flags - [x] Move doc into a more reasonable chapter. - [x] Check for compile warnings.
2020-07-22 09:34:55 +00:00
disable_auto_discover: false,
init_slashing_protection: false,
use_long_timeouts: false,
graffiti: None,
graffiti_file: None,
fee_recipient: None,
http_api: <_>::default(),
http_metrics: <_>::default(),
monitoring_api: None,
enable_doppelganger_protection: false,
beacon_nodes_tls_certs: None,
Enable proposer boost re-orging (#2860) ## Proposed Changes With proposer boosting implemented (#2822) we have an opportunity to re-org out late blocks. This PR adds three flags to the BN to control this behaviour: * `--disable-proposer-reorgs`: turn aggressive re-orging off (it's on by default). * `--proposer-reorg-threshold N`: attempt to orphan blocks with less than N% of the committee vote. If this parameter isn't set then N defaults to 20% when the feature is enabled. * `--proposer-reorg-epochs-since-finalization N`: only attempt to re-org late blocks when the number of epochs since finalization is less than or equal to N. The default is 2 epochs, meaning re-orgs will only be attempted when the chain is finalizing optimally. For safety Lighthouse will only attempt a re-org under very specific conditions: 1. The block being proposed is 1 slot after the canonical head, and the canonical head is 1 slot after its parent. i.e. at slot `n + 1` rather than building on the block from slot `n` we build on the block from slot `n - 1`. 2. The current canonical head received less than N% of the committee vote. N should be set depending on the proposer boost fraction itself, the fraction of the network that is believed to be applying it, and the size of the largest entity that could be hoarding votes. 3. The current canonical head arrived after the attestation deadline from our perspective. This condition was only added to support suppression of forkchoiceUpdated messages, but makes intuitive sense. 4. The block is being proposed in the first 2 seconds of the slot. This gives it time to propagate and receive the proposer boost. ## Additional Info For the initial idea and background, see: https://github.com/ethereum/consensus-specs/pull/2353#issuecomment-950238004 There is also a specification for this feature here: https://github.com/ethereum/consensus-specs/pull/3034 Co-authored-by: Michael Sproul <micsproul@gmail.com> Co-authored-by: pawan <pawandhananjay@gmail.com>
2022-12-13 09:57:26 +00:00
block_delay: None,
builder_proposals: false,
builder_registration_timestamp_override: None,
gas_limit: None,
disable_run_on_all: false,
}
}
}
impl Config {
Validator client refactor (#618) * Update to spec v0.9.0 * Update to v0.9.1 * Bump spec tags for v0.9.1 * Formatting, fix CI failures * Resolve accidental KeyPair merge conflict * Document new BeaconState functions * Add `validator` changes from `validator-to-rest` * Add initial (failing) REST api tests * Fix signature parsing * Add more tests * Refactor http router * Add working tests for publish beacon block * Add validator duties tests * Move account_manager under `lighthouse` binary * Unify logfile handling in `environment` crate. * Fix incorrect cache drops in `advance_caches` * Update fork choice for v0.9.1 * Add `deposit_contract` crate * Add progress on validator onboarding * Add unfinished attesation code * Update account manager CLI * Write eth1 data file as hex string * Integrate ValidatorDirectory with validator_client * Move ValidatorDirectory into validator_client * Clean up some FIXMEs * Add beacon_chain_sim * Fix a few docs/logs * Expand `beacon_chain_sim` * Fix spec for `beacon_chain_sim * More testing for api * Start work on attestation endpoint * Reject empty attestations * Allow attestations to genesis block * Add working tests for `rest_api` validator endpoint * Remove grpc from beacon_node * Start heavy refactor of validator client - Block production is working * Prune old validator client files * Start works on attestation service * Add attestation service to validator client * Use full pubkey for validator directories * Add validator duties post endpoint * Use par_iter for keypair generation * Use bulk duties request in validator client * Add version http endpoint tests * Add interop keys and startup wait * Ensure a prompt exit * Add duties pruning * Fix compile error in beacon node tests * Add github workflow * Modify rust.yaml * Modify gitlab actions * Add to CI file * Add sudo to CI npm install * Move cargo fmt to own job in tests * Fix cargo fmt in CI * Add rustup update before cargo fmt * Change name of CI job * Make other CI jobs require cargo fmt * Add CI badge * Remove gitlab and travis files * Add different http timeout for debug * Update docker file, use makefile in CI * Use make in the dockerfile, skip the test * Use the makefile for debug GI test * Update book * Tidy grpc and misc things * Apply discv5 fixes * Address other minor issues * Fix warnings * Attempt fix for addr parsing * Tidy validator config, CLIs * Tidy comments * Tidy signing, reduce ForkService duplication * Fail if skipping too many slots * Set default recent genesis time to 0 * Add custom http timeout to validator * Fix compile bug in node_test_rig * Remove old bootstrap flag from val CLI * Update docs * Tidy val client * Change val client log levels * Add comments, more validity checks * Fix compile error, add comments * Undo changes to eth2-libp2p/src * Reduce duplication of keypair generation * Add more logging for validator duties * Fix beacon_chain_sim, nitpicks * Fix compile error, minor nits * Address Michael's comments
2019-11-25 04:48:24 +00:00
/// Returns a `Default` implementation of `Self` with some parameters modified by the supplied
/// `cli_args`.
pub fn from_cli(cli_args: &ArgMatches, log: &Logger) -> Result<Config, String> {
Validator client refactor (#618) * Update to spec v0.9.0 * Update to v0.9.1 * Bump spec tags for v0.9.1 * Formatting, fix CI failures * Resolve accidental KeyPair merge conflict * Document new BeaconState functions * Add `validator` changes from `validator-to-rest` * Add initial (failing) REST api tests * Fix signature parsing * Add more tests * Refactor http router * Add working tests for publish beacon block * Add validator duties tests * Move account_manager under `lighthouse` binary * Unify logfile handling in `environment` crate. * Fix incorrect cache drops in `advance_caches` * Update fork choice for v0.9.1 * Add `deposit_contract` crate * Add progress on validator onboarding * Add unfinished attesation code * Update account manager CLI * Write eth1 data file as hex string * Integrate ValidatorDirectory with validator_client * Move ValidatorDirectory into validator_client * Clean up some FIXMEs * Add beacon_chain_sim * Fix a few docs/logs * Expand `beacon_chain_sim` * Fix spec for `beacon_chain_sim * More testing for api * Start work on attestation endpoint * Reject empty attestations * Allow attestations to genesis block * Add working tests for `rest_api` validator endpoint * Remove grpc from beacon_node * Start heavy refactor of validator client - Block production is working * Prune old validator client files * Start works on attestation service * Add attestation service to validator client * Use full pubkey for validator directories * Add validator duties post endpoint * Use par_iter for keypair generation * Use bulk duties request in validator client * Add version http endpoint tests * Add interop keys and startup wait * Ensure a prompt exit * Add duties pruning * Fix compile error in beacon node tests * Add github workflow * Modify rust.yaml * Modify gitlab actions * Add to CI file * Add sudo to CI npm install * Move cargo fmt to own job in tests * Fix cargo fmt in CI * Add rustup update before cargo fmt * Change name of CI job * Make other CI jobs require cargo fmt * Add CI badge * Remove gitlab and travis files * Add different http timeout for debug * Update docker file, use makefile in CI * Use make in the dockerfile, skip the test * Use the makefile for debug GI test * Update book * Tidy grpc and misc things * Apply discv5 fixes * Address other minor issues * Fix warnings * Attempt fix for addr parsing * Tidy validator config, CLIs * Tidy comments * Tidy signing, reduce ForkService duplication * Fail if skipping too many slots * Set default recent genesis time to 0 * Add custom http timeout to validator * Fix compile bug in node_test_rig * Remove old bootstrap flag from val CLI * Update docs * Tidy val client * Change val client log levels * Add comments, more validity checks * Fix compile error, add comments * Undo changes to eth2-libp2p/src * Reduce duplication of keypair generation * Add more logging for validator duties * Fix beacon_chain_sim, nitpicks * Fix compile error, minor nits * Address Michael's comments
2019-11-25 04:48:24 +00:00
let mut config = Config::default();
let default_root_dir = dirs::home_dir()
.map(|home| home.join(DEFAULT_ROOT_DIR))
.unwrap_or_else(|| PathBuf::from("."));
let (mut validator_dir, mut secrets_dir) = (None, None);
if cli_args.value_of("datadir").is_some() {
let base_dir: PathBuf = parse_required(cli_args, "datadir")?;
validator_dir = Some(base_dir.join(DEFAULT_VALIDATOR_DIR));
secrets_dir = Some(base_dir.join(DEFAULT_SECRET_DIR));
}
if cli_args.value_of("validators-dir").is_some() {
validator_dir = Some(parse_required(cli_args, "validators-dir")?);
}
if cli_args.value_of("secrets-dir").is_some() {
secrets_dir = Some(parse_required(cli_args, "secrets-dir")?);
}
config.validator_dir = validator_dir.unwrap_or_else(|| {
default_root_dir
.join(get_network_dir(cli_args))
.join(DEFAULT_VALIDATOR_DIR)
});
config.secrets_dir = secrets_dir.unwrap_or_else(|| {
default_root_dir
.join(get_network_dir(cli_args))
.join(DEFAULT_SECRET_DIR)
});
Prepare for public testnet (#628) * Update to spec v0.9.0 * Update to v0.9.1 * Bump spec tags for v0.9.1 * Formatting, fix CI failures * Resolve accidental KeyPair merge conflict * Document new BeaconState functions * Add `validator` changes from `validator-to-rest` * Add initial (failing) REST api tests * Fix signature parsing * Add more tests * Refactor http router * Add working tests for publish beacon block * Add validator duties tests * Move account_manager under `lighthouse` binary * Unify logfile handling in `environment` crate. * Fix incorrect cache drops in `advance_caches` * Update fork choice for v0.9.1 * Add `deposit_contract` crate * Add progress on validator onboarding * Add unfinished attesation code * Update account manager CLI * Write eth1 data file as hex string * Integrate ValidatorDirectory with validator_client * Move ValidatorDirectory into validator_client * Clean up some FIXMEs * Add beacon_chain_sim * Fix a few docs/logs * Expand `beacon_chain_sim` * Fix spec for `beacon_chain_sim * More testing for api * Start work on attestation endpoint * Reject empty attestations * Allow attestations to genesis block * Add working tests for `rest_api` validator endpoint * Remove grpc from beacon_node * Start heavy refactor of validator client - Block production is working * Prune old validator client files * Start works on attestation service * Add attestation service to validator client * Use full pubkey for validator directories * Add validator duties post endpoint * Use par_iter for keypair generation * Use bulk duties request in validator client * Add version http endpoint tests * Add interop keys and startup wait * Ensure a prompt exit * Add duties pruning * Fix compile error in beacon node tests * Add github workflow * Modify rust.yaml * Modify gitlab actions * Add to CI file * Add sudo to CI npm install * Move cargo fmt to own job in tests * Fix cargo fmt in CI * Add rustup update before cargo fmt * Change name of CI job * Make other CI jobs require cargo fmt * Add CI badge * Remove gitlab and travis files * Add different http timeout for debug * Update docker file, use makefile in CI * Use make in the dockerfile, skip the test * Use the makefile for debug GI test * Update book * Tidy grpc and misc things * Apply discv5 fixes * Address other minor issues * Fix warnings * Attempt fix for addr parsing * Tidy validator config, CLIs * Tidy comments * Tidy signing, reduce ForkService duplication * Fail if skipping too many slots * Set default recent genesis time to 0 * Add custom http timeout to validator * Fix compile bug in node_test_rig * Remove old bootstrap flag from val CLI * Update docs * Tidy val client * Change val client log levels * Add comments, more validity checks * Fix compile error, add comments * Undo changes to eth2-libp2p/src * Reduce duplication of keypair generation * Add more logging for validator duties * Fix beacon_chain_sim, nitpicks * Fix compile error, minor nits * Update to use v0.9.2 version of deposit contract * Add efforts to automate eth1 testnet deployment * Fix lcli testnet deployer * Modify bn CLI to parse eth2_testnet_dir * Progress with account_manager deposit tools * Make account manager submit deposits * Add password option for submitting deposits * Allow custom deposit amount * Add long names to lcli clap * Add password option to lcli deploy command * Fix minor bugs whilst testing * Address Michael's comments * Add refund-deposit-contract to lcli * Use time instead of skip count for denying long skips * Improve logging for eth1 * Fix bug with validator services exiting on error * Drop the block cache after genesis * Modify eth1 testnet config * Improve eth1 logging * Make validator wait until genesis time * Fix bug in eth1 voting * Add more logging to eth1 voting * Handle errors in eth1 http module * Set SECONDS_PER_DAY to sensible minimum * Shorten delay before testnet start * Ensure eth1 block is produced without any votes * Improve eth1 logging * Fix broken tests in eth1 * Tidy code in rest_api * Fix failing test in deposit_contract * Make CLI args more consistent * Change validator/duties endpoint * Add time-based skip slot limiting * Add new error type missed in previous commit * Add log when waiting for genesis * Refactor beacon node CLI * Remove unused dep * Add lcli eth1-genesis command * Fix bug in master merge * Apply clippy lints to beacon node * Add support for YamlConfig in Eth2TestnetDir * Upgrade tesnet deposit contract version * Remove unnecessary logging and correct formatting * Add a hardcoded eth2 testnet config * Ensure http server flag works. Overwrite configs with flags. * Ensure boot nodes are loaded from testnet dir * Fix account manager CLI bugs * Fix bugs with beacon node cli * Allow testnet dir without boot nodes * Write genesis state as SSZ * Remove ---/n from the start of testnet_dir files * Set default libp2p address * Tidy account manager CLI, add logging * Add check to see if testnet dir exists * Apply reviewers suggestions * Add HeadTracker struct * Add fork choice persistence * Shorten slot time for simulator * Add the /beacon/heads API endpoint * Update hardcoded testnet * Add tests for BeaconChain persistence + fix bugs * Extend BeaconChain persistence testing * Ensure chain is finalized b4 persistence tests * Ensure boot_enr.yaml is include in binary * Refactor beacon_chain_sim * Move files about in beacon sim * Update beacon_chain_sim * Fix bug with deposit inclusion * Increase log in genesis service, fix todo * Tidy sim, fix broken rest_api tests * Fix more broken tests * Update testnet * Fix broken rest api test * Tidy account manager CLI * Use tempdir for account manager * Stop hardcoded testnet dir from creating dir * Rename Eth2TestnetDir to Eth2TestnetConfig * Change hardcoded -> hard_coded * Tidy account manager * Add log to account manager * Tidy, ensure head tracker is loaded from disk * Tidy beacon chain builder * Tidy eth1_chain * Adds log support for simulator * Revert "Adds log support for simulator" This reverts commit ec77c66a052350f551db145cf20f213823428dd3. * Adds log support for simulator * Tidy after self-review * Change default log level * Address Michael's delicious PR comments * Fix off-by-one in tests
2019-12-03 04:28:57 +00:00
if !config.validator_dir.exists() {
fs::create_dir_all(&config.validator_dir)
.map_err(|e| format!("Failed to create {:?}: {:?}", config.validator_dir, e))?;
2019-09-01 09:33:43 +00:00
}
if let Some(beacon_nodes) = parse_optional::<String>(cli_args, "beacon-nodes")? {
config.beacon_nodes = beacon_nodes
.split(',')
.map(SensitiveUrl::parse)
.collect::<Result<_, _>>()
.map_err(|e| format!("Unable to parse beacon node URL: {:?}", e))?;
}
// To be deprecated.
else if let Some(beacon_node) = parse_optional::<String>(cli_args, "beacon-node")? {
warn!(
log,
"The --beacon-node flag is deprecated";
"msg" => "please use --beacon-nodes instead"
);
config.beacon_nodes = vec![SensitiveUrl::parse(&beacon_node)
.map_err(|e| format!("Unable to parse beacon node URL: {:?}", e))?];
}
// To be deprecated.
else if let Some(server) = parse_optional::<String>(cli_args, "server")? {
warn!(
log,
"The --server flag is deprecated";
"msg" => "please use --beacon-nodes instead"
);
config.beacon_nodes = vec![SensitiveUrl::parse(&server)
.map_err(|e| format!("Unable to parse beacon node URL: {:?}", e))?];
Wallet-based, encrypted key management (#1138) * Update hashmap hashset to stable futures * Adds panic test to hashset delay * Port remote_beacon_node to stable futures * Fix lcli merge conflicts * Non rpc stuff compiles * Remove padding * Add error enum, zeroize more things * Fix comment * protocol.rs compiles * Port websockets, timer and notifier to stable futures (#1035) * Fix lcli * Port timer to stable futures * Fix timer * Port websocket_server to stable futures * Port notifier to stable futures * Add TODOS * Port remote_beacon_node to stable futures * Partial eth2-libp2p stable future upgrade * Finished first round of fighting RPC types * Further progress towards porting eth2-libp2p adds caching to discovery * Update behaviour * Add keystore builder * Remove keystore stuff from val client * Add more tests, comments * RPC handler to stable futures * Update RPC to master libp2p * Add more comments, test vectors * Network service additions * Progress on improving JSON validation * More JSON verification * Start moving JSON into own mod * Remove old code * Add more tests, reader/writers * Tidy * Move keystore into own file * Move more logic into keystore file * Tidy * Tidy * Fix the fallback transport construction (#1102) * Allow for odd-character hex * Correct warning * Remove hashmap delay * Compiling version of eth2-libp2p * Update all crates versions * Fix conversion function and add tests (#1113) * Add more json missing field checks * Use scrypt by default * Tidy, address comments * Test path and uuid in vectors * Fix comment * Add checks for kdf params * Enforce empty kdf message * Port validator_client to stable futures (#1114) * Add PH & MS slot clock changes * Account for genesis time * Add progress on duties refactor * Add simple is_aggregator bool to val subscription * Start work on attestation_verification.rs * Add progress on ObservedAttestations * Progress with ObservedAttestations * Fix tests * Add observed attestations to the beacon chain * Add attestation observation to processing code * Add progress on attestation verification * Add first draft of ObservedAttesters * Add more tests * Add observed attesters to beacon chain * Add observers to attestation processing * Add more attestation verification * Create ObservedAggregators map * Remove commented-out code * Add observed aggregators into chain * Add progress * Finish adding features to attestation verification * Ensure beacon chain compiles * Link attn verification into chain * Integrate new attn verification in chain * Remove old attestation processing code * Start trying to fix beacon_chain tests * Split adding into pools into two functions * Add aggregation to harness * Get test harness working again * Adjust the number of aggregators for test harness * Fix edge-case in harness * Integrate new attn processing in network * Fix compile bug in validator_client * Update validator API endpoints * Fix aggreagation in test harness * Fix enum thing * Fix attestation observation bug: * Patch failing API tests * Start adding comments to attestation verification * Remove unused attestation field * Unify "is block known" logic * Update comments * Supress fork choice errors for network processing * Add todos * Tidy * Add gossip attn tests * Disallow test harness to produce old attns * Comment out in-progress tests * Partially address pruning tests * Fix failing store test * Add aggregate tests * Add comments about which spec conditions we check * Dont re-aggregate * Split apart test harness attn production * Fix compile error in network * Make progress on commented-out test * Fix skipping attestation test * Add fork choice verification tests * Tidy attn tests, remove dead code * Remove some accidentally added code * Fix clippy lint * Rename test file * Add block tests, add cheap block proposer check * Rename block testing file * Add observed_block_producers * Tidy * Switch around block signature verification * Finish block testing * Remove gossip from signature tests * First pass of self review * Fix deviation in spec * Update test spec tags * Start moving over to hashset * Finish moving observed attesters to hashmap * Move aggregation pool over to hashmap * Make fc attn borrow again * Fix rest_api compile error * Fix missing comments * Fix monster test * Uncomment increasing slots test * Address remaining comments * Remove unsafe, use cfg test * Remove cfg test flag * Fix dodgy comment * Revert "Update hashmap hashset to stable futures" This reverts commit d432378a3cc5cd67fc29c0b15b96b886c1323554. * Revert "Adds panic test to hashset delay" This reverts commit 281502396fc5b90d9c421a309c2c056982c9525b. * Ported attestation_service * Ported duties_service * Ported fork_service * More ports * Port block_service * Minor fixes * VC compiles * Update TODOS * Borrow self where possible * Ignore aggregates that are already known. * Unify aggregator modulo logic * Fix typo in logs * Refactor validator subscription logic * Avoid reproducing selection proof * Skip HTTP call if no subscriptions * Rename DutyAndState -> DutyAndProof * Tidy logs * Print root as dbg * Fix compile errors in tests * Fix compile error in test * Re-Fix attestation and duties service * Minor fixes Co-authored-by: Paul Hauner <paul@paulhauner.com> * Expose json_keystore mod * First commits on path derivation * Progress with implementation * More progress * Passing intermediate test vectors * Tidy, add comments * Add DerivedKey structs * Move key derivation into own crate * Add zeroize structs * Return error for empty seed * Add tests * Tidy * First commits on path derivation * Progress with implementation * Move key derivation into own crate * Start defining JSON wallet * Add progress * Split out encrypt/decrypt * First commits on path derivation * Progress with implementation * More progress * Passing intermediate test vectors * Tidy, add comments * Add DerivedKey structs * Move key derivation into own crate * Add zeroize structs * Return error for empty seed * Add tests * Tidy * Add progress * Replace some password usage with slice * First commits on path derivation * Progress with implementation * More progress * Passing intermediate test vectors * Tidy, add comments * Add DerivedKey structs * Move key derivation into own crate * Add zeroize structs * Return error for empty seed * Add tests * Tidy * Add progress * Expose PlainText struct * First commits on path derivation * Progress with implementation * More progress * Passing intermediate test vectors * Tidy, add comments * Add DerivedKey structs * Move key derivation into own crate * Add zeroize structs * Return error for empty seed * Add tests * Tidy * Add builder * Expose consts, remove Password * Minor progress * Expose SALT_SIZE * First compiling version * Add test vectors * Network crate update to stable futures * Move dbg assert statement * Port account_manager to stable futures (#1121) * Port account_manager to stable futures * Run async fns in tokio environment * Port rest_api crate to stable futures (#1118) * Port rest_api lib to stable futures * Reduce tokio features * Update notifier to stable futures * Builder update * Further updates * Add mnemonic, tidy * Convert self referential async functions * Tidy * Add testing * Add first attempt at validator_dir * Present pubkey field * stable futures fixes (#1124) * Fix eth1 update functions * Fix genesis and client * Fix beacon node lib * Return appropriate runtimes from environment * Fix test rig * Refactor eth1 service update * Upgrade simulator to stable futures * Lighthouse compiles on stable futures * Add first pass of wallet manager * Progress with CLI * Remove println debugging statement * Tidy output * Tidy 600 perms * Update libp2p service, start rpc test upgrade * Add validator creation flow * Update network crate for new libp2p * Start tidying, adding comments * Update tokio::codec to futures_codec (#1128) * Further work towards RPC corrections * Correct http timeout and network service select * Add wallet mgr testing * Shift LockedWallet into own file * Add comments to fs * Start integration into VC * Use tokio runtime for libp2p * Revert "Update tokio::codec to futures_codec (#1128)" This reverts commit e57aea924acf5cbabdcea18895ac07e38a425ed7. * Upgrade RPC libp2p tests * Upgrade secio fallback test * Add lcli keypair upgrade command * Upgrade gossipsub examples * Clean up RPC protocol * Test fixes (#1133) * Correct websocket timeout and run on os thread * Fix network test * Add --secrets-dir to VC * Remove --legacy-keys from VC * Clean up PR * Correct tokio tcp move attestation service tests * Upgrade attestation service tests * Fix sim * Correct network test * Correct genesis test * Start docs * Add progress for validator generation * Tidy error messages * Test corrections * Log info when block is received * Modify logs and update attester service events * Stable futures: fixes to vc, eth1 and account manager (#1142) * Add local testnet scripts * Remove whiteblock script * Rename local testnet script * Move spawns onto handle * Fix VC panic * Initial fix to block production issue * Tidy block producer fix * Tidy further * Add local testnet clean script * Run cargo fmt * Tidy duties service * Tidy fork service * Tidy ForkService * Tidy AttestationService * Tidy notifier * Ensure await is not suppressed in eth1 * Ensure await is not suppressed in account_manager * Use .ok() instead of .unwrap_or(()) * RPC decoding test for proto * Update discv5 and eth2-libp2p deps * Run cargo fmt * Pre-build keystores for sim * Fix lcli double runtime issue (#1144) * Handle stream termination and dialing peer errors * Correct peer_info variant types * Add progress on new deposit flow * Remove unnecessary warnings * Handle subnet unsubscription removal and improve logigng * Add logs around ping * Upgrade discv5 and improve logging * Handle peer connection status for multiple connections * Improve network service logging * Add more incomplete progress * Improve logging around peer manager * Upgrade swarm poll centralise peer management * Identify clients on error * Fix `remove_peer` in sync (#1150) * remove_peer removes from all chains * Remove logs * Fix early return from loop * Improved logging, fix panic * Partially correct tests * Add deposit command * Remove old validator directory * Start adding AM tests * Stable futures: Vc sync (#1149) * Improve syncing heuristic * Add comments * Use safer method for tolerance * Fix tests * Binary testing progress * Progress with CLI tests * Use constants for flags * More account manager testing * Improve CLI tests * Move upgrade-legacy-keypairs into account man * Use rayon for VC key generation * Add comments to `validator_dir` * Add testing to validator_dir * Add fix to eth1-sim * Check errors in eth1-sim * Fix mutability issue * Ensure password file ends in .pass * Add more tests to wallet manager * Tidy deposit * Tidy account manager * Tidy account manager * Remove panic * Generate keypairs earlier in sim * Tidy eth1-sime * Try to fix eth1 sim * Address review comments * Fix typo in CLI command * Update docs * Disable eth1 sim * Remove eth1 sim completely Co-authored-by: Age Manning <Age@AgeManning.com> Co-authored-by: pawanjay176 <pawandhananjay@gmail.com>
2020-05-18 09:01:45 +00:00
}
if cli_args.is_present("delete-lockfiles") {
warn!(
log,
"The --delete-lockfiles flag is deprecated";
"msg" => "it is no longer necessary, and no longer has any effect",
);
}
config.allow_unsynced_beacon_node = cli_args.is_present("allow-unsynced");
config.disable_run_on_all = cli_args.is_present("disable-run-on-all");
Introduce validator definition file for VC (#1357) ## Issue Addressed NA ## Proposed Changes - Introduces the `valdiator_definitions.yml` file which serves as an explicit list of validators that should be run by the validator client. - Removes `--strict` flag, split into `--strict-lockfiles` and `--disable-auto-discover` - Adds a "Validator Management" page to the book. - Adds the `common/account_utils` crate which contains some logic that was starting to duplicate across the codebase. The new docs for this feature are the best description of it (apart from the code, I guess): https://github.com/sigp/lighthouse/blob/9cb87e93ce239bbe06a9d01e92126f978356dd13/book/src/validator-management.md ## API Changes This change should be transparent for *most* existing users. If the `valdiator_definitions.yml` doesn't exist then it will be automatically generated using a method that will detect all the validators in their `validators_dir`. Users will have issues if they are: 1. Using `--strict`. 1. Have keystores in their `~/.lighthouse/validators` directory that weren't being detected by the current keystore discovery method. For users with (1), the VC will refuse to start because the `--strict` flag has been removed. They will be forced to review `--help` and choose an equivalent flag. For users with (2), this seems fairly unlikely and since we're only in testnets there's no *real* value on the line here. I'm happy to take the risk, it would be a different case for mainnet. ## Additional Info This PR adds functionality we will need for #1347. ## TODO - [x] Reconsider flags - [x] Move doc into a more reasonable chapter. - [x] Check for compile warnings.
2020-07-22 09:34:55 +00:00
config.disable_auto_discover = cli_args.is_present("disable-auto-discover");
config.init_slashing_protection = cli_args.is_present("init-slashing-protection");
config.use_long_timeouts = cli_args.is_present("use-long-timeouts");
if let Some(graffiti_file_path) = cli_args.value_of("graffiti-file") {
let mut graffiti_file = GraffitiFile::new(graffiti_file_path.into());
graffiti_file
.read_graffiti_file()
.map_err(|e| format!("Error reading graffiti file: {:?}", e))?;
config.graffiti_file = Some(graffiti_file);
info!(log, "Successfully loaded graffiti file"; "path" => graffiti_file_path);
}
if let Some(input_graffiti) = cli_args.value_of("graffiti") {
let graffiti_bytes = input_graffiti.as_bytes();
if graffiti_bytes.len() > GRAFFITI_BYTES_LEN {
return Err(format!(
"Your graffiti is too long! {} bytes maximum!",
GRAFFITI_BYTES_LEN
));
} else {
Implement standard eth2.0 API (#1569) - Resolves #1550 - Resolves #824 - Resolves #825 - Resolves #1131 - Resolves #1411 - Resolves #1256 - Resolve #1177 - Includes the `ShufflingId` struct initially defined in #1492. That PR is now closed and the changes are included here, with significant bug fixes. - Implement the https://github.com/ethereum/eth2.0-APIs in a new `http_api` crate using `warp`. This replaces the `rest_api` crate. - Add a new `common/eth2` crate which provides a wrapper around `reqwest`, providing the HTTP client that is used by the validator client and for testing. This replaces the `common/remote_beacon_node` crate. - Create a `http_metrics` crate which is a dedicated server for Prometheus metrics (they are no longer served on the same port as the REST API). We now have flags for `--metrics`, `--metrics-address`, etc. - Allow the `subnet_id` to be an optional parameter for `VerifiedUnaggregatedAttestation::verify`. This means it does not need to be provided unnecessarily by the validator client. - Move `fn map_attestation_committee` in `mod beacon_chain::attestation_verification` to a new `fn with_committee_cache` on the `BeaconChain` so the same cache can be used for obtaining validator duties. - Add some other helpers to `BeaconChain` to assist with common API duties (e.g., `block_root_at_slot`, `head_beacon_block_root`). - Change the `NaiveAggregationPool` so it can index attestations by `hash_tree_root(attestation.data)`. This is a requirement of the API. - Add functions to `BeaconChainHarness` to allow it to create slashings and exits. - Allow for `eth1::Eth1NetworkId` to go to/from a `String`. - Add functions to the `OperationPool` to allow getting all objects in the pool. - Add function to `BeaconState` to check if a committee cache is initialized. - Fix bug where `seconds_per_eth1_block` was not transferring over from `YamlConfig` to `ChainSpec`. - Add the `deposit_contract_address` to `YamlConfig` and `ChainSpec`. We needed to be able to return it in an API response. - Change some uses of serde `serialize_with` and `deserialize_with` to a single use of `with` (code quality). - Impl `Display` and `FromStr` for several BLS fields. - Check for clock discrepancy when VC polls BN for sync state (with +/- 1 slot tolerance). This is not intended to be comprehensive, it was just easy to do. - See #1434 for a per-endpoint overview. - Seeking clarity here: https://github.com/ethereum/eth2.0-APIs/issues/75 - [x] Add docs for prom port to close #1256 - [x] Follow up on this #1177 - [x] ~~Follow up with #1424~~ Will fix in future PR. - [x] Follow up with #1411 - [x] ~~Follow up with #1260~~ Will fix in future PR. - [x] Add quotes to all integers. - [x] Remove `rest_types` - [x] Address missing beacon block error. (#1629) - [x] ~~Add tests for lighthouse/peers endpoints~~ Wontfix - [x] ~~Follow up with validator status proposal~~ Tracked in #1434 - [x] Unify graffiti structs - [x] ~~Start server when waiting for genesis?~~ Will fix in future PR. - [x] TODO in http_api tests - [x] Move lighthouse endpoints off /eth/v1 - [x] Update docs to link to standard - ~~Blocked on #1586~~ Co-authored-by: Michael Sproul <michael@sigmaprime.io>
2020-09-29 03:46:54 +00:00
let mut graffiti = [0; 32];
// Copy the provided bytes over.
//
// Panic-free because `graffiti_bytes.len()` <= `GRAFFITI_BYTES_LEN`.
graffiti[..graffiti_bytes.len()].copy_from_slice(graffiti_bytes);
Implement standard eth2.0 API (#1569) - Resolves #1550 - Resolves #824 - Resolves #825 - Resolves #1131 - Resolves #1411 - Resolves #1256 - Resolve #1177 - Includes the `ShufflingId` struct initially defined in #1492. That PR is now closed and the changes are included here, with significant bug fixes. - Implement the https://github.com/ethereum/eth2.0-APIs in a new `http_api` crate using `warp`. This replaces the `rest_api` crate. - Add a new `common/eth2` crate which provides a wrapper around `reqwest`, providing the HTTP client that is used by the validator client and for testing. This replaces the `common/remote_beacon_node` crate. - Create a `http_metrics` crate which is a dedicated server for Prometheus metrics (they are no longer served on the same port as the REST API). We now have flags for `--metrics`, `--metrics-address`, etc. - Allow the `subnet_id` to be an optional parameter for `VerifiedUnaggregatedAttestation::verify`. This means it does not need to be provided unnecessarily by the validator client. - Move `fn map_attestation_committee` in `mod beacon_chain::attestation_verification` to a new `fn with_committee_cache` on the `BeaconChain` so the same cache can be used for obtaining validator duties. - Add some other helpers to `BeaconChain` to assist with common API duties (e.g., `block_root_at_slot`, `head_beacon_block_root`). - Change the `NaiveAggregationPool` so it can index attestations by `hash_tree_root(attestation.data)`. This is a requirement of the API. - Add functions to `BeaconChainHarness` to allow it to create slashings and exits. - Allow for `eth1::Eth1NetworkId` to go to/from a `String`. - Add functions to the `OperationPool` to allow getting all objects in the pool. - Add function to `BeaconState` to check if a committee cache is initialized. - Fix bug where `seconds_per_eth1_block` was not transferring over from `YamlConfig` to `ChainSpec`. - Add the `deposit_contract_address` to `YamlConfig` and `ChainSpec`. We needed to be able to return it in an API response. - Change some uses of serde `serialize_with` and `deserialize_with` to a single use of `with` (code quality). - Impl `Display` and `FromStr` for several BLS fields. - Check for clock discrepancy when VC polls BN for sync state (with +/- 1 slot tolerance). This is not intended to be comprehensive, it was just easy to do. - See #1434 for a per-endpoint overview. - Seeking clarity here: https://github.com/ethereum/eth2.0-APIs/issues/75 - [x] Add docs for prom port to close #1256 - [x] Follow up on this #1177 - [x] ~~Follow up with #1424~~ Will fix in future PR. - [x] Follow up with #1411 - [x] ~~Follow up with #1260~~ Will fix in future PR. - [x] Add quotes to all integers. - [x] Remove `rest_types` - [x] Address missing beacon block error. (#1629) - [x] ~~Add tests for lighthouse/peers endpoints~~ Wontfix - [x] ~~Follow up with validator status proposal~~ Tracked in #1434 - [x] Unify graffiti structs - [x] ~~Start server when waiting for genesis?~~ Will fix in future PR. - [x] TODO in http_api tests - [x] Move lighthouse endpoints off /eth/v1 - [x] Update docs to link to standard - ~~Blocked on #1586~~ Co-authored-by: Michael Sproul <michael@sigmaprime.io>
2020-09-29 03:46:54 +00:00
config.graffiti = Some(graffiti.into());
}
}
if let Some(input_fee_recipient) =
parse_optional::<Address>(cli_args, "suggested-fee-recipient")?
{
config.fee_recipient = Some(input_fee_recipient);
}
if let Some(tls_certs) = parse_optional::<String>(cli_args, "beacon-nodes-tls-certs")? {
config.beacon_nodes_tls_certs = Some(tls_certs.split(',').map(PathBuf::from).collect());
}
/*
* Http API server
*/
if cli_args.is_present("http") {
config.http_api.enabled = true;
}
if let Some(address) = cli_args.value_of("http-address") {
if cli_args.is_present("unencrypted-http-transport") {
config.http_api.listen_addr = address
.parse::<IpAddr>()
.map_err(|_| "http-address is not a valid IP address.")?;
} else {
return Err(
"While using `--http-address`, you must also use `--unencrypted-http-transport`."
.to_string(),
);
}
}
if let Some(port) = cli_args.value_of("http-port") {
config.http_api.listen_port = port
.parse::<u16>()
.map_err(|_| "http-port is not a valid u16.")?;
}
if let Some(allow_origin) = cli_args.value_of("http-allow-origin") {
// Pre-validate the config value to give feedback to the user on node startup, instead of
// as late as when the first API response is produced.
hyper::header::HeaderValue::from_str(allow_origin)
.map_err(|_| "Invalid allow-origin value")?;
config.http_api.allow_origin = Some(allow_origin.to_string());
}
/*
* Prometheus metrics HTTP server
*/
if cli_args.is_present("metrics") {
config.http_metrics.enabled = true;
}
if let Some(address) = cli_args.value_of("metrics-address") {
config.http_metrics.listen_addr = address
.parse::<IpAddr>()
.map_err(|_| "metrics-address is not a valid IP address.")?;
}
if let Some(port) = cli_args.value_of("metrics-port") {
config.http_metrics.listen_port = port
.parse::<u16>()
.map_err(|_| "metrics-port is not a valid u16.")?;
}
if let Some(allow_origin) = cli_args.value_of("metrics-allow-origin") {
// Pre-validate the config value to give feedback to the user on node startup, instead of
// as late as when the first API response is produced.
hyper::header::HeaderValue::from_str(allow_origin)
.map_err(|_| "Invalid allow-origin value")?;
config.http_metrics.allow_origin = Some(allow_origin.to_string());
}
if cli_args.is_present(DISABLE_MALLOC_TUNING_FLAG) {
config.http_metrics.allocator_metrics_enabled = false;
}
/*
* Explorer metrics
*/
if let Some(monitoring_endpoint) = cli_args.value_of("monitoring-endpoint") {
let update_period_secs =
clap_utils::parse_optional(cli_args, "monitoring-endpoint-period")?;
config.monitoring_api = Some(monitoring_api::Config {
db_path: None,
freezer_db_path: None,
update_period_secs,
monitoring_endpoint: monitoring_endpoint.to_string(),
});
}
if cli_args.is_present("enable-doppelganger-protection") {
config.enable_doppelganger_protection = true;
}
if cli_args.is_present("builder-proposals") {
config.builder_proposals = true;
}
config.gas_limit = cli_args
.value_of("gas-limit")
.map(|gas_limit| {
gas_limit
.parse::<u64>()
.map_err(|_| "gas-limit is not a valid u64.")
})
.transpose()?;
if let Some(registration_timestamp_override) =
cli_args.value_of("builder-registration-timestamp-override")
{
config.builder_registration_timestamp_override = Some(
registration_timestamp_override
.parse::<u64>()
.map_err(|_| "builder-registration-timestamp-override is not a valid u64.")?,
);
}
if cli_args.is_present("strict-fee-recipient") {
warn!(
log,
"The flag `--strict-fee-recipient` has been deprecated due to a bug causing \
missed proposals. The flag will be ignored."
);
}
Enable proposer boost re-orging (#2860) ## Proposed Changes With proposer boosting implemented (#2822) we have an opportunity to re-org out late blocks. This PR adds three flags to the BN to control this behaviour: * `--disable-proposer-reorgs`: turn aggressive re-orging off (it's on by default). * `--proposer-reorg-threshold N`: attempt to orphan blocks with less than N% of the committee vote. If this parameter isn't set then N defaults to 20% when the feature is enabled. * `--proposer-reorg-epochs-since-finalization N`: only attempt to re-org late blocks when the number of epochs since finalization is less than or equal to N. The default is 2 epochs, meaning re-orgs will only be attempted when the chain is finalizing optimally. For safety Lighthouse will only attempt a re-org under very specific conditions: 1. The block being proposed is 1 slot after the canonical head, and the canonical head is 1 slot after its parent. i.e. at slot `n + 1` rather than building on the block from slot `n` we build on the block from slot `n - 1`. 2. The current canonical head received less than N% of the committee vote. N should be set depending on the proposer boost fraction itself, the fraction of the network that is believed to be applying it, and the size of the largest entity that could be hoarding votes. 3. The current canonical head arrived after the attestation deadline from our perspective. This condition was only added to support suppression of forkchoiceUpdated messages, but makes intuitive sense. 4. The block is being proposed in the first 2 seconds of the slot. This gives it time to propagate and receive the proposer boost. ## Additional Info For the initial idea and background, see: https://github.com/ethereum/consensus-specs/pull/2353#issuecomment-950238004 There is also a specification for this feature here: https://github.com/ethereum/consensus-specs/pull/3034 Co-authored-by: Michael Sproul <micsproul@gmail.com> Co-authored-by: pawan <pawandhananjay@gmail.com>
2022-12-13 09:57:26 +00:00
/*
* Experimental
*/
if let Some(delay_ms) = parse_optional::<u64>(cli_args, "block-delay-ms")? {
config.block_delay = Some(Duration::from_millis(delay_ms));
}
Wallet-based, encrypted key management (#1138) * Update hashmap hashset to stable futures * Adds panic test to hashset delay * Port remote_beacon_node to stable futures * Fix lcli merge conflicts * Non rpc stuff compiles * Remove padding * Add error enum, zeroize more things * Fix comment * protocol.rs compiles * Port websockets, timer and notifier to stable futures (#1035) * Fix lcli * Port timer to stable futures * Fix timer * Port websocket_server to stable futures * Port notifier to stable futures * Add TODOS * Port remote_beacon_node to stable futures * Partial eth2-libp2p stable future upgrade * Finished first round of fighting RPC types * Further progress towards porting eth2-libp2p adds caching to discovery * Update behaviour * Add keystore builder * Remove keystore stuff from val client * Add more tests, comments * RPC handler to stable futures * Update RPC to master libp2p * Add more comments, test vectors * Network service additions * Progress on improving JSON validation * More JSON verification * Start moving JSON into own mod * Remove old code * Add more tests, reader/writers * Tidy * Move keystore into own file * Move more logic into keystore file * Tidy * Tidy * Fix the fallback transport construction (#1102) * Allow for odd-character hex * Correct warning * Remove hashmap delay * Compiling version of eth2-libp2p * Update all crates versions * Fix conversion function and add tests (#1113) * Add more json missing field checks * Use scrypt by default * Tidy, address comments * Test path and uuid in vectors * Fix comment * Add checks for kdf params * Enforce empty kdf message * Port validator_client to stable futures (#1114) * Add PH & MS slot clock changes * Account for genesis time * Add progress on duties refactor * Add simple is_aggregator bool to val subscription * Start work on attestation_verification.rs * Add progress on ObservedAttestations * Progress with ObservedAttestations * Fix tests * Add observed attestations to the beacon chain * Add attestation observation to processing code * Add progress on attestation verification * Add first draft of ObservedAttesters * Add more tests * Add observed attesters to beacon chain * Add observers to attestation processing * Add more attestation verification * Create ObservedAggregators map * Remove commented-out code * Add observed aggregators into chain * Add progress * Finish adding features to attestation verification * Ensure beacon chain compiles * Link attn verification into chain * Integrate new attn verification in chain * Remove old attestation processing code * Start trying to fix beacon_chain tests * Split adding into pools into two functions * Add aggregation to harness * Get test harness working again * Adjust the number of aggregators for test harness * Fix edge-case in harness * Integrate new attn processing in network * Fix compile bug in validator_client * Update validator API endpoints * Fix aggreagation in test harness * Fix enum thing * Fix attestation observation bug: * Patch failing API tests * Start adding comments to attestation verification * Remove unused attestation field * Unify "is block known" logic * Update comments * Supress fork choice errors for network processing * Add todos * Tidy * Add gossip attn tests * Disallow test harness to produce old attns * Comment out in-progress tests * Partially address pruning tests * Fix failing store test * Add aggregate tests * Add comments about which spec conditions we check * Dont re-aggregate * Split apart test harness attn production * Fix compile error in network * Make progress on commented-out test * Fix skipping attestation test * Add fork choice verification tests * Tidy attn tests, remove dead code * Remove some accidentally added code * Fix clippy lint * Rename test file * Add block tests, add cheap block proposer check * Rename block testing file * Add observed_block_producers * Tidy * Switch around block signature verification * Finish block testing * Remove gossip from signature tests * First pass of self review * Fix deviation in spec * Update test spec tags * Start moving over to hashset * Finish moving observed attesters to hashmap * Move aggregation pool over to hashmap * Make fc attn borrow again * Fix rest_api compile error * Fix missing comments * Fix monster test * Uncomment increasing slots test * Address remaining comments * Remove unsafe, use cfg test * Remove cfg test flag * Fix dodgy comment * Revert "Update hashmap hashset to stable futures" This reverts commit d432378a3cc5cd67fc29c0b15b96b886c1323554. * Revert "Adds panic test to hashset delay" This reverts commit 281502396fc5b90d9c421a309c2c056982c9525b. * Ported attestation_service * Ported duties_service * Ported fork_service * More ports * Port block_service * Minor fixes * VC compiles * Update TODOS * Borrow self where possible * Ignore aggregates that are already known. * Unify aggregator modulo logic * Fix typo in logs * Refactor validator subscription logic * Avoid reproducing selection proof * Skip HTTP call if no subscriptions * Rename DutyAndState -> DutyAndProof * Tidy logs * Print root as dbg * Fix compile errors in tests * Fix compile error in test * Re-Fix attestation and duties service * Minor fixes Co-authored-by: Paul Hauner <paul@paulhauner.com> * Expose json_keystore mod * First commits on path derivation * Progress with implementation * More progress * Passing intermediate test vectors * Tidy, add comments * Add DerivedKey structs * Move key derivation into own crate * Add zeroize structs * Return error for empty seed * Add tests * Tidy * First commits on path derivation * Progress with implementation * Move key derivation into own crate * Start defining JSON wallet * Add progress * Split out encrypt/decrypt * First commits on path derivation * Progress with implementation * More progress * Passing intermediate test vectors * Tidy, add comments * Add DerivedKey structs * Move key derivation into own crate * Add zeroize structs * Return error for empty seed * Add tests * Tidy * Add progress * Replace some password usage with slice * First commits on path derivation * Progress with implementation * More progress * Passing intermediate test vectors * Tidy, add comments * Add DerivedKey structs * Move key derivation into own crate * Add zeroize structs * Return error for empty seed * Add tests * Tidy * Add progress * Expose PlainText struct * First commits on path derivation * Progress with implementation * More progress * Passing intermediate test vectors * Tidy, add comments * Add DerivedKey structs * Move key derivation into own crate * Add zeroize structs * Return error for empty seed * Add tests * Tidy * Add builder * Expose consts, remove Password * Minor progress * Expose SALT_SIZE * First compiling version * Add test vectors * Network crate update to stable futures * Move dbg assert statement * Port account_manager to stable futures (#1121) * Port account_manager to stable futures * Run async fns in tokio environment * Port rest_api crate to stable futures (#1118) * Port rest_api lib to stable futures * Reduce tokio features * Update notifier to stable futures * Builder update * Further updates * Add mnemonic, tidy * Convert self referential async functions * Tidy * Add testing * Add first attempt at validator_dir * Present pubkey field * stable futures fixes (#1124) * Fix eth1 update functions * Fix genesis and client * Fix beacon node lib * Return appropriate runtimes from environment * Fix test rig * Refactor eth1 service update * Upgrade simulator to stable futures * Lighthouse compiles on stable futures * Add first pass of wallet manager * Progress with CLI * Remove println debugging statement * Tidy output * Tidy 600 perms * Update libp2p service, start rpc test upgrade * Add validator creation flow * Update network crate for new libp2p * Start tidying, adding comments * Update tokio::codec to futures_codec (#1128) * Further work towards RPC corrections * Correct http timeout and network service select * Add wallet mgr testing * Shift LockedWallet into own file * Add comments to fs * Start integration into VC * Use tokio runtime for libp2p * Revert "Update tokio::codec to futures_codec (#1128)" This reverts commit e57aea924acf5cbabdcea18895ac07e38a425ed7. * Upgrade RPC libp2p tests * Upgrade secio fallback test * Add lcli keypair upgrade command * Upgrade gossipsub examples * Clean up RPC protocol * Test fixes (#1133) * Correct websocket timeout and run on os thread * Fix network test * Add --secrets-dir to VC * Remove --legacy-keys from VC * Clean up PR * Correct tokio tcp move attestation service tests * Upgrade attestation service tests * Fix sim * Correct network test * Correct genesis test * Start docs * Add progress for validator generation * Tidy error messages * Test corrections * Log info when block is received * Modify logs and update attester service events * Stable futures: fixes to vc, eth1 and account manager (#1142) * Add local testnet scripts * Remove whiteblock script * Rename local testnet script * Move spawns onto handle * Fix VC panic * Initial fix to block production issue * Tidy block producer fix * Tidy further * Add local testnet clean script * Run cargo fmt * Tidy duties service * Tidy fork service * Tidy ForkService * Tidy AttestationService * Tidy notifier * Ensure await is not suppressed in eth1 * Ensure await is not suppressed in account_manager * Use .ok() instead of .unwrap_or(()) * RPC decoding test for proto * Update discv5 and eth2-libp2p deps * Run cargo fmt * Pre-build keystores for sim * Fix lcli double runtime issue (#1144) * Handle stream termination and dialing peer errors * Correct peer_info variant types * Add progress on new deposit flow * Remove unnecessary warnings * Handle subnet unsubscription removal and improve logigng * Add logs around ping * Upgrade discv5 and improve logging * Handle peer connection status for multiple connections * Improve network service logging * Add more incomplete progress * Improve logging around peer manager * Upgrade swarm poll centralise peer management * Identify clients on error * Fix `remove_peer` in sync (#1150) * remove_peer removes from all chains * Remove logs * Fix early return from loop * Improved logging, fix panic * Partially correct tests * Add deposit command * Remove old validator directory * Start adding AM tests * Stable futures: Vc sync (#1149) * Improve syncing heuristic * Add comments * Use safer method for tolerance * Fix tests * Binary testing progress * Progress with CLI tests * Use constants for flags * More account manager testing * Improve CLI tests * Move upgrade-legacy-keypairs into account man * Use rayon for VC key generation * Add comments to `validator_dir` * Add testing to validator_dir * Add fix to eth1-sim * Check errors in eth1-sim * Fix mutability issue * Ensure password file ends in .pass * Add more tests to wallet manager * Tidy deposit * Tidy account manager * Tidy account manager * Remove panic * Generate keypairs earlier in sim * Tidy eth1-sime * Try to fix eth1 sim * Address review comments * Fix typo in CLI command * Update docs * Disable eth1 sim * Remove eth1 sim completely Co-authored-by: Age Manning <Age@AgeManning.com> Co-authored-by: pawanjay176 <pawandhananjay@gmail.com>
2020-05-18 09:01:45 +00:00
Ok(config)
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
// Ensures the default config does not panic.
fn default_config() {
Config::default();
}
}