Add Holesky (#4653)

## Issue Addressed

NA

## Proposed Changes

Add the Holesky network config as per 36e4ff2d51/custom_config_data.

Since the genesis state is ~190MB, I've opted to *not* include it in the binary and instead download it at runtime (see #4564 for context). To download this file we have:

- A hard-coded URL for a SigP-hosted S3 bucket with the Holesky genesis state. Assuming this download works correctly, users will be none the wiser that the state wasn't included in the binary (apart from some additional logs)
- If the user provides a `--checkpoint-sync-url` flag, then LH will download the genesis state from that server rather than our S3 bucket.
- If the user provides a `--genesis-state-url` flag, then LH will download the genesis state from that server regardless of the S3 bucket or `--checkpoint-sync-url` flag.
- Whenever a genesis state is downloaded it is checked against a checksum baked into the binary.
- A genesis state will never be downloaded if it's already included in the binary.
- There is a `--genesis-state-url-timeout` flag to tweak the timeout for downloading the genesis state file.

## Log Output

Example of log output when a state is downloaded:

```bash
Aug 23 05:40:13.424 INFO Logging to file                         path: "/Users/paul/.lighthouse/holesky/beacon/logs/beacon.log"
Aug 23 05:40:13.425 INFO Lighthouse started                      version: Lighthouse/v4.3.0-bd9931f+
Aug 23 05:40:13.425 INFO Configured for network                  name: holesky
Aug 23 05:40:13.426 INFO Data directory initialised              datadir: /Users/paul/.lighthouse/holesky
Aug 23 05:40:13.427 INFO Deposit contract                        address: 0x4242424242424242424242424242424242424242, deploy_block: 0
Aug 23 05:40:13.427 INFO Downloading genesis state               info: this may take some time on testnets with large validator counts, timeout: 60s, server: https://sigp-public-genesis-states.s3.ap-southeast-2.amazonaws.com/
Aug 23 05:40:29.895 INFO Starting from known genesis state       service: beacon
```

Example of log output when there are no URLs specified:

```
Aug 23 06:29:51.645 INFO Logging to file                         path: "/Users/paul/.lighthouse/goerli/beacon/logs/beacon.log"
Aug 23 06:29:51.646 INFO Lighthouse started                      version: Lighthouse/v4.3.0-666a39c+
Aug 23 06:29:51.646 INFO Configured for network                  name: goerli
Aug 23 06:29:51.647 INFO Data directory initialised              datadir: /Users/paul/.lighthouse/goerli
Aug 23 06:29:51.647 INFO Deposit contract                        address: 0xff50ed3d0ec03ac01d4c79aad74928bff48a7b2b, deploy_block: 4367322
The genesis state is not present in the binary and there are no known download URLs. Please use --checkpoint-sync-url or --genesis-state-url.
```

## Additional Info

I tested the `--genesis-state-url` flag with all 9 Goerli checkpoint sync servers on https://eth-clients.github.io/checkpoint-sync-endpoints/ and they all worked 🎉 

My IDE eagerly formatted some `Cargo.toml`. I've disabled it but I don't see the value in spending time reverting the changes that are already there.

I also added the `GenesisStateBytes` enum to avoid an unnecessary clone on the genesis state bytes baked into the binary. This is not a huge deal on Mainnet, but will become more relevant when testing with big genesis states.

When we do a fresh checkpoint sync we're downloading the genesis state to check the `genesis_validators_root` against the finalised state we receive. This is not *entirely* pointless, since we verify the checksum when we download the genesis state so we are actually guaranteeing that the finalised state is on the same network. There might be a smarter/less-download-y way to go about this, but I've run out of cycles to figure that out. Perhaps we can grab it in the next release?

Cargo.lock

@@ -33,6 +33,7 @@ dependencies = [
  "serde",
  "serde_json",
  "slashing_protection",
+ "slog",
  "slot_clock",
  "tempfile",
  "tokio",
@@ -2223,9 +2224,16 @@ dependencies = [
  "discv5",
  "eth2_config",
  "ethereum_ssz",
+ "logging",
+ "pretty_reqwest_error",
+ "reqwest",
+ "sensitive_url",
  "serde_yaml",
+ "sha2 0.10.7",
+ "slog",
  "tempfile",
  "types",
+ "url",
  "zip",
 ]
 

account_manager/Cargo.toml

@@ -1,7 +1,10 @@
 [package]
 name = "account_manager"
 version = "0.3.5"
-authors = ["Paul Hauner <paul@paulhauner.com>", "Luke Anderson <luke@sigmaprime.io>"]
+authors = [
+    "Paul Hauner <paul@paulhauner.com>",
+    "Luke Anderson <luke@sigmaprime.io>",
+]
 edition = "2021"
 
 [dependencies]
@@ -19,13 +22,14 @@ tokio = { version = "1.14.0", features = ["full"] }
 eth2_keystore = { path = "../crypto/eth2_keystore" }
 account_utils = { path = "../common/account_utils" }
 slashing_protection = { path = "../validator_client/slashing_protection" }
-eth2 = {path = "../common/eth2"}
+eth2 = { path = "../common/eth2" }
-safe_arith = {path = "../consensus/safe_arith"}
+safe_arith = { path = "../consensus/safe_arith" }
 slot_clock = { path = "../common/slot_clock" }
 filesystem = { path = "../common/filesystem" }
 sensitive_url = { path = "../common/sensitive_url" }
 serde = { version = "1.0.116", features = ["derive"] }
 serde_json = "1.0.58"
+slog = { version = "2.5.2" }
 
 [dev-dependencies]
 tempfile = "3.1.0"

account_manager/src/validator/exit.rs

@@ -10,6 +10,7 @@ use eth2_keystore::Keystore;
 use eth2_network_config::Eth2NetworkConfig;
 use safe_arith::SafeArith;
 use sensitive_url::SensitiveUrl;
+use slog::Logger;
 use slot_clock::{SlotClock, SystemTimeSlotClock};
 use std::path::{Path, PathBuf};
 use std::time::Duration;
@@ -78,6 +79,12 @@ pub fn cli_run<E: EthSpec>(matches: &ArgMatches, env: Environment<E>) -> Result<
     let password_file_path: Option<PathBuf> =
         clap_utils::parse_optional(matches, PASSWORD_FILE_FLAG)?;
 
+    let genesis_state_url: Option<String> =
+        clap_utils::parse_optional(matches, "genesis-state-url")?;
+    let genesis_state_url_timeout =
+        clap_utils::parse_required(matches, "genesis-state-url-timeout")
+            .map(Duration::from_secs)?;
+
     let stdin_inputs = cfg!(windows) || matches.is_present(STDIN_INPUTS_FLAG);
     let no_wait = matches.is_present(NO_WAIT);
     let no_confirmation = matches.is_present(NO_CONFIRMATION);
@@ -104,6 +111,9 @@ pub fn cli_run<E: EthSpec>(matches: &ArgMatches, env: Environment<E>) -> Result<
         &eth2_network_config,
         no_wait,
         no_confirmation,
+        genesis_state_url,
+        genesis_state_url_timeout,
+        env.core_context().log(),
     ))?;
 
     Ok(())
@@ -120,13 +130,14 @@ async fn publish_voluntary_exit<E: EthSpec>(
     eth2_network_config: &Eth2NetworkConfig,
     no_wait: bool,
     no_confirmation: bool,
+    genesis_state_url: Option<String>,
+    genesis_state_url_timeout: Duration,
+    log: &Logger,
 ) -> Result<(), String> {
     let genesis_data = get_geneisis_data(client).await?;
     let testnet_genesis_root = eth2_network_config
-        .beacon_state::<E>()
+        .genesis_validators_root::<E>(genesis_state_url.as_deref(), genesis_state_url_timeout, log)?
-        .as_ref()
+        .ok_or("Genesis state is unknown")?;
-        .expect("network should have valid genesis state")
-        .genesis_validators_root();
 
     // Verify that the beacon node and validator being exited are on the same network.
     if genesis_data.genesis_validators_root != testnet_genesis_root {

account_manager/src/validator/slashing_protection.rs

@@ -7,7 +7,8 @@ use slashing_protection::{
 use std::fs::File;
 use std::path::PathBuf;
 use std::str::FromStr;
-use types::{BeaconState, Epoch, EthSpec, PublicKeyBytes, Slot};
+use std::time::Duration;
+use types::{Epoch, EthSpec, PublicKeyBytes, Slot};
 
 pub const CMD: &str = "slashing-protection";
 pub const IMPORT_CMD: &str = "import";
@@ -82,19 +83,24 @@ pub fn cli_run<T: EthSpec>(
 ) -> Result<(), String> {
     let slashing_protection_db_path = validator_base_dir.join(SLASHING_PROTECTION_FILENAME);
 
+    let genesis_state_url: Option<String> =
+        clap_utils::parse_optional(matches, "genesis-state-url")?;
+    let genesis_state_url_timeout =
+        clap_utils::parse_required(matches, "genesis-state-url-timeout")
+            .map(Duration::from_secs)?;
+
+    let context = env.core_context();
     let eth2_network_config = env
         .eth2_network_config
         .ok_or("Unable to get testnet configuration from the environment")?;
 
     let genesis_validators_root = eth2_network_config
-        .beacon_state::<T>()
+        .genesis_validators_root::<T>(
-        .map(|state: BeaconState<T>| state.genesis_validators_root())
+            genesis_state_url.as_deref(),
-        .map_err(|e| {
+            genesis_state_url_timeout,
-            format!(
+            context.log(),
-                "Unable to get genesis state, has genesis occurred? Detail: {:?}",
+        )?
-                e
+        .ok_or_else(|| "Unable to get genesis state, has genesis occurred?".to_string())?;
-            )
-        })?;
 
     match matches.subcommand() {
         (IMPORT_CMD, Some(matches)) => {

beacon_node/Cargo.toml

@@ -1,7 +1,10 @@
 [package]
 name = "beacon_node"
 version = "4.3.0"
-authors = ["Paul Hauner <paul@paulhauner.com>", "Age Manning <Age@AgeManning.com"]
+authors = [
+    "Paul Hauner <paul@paulhauner.com>",
+    "Age Manning <Age@AgeManning.com",
+]
 edition = "2021"
 
 [lib]
@@ -12,7 +15,9 @@ path = "src/lib.rs"
 node_test_rig = { path = "../testing/node_test_rig" }
 
 [features]
-write_ssz_files = ["beacon_chain/write_ssz_files"]  # Writes debugging .ssz files to /tmp during block processing.
+write_ssz_files = [
+    "beacon_chain/write_ssz_files",
+] # Writes debugging .ssz files to /tmp during block processing.
 
 [dependencies]
 eth2_config = { path = "../common/eth2_config" }
@@ -21,9 +26,12 @@ types = { path = "../consensus/types" }
 store = { path = "./store" }
 client = { path = "client" }
 clap = "2.33.3"
-slog = { version = "2.5.2", features = ["max_level_trace", "release_max_level_trace"] }
+slog = { version = "2.5.2", features = [
+    "max_level_trace",
+    "release_max_level_trace",
+] }
 dirs = "3.0.1"
-directory = {path = "../common/directory"}
+directory = { path = "../common/directory" }
 futures = "0.3.7"
 environment = { path = "../lighthouse/environment" }
 task_executor = { path = "../common/task_executor" }

beacon_node/client/src/builder.rs

@@ -154,6 +154,7 @@ where
         let runtime_context =
             runtime_context.ok_or("beacon_chain_start_method requires a runtime context")?;
         let context = runtime_context.service_context("beacon".into());
+        let log = context.log();
         let spec = chain_spec.ok_or("beacon_chain_start_method requires a chain spec")?;
         let event_handler = if self.http_api_config.enabled {
             Some(ServerSentEventHandler::new(
@@ -164,7 +165,7 @@ where
             None
         };
 
-        let execution_layer = if let Some(config) = config.execution_layer {
+        let execution_layer = if let Some(config) = config.execution_layer.clone() {
             let context = runtime_context.service_context("exec".into());
             let execution_layer = ExecutionLayer::from_config(
                 config,
@@ -249,23 +250,19 @@ where
                 )?;
                 builder.genesis_state(genesis_state).map(|v| (v, None))?
             }
-            ClientGenesis::SszBytes {
+            ClientGenesis::GenesisState => {
-                genesis_state_bytes,
-            } => {
                 info!(
                     context.log(),
                     "Starting from known genesis state";
                 );
 
-                let genesis_state = BeaconState::from_ssz_bytes(&genesis_state_bytes, &spec)
+                let genesis_state = genesis_state(&runtime_context, &config, log)?;
-                    .map_err(|e| format!("Unable to parse genesis state SSZ: {:?}", e))?;
 
                 builder.genesis_state(genesis_state).map(|v| (v, None))?
             }
             ClientGenesis::WeakSubjSszBytes {
                 anchor_state_bytes,
                 anchor_block_bytes,
-                genesis_state_bytes,
             } => {
                 info!(context.log(), "Starting checkpoint sync");
                 if config.chain.genesis_backfill {
@@ -279,17 +276,13 @@ where
                     .map_err(|e| format!("Unable to parse weak subj state SSZ: {:?}", e))?;
                 let anchor_block = SignedBeaconBlock::from_ssz_bytes(&anchor_block_bytes, &spec)
                     .map_err(|e| format!("Unable to parse weak subj block SSZ: {:?}", e))?;
-                let genesis_state = BeaconState::from_ssz_bytes(&genesis_state_bytes, &spec)
+                let genesis_state = genesis_state(&runtime_context, &config, log)?;
-                    .map_err(|e| format!("Unable to parse genesis state SSZ: {:?}", e))?;
 
                 builder
                     .weak_subjectivity_state(anchor_state, anchor_block, genesis_state)
                     .map(|v| (v, None))?
             }
-            ClientGenesis::CheckpointSyncUrl {
+            ClientGenesis::CheckpointSyncUrl { url } => {
-                genesis_state_bytes,
-                url,
-            } => {
                 info!(
                     context.log(),
                     "Starting checkpoint sync";
@@ -384,8 +377,7 @@ where
 
                 debug!(context.log(), "Downloaded finalized block");
 
-                let genesis_state = BeaconState::from_ssz_bytes(&genesis_state_bytes, &spec)
+                let genesis_state = genesis_state(&runtime_context, &config, log)?;
-                    .map_err(|e| format!("Unable to parse genesis state SSZ: {:?}", e))?;
 
                 info!(
                     context.log(),
@@ -1089,3 +1081,22 @@ where
         Ok(self)
     }
 }
+
+/// Obtain the genesis state from the `eth2_network_config` in `context`.
+fn genesis_state<T: EthSpec>(
+    context: &RuntimeContext<T>,
+    config: &ClientConfig,
+    log: &Logger,
+) -> Result<BeaconState<T>, String> {
+    let eth2_network_config = context
+        .eth2_network_config
+        .as_ref()
+        .ok_or("An eth2_network_config is required to obtain the genesis state")?;
+    eth2_network_config
+        .genesis_state::<T>(
+            config.genesis_state_url.as_deref(),
+            config.genesis_state_url_timeout,
+            log,
+        )?
+        .ok_or_else(|| "Genesis state is unknown".to_string())
+}

beacon_node/client/src/config.rs

@@ -7,6 +7,7 @@ use sensitive_url::SensitiveUrl;
 use serde_derive::{Deserialize, Serialize};
 use std::fs;
 use std::path::PathBuf;
+use std::time::Duration;
 use types::{Graffiti, PublicKeyBytes};
 /// Default directory name for the freezer database under the top-level data dir.
 const DEFAULT_FREEZER_DB_DIR: &str = "freezer_db";
@@ -25,18 +26,13 @@ pub enum ClientGenesis {
     /// contract.
     #[default]
     DepositContract,
-    /// Loads the genesis state from SSZ-encoded `BeaconState` bytes.
+    /// Loads the genesis state from the genesis state in the `Eth2NetworkConfig`.
-    ///
+    GenesisState,
-    /// We include the bytes instead of the `BeaconState<E>` because the `EthSpec` type
-    /// parameter would be very annoying.
-    SszBytes { genesis_state_bytes: Vec<u8> },
     WeakSubjSszBytes {
-        genesis_state_bytes: Vec<u8>,
         anchor_state_bytes: Vec<u8>,
         anchor_block_bytes: Vec<u8>,
     },
     CheckpointSyncUrl {
-        genesis_state_bytes: Vec<u8>,
         url: SensitiveUrl,
     },
 }
@@ -81,6 +77,8 @@ pub struct Config {
     pub slasher: Option<slasher::Config>,
     pub logger_config: LoggerConfig,
     pub beacon_processor: BeaconProcessorConfig,
+    pub genesis_state_url: Option<String>,
+    pub genesis_state_url_timeout: Duration,
 }
 
 impl Default for Config {
@@ -108,6 +106,9 @@ impl Default for Config {
             validator_monitor_individual_tracking_threshold: DEFAULT_INDIVIDUAL_TRACKING_THRESHOLD,
             logger_config: LoggerConfig::default(),
             beacon_processor: <_>::default(),
+            genesis_state_url: <_>::default(),
+            // This default value should always be overwritten by the CLI default value.
+            genesis_state_url_timeout: Duration::from_secs(60),
         }
     }
 }

beacon_node/src/config.rs

@@ -471,9 +471,30 @@ pub fn get_config<E: EthSpec>(
     client_config.chain.checkpoint_sync_url_timeout =
         clap_utils::parse_required::<u64>(cli_args, "checkpoint-sync-url-timeout")?;
 
-    client_config.genesis = if let Some(genesis_state_bytes) =
+    client_config.genesis_state_url_timeout =
-        eth2_network_config.genesis_state_bytes.clone()
+        clap_utils::parse_required(cli_args, "genesis-state-url-timeout")
-    {
+            .map(Duration::from_secs)?;
+
+    let genesis_state_url_opt =
+        clap_utils::parse_optional::<String>(cli_args, "genesis-state-url")?;
+    let checkpoint_sync_url_opt =
+        clap_utils::parse_optional::<String>(cli_args, "checkpoint-sync-url")?;
+
+    // If the `--genesis-state-url` is defined, use that to download the
+    // genesis state bytes. If it's not defined, try `--checkpoint-sync-url`.
+    client_config.genesis_state_url = if let Some(genesis_state_url) = genesis_state_url_opt {
+        Some(genesis_state_url)
+    } else if let Some(checkpoint_sync_url) = checkpoint_sync_url_opt {
+        // If the checkpoint sync URL is going to be used to download the
+        // genesis state, adopt the timeout from the checkpoint sync URL too.
+        client_config.genesis_state_url_timeout =
+            Duration::from_secs(client_config.chain.checkpoint_sync_url_timeout);
+        Some(checkpoint_sync_url)
+    } else {
+        None
+    };
+
+    client_config.genesis = if eth2_network_config.genesis_state_is_known() {
         // Set up weak subjectivity sync, or start from the hardcoded genesis state.
         if let (Some(initial_state_path), Some(initial_block_path)) = (
             cli_args.value_of("checkpoint-state"),
@@ -495,7 +516,6 @@ pub fn get_config<E: EthSpec>(
             let anchor_block_bytes = read(initial_block_path)?;
 
             ClientGenesis::WeakSubjSszBytes {
-                genesis_state_bytes,
                 anchor_state_bytes,
                 anchor_block_bytes,
             }
@@ -503,17 +523,9 @@ pub fn get_config<E: EthSpec>(
             let url = SensitiveUrl::parse(remote_bn_url)
                 .map_err(|e| format!("Invalid checkpoint sync URL: {:?}", e))?;
 
-            ClientGenesis::CheckpointSyncUrl {
+            ClientGenesis::CheckpointSyncUrl { url }
-                genesis_state_bytes,
-                url,
-            }
         } else {
-            // Note: re-serializing the genesis state is not so efficient, however it avoids adding
+            ClientGenesis::GenesisState
-            // trait bounds to the `ClientGenesis` enum. This would have significant flow-on
-            // effects.
-            ClientGenesis::SszBytes {
-                genesis_state_bytes,
-            }
         }
     } else {
         if cli_args.is_present("checkpoint-state") || cli_args.is_present("checkpoint-sync-url") {

boot_node/src/config.rs

@@ -10,6 +10,7 @@ use lighthouse_network::{
 use serde_derive::{Deserialize, Serialize};
 use ssz::Encode;
 use std::net::{SocketAddrV4, SocketAddrV6};
+use std::time::Duration;
 use std::{marker::PhantomData, path::PathBuf};
 use types::EthSpec;
 
@@ -90,8 +91,19 @@ impl<T: EthSpec> BootNodeConfig<T> {
             let enr_fork = {
                 let spec = eth2_network_config.chain_spec::<T>()?;
 
-                if eth2_network_config.beacon_state_is_known() {
+                let genesis_state_url: Option<String> =
-                    let genesis_state = eth2_network_config.beacon_state::<T>()?;
+                    clap_utils::parse_optional(matches, "genesis-state-url")?;
+                let genesis_state_url_timeout =
+                    clap_utils::parse_required(matches, "genesis-state-url-timeout")
+                        .map(Duration::from_secs)?;
+
+                if eth2_network_config.genesis_state_is_known() {
+                    let genesis_state = eth2_network_config
+                        .genesis_state::<T>(genesis_state_url.as_deref(), genesis_state_url_timeout, &logger)?
+                        .ok_or_else(|| {
+                            "The genesis state for this network is not known, this is an unsupported mode"
+                                .to_string()
+                        })?;
 
                     slog::info!(logger, "Genesis state found"; "root" => genesis_state.canonical_root().to_string());
                     let enr_fork = spec.enr_fork_id::<T>(

common/eth2_config/src/lib.rs

@@ -23,6 +23,16 @@ pub const PREDEFINED_NETWORKS_DIR: &str = predefined_networks_dir!();
 pub const GENESIS_FILE_NAME: &str = "genesis.ssz";
 pub const GENESIS_ZIP_FILE_NAME: &str = "genesis.ssz.zip";
 
+const HOLESKY_GENESIS_STATE_SOURCE: GenesisStateSource = GenesisStateSource::Url {
+    urls: &[
+        // This is an AWS S3 bucket hosted by Sigma Prime. See Paul Hauner for
+        // more details.
+        "https://sigp-public-genesis-states.s3.ap-southeast-2.amazonaws.com/holesky/",
+    ],
+    checksum: "0x76631cd0b9ddc5b2c766b496e23f16759ce1181446a4efb40e5540cd15b78a07",
+    genesis_validators_root: "0x9143aa7c615a7f7115e2b6aac319c03529df8242ae705fba9df39b79c59fa8b1",
+};
+
 /// The core configuration of a Lighthouse beacon node.
 #[derive(Debug, Clone)]
 pub struct Eth2Config {
@@ -62,6 +72,32 @@ impl Eth2Config {
     }
 }
 
+/// Describes how a genesis state may be obtained.
+#[derive(Copy, Clone, Debug, PartialEq)]
+pub enum GenesisStateSource {
+    /// The genesis state for this network is not yet known.
+    Unknown,
+    /// The genesis state for this network is included in the binary via
+    /// `include_bytes!` or by loading from a testnet dir.
+    IncludedBytes,
+    /// The genesis state for this network should be downloaded from a URL.
+    Url {
+        /// URLs to try to download the file from, in order.
+        urls: &'static [&'static str],
+        /// The SHA256 of the genesis state bytes. This is *not* a hash tree
+        /// root to simplify the types (i.e., to avoid getting EthSpec
+        /// involved).
+        ///
+        /// The format should be 0x-prefixed ASCII bytes.
+        checksum: &'static str,
+        /// The `genesis_validators_root` of the genesis state. Used to avoid
+        /// downloading the state for simple signing operations.
+        ///
+        /// The format should be 0x-prefixed ASCII bytes.
+        genesis_validators_root: &'static str,
+    },
+}
+
 /// A directory that can be built by downloading files via HTTP.
 ///
 /// Used by the `eth2_network_config` crate to initialize the network directories during build and
@@ -70,7 +106,7 @@ impl Eth2Config {
 pub struct Eth2NetArchiveAndDirectory<'a> {
     pub name: &'a str,
     pub config_dir: &'a str,
-    pub genesis_is_known: bool,
+    pub genesis_state_source: GenesisStateSource,
 }
 
 impl<'a> Eth2NetArchiveAndDirectory<'a> {
@@ -89,15 +125,11 @@ impl<'a> Eth2NetArchiveAndDirectory<'a> {
     }
 }
 
-/// Indicates that the `genesis.ssz.zip` file is present on the filesystem. This means that the
+#[derive(Clone, Debug, PartialEq)]
-/// deposit ceremony has concluded and the final genesis `BeaconState` is known.
-const GENESIS_STATE_IS_KNOWN: bool = true;
-
-#[derive(Copy, Clone, Debug, PartialEq)]
 pub struct HardcodedNet {
     pub name: &'static str,
     pub config_dir: &'static str,
-    pub genesis_is_known: bool,
+    pub genesis_state_source: GenesisStateSource,
     pub config: &'static [u8],
     pub deploy_block: &'static [u8],
     pub boot_enr: &'static [u8],
@@ -109,7 +141,7 @@ pub struct HardcodedNet {
 /// It also defines a `include_<title>_file!` macro which provides a wrapper around
 /// `std::include_bytes`, allowing the inclusion of bytes from the specific testnet directory.
 macro_rules! define_archive {
-    ($name_ident: ident, $config_dir: tt, $genesis_is_known: ident) => {
+    ($name_ident: ident, $config_dir: tt, $genesis_state_source: path) => {
         paste! {
             #[macro_use]
             pub mod $name_ident {
@@ -118,7 +150,7 @@ macro_rules! define_archive {
                 pub const ETH2_NET_DIR: Eth2NetArchiveAndDirectory = Eth2NetArchiveAndDirectory {
                     name: stringify!($name_ident),
                     config_dir: $config_dir,
-                    genesis_is_known: $genesis_is_known,
+                    genesis_state_source: $genesis_state_source,
                 };
 
                 /// A wrapper around `std::include_bytes` which includes a file from a specific network
@@ -151,7 +183,7 @@ macro_rules! define_net {
         $this_crate::HardcodedNet {
             name: ETH2_NET_DIR.name,
             config_dir: ETH2_NET_DIR.config_dir,
-            genesis_is_known: ETH2_NET_DIR.genesis_is_known,
+            genesis_state_source: ETH2_NET_DIR.genesis_state_source,
             config: $this_crate::$include_file!($this_crate, "../", "config.yaml"),
             deploy_block: $this_crate::$include_file!($this_crate, "../", "deploy_block.txt"),
             boot_enr: $this_crate::$include_file!($this_crate, "../", "boot_enr.yaml"),
@@ -199,9 +231,9 @@ macro_rules! define_nets {
 /// `build.rs` which will unzip the genesis states. Then, that `eth2_network_configs` crate can
 /// perform the final step of using `std::include_bytes` to bake the files (bytes) into the binary.
 macro_rules! define_hardcoded_nets {
-    ($(($name_ident: ident, $config_dir: tt, $genesis_is_known: ident)),+) => {
+    ($(($name_ident: ident, $config_dir: tt, $genesis_state_source: path)),+) => {
         $(
-        define_archive!($name_ident, $config_dir, $genesis_is_known);
+        define_archive!($name_ident, $config_dir, $genesis_state_source);
         )+
 
         pub const ETH2_NET_DIRS: &[Eth2NetArchiveAndDirectory<'static>] = &[$($name_ident::ETH2_NET_DIR,)+];
@@ -242,9 +274,8 @@ define_hardcoded_nets!(
         // The name of the directory in the `eth2_network_config/built_in_network_configs`
         // directory where the configuration files are located for this network.
         "mainnet",
-        // Set to `true` if the genesis state can be found in the `built_in_network_configs`
+        // Describes how the genesis state can be obtained.
-        // directory.
+        GenesisStateSource::IncludedBytes
-        GENESIS_STATE_IS_KNOWN
     ),
     (
         // Network name (must be unique among all networks).
@@ -252,9 +283,8 @@ define_hardcoded_nets!(
         // The name of the directory in the `eth2_network_config/built_in_network_configs`
         // directory where the configuration files are located for this network.
         "prater",
-        // Set to `true` if the genesis state can be found in the `built_in_network_configs`
+        // Describes how the genesis state can be obtained.
-        // directory.
+        GenesisStateSource::IncludedBytes
-        GENESIS_STATE_IS_KNOWN
     ),
     (
         // Network name (must be unique among all networks).
@@ -264,9 +294,8 @@ define_hardcoded_nets!(
         //
         // The Goerli network is effectively an alias to Prater.
         "prater",
-        // Set to `true` if the genesis state can be found in the `built_in_network_configs`
+        // Describes how the genesis state can be obtained.
-        // directory.
+        GenesisStateSource::IncludedBytes
-        GENESIS_STATE_IS_KNOWN
     ),
     (
         // Network name (must be unique among all networks).
@@ -274,9 +303,8 @@ define_hardcoded_nets!(
         // The name of the directory in the `eth2_network_config/built_in_network_configs`
         // directory where the configuration files are located for this network.
         "gnosis",
-        // Set to `true` if the genesis state can be found in the `built_in_network_configs`
+        // Describes how the genesis state can be obtained.
-        // directory.
+        GenesisStateSource::IncludedBytes
-        GENESIS_STATE_IS_KNOWN
     ),
     (
         // Network name (must be unique among all networks).
@@ -284,8 +312,16 @@ define_hardcoded_nets!(
         // The name of the directory in the `eth2_network_config/built_in_network_configs`
         // directory where the configuration files are located for this network.
         "sepolia",
-        // Set to `true` if the genesis state can be found in the `built_in_network_configs`
+        // Describes how the genesis state can be obtained.
-        // directory.
+        GenesisStateSource::IncludedBytes
-        GENESIS_STATE_IS_KNOWN
+    ),
+    (
+        // Network name (must be unique among all networks).
+        holesky,
+        // The name of the directory in the `eth2_network_config/built_in_network_configs`
+        // directory where the configuration files are located for this network.
+        "holesky",
+        // Describes how the genesis state can be obtained.
+        HOLESKY_GENESIS_STATE_SOURCE
     )
 );

common/eth2_network_config/Cargo.toml

@@ -8,14 +8,21 @@ build = "build.rs"
 
 [build-dependencies]
 zip = "0.6"
-eth2_config = { path = "../eth2_config"}
+eth2_config = { path = "../eth2_config" }
 
 [dev-dependencies]
 tempfile = "3.1.0"
 
 [dependencies]
 serde_yaml = "0.8.13"
-types = { path = "../../consensus/types"}
+types = { path = "../../consensus/types" }
 ethereum_ssz = "0.5.0"
-eth2_config = { path = "../eth2_config"}
+eth2_config = { path = "../eth2_config" }
 discv5 = "0.3.1"
+reqwest = { version = "0.11.0", features = ["blocking"] }
+pretty_reqwest_error = { path = "../pretty_reqwest_error" }
+sha2 = "0.10"
+url = "2.2.2"
+sensitive_url = { path = "../sensitive_url" }
+slog = "2.5.2"
+logging = { path = "../logging" }

common/eth2_network_config/build.rs

@@ -1,5 +1,7 @@
 //! Extracts zipped genesis states on first run.
-use eth2_config::{Eth2NetArchiveAndDirectory, ETH2_NET_DIRS, GENESIS_FILE_NAME};
+use eth2_config::{
+    Eth2NetArchiveAndDirectory, GenesisStateSource, ETH2_NET_DIRS, GENESIS_FILE_NAME,
+};
 use std::fs::File;
 use std::io;
 use zip::ZipArchive;
@@ -26,7 +28,7 @@ fn uncompress_state(network: &Eth2NetArchiveAndDirectory<'static>) -> Result<(),
         return Ok(());
     }
 
-    if network.genesis_is_known {
+    if network.genesis_state_source == GenesisStateSource::IncludedBytes {
         // Extract genesis state from genesis.ssz.zip
         let archive_path = network.genesis_state_archive();
         let archive_file = File::open(&archive_path)
@@ -46,7 +48,8 @@ fn uncompress_state(network: &Eth2NetArchiveAndDirectory<'static>) -> Result<(),
         io::copy(&mut file, &mut outfile)
             .map_err(|e| format!("Error writing file {:?}: {}", genesis_ssz_path, e))?;
     } else {
-        // Create empty genesis.ssz if genesis is unknown
+        // Create empty genesis.ssz if genesis is unknown or to be downloaded via URL.
+        // This is a bit of a hack to make `include_bytes!` easier to deal with.
         File::create(genesis_ssz_path)
             .map_err(|e| format!("Failed to create {}: {}", GENESIS_FILE_NAME, e))?;
     }

common/eth2_network_config/built_in_network_configs/holesky/boot_enr.yaml

@@ -0,0 +1,8 @@
+# EF
+- enr:-Iq4QJk4WqRkjsX5c2CXtOra6HnxN-BMXnWhmhEQO9Bn9iABTJGdjUOurM7Btj1ouKaFkvTRoju5vz2GPmVON2dffQKGAX53x8JigmlkgnY0gmlwhLKAlv6Jc2VjcDI1NmsxoQK6S-Cii_KmfFdUJL2TANL3ksaKUnNXvTCv1tLwXs0QgIN1ZHCCIyk
+- enr:-KG4QF6d6vMSboSujAXTI4vYqArccm0eIlXfcxf2Lx_VE1q6IkQo_2D5LAO3ZSBVUs0w5rrVDmABJZuMzISe_pZundADhGV0aDKQqX6DZjABcAAAAQAAAAAAAIJpZIJ2NIJpcISygIjpiXNlY3AyNTZrMaEDF3aSa7QSCvdqLpANNd8GML4PLEZVg45fKQwMWhDZjd2DdGNwgiMog3VkcIIjKA
+- enr:-Ly4QJLXSSAj3ggPBIcodvBU6IyfpU_yW7E9J-5syoJorBuvcYj_Fokcjr303bQoTdWXADf8po0ssh75Mr5wVGzZZsMBh2F0dG5ldHOIAAAAAAAAAACEZXRoMpCpfoNmMAFwAAABAAAAAAAAgmlkgnY0gmlwhJK-DYCJc2VjcDI1NmsxoQJrIlXIQDvQ6t9yDySqJYDXgZgLXzTvq8W7OI51jfmxJohzeW5jbmV0cwCDdGNwgiMog3VkcIIjKA
+# Teku
+- enr:-LK4QMlzEff6d-M0A1pSFG5lJ2c56i_I-ZftdojZbW3ehkGNM4pkQuHQqzVvF1BG9aDjIakjnmO23mCBFFZ2w5zOsugEh2F0dG5ldHOIAAAAAAYAAACEZXRoMpCpfoNmMAFwAAABAAAAAAAAgmlkgnY0gmlwhKyuI_mJc2VjcDI1NmsxoQIH1kQRCZW-4AIVyAeXj5o49m_IqNFKRHp6tSpfXMUrSYN0Y3CCIyiDdWRwgiMo
+# Sigma Prime
+- enr:-Le4QI88slOwzz66Ksq8Vnz324DPb1BzSiY-WYPvnoJIl-lceW9bmSJnwDzgNbCjp5wsBigg76x4tValvGgQPxxSjrMBhGV0aDKQqX6DZjABcAAAAQAAAAAAAIJpZIJ2NIJpcIQ5gR6Wg2lwNpAgAUHQBwEQAAAAAAAAADR-iXNlY3AyNTZrMaEDPMSNdcL92uNIyCsS177Z6KTXlbZakQqxv3aQcWawNXeDdWRwgiMohHVkcDaCI4I

common/eth2_network_config/built_in_network_configs/holesky/config.yaml

@@ -0,0 +1,117 @@
+# Extends the mainnet preset
+PRESET_BASE: 'mainnet'
+CONFIG_NAME: holesky
+
+# Genesis
+# ---------------------------------------------------------------
+# `2**14` (= 16,384)
+MIN_GENESIS_ACTIVE_VALIDATOR_COUNT: 16384
+# Sep-15-2023 13:55:00 +UTC
+MIN_GENESIS_TIME: 1694786100
+GENESIS_FORK_VERSION: 0x00017000
+# Genesis delay 5 mins
+GENESIS_DELAY: 300
+
+
+# Forking
+# ---------------------------------------------------------------
+# Some forks are disabled for now:
+#  - These may be re-assigned to another fork-version later
+#  - Temporarily set to max uint64 value: 2**64 - 1
+
+# Altair
+ALTAIR_FORK_VERSION: 0x10017000
+ALTAIR_FORK_EPOCH: 0
+# Merge
+BELLATRIX_FORK_VERSION: 0x20017000
+BELLATRIX_FORK_EPOCH: 0
+TERMINAL_TOTAL_DIFFICULTY: 0
+TERMINAL_BLOCK_HASH: 0x0000000000000000000000000000000000000000000000000000000000000000
+TERMINAL_BLOCK_HASH_ACTIVATION_EPOCH: 18446744073709551615
+
+# Capella
+CAPELLA_FORK_VERSION: 0x30017000
+CAPELLA_FORK_EPOCH: 256
+
+# DENEB
+DENEB_FORK_VERSION: 0x40017000
+DENEB_FORK_EPOCH: 18446744073709551615
+
+# Time parameters
+# ---------------------------------------------------------------
+# 12 seconds
+SECONDS_PER_SLOT: 12
+# 14 (estimate from Eth1 mainnet)
+SECONDS_PER_ETH1_BLOCK: 14
+# 2**8 (= 256) epochs ~27 hours
+MIN_VALIDATOR_WITHDRAWABILITY_DELAY: 256
+# 2**8 (= 256) epochs ~27 hours
+SHARD_COMMITTEE_PERIOD: 256
+# 2**11 (= 2,048) Eth1 blocks ~8 hours
+ETH1_FOLLOW_DISTANCE: 2048
+
+
+# Validator cycle
+# ---------------------------------------------------------------
+# 2**2 (= 4)
+INACTIVITY_SCORE_BIAS: 4
+# 2**4 (= 16)
+INACTIVITY_SCORE_RECOVERY_RATE: 16
+# 28,000,000,000 Gwei to ensure quicker ejection
+EJECTION_BALANCE: 28000000000
+# 2**2 (= 4)
+MIN_PER_EPOCH_CHURN_LIMIT: 4
+# 2**16 (= 65,536)
+CHURN_LIMIT_QUOTIENT: 65536
+
+# Fork choice
+# ---------------------------------------------------------------
+# 40%
+PROPOSER_SCORE_BOOST: 40
+
+# Deposit contract
+# ---------------------------------------------------------------
+DEPOSIT_CHAIN_ID: 17000
+DEPOSIT_NETWORK_ID: 17000
+DEPOSIT_CONTRACT_ADDRESS: 0x4242424242424242424242424242424242424242
+
+# Networking
+# ---------------------------------------------------------------
+# `10 * 2**20` (= 10485760, 10 MiB)
+GOSSIP_MAX_SIZE: 10485760
+# `2**10` (= 1024)
+MAX_REQUEST_BLOCKS: 1024
+# `2**8` (= 256)
+EPOCHS_PER_SUBNET_SUBSCRIPTION: 256
+# `MIN_VALIDATOR_WITHDRAWABILITY_DELAY + CHURN_LIMIT_QUOTIENT // 2` (= 33024, ~5 months)
+MIN_EPOCHS_FOR_BLOCK_REQUESTS: 33024
+# `10 * 2**20` (=10485760, 10 MiB)
+MAX_CHUNK_SIZE: 10485760
+# 5s
+TTFB_TIMEOUT: 5
+# 10s
+RESP_TIMEOUT: 10
+ATTESTATION_PROPAGATION_SLOT_RANGE: 32
+# 500ms
+MAXIMUM_GOSSIP_CLOCK_DISPARITY: 500
+MESSAGE_DOMAIN_INVALID_SNAPPY: 0x00000000
+MESSAGE_DOMAIN_VALID_SNAPPY: 0x01000000
+# 2 subnets per node
+SUBNETS_PER_NODE: 2
+# 2**8 (= 64)
+ATTESTATION_SUBNET_COUNT: 64
+ATTESTATION_SUBNET_EXTRA_BITS: 0
+# ceillog2(ATTESTATION_SUBNET_COUNT) + ATTESTATION_SUBNET_EXTRA_BITS
+ATTESTATION_SUBNET_PREFIX_BITS: 6
+
+# Deneb
+# `2**7` (=128)
+MAX_REQUEST_BLOCKS_DENEB: 128
+# MAX_REQUEST_BLOCKS_DENEB * MAX_BLOBS_PER_BLOCK
+MAX_REQUEST_BLOB_SIDECARS: 768
+# `2**12` (= 4096 epochs, ~18 days)
+MIN_EPOCHS_FOR_BLOB_SIDECARS_REQUESTS: 4096
+# `6`
+BLOB_SIDECAR_SUBNET_COUNT: 6
+# `uint64(6)`
+MAX_BLOBS_PER_BLOCK: 6

common/eth2_network_config/built_in_network_configs/holesky/deploy_block.txt

@@ -0,0 +1 @@
+0

common/eth2_network_config/src/lib.rs

@@ -13,10 +13,20 @@
 
 use discv5::enr::{CombinedKey, Enr};
 use eth2_config::{instantiate_hardcoded_nets, HardcodedNet};
+use pretty_reqwest_error::PrettyReqwestError;
+use reqwest::blocking::Client;
+use sensitive_url::SensitiveUrl;
+use sha2::{Digest, Sha256};
+use slog::{info, warn, Logger};
 use std::fs::{create_dir_all, File};
 use std::io::{Read, Write};
 use std::path::PathBuf;
-use types::{BeaconState, ChainSpec, Config, EthSpec, EthSpecId};
+use std::str::FromStr;
+use std::time::Duration;
+use types::{BeaconState, ChainSpec, Config, EthSpec, EthSpecId, Hash256};
+use url::Url;
+
+pub use eth2_config::GenesisStateSource;
 
 pub const DEPLOY_BLOCK_FILE: &str = "deploy_block.txt";
 pub const BOOT_ENR_FILE: &str = "boot_enr.yaml";
@@ -32,6 +42,35 @@ instantiate_hardcoded_nets!(eth2_config);
 
 pub const DEFAULT_HARDCODED_NETWORK: &str = "mainnet";
 
+/// A simple slice-or-vec enum to avoid cloning the beacon state bytes in the
+/// binary whilst also supporting loading them from a file at runtime.
+#[derive(Clone, PartialEq, Debug)]
+pub enum GenesisStateBytes {
+    Slice(&'static [u8]),
+    Vec(Vec<u8>),
+}
+
+impl AsRef<[u8]> for GenesisStateBytes {
+    fn as_ref(&self) -> &[u8] {
+        match self {
+            GenesisStateBytes::Slice(slice) => slice,
+            GenesisStateBytes::Vec(vec) => vec.as_ref(),
+        }
+    }
+}
+
+impl From<&'static [u8]> for GenesisStateBytes {
+    fn from(slice: &'static [u8]) -> Self {
+        GenesisStateBytes::Slice(slice)
+    }
+}
+
+impl From<Vec<u8>> for GenesisStateBytes {
+    fn from(vec: Vec<u8>) -> Self {
+        GenesisStateBytes::Vec(vec)
+    }
+}
+
 /// Specifies an Eth2 network.
 ///
 /// See the crate-level documentation for more details.
@@ -41,7 +80,8 @@ pub struct Eth2NetworkConfig {
     /// value to be the block number where the first deposit occurs.
     pub deposit_contract_deploy_block: u64,
     pub boot_enr: Option<Vec<Enr<CombinedKey>>>,
-    pub genesis_state_bytes: Option<Vec<u8>>,
+    pub genesis_state_source: GenesisStateSource,
+    pub genesis_state_bytes: Option<GenesisStateBytes>,
     pub config: Config,
 }
 
@@ -65,8 +105,10 @@ impl Eth2NetworkConfig {
                 serde_yaml::from_reader(net.boot_enr)
                     .map_err(|e| format!("Unable to parse boot enr: {:?}", e))?,
             ),
-            genesis_state_bytes: Some(net.genesis_state_bytes.to_vec())
+            genesis_state_source: net.genesis_state_source,
-                .filter(|bytes| !bytes.is_empty()),
+            genesis_state_bytes: Some(net.genesis_state_bytes)
+                .filter(|bytes| !bytes.is_empty())
+                .map(Into::into),
             config: serde_yaml::from_reader(net.config)
                 .map_err(|e| format!("Unable to parse yaml config: {:?}", e))?,
         })
@@ -81,8 +123,37 @@ impl Eth2NetworkConfig {
     }
 
     /// Returns `true` if this configuration contains a `BeaconState`.
-    pub fn beacon_state_is_known(&self) -> bool {
+    pub fn genesis_state_is_known(&self) -> bool {
-        self.genesis_state_bytes.is_some()
+        self.genesis_state_source != GenesisStateSource::Unknown
+    }
+
+    /// The `genesis_validators_root` of the genesis state. May download the
+    /// genesis state if the value is not already available.
+    pub fn genesis_validators_root<E: EthSpec>(
+        &self,
+        genesis_state_url: Option<&str>,
+        timeout: Duration,
+        log: &Logger,
+    ) -> Result<Option<Hash256>, String> {
+        if let GenesisStateSource::Url {
+            genesis_validators_root,
+            ..
+        } = self.genesis_state_source
+        {
+            Hash256::from_str(genesis_validators_root)
+                .map(Option::Some)
+                .map_err(|e| {
+                    format!(
+                        "Unable to parse genesis state genesis_validators_root: {:?}",
+                        e
+                    )
+                })
+        } else {
+            self.genesis_state::<E>(genesis_state_url, timeout, log)?
+                .map(|state| state.genesis_validators_root())
+                .map(Result::Ok)
+                .transpose()
+        }
     }
 
     /// Construct a consolidated `ChainSpec` from the YAML config.
@@ -96,15 +167,65 @@ impl Eth2NetworkConfig {
     }
 
     /// Attempts to deserialize `self.beacon_state`, returning an error if it's missing or invalid.
-    pub fn beacon_state<E: EthSpec>(&self) -> Result<BeaconState<E>, String> {
+    ///
+    /// If the genesis state is configured to be downloaded from a URL, then the
+    /// `genesis_state_url` will override the built-in list of download URLs.
+    pub fn genesis_state<E: EthSpec>(
+        &self,
+        genesis_state_url: Option<&str>,
+        timeout: Duration,
+        log: &Logger,
+    ) -> Result<Option<BeaconState<E>>, String> {
         let spec = self.chain_spec::<E>()?;
-        let genesis_state_bytes = self
+        match &self.genesis_state_source {
+            GenesisStateSource::Unknown => Ok(None),
+            GenesisStateSource::IncludedBytes => {
+                let state = self
                     .genesis_state_bytes
                     .as_ref()
-            .ok_or("Genesis state is unknown")?;
+                    .map(|bytes| {
+                        BeaconState::from_ssz_bytes(bytes.as_ref(), &spec).map_err(|e| {
+                            format!("Built-in genesis state SSZ bytes are invalid: {:?}", e)
+                        })
+                    })
+                    .ok_or("Genesis state bytes missing from Eth2NetworkConfig")??;
+                Ok(Some(state))
+            }
+            GenesisStateSource::Url {
+                urls: built_in_urls,
+                checksum,
+                genesis_validators_root,
+            } => {
+                let checksum = Hash256::from_str(checksum).map_err(|e| {
+                    format!("Unable to parse genesis state bytes checksum: {:?}", e)
+                })?;
+                let bytes = if let Some(specified_url) = genesis_state_url {
+                    download_genesis_state(&[specified_url], timeout, checksum, log)
+                } else {
+                    download_genesis_state(built_in_urls, timeout, checksum, log)
+                }?;
+                let state = BeaconState::from_ssz_bytes(bytes.as_ref(), &spec).map_err(|e| {
+                    format!("Downloaded genesis state SSZ bytes are invalid: {:?}", e)
+                })?;
 
-        BeaconState::from_ssz_bytes(genesis_state_bytes, &spec)
+                let genesis_validators_root =
-            .map_err(|e| format!("Genesis state SSZ bytes are invalid: {:?}", e))
+                    Hash256::from_str(genesis_validators_root).map_err(|e| {
+                        format!(
+                            "Unable to parse genesis state genesis_validators_root: {:?}",
+                            e
+                        )
+                    })?;
+                if state.genesis_validators_root() != genesis_validators_root {
+                    return Err(format!(
+                        "Downloaded genesis validators root {:?} does not match expected {:?}",
+                        state.genesis_validators_root(),
+                        genesis_validators_root
+                    ));
+                }
+
+                Ok(Some(state))
+            }
+        }
     }
 
     /// Write the files to the directory.
@@ -162,7 +283,7 @@ impl Eth2NetworkConfig {
             File::create(&file)
                 .map_err(|e| format!("Unable to create {:?}: {:?}", file, e))
                 .and_then(|mut file| {
-                    file.write_all(genesis_state_bytes)
+                    file.write_all(genesis_state_bytes.as_ref())
                         .map_err(|e| format!("Unable to write {:?}: {:?}", file, e))
                 })?;
         }
@@ -198,7 +319,7 @@ impl Eth2NetworkConfig {
 
         // The genesis state is a special case because it uses SSZ, not YAML.
         let genesis_file_path = base_dir.join(GENESIS_STATE_FILE);
-        let genesis_state_bytes = if genesis_file_path.exists() {
+        let (genesis_state_bytes, genesis_state_source) = if genesis_file_path.exists() {
             let mut bytes = vec![];
             File::open(&genesis_file_path)
                 .map_err(|e| format!("Unable to open {:?}: {:?}", genesis_file_path, e))
@@ -207,20 +328,105 @@ impl Eth2NetworkConfig {
                         .map_err(|e| format!("Unable to read {:?}: {:?}", file, e))
                 })?;
 
-            Some(bytes).filter(|bytes| !bytes.is_empty())
+            let state = Some(bytes).filter(|bytes| !bytes.is_empty());
+            let genesis_state_source = if state.is_some() {
+                GenesisStateSource::IncludedBytes
             } else {
-            None
+                GenesisStateSource::Unknown
+            };
+            (state, genesis_state_source)
+        } else {
+            (None, GenesisStateSource::Unknown)
         };
 
         Ok(Self {
             deposit_contract_deploy_block,
             boot_enr,
-            genesis_state_bytes,
+            genesis_state_source,
+            genesis_state_bytes: genesis_state_bytes.map(Into::into),
             config,
         })
     }
 }
 
+/// Try to download a genesis state from each of the `urls` in the order they
+/// are defined. Return `Ok` if any url returns a response that matches the
+/// given `checksum`.
+fn download_genesis_state(
+    urls: &[&str],
+    timeout: Duration,
+    checksum: Hash256,
+    log: &Logger,
+) -> Result<Vec<u8>, String> {
+    if urls.is_empty() {
+        return Err(
+            "The genesis state is not present in the binary and there are no known download URLs. \
+            Please use --checkpoint-sync-url or --genesis-state-url."
+                .to_string(),
+        );
+    }
+
+    let mut errors = vec![];
+    for url in urls {
+        // URLs are always expected to be the base URL of a server that supports
+        // the beacon-API.
+        let url = parse_state_download_url(url)?;
+        let redacted_url = SensitiveUrl::new(url.clone())
+            .map(|url| url.to_string())
+            .unwrap_or_else(|_| "<REDACTED>".to_string());
+
+        info!(
+            log,
+            "Downloading genesis state";
+            "server" => &redacted_url,
+            "timeout" => ?timeout,
+            "info" => "this may take some time on testnets with large validator counts"
+        );
+
+        let client = Client::new();
+        let response = client
+            .get(url)
+            .header("Accept", "application/octet-stream")
+            .timeout(timeout)
+            .send()
+            .and_then(|r| r.error_for_status().and_then(|r| r.bytes()));
+
+        match response {
+            Ok(bytes) => {
+                // Check the server response against our local checksum.
+                if Sha256::digest(bytes.as_ref())[..] == checksum[..] {
+                    return Ok(bytes.into());
+                } else {
+                    warn!(
+                        log,
+                        "Genesis state download failed";
+                        "server" => &redacted_url,
+                        "timeout" => ?timeout,
+                    );
+                    errors.push(format!(
+                        "Response from {} did not match local checksum",
+                        redacted_url
+                    ))
+                }
+            }
+            Err(e) => errors.push(PrettyReqwestError::from(e).to_string()),
+        }
+    }
+    Err(format!(
+        "Unable to download a genesis state from {} source(s): {}",
+        errors.len(),
+        errors.join(",")
+    ))
+}
+
+/// Parses the `url` and joins the necessary state download path.
+fn parse_state_download_url(url: &str) -> Result<Url, String> {
+    Url::parse(url)
+        .map_err(|e| format!("Invalid genesis state URL: {:?}", e))?
+        .join("eth/v2/debug/beacon/states/genesis")
+        .map_err(|e| format!("Failed to append genesis state path to URL: {:?}", e))
+}
+
 #[cfg(test)]
 mod tests {
     use super::*;
@@ -260,7 +466,9 @@ mod tests {
     #[test]
     fn mainnet_genesis_state() {
         let config = Eth2NetworkConfig::from_hardcoded_net(&MAINNET).unwrap();
-        config.beacon_state::<E>().expect("beacon state can decode");
+        config
+            .genesis_state::<E>(None, Duration::from_secs(1), &logging::test_logger())
+            .expect("beacon state can decode");
     }
 
     #[test]
@@ -285,10 +493,25 @@ mod tests {
 
             assert_eq!(
                 config.genesis_state_bytes.is_some(),
-                net.genesis_is_known,
+                net.genesis_state_source == GenesisStateSource::IncludedBytes,
                 "{:?}",
                 net.name
             );
+
+            if let GenesisStateSource::Url {
+                urls,
+                checksum,
+                genesis_validators_root,
+            } = net.genesis_state_source
+            {
+                Hash256::from_str(checksum).expect("the checksum must be a valid 32-byte value");
+                Hash256::from_str(genesis_validators_root)
+                    .expect("the GVR must be a valid 32-byte value");
+                for url in urls {
+                    parse_state_download_url(url).expect("url must be valid");
+                }
+            }
+
             assert_eq!(config.config.config_name, Some(net.config_dir.to_string()));
         }
     }
@@ -324,10 +547,20 @@ mod tests {
         let base_dir = temp_dir.path().join("my_testnet");
         let deposit_contract_deploy_block = 42;
 
+        let genesis_state_source = if genesis_state.is_some() {
+            GenesisStateSource::IncludedBytes
+        } else {
+            GenesisStateSource::Unknown
+        };
+
         let testnet: Eth2NetworkConfig = Eth2NetworkConfig {
             deposit_contract_deploy_block,
             boot_enr,
-            genesis_state_bytes: genesis_state.as_ref().map(Encode::as_ssz_bytes),
+            genesis_state_source,
+            genesis_state_bytes: genesis_state
+                .as_ref()
+                .map(Encode::as_ssz_bytes)
+                .map(Into::into),
             config,
         };
 

common/pretty_reqwest_error/src/lib.rs

@@ -55,6 +55,12 @@ impl fmt::Debug for PrettyReqwestError {
     }
 }
 
+impl fmt::Display for PrettyReqwestError {
+    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
+        write!(f, "{:?}", self)
+    }
+}
+
 impl From<reqwest::Error> for PrettyReqwestError {
     fn from(inner: reqwest::Error) -> Self {
         Self(inner)

lcli/src/eth1_genesis.rs

@@ -49,7 +49,7 @@ pub fn run<T: EthSpec>(
             .wait_for_genesis_state::<T>(ETH1_GENESIS_UPDATE_INTERVAL, spec)
             .await
             .map(move |genesis_state| {
-                eth2_network_config.genesis_state_bytes = Some(genesis_state.as_ssz_bytes());
+                eth2_network_config.genesis_state_bytes = Some(genesis_state.as_ssz_bytes().into());
                 eth2_network_config.force_write_to_file(testnet_dir)
             })
             .map_err(|e| format!("Failed to find genesis: {}", e))?;

lcli/src/interop_genesis.rs

@@ -42,7 +42,7 @@ pub fn run<T: EthSpec>(testnet_dir: PathBuf, matches: &ArgMatches) -> Result<(),
         &spec,
     )?;
 
-    eth2_network_config.genesis_state_bytes = Some(genesis_state.as_ssz_bytes());
+    eth2_network_config.genesis_state_bytes = Some(genesis_state.as_ssz_bytes().into());
     eth2_network_config.force_write_to_file(testnet_dir)?;
 
     Ok(())

lcli/src/new_testnet.rs

@@ -1,7 +1,7 @@
 use account_utils::eth2_keystore::keypair_from_secret;
 use clap::ArgMatches;
 use clap_utils::{parse_optional, parse_required, parse_ssz_optional};
-use eth2_network_config::Eth2NetworkConfig;
+use eth2_network_config::{Eth2NetworkConfig, GenesisStateSource};
 use eth2_wallet::bip39::Seed;
 use eth2_wallet::bip39::{Language, Mnemonic};
 use eth2_wallet::{recover_validator_secret_from_mnemonic, KeyType};
@@ -190,7 +190,8 @@ pub fn run<T: EthSpec>(testnet_dir_path: PathBuf, matches: &ArgMatches) -> Resul
     let testnet = Eth2NetworkConfig {
         deposit_contract_deploy_block,
         boot_enr: Some(vec![]),
-        genesis_state_bytes,
+        genesis_state_bytes: genesis_state_bytes.map(Into::into),
+        genesis_state_source: GenesisStateSource::IncludedBytes,
         config: Config::from_chain_spec::<T>(&spec),
     };
 

lighthouse/src/main.rs

@@ -324,6 +324,30 @@ fn main() {
                 .takes_value(true)
                 .global(true)
         )
+        .arg(
+            Arg::with_name("genesis-state-url")
+                .long("genesis-state-url")
+                .value_name("URL")
+                .help(
+                    "A URL of a beacon-API compatible server from which to download the genesis state. \
+                    Checkpoint sync server URLs can generally be used with this flag. \
+                    If not supplied, a default URL or the --checkpoint-sync-url may be used. \
+                    If the genesis state is already included in this binary then this value will be ignored.",
+                )
+                .takes_value(true)
+                .global(true),
+        )
+        .arg(
+            Arg::with_name("genesis-state-url-timeout")
+                .long("genesis-state-url-timeout")
+                .value_name("SECONDS")
+                .help(
+                    "The timeout in seconds for the request to --genesis-state-url.",
+                )
+                .takes_value(true)
+                .default_value("180")
+                .global(true),
+        )
         .subcommand(beacon_node::cli_app())
         .subcommand(boot_node::cli_app())
         .subcommand(validator_client::cli_app())

lighthouse/tests/beacon_node.rs

@@ -2406,3 +2406,28 @@ fn http_duplicate_block_status_override() {
             assert_eq!(config.http_api.duplicate_block_status_code.as_u16(), 301)
         });
 }
+
+#[test]
+fn genesis_state_url_default() {
+    CommandLineTest::new()
+        .run_with_zero_port()
+        .with_config(|config| {
+            assert_eq!(config.genesis_state_url, None);
+            assert_eq!(config.genesis_state_url_timeout, Duration::from_secs(180));
+        });
+}
+
+#[test]
+fn genesis_state_url_value() {
+    CommandLineTest::new()
+        .flag("genesis-state-url", Some("http://genesis.com"))
+        .flag("genesis-state-url-timeout", Some("42"))
+        .run_with_zero_port()
+        .with_config(|config| {
+            assert_eq!(
+                config.genesis_state_url.as_deref(),
+                Some("http://genesis.com")
+            );
+            assert_eq!(config.genesis_state_url_timeout, Duration::from_secs(42));
+        });
+}