Separate execution payloads in the DB (#3157)

## Proposed Changes

Reduce post-merge disk usage by not storing finalized execution payloads in Lighthouse's database.

⚠️ **This is achieved in a backwards-incompatible way for networks that have already merged** ⚠️. Kiln users and shadow fork enjoyers will be unable to downgrade after running the code from this PR. The upgrade migration may take several minutes to run, and can't be aborted after it begins.

The main changes are:

- New column in the database called `ExecPayload`, keyed by beacon block root.
- The `BeaconBlock` column now stores blinded blocks only.
- Lots of places that previously used full blocks now use blinded blocks, e.g. analytics APIs, block replay in the DB, etc.
- On finalization:
    - `prune_abanonded_forks` deletes non-canonical payloads whilst deleting non-canonical blocks.
    - `migrate_db` deletes finalized canonical payloads whilst deleting finalized states.
- Conversions between blinded and full blocks are implemented in a compositional way, duplicating some work from Sean's PR #3134.
- The execution layer has a new `get_payload_by_block_hash` method that reconstructs a payload using the EE's `eth_getBlockByHash` call.
   - I've tested manually that it works on Kiln, using Geth and Nethermind.
   - This isn't necessarily the most efficient method, and new engine APIs are being discussed to improve this: https://github.com/ethereum/execution-apis/pull/146.
   - We're depending on the `ethers` master branch, due to lots of recent changes. We're also using a workaround for https://github.com/gakonst/ethers-rs/issues/1134.
- Payload reconstruction is used in the HTTP API via `BeaconChain::get_block`, which is now `async`. Due to the `async` fn, the `blocking_json` wrapper has been removed.
- Payload reconstruction is used in network RPC to serve blocks-by-{root,range} responses. Here the `async` adjustment is messier, although I think I've managed to come up with a reasonable compromise: the handlers take the `SendOnDrop` by value so that they can drop it on _task completion_ (after the `fn` returns). Still, this is introducing disk reads onto core executor threads, which may have a negative performance impact (thoughts appreciated).

## Additional Info

- [x] For performance it would be great to remove the cloning of full blocks when converting them to blinded blocks to write to disk. I'm going to experiment with a `put_block` API that takes the block by value, breaks it into a blinded block and a payload, stores the blinded block, and then re-assembles the full block for the caller.
- [x] We should measure the latency of blocks-by-root and blocks-by-range responses.
- [x] We should add integration tests that stress the payload reconstruction (basic tests done, issue for more extensive tests: https://github.com/sigp/lighthouse/issues/3159)
- [x] We should (manually) test the schema v9 migration from several prior versions, particularly as blocks have changed on disk and some migrations rely on being able to load blocks.

Co-authored-by: Paul Hauner <paul@paulhauner.com>

beacon_node/beacon_chain/Cargo.toml

@@ -59,7 +59,8 @@ strum = { version = "0.24.0", features = ["derive"] }
 logging = { path = "../../common/logging" }
 execution_layer = { path = "../execution_layer" }
 sensitive_url = { path = "../../common/sensitive_url" }
-superstruct = "0.4.1"
+superstruct = "0.5.0"
+hex = "0.4.2"
 
 [[test]]
 name = "beacon_chain_tests"

beacon_node/beacon_chain/src/beacon_chain.rs

@@ -83,8 +83,11 @@ use std::marker::PhantomData;
 use std::sync::Arc;
 use std::time::{Duration, Instant};
 use store::iter::{BlockRootsIterator, ParentRootBlockIterator, StateRootsIterator};
-use store::{Error as DBError, HotColdDB, KeyValueStore, KeyValueStoreOp, StoreItem, StoreOp};
+use store::{
+    DatabaseBlock, Error as DBError, HotColdDB, KeyValueStore, KeyValueStoreOp, StoreItem, StoreOp,
+};
 use task_executor::ShutdownReason;
+use tree_hash::TreeHash;
 use types::beacon_state::CloneConfig;
 use types::*;
 
@@ -587,7 +590,7 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
         block_root: Hash256,
     ) -> Result<impl Iterator<Item = Result<(Hash256, Slot), Error>> + '_, Error> {
         let block = self
-            .get_block(&block_root)?
+            .get_blinded_block(&block_root)?
             .ok_or(Error::MissingBeaconBlock(block_root))?;
         let state = self
             .get_state(&block.state_root(), Some(block.slot()))?
@@ -752,11 +755,11 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
         &self,
         request_slot: Slot,
         skips: WhenSlotSkipped,
-    ) -> Result<Option<SignedBeaconBlock<T::EthSpec>>, Error> {
+    ) -> Result<Option<SignedBlindedBeaconBlock<T::EthSpec>>, Error> {
         let root = self.block_root_at_slot(request_slot, skips)?;
 
         if let Some(block_root) = root {
-            Ok(self.store.get_block(&block_root)?)
+            Ok(self.store.get_blinded_block(&block_root)?)
         } else {
             Ok(None)
         }
@@ -961,16 +964,14 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
     /// ## Errors
     ///
     /// May return a database error.
-    pub fn get_block_checking_early_attester_cache(
+    pub async fn get_block_checking_early_attester_cache(
         &self,
         block_root: &Hash256,
     ) -> Result<Option<SignedBeaconBlock<T::EthSpec>>, Error> {
-        let block_opt = self
-            .store
-            .get_block(block_root)?
-            .or_else(|| self.early_attester_cache.get_block(*block_root));
-
-        Ok(block_opt)
+        if let Some(block) = self.early_attester_cache.get_block(*block_root) {
+            return Ok(Some(block));
+        }
+        self.get_block(block_root).await
     }
 
     /// Returns the block at the given root, if any.
@@ -978,11 +979,69 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
     /// ## Errors
     ///
     /// May return a database error.
-    pub fn get_block(
+    pub async fn get_block(
         &self,
         block_root: &Hash256,
     ) -> Result<Option<SignedBeaconBlock<T::EthSpec>>, Error> {
-        Ok(self.store.get_block(block_root)?)
+        // Load block from database, returning immediately if we have the full block w payload
+        // stored.
+        let blinded_block = match self.store.try_get_full_block(block_root)? {
+            Some(DatabaseBlock::Full(block)) => return Ok(Some(block)),
+            Some(DatabaseBlock::Blinded(block)) => block,
+            None => return Ok(None),
+        };
+
+        // If we only have a blinded block, load the execution payload from the EL.
+        let block_message = blinded_block.message();
+        let execution_payload_header = &block_message
+            .execution_payload()
+            .map_err(|_| Error::BlockVariantLacksExecutionPayload(*block_root))?
+            .execution_payload_header;
+
+        let exec_block_hash = execution_payload_header.block_hash;
+
+        let execution_payload = self
+            .execution_layer
+            .as_ref()
+            .ok_or(Error::ExecutionLayerMissing)?
+            .get_payload_by_block_hash(exec_block_hash)
+            .await
+            .map_err(|e| Error::ExecutionLayerErrorPayloadReconstruction(exec_block_hash, e))?
+            .ok_or(Error::BlockHashMissingFromExecutionLayer(exec_block_hash))?;
+
+        // Verify payload integrity.
+        let header_from_payload = ExecutionPayloadHeader::from(&execution_payload);
+        if header_from_payload != *execution_payload_header {
+            for txn in &execution_payload.transactions {
+                debug!(
+                    self.log,
+                    "Reconstructed txn";
+                    "bytes" => format!("0x{}", hex::encode(&**txn)),
+                );
+            }
+
+            return Err(Error::InconsistentPayloadReconstructed {
+                slot: blinded_block.slot(),
+                exec_block_hash,
+                canonical_payload_root: execution_payload_header.tree_hash_root(),
+                reconstructed_payload_root: header_from_payload.tree_hash_root(),
+                canonical_transactions_root: execution_payload_header.transactions_root,
+                reconstructed_transactions_root: header_from_payload.transactions_root,
+            });
+        }
+
+        // Add the payload to the block to form a full block.
+        blinded_block
+            .try_into_full_block(Some(execution_payload))
+            .ok_or(Error::AddPayloadLogicError)
+            .map(Some)
+    }
+
+    pub fn get_blinded_block(
+        &self,
+        block_root: &Hash256,
+    ) -> Result<Option<SignedBlindedBeaconBlock<T::EthSpec>>, Error> {
+        Ok(self.store.get_blinded_block(block_root)?)
     }
 
     /// Returns the state at the given root, if any.
@@ -3373,7 +3432,8 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
             .map::<Result<_, Error>, _>(Ok)
             .unwrap_or_else(|| {
                 let beacon_block = self
-                    .get_block(&beacon_block_root)?
+                    .store
+                    .get_full_block(&beacon_block_root)?
                     .ok_or(Error::MissingBeaconBlock(beacon_block_root))?;
 
                 let beacon_state_root = beacon_block.state_root();
@@ -4525,11 +4585,14 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
     ///
     /// This could be a very expensive operation and should only be done in testing/analysis
     /// activities.
-    pub fn chain_dump(&self) -> Result<Vec<BeaconSnapshot<T::EthSpec>>, Error> {
+    #[allow(clippy::type_complexity)]
+    pub fn chain_dump(
+        &self,
+    ) -> Result<Vec<BeaconSnapshot<T::EthSpec, BlindedPayload<T::EthSpec>>>, Error> {
         let mut dump = vec![];
 
         let mut last_slot = BeaconSnapshot {
-            beacon_block: self.head()?.beacon_block,
+            beacon_block: self.head()?.beacon_block.into(),
             beacon_block_root: self.head()?.beacon_block_root,
             beacon_state: self.head()?.beacon_state,
         };
@@ -4543,9 +4606,12 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
                 break; // Genesis has been reached.
             }
 
-            let beacon_block = self.store.get_block(&beacon_block_root)?.ok_or_else(|| {
-                Error::DBInconsistent(format!("Missing block {}", beacon_block_root))
-            })?;
+            let beacon_block = self
+                .store
+                .get_blinded_block(&beacon_block_root)?
+                .ok_or_else(|| {
+                    Error::DBInconsistent(format!("Missing block {}", beacon_block_root))
+                })?;
             let beacon_state_root = beacon_block.state_root();
             let beacon_state = self
                 .store
@@ -4630,7 +4696,7 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
                 visited.insert(block_hash);
 
                 if signed_beacon_block.slot() % T::EthSpec::slots_per_epoch() == 0 {
-                    let block = self.get_block(&block_hash).unwrap().unwrap();
+                    let block = self.get_blinded_block(&block_hash).unwrap().unwrap();
                     let state = self
                         .get_state(&block.state_root(), Some(block.slot()))
                         .unwrap()

beacon_node/beacon_chain/src/beacon_fork_choice_store.rs

@@ -13,7 +13,8 @@ use std::sync::Arc;
 use store::{Error as StoreError, HotColdDB, ItemStore};
 use superstruct::superstruct;
 use types::{
-    BeaconBlock, BeaconState, BeaconStateError, Checkpoint, Epoch, EthSpec, Hash256, Slot,
+    BeaconBlock, BeaconState, BeaconStateError, Checkpoint, Epoch, EthSpec, ExecPayload, Hash256,
+    Slot,
 };
 
 #[derive(Debug)]
@@ -254,9 +255,9 @@ where
         self.time = slot
     }
 
-    fn on_verified_block(
+    fn on_verified_block<Payload: ExecPayload<E>>(
         &mut self,
-        _block: &BeaconBlock<E>,
+        _block: &BeaconBlock<E, Payload>,
         block_root: Hash256,
         state: &BeaconState<E>,
     ) -> Result<(), Self::Error> {
@@ -300,7 +301,7 @@ where
             metrics::inc_counter(&metrics::BALANCES_CACHE_MISSES);
             let justified_block = self
                 .store
-                .get_block(&self.justified_checkpoint.root)
+                .get_blinded_block(&self.justified_checkpoint.root)
                 .map_err(Error::FailedToReadBlock)?
                 .ok_or(Error::MissingBlock(self.justified_checkpoint.root))?
                 .deconstruct()

beacon_node/beacon_chain/src/beacon_snapshot.rs

@@ -1,19 +1,22 @@
 use serde_derive::Serialize;
-use types::{beacon_state::CloneConfig, BeaconState, EthSpec, Hash256, SignedBeaconBlock};
+use types::{
+    beacon_state::CloneConfig, BeaconState, EthSpec, ExecPayload, FullPayload, Hash256,
+    SignedBeaconBlock,
+};
 
 /// Represents some block and its associated state. Generally, this will be used for tracking the
 /// head, justified head and finalized head.
 #[derive(Clone, Serialize, PartialEq, Debug)]
-pub struct BeaconSnapshot<E: EthSpec> {
-    pub beacon_block: SignedBeaconBlock<E>,
+pub struct BeaconSnapshot<E: EthSpec, Payload: ExecPayload<E> = FullPayload<E>> {
+    pub beacon_block: SignedBeaconBlock<E, Payload>,
     pub beacon_block_root: Hash256,
     pub beacon_state: BeaconState<E>,
 }
 
-impl<E: EthSpec> BeaconSnapshot<E> {
+impl<E: EthSpec, Payload: ExecPayload<E>> BeaconSnapshot<E, Payload> {
     /// Create a new checkpoint.
     pub fn new(
-        beacon_block: SignedBeaconBlock<E>,
+        beacon_block: SignedBeaconBlock<E, Payload>,
         beacon_block_root: Hash256,
         beacon_state: BeaconState<E>,
     ) -> Self {
@@ -36,7 +39,7 @@ impl<E: EthSpec> BeaconSnapshot<E> {
     /// Update all fields of the checkpoint.
     pub fn update(
         &mut self,
-        beacon_block: SignedBeaconBlock<E>,
+        beacon_block: SignedBeaconBlock<E, Payload>,
         beacon_block_root: Hash256,
         beacon_state: BeaconState<E>,
     ) {

beacon_node/beacon_chain/src/block_reward.rs

@@ -2,12 +2,12 @@ use crate::{BeaconChain, BeaconChainError, BeaconChainTypes};
 use eth2::lighthouse::{AttestationRewards, BlockReward, BlockRewardMeta};
 use operation_pool::{AttMaxCover, MaxCover};
 use state_processing::per_block_processing::altair::sync_committee::compute_sync_aggregate_rewards;
-use types::{BeaconBlockRef, BeaconState, EthSpec, Hash256, RelativeEpoch};
+use types::{BeaconBlockRef, BeaconState, EthSpec, ExecPayload, Hash256, RelativeEpoch};
 
 impl<T: BeaconChainTypes> BeaconChain<T> {
-    pub fn compute_block_reward(
+    pub fn compute_block_reward<Payload: ExecPayload<T::EthSpec>>(
         &self,
-        block: BeaconBlockRef<'_, T::EthSpec>,
+        block: BeaconBlockRef<'_, T::EthSpec, Payload>,
         block_root: Hash256,
         state: &BeaconState<T::EthSpec>,
     ) -> Result<BlockReward, BeaconChainError> {

beacon_node/beacon_chain/src/block_verification.rs

@@ -78,9 +78,9 @@ use store::{Error as DBError, HotColdDB, HotStateSummary, KeyValueStore, StoreOp
 use tree_hash::TreeHash;
 use types::ExecPayload;
 use types::{
-    BeaconBlockRef, BeaconState, BeaconStateError, ChainSpec, CloneConfig, Epoch, EthSpec,
-    ExecutionBlockHash, Hash256, InconsistentFork, PublicKey, PublicKeyBytes, RelativeEpoch,
-    SignedBeaconBlock, SignedBeaconBlockHeader, Slot,
+    BeaconBlockRef, BeaconState, BeaconStateError, BlindedPayload, ChainSpec, CloneConfig, Epoch,
+    EthSpec, ExecutionBlockHash, Hash256, InconsistentFork, PublicKey, PublicKeyBytes,
+    RelativeEpoch, SignedBeaconBlock, SignedBeaconBlockHeader, Slot,
 };
 
 const POS_PANDA_BANNER: &str = r#"
@@ -567,7 +567,7 @@ pub struct FullyVerifiedBlock<'a, T: BeaconChainTypes> {
     pub block: SignedBeaconBlock<T::EthSpec>,
     pub block_root: Hash256,
     pub state: BeaconState<T::EthSpec>,
-    pub parent_block: SignedBeaconBlock<T::EthSpec>,
+    pub parent_block: SignedBeaconBlock<T::EthSpec, BlindedPayload<T::EthSpec>>,
     pub confirmation_db_batch: Vec<StoreOp<'a, T::EthSpec>>,
     pub payload_verification_status: PayloadVerificationStatus,
 }
@@ -1569,7 +1569,7 @@ fn load_parent<T: BeaconChainTypes>(
         // indicate that we don't yet know the parent.
         let root = block.parent_root();
         let parent_block = chain
-            .get_block(&block.parent_root())
+            .get_blinded_block(&block.parent_root())
             .map_err(BlockError::BeaconChainError)?
             .ok_or_else(|| {
                 // Return a `MissingBeaconBlock` error instead of a `ParentUnknown` error since

beacon_node/beacon_chain/src/builder.rs

@@ -240,7 +240,7 @@ where
             .ok_or("Fork choice not found in store")?;
 
         let genesis_block = store
-            .get_block(&chain.genesis_block_root)
+            .get_blinded_block(&chain.genesis_block_root)
             .map_err(|e| descriptive_db_error("genesis block", &e))?
             .ok_or("Genesis block not found in store")?;
         let genesis_state = store
@@ -588,7 +588,7 @@ where
         // Try to decode the head block according to the current fork, if that fails, try
         // to backtrack to before the most recent fork.
         let (head_block_root, head_block, head_reverted) =
-            match store.get_block(&initial_head_block_root) {
+            match store.get_full_block(&initial_head_block_root) {
                 Ok(Some(block)) => (initial_head_block_root, block, false),
                 Ok(None) => return Err("Head block not found in store".into()),
                 Err(StoreError::SszDecodeError(_)) => {
@@ -986,10 +986,10 @@ mod test {
         assert_eq!(
             chain
                 .store
-                .get_block(&Hash256::zero())
+                .get_blinded_block(&Hash256::zero())
                 .expect("should read db")
                 .expect("should find genesis block"),
-            block,
+            block.clone().into(),
             "should store genesis block under zero hash alias"
         );
         assert_eq!(

beacon_node/beacon_chain/src/errors.rs

@@ -139,6 +139,18 @@ pub enum BeaconChainError {
     },
     AltairForkDisabled,
     ExecutionLayerMissing,
+    BlockVariantLacksExecutionPayload(Hash256),
+    ExecutionLayerErrorPayloadReconstruction(ExecutionBlockHash, execution_layer::Error),
+    BlockHashMissingFromExecutionLayer(ExecutionBlockHash),
+    InconsistentPayloadReconstructed {
+        slot: Slot,
+        exec_block_hash: ExecutionBlockHash,
+        canonical_payload_root: Hash256,
+        reconstructed_payload_root: Hash256,
+        canonical_transactions_root: Hash256,
+        reconstructed_transactions_root: Hash256,
+    },
+    AddPayloadLogicError,
     ExecutionForkChoiceUpdateFailed(execution_layer::Error),
     PrepareProposerBlockingFailed(execution_layer::Error),
     ExecutionForkChoiceUpdateInvalid {

beacon_node/beacon_chain/src/execution_payload.rs

@@ -333,7 +333,7 @@ pub async fn prepare_execution_payload<T: BeaconChainTypes, Payload: ExecPayload
         } else {
             chain
                 .store
-                .get_block(&finalized_root)
+                .get_blinded_block(&finalized_root)
                 .map_err(BlockProductionError::FailedToReadFinalizedBlock)?
                 .ok_or(BlockProductionError::MissingFinalizedBlock(finalized_root))?
                 .message()

beacon_node/beacon_chain/src/fork_revert.rs

@@ -48,7 +48,7 @@ pub fn revert_to_fork_boundary<E: EthSpec, Hot: ItemStore<E>, Cold: ItemStore<E>
     );
     let block_iter = ParentRootBlockIterator::fork_tolerant(&store, head_block_root);
 
-    process_results(block_iter, |mut iter| {
+    let (block_root, blinded_block) = process_results(block_iter, |mut iter| {
         iter.find_map(|(block_root, block)| {
             if block.slot() < fork_epoch.start_slot(E::slots_per_epoch()) {
                 Some((block_root, block))
@@ -69,7 +69,13 @@ pub fn revert_to_fork_boundary<E: EthSpec, Hot: ItemStore<E>, Cold: ItemStore<E>
             e, CORRUPT_DB_MESSAGE
         )
     })?
-    .ok_or_else(|| format!("No pre-fork blocks found. {}", CORRUPT_DB_MESSAGE))
+    .ok_or_else(|| format!("No pre-fork blocks found. {}", CORRUPT_DB_MESSAGE))?;
+
+    let block = store
+        .make_full_block(&block_root, blinded_block)
+        .map_err(|e| format!("Unable to add payload to new head block: {:?}", e))?;
+
+    Ok((block_root, block))
 }
 
 /// Reset fork choice to the finalized checkpoint of the supplied head state.
@@ -97,7 +103,7 @@ pub fn reset_fork_choice_to_finalization<E: EthSpec, Hot: ItemStore<E>, Cold: It
     let finalized_checkpoint = head_state.finalized_checkpoint();
     let finalized_block_root = finalized_checkpoint.root;
     let finalized_block = store
-        .get_block(&finalized_block_root)
+        .get_full_block(&finalized_block_root)
         .map_err(|e| format!("Error loading finalized block: {:?}", e))?
         .ok_or_else(|| {
             format!(

beacon_node/beacon_chain/src/historical_blocks.rs

@@ -9,7 +9,7 @@ use std::borrow::Cow;
 use std::iter;
 use std::time::Duration;
 use store::{chunked_vector::BlockRoots, AnchorInfo, ChunkWriter, KeyValueStore};
-use types::{Hash256, SignedBeaconBlock, Slot};
+use types::{Hash256, SignedBlindedBeaconBlock, Slot};
 
 /// Use a longer timeout on the pubkey cache.
 ///
@@ -58,7 +58,7 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
     /// Return the number of blocks successfully imported.
     pub fn import_historical_block_batch(
         &self,
-        blocks: &[SignedBeaconBlock<T::EthSpec>],
+        blocks: Vec<SignedBlindedBeaconBlock<T::EthSpec>>,
     ) -> Result<usize, Error> {
         let anchor_info = self
             .store
@@ -106,8 +106,9 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
                 .into());
             }
 
-            // Store block in the hot database.
-            hot_batch.push(self.store.block_as_kv_store_op(&block_root, block));
+            // Store block in the hot database without payload.
+            self.store
+                .blinded_block_as_kv_store_ops(&block_root, block, &mut hot_batch);
 
             // Store block roots, including at all skip slots in the freezer DB.
             for slot in (block.slot().as_usize()..prev_block_slot.as_usize()).rev() {

beacon_node/beacon_chain/src/migrate.rs

@@ -391,7 +391,7 @@ impl<E: EthSpec, Hot: ItemStore<E>, Cold: ItemStore<E>> BackgroundMigrator<E, Ho
             // so delete it from the head tracker but leave it and its states in the database
             // This is suboptimal as it wastes disk space, but it's difficult to fix. A re-sync
             // can be used to reclaim the space.
-            let head_state_root = match store.get_block(&head_hash) {
+            let head_state_root = match store.get_blinded_block(&head_hash) {
                 Ok(Some(block)) => block.state_root(),
                 Ok(None) => {
                     return Err(BeaconStateError::MissingBeaconBlock(head_hash.into()).into())
@@ -535,7 +535,12 @@ impl<E: EthSpec, Hot: ItemStore<E>, Cold: ItemStore<E>> BackgroundMigrator<E, Ho
         let batch: Vec<StoreOp<E>> = abandoned_blocks
             .into_iter()
             .map(Into::into)
-            .map(StoreOp::DeleteBlock)
+            .flat_map(|block_root: Hash256| {
+                [
+                    StoreOp::DeleteBlock(block_root),
+                    StoreOp::DeleteExecutionPayload(block_root),
+                ]
+            })
             .chain(
                 abandoned_states
                     .into_iter()
@@ -543,7 +548,7 @@ impl<E: EthSpec, Hot: ItemStore<E>, Cold: ItemStore<E>> BackgroundMigrator<E, Ho
             )
             .collect();
 
-        let mut kv_batch = store.convert_to_kv_batch(&batch)?;
+        let mut kv_batch = store.convert_to_kv_batch(batch)?;
 
         // Persist the head in case the process is killed or crashes here. This prevents
         // the head tracker reverting after our mutation above.

beacon_node/beacon_chain/src/pre_finalization_cache.rs

@@ -71,7 +71,7 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
         }
 
         // 2. Check on disk.
-        if self.store.get_block(&block_root)?.is_some() {
+        if self.store.get_blinded_block(&block_root)?.is_some() {
             cache.block_roots.put(block_root, ());
             return Ok(true);
         }

beacon_node/beacon_chain/src/schema_change.rs

@@ -2,6 +2,7 @@
 mod migration_schema_v6;
 mod migration_schema_v7;
 mod migration_schema_v8;
+mod migration_schema_v9;
 mod types;
 
 use crate::beacon_chain::{BeaconChainTypes, FORK_CHOICE_DB_KEY, OP_POOL_DB_KEY};
@@ -32,7 +33,7 @@ pub fn migrate_schema<T: BeaconChainTypes>(
     match (from, to) {
         // Migrating from the current schema version to iself is always OK, a no-op.
         (_, _) if from == to && to == CURRENT_SCHEMA_VERSION => Ok(()),
-        // Migrate across multiple versions by recursively migrating one step at a time.
+        // Upgrade across multiple versions by recursively migrating one step at a time.
         (_, _) if from.as_u64() + 1 < to.as_u64() => {
             let next = SchemaVersion(from.as_u64() + 1);
             migrate_schema::<T>(db.clone(), datadir, from, next, log.clone())?;
@@ -181,6 +182,17 @@ pub fn migrate_schema<T: BeaconChainTypes>(
 
             Ok(())
         }
+        // Upgrade from v8 to v9 to separate the execution payloads into their own column.
+        (SchemaVersion(8), SchemaVersion(9)) => {
+            migration_schema_v9::upgrade_to_v9::<T>(db.clone(), log)?;
+            db.store_schema_version(to)
+        }
+        // Downgrade from v9 to v8 to ignore the separation of execution payloads
+        // NOTE: only works before the Bellatrix fork epoch.
+        (SchemaVersion(9), SchemaVersion(8)) => {
+            migration_schema_v9::downgrade_from_v9::<T>(db.clone(), log)?;
+            db.store_schema_version(to)
+        }
         // Anything else is an error.
         (_, _) => Err(HotColdDBError::UnsupportedSchemaVersion {
             target_version: to,

beacon_node/beacon_chain/src/schema_change/migration_schema_v7.rs

@@ -31,7 +31,7 @@ pub(crate) fn update_with_reinitialized_fork_choice<T: BeaconChainTypes>(
         .finalized_checkpoint
         .root;
     let anchor_block = db
-        .get_block(&anchor_block_root)
+        .get_full_block_prior_to_v9(&anchor_block_root)
         .map_err(|e| format!("{:?}", e))?
         .ok_or_else(|| "Missing anchor beacon block".to_string())?;
     let anchor_state = db

beacon_node/beacon_chain/src/schema_change/migration_schema_v8.rs

@@ -34,7 +34,7 @@ pub fn update_fork_choice<T: BeaconChainTypes>(
     // before schema v8 the cache would always miss on skipped slots.
     for item in balances_cache.items {
         // Drop any blocks that aren't found, they're presumably too old and this is only a cache.
-        if let Some(block) = db.get_block(&item.block_root)? {
+        if let Some(block) = db.get_full_block_prior_to_v9(&item.block_root)? {
             fork_choice_store.balances_cache.items.push(CacheItemV8 {
                 block_root: item.block_root,
                 epoch: block.slot().epoch(T::EthSpec::slots_per_epoch()),

beacon_node/beacon_chain/src/schema_change/migration_schema_v9.rs

@@ -0,0 +1,176 @@
+use crate::beacon_chain::BeaconChainTypes;
+use slog::{debug, error, info, Logger};
+use slot_clock::SlotClock;
+use std::sync::Arc;
+use std::time::Duration;
+use store::{DBColumn, Error, HotColdDB, KeyValueStore};
+use types::{EthSpec, Hash256, Slot};
+
+const OPS_PER_BLOCK_WRITE: usize = 2;
+
+/// The slot clock isn't usually available before the database is initialized, so we construct a
+/// temporary slot clock by reading the genesis state. It should always exist if the database is
+/// initialized at a prior schema version, however we still handle the lack of genesis state
+/// gracefully.
+fn get_slot_clock<T: BeaconChainTypes>(
+    db: &HotColdDB<T::EthSpec, T::HotStore, T::ColdStore>,
+    log: &Logger,
+) -> Result<Option<T::SlotClock>, Error> {
+    // At schema v8 the genesis block must be a *full* block (with payload). In all likeliness it
+    // actually has no payload.
+    let spec = db.get_chain_spec();
+    let genesis_block = if let Some(block) = db.get_full_block_prior_to_v9(&Hash256::zero())? {
+        block
+    } else {
+        error!(log, "Missing genesis block");
+        return Ok(None);
+    };
+    let genesis_state =
+        if let Some(state) = db.get_state(&genesis_block.state_root(), Some(Slot::new(0)))? {
+            state
+        } else {
+            error!(log, "Missing genesis state"; "state_root" => ?genesis_block.state_root());
+            return Ok(None);
+        };
+    Ok(Some(T::SlotClock::new(
+        spec.genesis_slot,
+        Duration::from_secs(genesis_state.genesis_time()),
+        Duration::from_secs(spec.seconds_per_slot),
+    )))
+}
+
+pub fn upgrade_to_v9<T: BeaconChainTypes>(
+    db: Arc<HotColdDB<T::EthSpec, T::HotStore, T::ColdStore>>,
+    log: Logger,
+) -> Result<(), Error> {
+    // This upgrade is a no-op if the Bellatrix fork epoch has not already passed. This migration
+    // was implemented before the activation of Bellatrix on all networks except Kiln, so the only
+    // users who will need to wait for the slow copying migration are Kiln users.
+    let slot_clock = if let Some(slot_clock) = get_slot_clock::<T>(&db, &log)? {
+        slot_clock
+    } else {
+        error!(
+            log,
+            "Unable to complete migration because genesis state or genesis block is missing"
+        );
+        return Err(Error::SlotClockUnavailableForMigration);
+    };
+
+    let current_epoch = if let Some(slot) = slot_clock.now() {
+        slot.epoch(T::EthSpec::slots_per_epoch())
+    } else {
+        return Ok(());
+    };
+
+    let bellatrix_fork_epoch = if let Some(fork_epoch) = db.get_chain_spec().bellatrix_fork_epoch {
+        fork_epoch
+    } else {
+        info!(
+            log,
+            "Upgrading database schema to v9 (no-op)";
+            "info" => "To downgrade before the merge run `lighthouse db migrate`"
+        );
+        return Ok(());
+    };
+
+    if current_epoch >= bellatrix_fork_epoch {
+        info!(
+            log,
+            "Upgrading database schema to v9";
+            "info" => "This will take several minutes. Each block will be read from and \
+                       re-written to the database. You may safely exit now (Ctrl-C) and resume \
+                       the migration later. Downgrading is no longer possible."
+        );
+
+        for res in db.hot_db.iter_column_keys(DBColumn::BeaconBlock) {
+            let block_root = res?;
+            let block = match db.get_full_block_prior_to_v9(&block_root) {
+                // A pre-v9 block is present.
+                Ok(Some(block)) => block,
+                // A block is missing.
+                Ok(None) => return Err(Error::BlockNotFound(block_root)),
+                // There was an error reading a pre-v9 block. Try reading it as a post-v9 block.
+                Err(_) => {
+                    if db.try_get_full_block(&block_root)?.is_some() {
+                        // The block is present as a post-v9 block, assume that it was already
+                        // correctly migrated.
+                        continue;
+                    } else {
+                        // This scenario should not be encountered since a prior check has ensured
+                        // that this block exists.
+                        return Err(Error::V9MigrationFailure(block_root));
+                    }
+                }
+            };
+
+            if block.message().execution_payload().is_ok() {
+                // Overwrite block with blinded block and store execution payload separately.
+                debug!(
+                    log,
+                    "Rewriting Bellatrix block";
+                    "block_root" => ?block_root,
+                );
+
+                let mut kv_batch = Vec::with_capacity(OPS_PER_BLOCK_WRITE);
+                db.block_as_kv_store_ops(&block_root, block, &mut kv_batch)?;
+                db.hot_db.do_atomically(kv_batch)?;
+            }
+        }
+    } else {
+        info!(
+            log,
+            "Upgrading database schema to v9 (no-op)";
+            "info" => "To downgrade before the merge run `lighthouse db migrate`"
+        );
+    }
+
+    Ok(())
+}
+
+// This downgrade is conditional and will only succeed if the Bellatrix fork epoch hasn't been
+// reached.
+pub fn downgrade_from_v9<T: BeaconChainTypes>(
+    db: Arc<HotColdDB<T::EthSpec, T::HotStore, T::ColdStore>>,
+    log: Logger,
+) -> Result<(), Error> {
+    let slot_clock = if let Some(slot_clock) = get_slot_clock::<T>(&db, &log)? {
+        slot_clock
+    } else {
+        error!(
+            log,
+            "Unable to complete migration because genesis state or genesis block is missing"
+        );
+        return Err(Error::SlotClockUnavailableForMigration);
+    };
+
+    let current_epoch = if let Some(slot) = slot_clock.now() {
+        slot.epoch(T::EthSpec::slots_per_epoch())
+    } else {
+        return Ok(());
+    };
+
+    let bellatrix_fork_epoch = if let Some(fork_epoch) = db.get_chain_spec().bellatrix_fork_epoch {
+        fork_epoch
+    } else {
+        info!(
+            log,
+            "Downgrading database schema from v9";
+            "info" => "You need to upgrade to v9 again before the merge"
+        );
+        return Ok(());
+    };
+
+    if current_epoch >= bellatrix_fork_epoch {
+        error!(
+            log,
+            "Downgrading from schema v9 after the Bellatrix fork epoch is not supported";
+            "current_epoch" => current_epoch,
+            "bellatrix_fork_epoch" => bellatrix_fork_epoch,
+            "reason" => "You need a v9 schema database to run on a merged version of Prater or \
+                         mainnet. On Kiln, you have to re-sync",
+        );
+        Err(Error::ResyncRequiredForExecutionPayloadSeparation)
+    } else {
+        Ok(())
+    }
+}

beacon_node/beacon_chain/src/snapshot_cache.rs

@@ -3,8 +3,8 @@ use itertools::process_results;
 use std::cmp;
 use std::time::Duration;
 use types::{
-    beacon_state::CloneConfig, BeaconState, ChainSpec, Epoch, EthSpec, Hash256, SignedBeaconBlock,
-    Slot,
+    beacon_state::CloneConfig, BeaconState, BlindedPayload, ChainSpec, Epoch, EthSpec, Hash256,
+    SignedBeaconBlock, Slot,
 };
 
 /// The default size of the cache.
@@ -23,7 +23,7 @@ pub struct PreProcessingSnapshot<T: EthSpec> {
     pub pre_state: BeaconState<T>,
     /// This value is only set to `Some` if the `pre_state` was *not* advanced forward.
     pub beacon_state_root: Option<Hash256>,
-    pub beacon_block: SignedBeaconBlock<T>,
+    pub beacon_block: SignedBeaconBlock<T, BlindedPayload<T>>,
     pub beacon_block_root: Hash256,
 }
 
@@ -33,7 +33,7 @@ impl<T: EthSpec> From<BeaconSnapshot<T>> for PreProcessingSnapshot<T> {
         Self {
             pre_state: snapshot.beacon_state,
             beacon_state_root,
-            beacon_block: snapshot.beacon_block,
+            beacon_block: snapshot.beacon_block.into(),
             beacon_block_root: snapshot.beacon_block_root,
         }
     }
@@ -63,7 +63,7 @@ impl<T: EthSpec> CacheItem<T> {
             Some(self.beacon_block.state_root()).filter(|_| self.pre_state.is_none());
 
         PreProcessingSnapshot {
-            beacon_block: self.beacon_block,
+            beacon_block: self.beacon_block.into(),
             beacon_block_root: self.beacon_block_root,
             pre_state: self.pre_state.unwrap_or(self.beacon_state),
             beacon_state_root,
@@ -76,7 +76,7 @@ impl<T: EthSpec> CacheItem<T> {
             Some(self.beacon_block.state_root()).filter(|_| self.pre_state.is_none());
 
         PreProcessingSnapshot {
-            beacon_block: self.beacon_block.clone(),
+            beacon_block: self.beacon_block.clone().into(),
             beacon_block_root: self.beacon_block_root,
             pre_state: self
                 .pre_state

beacon_node/beacon_chain/src/test_utils.rs

@@ -528,8 +528,11 @@ where
         self.chain.slot().unwrap()
     }
 
-    pub fn get_block(&self, block_hash: SignedBeaconBlockHash) -> Option<SignedBeaconBlock<E>> {
-        self.chain.get_block(&block_hash.into()).unwrap()
+    pub fn get_block(
+        &self,
+        block_hash: SignedBeaconBlockHash,
+    ) -> Option<SignedBeaconBlock<E, BlindedPayload<E>>> {
+        self.chain.get_blinded_block(&block_hash.into()).unwrap()
     }
 
     pub fn block_exists(&self, block_hash: SignedBeaconBlockHash) -> bool {

beacon_node/beacon_chain/tests/attestation_production.rs

@@ -55,11 +55,15 @@ fn produces_attestations() {
             Slot::from(num_blocks_produced)
         };
 
-        let block = chain
+        let blinded_block = chain
             .block_at_slot(block_slot, WhenSlotSkipped::Prev)
             .expect("should get block")
             .expect("block should not be skipped");
-        let block_root = block.message().tree_hash_root();
+        let block_root = blinded_block.message().tree_hash_root();
+        let block = chain
+            .store
+            .make_full_block(&block_root, blinded_block)
+            .unwrap();
 
         let epoch_boundary_slot = state
             .current_epoch()

beacon_node/beacon_chain/tests/attestation_verification.rs

@@ -975,7 +975,7 @@ fn attestation_that_skips_epochs() {
     let block_slot = harness
         .chain
         .store
-        .get_block(&block_root)
+        .get_blinded_block(&block_root)
         .expect("should not error getting block")
         .expect("should find attestation block")
         .message()

beacon_node/beacon_chain/tests/block_verification.rs

@@ -46,6 +46,18 @@ fn get_chain_segment() -> Vec<BeaconSnapshot<E>> {
         .chain_dump()
         .expect("should dump chain")
         .into_iter()
+        .map(|snapshot| {
+            let full_block = harness
+                .chain
+                .store
+                .make_full_block(&snapshot.beacon_block_root, snapshot.beacon_block)
+                .unwrap();
+            BeaconSnapshot {
+                beacon_block_root: snapshot.beacon_block_root,
+                beacon_block: full_block,
+                beacon_state: snapshot.beacon_state,
+            }
+        })
         .skip(1)
         .collect()
 }

beacon_node/beacon_chain/tests/payload_invalidation.rs

@@ -71,7 +71,7 @@ impl InvalidPayloadRig {
     fn block_hash(&self, block_root: Hash256) -> ExecutionBlockHash {
         self.harness
             .chain
-            .get_block(&block_root)
+            .get_blinded_block(&block_root)
             .unwrap()
             .unwrap()
             .message()
@@ -273,7 +273,12 @@ impl InvalidPayloadRig {
                 }
 
                 assert_eq!(
-                    self.harness.chain.get_block(&block_root).unwrap().unwrap(),
+                    self.harness
+                        .chain
+                        .store
+                        .get_full_block(&block_root)
+                        .unwrap()
+                        .unwrap(),
                     block,
                     "block from db must match block imported"
                 );
@@ -311,7 +316,11 @@ impl InvalidPayloadRig {
                     assert_eq!(block_in_forkchoice, None);
 
                     assert!(
-                        self.harness.chain.get_block(&block_root).unwrap().is_none(),
+                        self.harness
+                            .chain
+                            .get_blinded_block(&block_root)
+                            .unwrap()
+                            .is_none(),
                         "invalid block cannot be accessed via get_block"
                     );
                 } else {
@@ -427,7 +436,7 @@ fn justified_checkpoint_becomes_invalid() {
     let parent_root_of_justified = rig
         .harness
         .chain
-        .get_block(&justified_checkpoint.root)
+        .get_blinded_block(&justified_checkpoint.root)
         .unwrap()
         .unwrap()
         .parent_root();
@@ -643,7 +652,13 @@ fn invalidates_all_descendants() {
     assert!(rig.execution_status(fork_block_root).is_invalid());
 
     for root in blocks {
-        let slot = rig.harness.chain.get_block(&root).unwrap().unwrap().slot();
+        let slot = rig
+            .harness
+            .chain
+            .get_blinded_block(&root)
+            .unwrap()
+            .unwrap()
+            .slot();
 
         // Fork choice doesn't have info about pre-finalization, nothing to check here.
         if slot < finalized_slot {
@@ -707,7 +722,13 @@ fn switches_heads() {
     assert!(rig.execution_status(fork_block_root).is_optimistic());
 
     for root in blocks {
-        let slot = rig.harness.chain.get_block(&root).unwrap().unwrap().slot();
+        let slot = rig
+            .harness
+            .chain
+            .get_blinded_block(&root)
+            .unwrap()
+            .unwrap()
+            .slot();
 
         // Fork choice doesn't have info about pre-finalization, nothing to check here.
         if slot < finalized_slot {
@@ -739,9 +760,17 @@ fn invalid_during_processing() {
     ];
 
     // 0 should be present in the chain.
-    assert!(rig.harness.chain.get_block(&roots[0]).unwrap().is_some());
+    assert!(rig
+        .harness
+        .chain
+        .get_blinded_block(&roots[0])
+        .unwrap()
+        .is_some());
     // 1 should *not* be present in the chain.
-    assert_eq!(rig.harness.chain.get_block(&roots[1]).unwrap(), None);
+    assert_eq!(
+        rig.harness.chain.get_blinded_block(&roots[1]).unwrap(),
+        None
+    );
     // 2 should be the head.
     let head = rig.harness.chain.head_info().unwrap();
     assert_eq!(head.block_root, roots[2]);
@@ -760,7 +789,7 @@ fn invalid_after_optimistic_sync() {
     ];
 
     for root in &roots {
-        assert!(rig.harness.chain.get_block(root).unwrap().is_some());
+        assert!(rig.harness.chain.get_blinded_block(root).unwrap().is_some());
     }
 
     // 2 should be the head.

beacon_node/beacon_chain/tests/store_tests.rs

@@ -313,7 +313,10 @@ fn epoch_boundary_state_attestation_processing() {
     for (attestation, subnet_id) in late_attestations.into_iter().flatten() {
         // load_epoch_boundary_state is idempotent!
         let block_root = attestation.data.beacon_block_root;
-        let block = store.get_block(&block_root).unwrap().expect("block exists");
+        let block = store
+            .get_blinded_block(&block_root)
+            .unwrap()
+            .expect("block exists");
         let epoch_boundary_state = store
             .load_epoch_boundary_state(&block.state_root())
             .expect("no error")
@@ -603,7 +606,7 @@ fn delete_blocks_and_states() {
     );
 
     let faulty_head_block = store
-        .get_block(&faulty_head.into())
+        .get_blinded_block(&faulty_head.into())
         .expect("no errors")
         .expect("faulty head block exists");
 
@@ -645,7 +648,7 @@ fn delete_blocks_and_states() {
             break;
         }
         store.delete_block(&block_root).unwrap();
-        assert_eq!(store.get_block(&block_root).unwrap(), None);
+        assert_eq!(store.get_blinded_block(&block_root).unwrap(), None);
     }
 
     // Deleting frozen states should do nothing
@@ -890,7 +893,12 @@ fn shuffling_compatible_short_fork() {
 }
 
 fn get_state_for_block(harness: &TestHarness, block_root: Hash256) -> BeaconState<E> {
-    let head_block = harness.chain.get_block(&block_root).unwrap().unwrap();
+    let head_block = harness
+        .chain
+        .store
+        .get_blinded_block(&block_root)
+        .unwrap()
+        .unwrap();
     harness
         .chain
         .get_state(&head_block.state_root(), Some(head_block.slot()))
@@ -1695,7 +1703,7 @@ fn check_all_blocks_exist<'a>(
     blocks: impl Iterator<Item = &'a SignedBeaconBlockHash>,
 ) {
     for &block_hash in blocks {
-        let block = harness.chain.get_block(&block_hash.into()).unwrap();
+        let block = harness.chain.get_blinded_block(&block_hash.into()).unwrap();
         assert!(
             block.is_some(),
             "expected block {:?} to be in DB",
@@ -1742,7 +1750,7 @@ fn check_no_blocks_exist<'a>(
     blocks: impl Iterator<Item = &'a SignedBeaconBlockHash>,
 ) {
     for &block_hash in blocks {
-        let block = harness.chain.get_block(&block_hash.into()).unwrap();
+        let block = harness.chain.get_blinded_block(&block_hash.into()).unwrap();
         assert!(
             block.is_none(),
             "did not expect block {:?} to be in the DB",
@@ -1988,7 +1996,12 @@ fn weak_subjectivity_sync() {
         .unwrap()
         .unwrap();
     let wss_checkpoint = harness.chain.head_info().unwrap().finalized_checkpoint;
-    let wss_block = harness.get_block(wss_checkpoint.root.into()).unwrap();
+    let wss_block = harness
+        .chain
+        .store
+        .get_full_block(&wss_checkpoint.root)
+        .unwrap()
+        .unwrap();
     let wss_state = full_store
         .get_state(&wss_block.state_root(), None)
         .unwrap()
@@ -2042,8 +2055,14 @@ fn weak_subjectivity_sync() {
 
     for snapshot in new_blocks {
         let block = &snapshot.beacon_block;
+        let full_block = harness
+            .chain
+            .store
+            .make_full_block(&snapshot.beacon_block_root, block.clone())
+            .unwrap();
+
         beacon_chain.slot_clock.set_slot(block.slot().as_u64());
-        beacon_chain.process_block(block.clone()).unwrap();
+        beacon_chain.process_block(full_block).unwrap();
         beacon_chain.fork_choice().unwrap();
 
         // Check that the new block's state can be loaded correctly.
@@ -2085,13 +2104,13 @@ fn weak_subjectivity_sync() {
         .map(|s| s.beacon_block.clone())
         .collect::<Vec<_>>();
     beacon_chain
-        .import_historical_block_batch(&historical_blocks)
+        .import_historical_block_batch(historical_blocks.clone())
         .unwrap();
     assert_eq!(beacon_chain.store.get_oldest_block_slot(), 0);
 
     // Resupplying the blocks should not fail, they can be safely ignored.
     beacon_chain
-        .import_historical_block_batch(&historical_blocks)
+        .import_historical_block_batch(historical_blocks)
         .unwrap();
 
     // The forwards iterator should now match the original chain
@@ -2114,7 +2133,7 @@ fn weak_subjectivity_sync() {
         .unwrap()
         .map(Result::unwrap)
     {
-        let block = store.get_block(&block_root).unwrap().unwrap();
+        let block = store.get_blinded_block(&block_root).unwrap().unwrap();
         assert_eq!(block.slot(), slot);
     }
 
@@ -2574,7 +2593,7 @@ fn check_iterators(harness: &TestHarness) {
 }
 
 fn get_finalized_epoch_boundary_blocks(
-    dump: &[BeaconSnapshot<MinimalEthSpec>],
+    dump: &[BeaconSnapshot<MinimalEthSpec, BlindedPayload<MinimalEthSpec>>],
 ) -> HashSet<SignedBeaconBlockHash> {
     dump.iter()
         .cloned()
@@ -2582,7 +2601,9 @@ fn get_finalized_epoch_boundary_blocks(
         .collect()
 }
 
-fn get_blocks(dump: &[BeaconSnapshot<MinimalEthSpec>]) -> HashSet<SignedBeaconBlockHash> {
+fn get_blocks(
+    dump: &[BeaconSnapshot<MinimalEthSpec, BlindedPayload<MinimalEthSpec>>],
+) -> HashSet<SignedBeaconBlockHash> {
     dump.iter()
         .cloned()
         .map(|checkpoint| checkpoint.beacon_block_root.into())

beacon_node/beacon_chain/tests/tests.rs

@@ -744,7 +744,11 @@ fn block_roots_skip_slot_behaviour() {
                 "WhenSlotSkipped::Prev should accurately return the prior skipped block"
             );
 
-            let expected_block = harness.chain.get_block(&skipped_root).unwrap().unwrap();
+            let expected_block = harness
+                .chain
+                .get_blinded_block(&skipped_root)
+                .unwrap()
+                .unwrap();
 
             assert_eq!(
                 harness
@@ -782,7 +786,11 @@ fn block_roots_skip_slot_behaviour() {
                 "WhenSlotSkipped::None and WhenSlotSkipped::Prev should be equal on non-skipped slot"
             );
 
-            let expected_block = harness.chain.get_block(&skips_prev).unwrap().unwrap();
+            let expected_block = harness
+                .chain
+                .get_blinded_block(&skips_prev)
+                .unwrap()
+                .unwrap();
 
             assert_eq!(
                 harness