cerc-io/lighthouse
11
0
Fork 0
Code Issues 1 Pull Requests Packages Projects Releases Wiki Activity

Single blob lookups (#4152)

Browse Source 
* some blob reprocessing work

* remove ForceBlockLookup

* reorder enum match arms in sync manager

* a lot more reprocessing work

* impl logic for triggerng blob lookups along with block lookups

* deal with rpc blobs in groups per block in the da checker. don't cache missing blob ids in the da checker.

* make single block lookup generic

* more work

* add delayed processing logic and combine some requests

* start fixing some compile errors

* fix compilation in main block lookup mod

* much work

* get things compiling

* parent blob lookups

* fix compile

* revert red/stevie changes

* fix up sync manager delay message logic

* add peer usefulness enum

* should remove lookup refactor

* consolidate retry error handling

* improve peer scoring during certain failures in parent lookups

* improve retry code

* drop parent lookup if either req has a peer disconnect during download

* refactor single block processed method

* processing peer refactor

* smol bugfix

* fix some todos

* fix lints

* fix lints

* fix compile in lookup tests

* fix lints

* fix lints

* fix existing block lookup tests

* renamings

* fix after merge

* cargo fmt

* compilation fix in beacon chain tests

* fix

* refactor lookup tests to work with multiple forks and response types

* make tests into macros

* wrap availability check error

* fix compile after merge

* add random blobs

* start fixing up lookup verify error handling

* some bug fixes and the start of deneb only tests

* make tests work for all forks

* track information about peer source

* error refactoring

* improve peer scoring

* fix test compilation

* make sure blobs are sent for processing after stream termination, delete copied tests

* add some tests and fix a bug

* smol bugfixes and moar tests

* add tests and fix some things

* compile after merge

* lots of refactoring

* retry on invalid block/blob

* merge unknown parent messages before current slot lookup

* get tests compiling

* penalize blob peer on invalid blobs

* Check disk on in-memory cache miss

* Update beacon_node/beacon_chain/src/data_availability_checker/overflow_lru_cache.rs

* Update beacon_node/network/src/sync/network_context.rs

Co-authored-by: Divma <26765164+divagant-martian@users.noreply.github.com>

* fix bug in matching blocks and blobs in range sync

* pr feedback

* fix conflicts

* upgrade logs from warn to crit when we receive incorrect response in range

* synced_and_connected_within_tolerance -> should_search_for_block

* remove todo

* Fix Broken Overflow Tests

* fix merge conflicts

* checkpoint sync without alignment

* add import

* query for checkpoint state by slot rather than state root (teku doesn't serve by state root)

* get state first and query by most recent block root

* simplify delay logic

* rename unknown parent sync message variants

* rename parameter, block_slot -> slot

* add some docs to the lookup module

* use interval instead of sleep

* drop request if blocks and blobs requests both return `None` for `Id`

* clean up `find_single_lookup` logic

* add lookup source enum

* clean up `find_single_lookup` logic

* add docs to find_single_lookup_request

* move LookupSource our of param where unnecessary

* remove unnecessary todo

* query for block by `state.latest_block_header.slot`

* fix lint

* fix test

* fix test

* fix observed  blob sidecars test

* PR updates

* use optional params instead of a closure

* create lookup and trigger request in separate method calls

* remove `LookupSource`

* make sure duplicate lookups are not dropped

---------

Co-authored-by: Pawan Dhananjay <pawandhananjay@gmail.com>
Co-authored-by: Mark Mackey <mark@sigmaprime.io>
Co-authored-by: Divma <26765164+divagant-martian@users.noreply.github.com>
This commit is contained in:
realbigsean
2023-06-15 12:59:10 -04:00
committed by GitHub
co-authored by Divma Pawan Dhananjay Mark Mackey
parent 5428e68943
commit a62e52f319
47 changed files with 4981 additions and 1309 deletions
Download Patch File Download Diff File Expand all files Collapse all files
beacon_node/beacon_chain/src/beacon_chain.rs
+28 -14
View File
@@ -117,7 +117,7 @@ use tokio_stream::Stream;
use tree_hash::TreeHash;
use types::beacon_block_body::KzgCommitments;
use types::beacon_state::CloneConfig;
use types::blob_sidecar::{BlobIdentifier, BlobSidecarList, Blobs};
use types::blob_sidecar::{BlobSidecarList, Blobs};
use types::consts::deneb::MIN_EPOCHS_FOR_BLOB_SIDECARS_REQUESTS;
use types::*;
@@ -185,12 +185,10 @@ pub enum WhenSlotSkipped {
#[derive(Debug, PartialEq)]
pub enum AvailabilityProcessingStatus {
PendingBlobs(Vec<BlobIdentifier>),
PendingBlock(Hash256),
MissingComponents(Slot, Hash256),
Imported(Hash256),
}
//TODO(sean) using this in tests for now
impl TryInto<SignedBeaconBlockHash> for AvailabilityProcessingStatus {
type Error = ();
@@ -468,7 +466,7 @@ pub struct BeaconChain<T: BeaconChainTypes> {
/// The slot at which blocks are downloaded back to.
pub genesis_backfill_slot: Slot,
pub proposal_blob_cache: BlobCache<T::EthSpec>,
pub data_availability_checker: DataAvailabilityChecker<T>,
pub data_availability_checker: Arc<DataAvailabilityChecker<T>>,
pub kzg: Option<Arc<Kzg>>,
}
@@ -1985,8 +1983,7 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
self: &Arc<Self>,
blob_sidecar: SignedBlobSidecar<T::EthSpec>,
subnet_id: u64,
) -> Result<GossipVerifiedBlob<T::EthSpec>, BlobError> // TODO(pawan): make a GossipVerifedBlob type
{
) -> Result<GossipVerifiedBlob<T::EthSpec>, BlobError<T::EthSpec>> {
blob_verification::validate_blob_sidecar_for_gossip(blob_sidecar, subnet_id, self)
}
@@ -2674,7 +2671,24 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
)
.await
{
Ok(_) => imported_blocks += 1,
Ok(status) => {
match status {
AvailabilityProcessingStatus::Imported(_) => {
// The block was imported successfully.
imported_blocks += 1;
}
AvailabilityProcessingStatus::MissingComponents(slot, block_root) => {
warn!(self.log, "Blobs missing in response to range request";
"block_root" => ?block_root, "slot" => slot);
return ChainSegmentResult::Failed {
imported_blocks,
error: BlockError::AvailabilityCheck(
AvailabilityCheckError::MissingBlobs,
),
};
}
}
}
Err(error) => {
return ChainSegmentResult::Failed {
imported_blocks,
@@ -2748,6 +2762,7 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
count_unrealized: CountUnrealized,
) -> Result<AvailabilityProcessingStatus, BlockError<T::EthSpec>> {
self.check_availability_and_maybe_import(
blob.slot(),
|chain| chain.data_availability_checker.put_gossip_blob(blob),
count_unrealized,
)
@@ -2804,6 +2819,7 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
}
ExecutedBlock::AvailabilityPending(block) => {
self.check_availability_and_maybe_import(
block.block.slot(),
|chain| {
chain
.data_availability_checker
@@ -2907,6 +2923,7 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
/// (i.e., this function is not atomic).
pub async fn check_availability_and_maybe_import(
self: &Arc<Self>,
slot: Slot,
cache_fn: impl FnOnce(Arc<Self>) -> Result<Availability<T::EthSpec>, AvailabilityCheckError>,
count_unrealized: CountUnrealized,
) -> Result<AvailabilityProcessingStatus, BlockError<T::EthSpec>> {
@@ -2915,12 +2932,9 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
Availability::Available(block) => {
self.import_available_block(block, count_unrealized).await
}
Availability::PendingBlock(block_root) => {
Ok(AvailabilityProcessingStatus::PendingBlock(block_root))
}
Availability::PendingBlobs(blob_ids) => {
Ok(AvailabilityProcessingStatus::PendingBlobs(blob_ids))
}
Availability::MissingComponents(block_root) => Ok(
AvailabilityProcessingStatus::MissingComponents(slot, block_root),
),
}
}
beacon_node/beacon_chain/src/blob_verification.rs
+47 -36
View File
@@ -16,15 +16,17 @@ use eth2::types::BlockContentsTuple;
use kzg::Kzg;
use slog::{debug, warn};
use ssz_derive::{Decode, Encode};
use ssz_types::FixedVector;
use std::borrow::Cow;
use types::blob_sidecar::{BlobIdentifier, FixedBlobSidecarList};
use types::{
BeaconBlockRef, BeaconState, BeaconStateError, BlobSidecar, BlobSidecarList, ChainSpec,
CloneConfig, Epoch, EthSpec, FullPayload, Hash256, KzgCommitment, RelativeEpoch,
SignedBeaconBlock, SignedBeaconBlockHeader, SignedBlobSidecar, Slot,
BeaconBlockRef, BeaconState, BeaconStateError, BlobSidecar, ChainSpec, CloneConfig, Epoch,
EthSpec, FullPayload, Hash256, KzgCommitment, RelativeEpoch, SignedBeaconBlock,
SignedBeaconBlockHeader, SignedBlobSidecar, Slot,
};
#[derive(Debug)]
pub enum BlobError {
pub enum BlobError<T: EthSpec> {
/// The blob sidecar is from a slot that is later than the current slot (with respect to the
/// gossip clock disparity).
///
@@ -96,10 +98,7 @@ pub enum BlobError {
/// ## Peer scoring
///
/// We cannot process the blob without validating its parent, the peer isn't necessarily faulty.
BlobParentUnknown {
blob_root: Hash256,
blob_parent_root: Hash256,
},
BlobParentUnknown(Arc<BlobSidecar<T>>),
/// A blob has already been seen for the given `(sidecar.block_root, sidecar.index)` tuple
/// over gossip or no gossip sources.
@@ -114,13 +113,13 @@ pub enum BlobError {
},
}
impl From<BeaconChainError> for BlobError {
impl<T: EthSpec> From<BeaconChainError> for BlobError<T> {
fn from(e: BeaconChainError) -> Self {
BlobError::BeaconChainError(e)
}
}
impl From<BeaconStateError> for BlobError {
impl<T: EthSpec> From<BeaconStateError> for BlobError<T> {
fn from(e: BeaconStateError) -> Self {
BlobError::BeaconChainError(BeaconChainError::BeaconStateError(e))
}
@@ -128,27 +127,36 @@ impl From<BeaconStateError> for BlobError {
/// A wrapper around a `BlobSidecar` that indicates it has been approved for re-gossiping on
/// the p2p network.
#[derive(Debug)]
#[derive(Debug, Clone)]
pub struct GossipVerifiedBlob<T: EthSpec> {
blob: Arc<BlobSidecar<T>>,
}
impl<T: EthSpec> GossipVerifiedBlob<T> {
pub fn id(&self) -> BlobIdentifier {
self.blob.id()
}
pub fn block_root(&self) -> Hash256 {
self.blob.block_root
}
pub fn to_blob(self) -> Arc<BlobSidecar<T>> {
self.blob
}
pub fn slot(&self) -> Slot {
self.blob.slot
}
}
pub fn validate_blob_sidecar_for_gossip<T: BeaconChainTypes>(
signed_blob_sidecar: SignedBlobSidecar<T::EthSpec>,
subnet: u64,
chain: &BeaconChain<T>,
) -> Result<GossipVerifiedBlob<T::EthSpec>, BlobError> {
) -> Result<GossipVerifiedBlob<T::EthSpec>, BlobError<T::EthSpec>> {
let blob_slot = signed_blob_sidecar.message.slot;
let blob_index = signed_blob_sidecar.message.index;
let block_root = signed_blob_sidecar.message.block_root;
let block_parent_root = signed_blob_sidecar.message.block_parent_root;
let blob_proposer_index = signed_blob_sidecar.message.proposer_index;
let block_root = signed_blob_sidecar.message.block_root;
// Verify that the blob_sidecar was received on the correct subnet.
if blob_index != subnet {
@@ -211,10 +219,7 @@ pub fn validate_blob_sidecar_for_gossip<T: BeaconChainTypes>(
});
}
} else {
return Err(BlobError::BlobParentUnknown {
blob_root: block_root,
blob_parent_root: block_parent_root,
});
return Err(BlobError::BlobParentUnknown(signed_blob_sidecar.message));
}
// Note: We check that the proposer_index matches against the shuffling first to avoid
@@ -366,7 +371,7 @@ fn cheap_state_advance_to_obtain_committees<'a, E: EthSpec>(
state_root_opt: Option<Hash256>,
blob_slot: Slot,
spec: &ChainSpec,
) -> Result<Cow<'a, BeaconState<E>>, BlobError> {
) -> Result<Cow<'a, BeaconState<E>>, BlobError<E>> {
let block_epoch = blob_slot.epoch(E::slots_per_epoch());
if state.current_epoch() == block_epoch {
@@ -443,19 +448,14 @@ impl<T: EthSpec> KzgVerifiedBlob<T> {
///
/// Returns an error if the kzg verification check fails.
pub fn verify_kzg_for_blob<T: EthSpec>(
blob: GossipVerifiedBlob<T>,
blob: Arc<BlobSidecar<T>>,
kzg: &Kzg,
) -> Result<KzgVerifiedBlob<T>, AvailabilityCheckError> {
//TODO(sean) remove clone
if validate_blob::<T>(
kzg,
blob.blob.blob.clone(),
blob.blob.kzg_commitment,
blob.blob.kzg_proof,
)
.map_err(AvailabilityCheckError::Kzg)?
if validate_blob::<T>(kzg, blob.blob.clone(), blob.kzg_commitment, blob.kzg_proof)
.map_err(AvailabilityCheckError::Kzg)?
{
Ok(KzgVerifiedBlob { blob: blob.blob })
Ok(KzgVerifiedBlob { blob })
} else {
Err(AvailabilityCheckError::KzgVerificationFailed)
}
@@ -467,7 +467,7 @@ pub fn verify_kzg_for_blob<T: EthSpec>(
/// Note: This function should be preferred over calling `verify_kzg_for_blob`
/// in a loop since this function kzg verifies a list of blobs more efficiently.
pub fn verify_kzg_for_blob_list<T: EthSpec>(
blob_list: BlobSidecarList<T>,
blob_list: Vec<Arc<BlobSidecar<T>>>,
kzg: &Kzg,
) -> Result<KzgVerifiedBlobList<T>, AvailabilityCheckError> {
let (blobs, (commitments, proofs)): (Vec<_>, (Vec<_>, Vec<_>)) = blob_list
@@ -608,7 +608,16 @@ impl<E: EthSpec> AsBlock<E> for &MaybeAvailableBlock<E> {
#[derivative(Hash(bound = "E: EthSpec"))]
pub enum BlockWrapper<E: EthSpec> {
Block(Arc<SignedBeaconBlock<E>>),
BlockAndBlobs(Arc<SignedBeaconBlock<E>>, Vec<Arc<BlobSidecar<E>>>),
BlockAndBlobs(Arc<SignedBeaconBlock<E>>, FixedBlobSidecarList<E>),
}
impl<E: EthSpec> BlockWrapper<E> {
pub fn deconstruct(self) -> (Arc<SignedBeaconBlock<E>>, Option<FixedBlobSidecarList<E>>) {
match self {
BlockWrapper::Block(block) => (block, None),
BlockWrapper::BlockAndBlobs(block, blobs) => (block, Some(blobs)),
}
}
}
impl<E: EthSpec> AsBlock<E> for BlockWrapper<E> {
@@ -675,13 +684,15 @@ impl<E: EthSpec> From<SignedBeaconBlock<E>> for BlockWrapper<E> {
impl<E: EthSpec> From<BlockContentsTuple<E, FullPayload<E>>> for BlockWrapper<E> {
fn from(value: BlockContentsTuple<E, FullPayload<E>>) -> Self {
match value.1 {
Some(variable_list) => Self::BlockAndBlobs(
Arc::new(value.0),
Vec::from(variable_list)
.into_iter()
.map(|signed_blob| signed_blob.message)
.collect::<Vec<_>>(),
),
Some(variable_list) => {
let mut blobs = Vec::with_capacity(E::max_blobs_per_block());
for blob in variable_list {
if blob.message.index < E::max_blobs_per_block() as u64 {
blobs.insert(blob.message.index as usize, Some(blob.message));
}
}
Self::BlockAndBlobs(Arc::new(value.0), FixedVector::from(blobs))
}
None => Self::Block(Arc::new(value.0)),
}
}
beacon_node/beacon_chain/src/block_verification.rs
+22 -43
View File
@@ -70,7 +70,7 @@ use crate::{
use derivative::Derivative;
use eth2::types::EventKind;
use execution_layer::PayloadStatus;
use fork_choice::{AttestationFromBlock, PayloadVerificationStatus};
pub use fork_choice::{AttestationFromBlock, PayloadVerificationStatus};
use parking_lot::RwLockReadGuard;
use proto_array::Block as ProtoBlock;
use safe_arith::ArithError;
@@ -150,10 +150,7 @@ pub enum BlockError<T: EthSpec> {
/// its parent.
ParentUnknown(BlockWrapper<T>),
/// The block skips too many slots and is a DoS risk.
TooManySkippedSlots {
parent_slot: Slot,
block_slot: Slot,
},
TooManySkippedSlots { parent_slot: Slot, block_slot: Slot },
/// The block slot is greater than the present slot.
///
/// ## Peer scoring
@@ -168,10 +165,7 @@ pub enum BlockError<T: EthSpec> {
/// ## Peer scoring
///
/// The peer has incompatible state transition logic and is faulty.
StateRootMismatch {
block: Hash256,
local: Hash256,
},
StateRootMismatch { block: Hash256, local: Hash256 },
/// The block was a genesis block, these blocks cannot be re-imported.
GenesisBlock,
/// The slot is finalized, no need to import.
@@ -190,9 +184,7 @@ pub enum BlockError<T: EthSpec> {
///
/// It's unclear if this block is valid, but it conflicts with finality and shouldn't be
/// imported.
NotFinalizedDescendant {
block_parent_root: Hash256,
},
NotFinalizedDescendant { block_parent_root: Hash256 },
/// Block is already known, no need to re-import.
///
/// ## Peer scoring
@@ -205,10 +197,7 @@ pub enum BlockError<T: EthSpec> {
///
/// The `proposer` has already proposed a block at this slot. The existing block may or may not
/// be equal to the given block.
RepeatProposal {
proposer: u64,
slot: Slot,
},
RepeatProposal { proposer: u64, slot: Slot },
/// The block slot exceeds the MAXIMUM_BLOCK_SLOT_NUMBER.
///
/// ## Peer scoring
@@ -223,10 +212,7 @@ pub enum BlockError<T: EthSpec> {
/// ## Peer scoring
///
/// The block is invalid and the peer is faulty.
IncorrectBlockProposer {
block: u64,
local_shuffling: u64,
},
IncorrectBlockProposer { block: u64, local_shuffling: u64 },
/// The proposal signature in invalid.
///
/// ## Peer scoring
@@ -250,10 +236,7 @@ pub enum BlockError<T: EthSpec> {
/// ## Peer scoring
///
/// The block is invalid and the peer is faulty.
BlockIsNotLaterThanParent {
block_slot: Slot,
parent_slot: Slot,
},
BlockIsNotLaterThanParent { block_slot: Slot, parent_slot: Slot },
/// At least one block in the chain segment did not have it's parent root set to the root of
/// the prior block.
///
@@ -309,15 +292,15 @@ pub enum BlockError<T: EthSpec> {
/// If it's actually our fault (e.g. our execution node database is corrupt) we have bigger
/// problems to worry about than losing peers, and we're doing the network a favour by
/// disconnecting.
ParentExecutionPayloadInvalid {
parent_root: Hash256,
},
BlobValidation(BlobError),
ParentExecutionPayloadInvalid { parent_root: Hash256 },
/// A blob alone failed validation.
BlobValidation(BlobError<T>),
/// The block and blob together failed validation.
AvailabilityCheck(AvailabilityCheckError),
}
impl<T: EthSpec> From<BlobError> for BlockError<T> {
fn from(e: BlobError) -> Self {
impl<T: EthSpec> From<BlobError<T>> for BlockError<T> {
fn from(e: BlobError<T>) -> Self {
Self::BlobValidation(e)
}
}
@@ -785,21 +768,17 @@ impl<E: EthSpec> AvailabilityPendingExecutedBlock<E> {
}
pub fn get_all_blob_ids(&self) -> Vec<BlobIdentifier> {
self.get_filtered_blob_ids(|_| true)
let block_root = self.import_data.block_root;
self.block
.get_filtered_blob_ids(Some(block_root), |_, _| true)
}
pub fn get_filtered_blob_ids(&self, filter: impl Fn(u64) -> bool) -> Vec<BlobIdentifier> {
let num_blobs_expected = self.num_blobs_expected();
let mut blob_ids = Vec::with_capacity(num_blobs_expected);
for i in 0..num_blobs_expected as u64 {
if filter(i) {
blob_ids.push(BlobIdentifier {
block_root: self.import_data.block_root,
index: i,
});
}
}
blob_ids
pub fn get_filtered_blob_ids(
&self,
filter: impl Fn(usize, Hash256) -> bool,
) -> Vec<BlobIdentifier> {
self.block
.get_filtered_blob_ids(Some(self.import_data.block_root), filter)
}
}
beacon_node/beacon_chain/src/builder.rs
+21 -29
View File
@@ -419,23 +419,14 @@ where
let weak_subj_block_root = weak_subj_block.canonical_root();
let weak_subj_state_root = weak_subj_block.state_root();
// Check that the given block lies on an epoch boundary. Due to the database only storing
// Check that the given state lies on an epoch boundary. Due to the database only storing
// full states on epoch boundaries and at restore points it would be difficult to support
// starting from a mid-epoch state.
if weak_subj_slot % TEthSpec::slots_per_epoch() != 0 {
return Err(format!(
"Checkpoint block at slot {} is not aligned to epoch start. \
Please supply an aligned checkpoint with block.slot % 32 == 0",
weak_subj_block.slot(),
));
}
// Check that the block and state have consistent slots and state roots.
if weak_subj_state.slot() != weak_subj_block.slot() {
return Err(format!(
"Slot of snapshot block ({}) does not match snapshot state ({})",
weak_subj_block.slot(),
weak_subj_state.slot(),
"Checkpoint state at slot {} is not aligned to epoch start. \
Please supply an aligned checkpoint with state.slot % 32 == 0",
weak_subj_slot,
));
}
@@ -444,16 +435,21 @@ where
weak_subj_state
.build_all_caches(&self.spec)
.map_err(|e| format!("Error building caches on checkpoint state: {e:?}"))?;
let computed_state_root = weak_subj_state
weak_subj_state
.update_tree_hash_cache()
.map_err(|e| format!("Error computing checkpoint state root: {:?}", e))?;
if weak_subj_state_root != computed_state_root {
return Err(format!(
"Snapshot state root does not match block, expected: {:?}, got: {:?}",
weak_subj_state_root, computed_state_root
));
let latest_block_slot = weak_subj_state.latest_block_header().slot;
// We can only validate the block root if it exists in the state. We can't calculated it
// from the `latest_block_header` because the state root might be set to the zero hash.
if let Ok(state_slot_block_root) = weak_subj_state.get_block_root(latest_block_slot) {
if weak_subj_block_root != *state_slot_block_root {
return Err(format!(
"Snapshot state's most recent block root does not match block, expected: {:?}, got: {:?}",
weak_subj_block_root, state_slot_block_root
));
}
}
// Check that the checkpoint state is for the same network as the genesis state.
@@ -508,13 +504,12 @@ where
let fc_store = BeaconForkChoiceStore::get_forkchoice_store(store, &snapshot)
.map_err(|e| format!("Unable to initialize fork choice store: {e:?}"))?;
let current_slot = Some(snapshot.beacon_block.slot());
let fork_choice = ForkChoice::from_anchor(
fc_store,
snapshot.beacon_block_root,
&snapshot.beacon_block,
&snapshot.beacon_state,
current_slot,
Some(weak_subj_slot),
&self.spec,
)
.map_err(|e| format!("Unable to initialize ForkChoice: {:?}", e))?;
@@ -891,13 +886,10 @@ where
validator_monitor: RwLock::new(validator_monitor),
genesis_backfill_slot,
//TODO(sean) should we move kzg solely to the da checker?
data_availability_checker: DataAvailabilityChecker::new(
slot_clock,
kzg.clone(),
store,
self.spec,
)
.map_err(|e| format!("Error initializing DataAvailabiltyChecker: {:?}", e))?,
data_availability_checker: Arc::new(
DataAvailabilityChecker::new(slot_clock, kzg.clone(), store, self.spec)
.map_err(|e| format!("Error initializing DataAvailabiltyChecker: {:?}", e))?,
),
proposal_blob_cache: BlobCache::default(),
kzg,
};
beacon_node/beacon_chain/src/data_availability_checker.rs
+115 -38
View File
@@ -10,12 +10,14 @@ use kzg::Error as KzgError;
use kzg::Kzg;
use slog::{debug, error};
use slot_clock::SlotClock;
use ssz_types::{Error, VariableList};
use ssz_types::{Error, FixedVector, VariableList};
use state_processing::per_block_processing::deneb::deneb::verify_kzg_commitments_against_transactions;
use std::collections::HashSet;
use std::sync::Arc;
use strum::IntoStaticStr;
use task_executor::TaskExecutor;
use types::beacon_block_body::KzgCommitments;
use types::blob_sidecar::{BlobIdentifier, BlobSidecar};
use types::blob_sidecar::{BlobIdentifier, BlobSidecar, FixedBlobSidecarList};
use types::consts::deneb::MIN_EPOCHS_FOR_BLOB_SIDECARS_REQUESTS;
use types::ssz_tagged_signed_beacon_block;
use types::{
@@ -27,27 +29,29 @@ mod overflow_lru_cache;
pub const OVERFLOW_LRU_CAPACITY: usize = 1024;
#[derive(Debug)]
#[derive(Debug, IntoStaticStr)]
pub enum AvailabilityCheckError {
DuplicateBlob(Hash256),
Kzg(KzgError),
KzgVerificationFailed,
KzgNotInitialized,
KzgVerificationFailed,
SszTypes(ssz_types::Error),
MissingBlobs,
NumBlobsMismatch {
num_kzg_commitments: usize,
num_blobs: usize,
},
TxKzgCommitmentMismatch,
MissingBlobs,
TxKzgCommitmentMismatch(String),
KzgCommitmentMismatch {
blob_index: u64,
},
Pending,
IncorrectFork,
BlobIndexInvalid(u64),
StoreError(store::Error),
DecodeError(ssz::DecodeError),
BlockBlobRootMismatch {
block_root: Hash256,
blob_block_root: Hash256,
},
}
impl From<ssz_types::Error> for AvailabilityCheckError {
@@ -86,8 +90,7 @@ pub struct DataAvailabilityChecker<T: BeaconChainTypes> {
/// to "complete" the requirements for an `AvailableBlock`.
#[derive(Debug, PartialEq)]
pub enum Availability<T: EthSpec> {
PendingBlobs(Vec<BlobIdentifier>),
PendingBlock(Hash256),
MissingComponents(Hash256),
Available(Box<AvailableExecutedBlock<T>>),
}
@@ -119,6 +122,52 @@ impl<T: BeaconChainTypes> DataAvailabilityChecker<T> {
})
}
pub fn has_block(&self, block_root: &Hash256) -> bool {
self.availability_cache.has_block(block_root)
}
pub fn get_missing_blob_ids_checking_cache(
&self,
block_root: Hash256,
) -> Option<Vec<BlobIdentifier>> {
let (block, blob_indices) = self.availability_cache.get_missing_blob_info(block_root);
self.get_missing_blob_ids(block_root, block.as_ref(), Some(blob_indices))
}
/// A `None` indicates blobs are not required.
///
/// If there's no block, all possible ids will be returned that don't exist in the given blobs.
/// If there no blobs, all possible ids will be returned.
pub fn get_missing_blob_ids(
&self,
block_root: Hash256,
block_opt: Option<&Arc<SignedBeaconBlock<T::EthSpec>>>,
blobs_opt: Option<HashSet<usize>>,
) -> Option<Vec<BlobIdentifier>> {
let epoch = self.slot_clock.now()?.epoch(T::EthSpec::slots_per_epoch());
self.da_check_required(epoch).then(|| {
block_opt
.map(|block| {
block.get_filtered_blob_ids(Some(block_root), |i, _| {
blobs_opt.as_ref().map_or(true, |blobs| !blobs.contains(&i))
})
})
.unwrap_or_else(|| {
let mut blob_ids = Vec::with_capacity(T::EthSpec::max_blobs_per_block());
for i in 0..T::EthSpec::max_blobs_per_block() {
if blobs_opt.as_ref().map_or(true, |blobs| !blobs.contains(&i)) {
blob_ids.push(BlobIdentifier {
block_root,
index: i as u64,
});
}
}
blob_ids
})
})
}
/// Get a blob from the availability cache.
pub fn get_blob(
&self,
@@ -127,6 +176,23 @@ impl<T: BeaconChainTypes> DataAvailabilityChecker<T> {
self.availability_cache.peek_blob(blob_id)
}
pub fn put_rpc_blobs(
&self,
block_root: Hash256,
blobs: FixedBlobSidecarList<T::EthSpec>,
) -> Result<Availability<T::EthSpec>, AvailabilityCheckError> {
let mut verified_blobs = vec![];
if let Some(kzg) = self.kzg.as_ref() {
for blob in blobs.iter().flatten() {
verified_blobs.push(verify_kzg_for_blob(blob.clone(), kzg)?)
}
} else {
return Err(AvailabilityCheckError::KzgNotInitialized);
};
self.availability_cache
.put_kzg_verified_blobs(block_root, &verified_blobs)
}
/// This first validates the KZG commitments included in the blob sidecar.
/// Check if we've cached other blobs for this block. If it completes a set and we also
/// have a block cached, return the `Availability` variant triggering block import.
@@ -139,13 +205,13 @@ impl<T: BeaconChainTypes> DataAvailabilityChecker<T> {
) -> Result<Availability<T::EthSpec>, AvailabilityCheckError> {
// Verify the KZG commitments.
let kzg_verified_blob = if let Some(kzg) = self.kzg.as_ref() {
verify_kzg_for_blob(gossip_blob, kzg)?
verify_kzg_for_blob(gossip_blob.to_blob(), kzg)?
} else {
return Err(AvailabilityCheckError::KzgNotInitialized);
};
self.availability_cache
.put_kzg_verified_blob(kzg_verified_blob)
.put_kzg_verified_blobs(kzg_verified_blob.block_root(), &[kzg_verified_blob])
}
/// Check if we have all the blobs for a block. If we do, return the Availability variant that
@@ -171,7 +237,8 @@ impl<T: BeaconChainTypes> DataAvailabilityChecker<T> {
.kzg
.as_ref()
.ok_or(AvailabilityCheckError::KzgNotInitialized)?;
let verified_blobs = verify_kzg_for_blob_list(VariableList::new(blob_list)?, kzg)?;
let filtered_blobs = blob_list.iter().flatten().cloned().collect();
let verified_blobs = verify_kzg_for_blob_list(filtered_blobs, kzg)?;
Ok(MaybeAvailableBlock::Available(
self.check_availability_with_blobs(block, verified_blobs)?,
@@ -180,27 +247,6 @@ impl<T: BeaconChainTypes> DataAvailabilityChecker<T> {
}
}
/// Checks if a block is available, returning an error if the block is not immediately available.
/// Does not access the gossip cache.
pub fn try_check_availability(
&self,
block: BlockWrapper<T::EthSpec>,
) -> Result<AvailableBlock<T::EthSpec>, AvailabilityCheckError> {
match block {
BlockWrapper::Block(block) => {
let blob_requirements = self.get_blob_requirements(&block)?;
let blobs = match blob_requirements {
BlobRequirements::EmptyBlobs => VerifiedBlobs::EmptyBlobs,
BlobRequirements::NotRequired => VerifiedBlobs::NotRequired,
BlobRequirements::PreDeneb => VerifiedBlobs::PreDeneb,
BlobRequirements::Required => return Err(AvailabilityCheckError::MissingBlobs),
};
Ok(AvailableBlock { block, blobs })
}
BlockWrapper::BlockAndBlobs(_, _) => Err(AvailabilityCheckError::Pending),
}
}
/// Verifies a block against a set of KZG verified blobs. Returns an AvailableBlock if block's
/// commitments are consistent with the provided verified blob commitments.
pub fn check_availability_with_blobs(
@@ -254,9 +300,11 @@ impl<T: BeaconChainTypes> DataAvailabilityChecker<T> {
transactions,
block_kzg_commitments,
)
.map_err(|_| AvailabilityCheckError::TxKzgCommitmentMismatch)?;
.map_err(|e| AvailabilityCheckError::TxKzgCommitmentMismatch(format!("{e:?}")))?;
if !verified {
return Err(AvailabilityCheckError::TxKzgCommitmentMismatch);
return Err(AvailabilityCheckError::TxKzgCommitmentMismatch(
"a commitment and version didn't match".to_string(),
));
}
}
@@ -410,6 +458,27 @@ pub struct AvailabilityPendingBlock<E: EthSpec> {
block: Arc<SignedBeaconBlock<E>>,
}
impl<E: EthSpec> AvailabilityPendingBlock<E> {
pub fn slot(&self) -> Slot {
self.block.slot()
}
pub fn num_blobs_expected(&self) -> usize {
self.block.num_expected_blobs()
}
pub fn get_all_blob_ids(&self, block_root: Option<Hash256>) -> Vec<BlobIdentifier> {
self.block.get_expected_blob_ids(block_root)
}
pub fn get_filtered_blob_ids(
&self,
block_root: Option<Hash256>,
filter: impl Fn(usize, Hash256) -> bool,
) -> Vec<BlobIdentifier> {
self.block.get_filtered_blob_ids(block_root, filter)
}
}
impl<E: EthSpec> AvailabilityPendingBlock<E> {
pub fn to_block(self) -> Arc<SignedBeaconBlock<E>> {
self.block
@@ -429,7 +498,7 @@ impl<E: EthSpec> AvailabilityPendingBlock<E> {
}
/// Verifies an AvailabilityPendingBlock against a set of KZG verified blobs.
/// This does not check whether a block *should* have blobs, these checks should must have been
/// This does not check whether a block *should* have blobs, these checks should have been
/// completed when producing the `AvailabilityPendingBlock`.
pub fn make_available(
self,
@@ -485,6 +554,13 @@ impl<E: EthSpec> AvailableBlock<E> {
&self.block
}
pub fn da_check_required(&self) -> bool {
match self.blobs {
VerifiedBlobs::PreDeneb | VerifiedBlobs::NotRequired => false,
VerifiedBlobs::EmptyBlobs | VerifiedBlobs::Available(_) => true,
}
}
pub fn deconstruct(self) -> (Arc<SignedBeaconBlock<E>>, Option<BlobSidecarList<E>>) {
match self.blobs {
VerifiedBlobs::EmptyBlobs | VerifiedBlobs::NotRequired | VerifiedBlobs::PreDeneb => {
@@ -542,7 +618,8 @@ impl<E: EthSpec> AsBlock<E> for AvailableBlock<E> {
fn into_block_wrapper(self) -> BlockWrapper<E> {
let (block, blobs_opt) = self.deconstruct();
if let Some(blobs) = blobs_opt {
BlockWrapper::BlockAndBlobs(block, blobs.to_vec())
let blobs_vec = blobs.iter().cloned().map(Option::Some).collect::<Vec<_>>();
BlockWrapper::BlockAndBlobs(block, FixedVector::from(blobs_vec))
} else {
BlockWrapper::Block(block)
}
beacon_node/beacon_chain/src/data_availability_checker/overflow_lru_cache.rs
+163 -80
View File
@@ -11,7 +11,9 @@ use ssz_derive::{Decode, Encode};
use ssz_types::FixedVector;
use std::{collections::HashSet, sync::Arc};
use types::blob_sidecar::BlobIdentifier;
use types::{BlobSidecar, Epoch, EthSpec, Hash256};
use types::{BlobSidecar, Epoch, EthSpec, Hash256, SignedBeaconBlock};
type MissingBlobInfo<T> = (Option<Arc<SignedBeaconBlock<T>>>, HashSet<usize>);
/// Caches partially available blobs and execution verified blocks corresponding
/// to a given `block_hash` that are received over gossip.
@@ -25,10 +27,12 @@ pub struct PendingComponents<T: EthSpec> {
}
impl<T: EthSpec> PendingComponents<T> {
pub fn new_from_blob(blob: KzgVerifiedBlob<T>) -> Self {
pub fn new_from_blobs(blobs: &[KzgVerifiedBlob<T>]) -> Self {
let mut verified_blobs = FixedVector::<_, _>::default();
if let Some(mut_maybe_blob) = verified_blobs.get_mut(blob.blob_index() as usize) {
*mut_maybe_blob = Some(blob);
for blob in blobs {
if let Some(mut_maybe_blob) = verified_blobs.get_mut(blob.blob_index() as usize) {
*mut_maybe_blob = Some(blob.clone());
}
}
Self {
@@ -82,6 +86,20 @@ impl<T: EthSpec> PendingComponents<T> {
None
})
}
pub fn get_missing_blob_info(&self) -> MissingBlobInfo<T> {
let block_opt = self
.executed_block
.as_ref()
.map(|block| block.block.block.clone());
let blobs = self
.verified_blobs
.iter()
.enumerate()
.filter_map(|(i, maybe_blob)| maybe_blob.as_ref().map(|_| i))
.collect::<HashSet<_>>();
(block_opt, blobs)
}
}
#[derive(Debug, PartialEq)]
@@ -193,11 +211,27 @@ impl<T: BeaconChainTypes> OverflowStore<T> {
Ok(disk_keys)
}
pub fn load_block(
&self,
block_root: &Hash256,
) -> Result<Option<AvailabilityPendingExecutedBlock<T::EthSpec>>, AvailabilityCheckError> {
let key = OverflowKey::from_block_root(*block_root);
self.0
.hot_db
.get_bytes(DBColumn::OverflowLRUCache.as_str(), &key.as_ssz_bytes())?
.map(|block_bytes| {
AvailabilityPendingExecutedBlock::from_ssz_bytes(block_bytes.as_slice())
})
.transpose()
.map_err(|e| e.into())
}
pub fn load_blob(
&self,
blob_id: &BlobIdentifier,
) -> Result<Option<Arc<BlobSidecar<T::EthSpec>>>, AvailabilityCheckError> {
let key = OverflowKey::from_blob_id::<T::EthSpec>(blob_id.clone())?;
let key = OverflowKey::from_blob_id::<T::EthSpec>(*blob_id)?;
self.0
.hot_db
@@ -320,6 +354,41 @@ impl<T: BeaconChainTypes> OverflowLRUCache<T> {
})
}
pub fn has_block(&self, block_root: &Hash256) -> bool {
let read_lock = self.critical.read();
if read_lock
.in_memory
.peek(block_root)
.map_or(false, |cache| cache.executed_block.is_some())
{
true
} else if read_lock.store_keys.contains(block_root) {
drop(read_lock);
// If there's some kind of error reading from the store, we should just return false
self.overflow_store
.load_block(block_root)
.map_or(false, |maybe_block| maybe_block.is_some())
} else {
false
}
}
pub fn get_missing_blob_info(&self, block_root: Hash256) -> MissingBlobInfo<T::EthSpec> {
let read_lock = self.critical.read();
if let Some(cache) = read_lock.in_memory.peek(&block_root) {
cache.get_missing_blob_info()
} else if read_lock.store_keys.contains(&block_root) {
drop(read_lock);
// return default if there's an error reading from the store
match self.overflow_store.get_pending_components(block_root) {
Ok(Some(pending_components)) => pending_components.get_missing_blob_info(),
_ => Default::default(),
}
} else {
Default::default()
}
}
pub fn peek_blob(
&self,
blob_id: &BlobIdentifier,
@@ -335,27 +404,39 @@ impl<T: BeaconChainTypes> OverflowLRUCache<T> {
}
}
pub fn put_kzg_verified_blob(
pub fn put_kzg_verified_blobs(
&self,
kzg_verified_blob: KzgVerifiedBlob<T::EthSpec>,
block_root: Hash256,
kzg_verified_blobs: &[KzgVerifiedBlob<T::EthSpec>],
) -> Result<Availability<T::EthSpec>, AvailabilityCheckError> {
for blob in kzg_verified_blobs {
let blob_block_root = blob.block_root();
if blob_block_root != block_root {
return Err(AvailabilityCheckError::BlockBlobRootMismatch {
block_root,
blob_block_root,
});
}
}
let mut write_lock = self.critical.write();
let block_root = kzg_verified_blob.block_root();
let availability = if let Some(mut pending_components) =
write_lock.pop_pending_components(block_root, &self.overflow_store)?
{
let blob_index = kzg_verified_blob.blob_index();
*pending_components
.verified_blobs
.get_mut(blob_index as usize)
.ok_or(AvailabilityCheckError::BlobIndexInvalid(blob_index))? =
Some(kzg_verified_blob);
for kzg_verified_blob in kzg_verified_blobs {
let blob_index = kzg_verified_blob.blob_index() as usize;
if let Some(maybe_verified_blob) =
pending_components.verified_blobs.get_mut(blob_index)
{
*maybe_verified_blob = Some(kzg_verified_blob.clone())
} else {
return Err(AvailabilityCheckError::BlobIndexInvalid(blob_index as u64));
}
}
if let Some(executed_block) = pending_components.executed_block.take() {
self.check_block_availability_maybe_cache(
write_lock,
block_root,
pending_components,
executed_block,
)?
@@ -365,17 +446,17 @@ impl<T: BeaconChainTypes> OverflowLRUCache<T> {
pending_components,
&self.overflow_store,
)?;
Availability::PendingBlock(block_root)
Availability::MissingComponents(block_root)
}
} else {
// not in memory or store -> put new in memory
let new_pending_components = PendingComponents::new_from_blob(kzg_verified_blob);
let new_pending_components = PendingComponents::new_from_blobs(kzg_verified_blobs);
write_lock.put_pending_components(
block_root,
new_pending_components,
&self.overflow_store,
)?;
Availability::PendingBlock(block_root)
Availability::MissingComponents(block_root)
};
Ok(availability)
@@ -394,7 +475,6 @@ impl<T: BeaconChainTypes> OverflowLRUCache<T> {
match write_lock.pop_pending_components(block_root, &self.overflow_store)? {
Some(pending_components) => self.check_block_availability_maybe_cache(
write_lock,
block_root,
pending_components,
executed_block,
)?,
@@ -422,7 +502,7 @@ impl<T: BeaconChainTypes> OverflowLRUCache<T> {
new_pending_components,
&self.overflow_store,
)?;
Availability::PendingBlobs(all_blob_ids)
Availability::MissingComponents(block_root)
}
};
@@ -435,11 +515,10 @@ impl<T: BeaconChainTypes> OverflowLRUCache<T> {
/// Returns an error if there was an error when matching the block commitments against blob commitments.
///
/// Returns `Ok(Availability::Available(_))` if all blobs for the block are present in cache.
/// Returns `Ok(Availability::PendingBlobs(_))` if all corresponding blobs have not been received in the cache.
/// Returns `Ok(Availability::MissingComponents(_))` if all corresponding blobs have not been received in the cache.
fn check_block_availability_maybe_cache(
&self,
mut write_lock: RwLockWriteGuard<Critical<T>>,
block_root: Hash256,
mut pending_components: PendingComponents<T::EthSpec>,
executed_block: AvailabilityPendingExecutedBlock<T::EthSpec>,
) -> Result<Availability<T::EthSpec>, AvailabilityCheckError> {
@@ -451,11 +530,12 @@ impl<T: BeaconChainTypes> OverflowLRUCache<T> {
payload_verification_outcome,
} = executed_block;
let verified_blobs = Vec::from(pending_components.verified_blobs)
let Some(verified_blobs) = Vec::from(pending_components.verified_blobs)
.into_iter()
.take(num_blobs_expected)
.map(|maybe_blob| maybe_blob.ok_or(AvailabilityCheckError::MissingBlobs))
.collect::<Result<Vec<_>, _>>()?;
.collect::<Option<Vec<_>>>() else {
return Ok(Availability::MissingComponents(import_data.block_root))
};
let available_block = block.make_available(verified_blobs)?;
Ok(Availability::Available(Box::new(
@@ -466,14 +546,7 @@ impl<T: BeaconChainTypes> OverflowLRUCache<T> {
),
)))
} else {
let missing_blob_ids = executed_block.get_filtered_blob_ids(|index| {
pending_components
.verified_blobs
.get(index as usize)
.map(|maybe_blob| maybe_blob.is_none())
.unwrap_or(true)
});
let block_root = executed_block.import_data.block_root;
let _ = pending_components.executed_block.insert(executed_block);
write_lock.put_pending_components(
block_root,
@@ -481,7 +554,7 @@ impl<T: BeaconChainTypes> OverflowLRUCache<T> {
&self.overflow_store,
)?;
Ok(Availability::PendingBlobs(missing_blob_ids))
Ok(Availability::MissingComponents(block_root))
}
}
@@ -1080,7 +1153,7 @@ mod test {
);
} else {
assert!(
matches!(availability, Availability::PendingBlobs(_)),
matches!(availability, Availability::MissingComponents(_)),
"should be pending blobs"
);
assert_eq!(
@@ -1100,16 +1173,18 @@ mod test {
.as_ref()
.cloned()
.expect("kzg should exist");
let mut kzg_verified_blobs = Vec::new();
for (blob_index, gossip_blob) in blobs.into_iter().enumerate() {
let kzg_verified_blob =
verify_kzg_for_blob(gossip_blob, kzg.as_ref()).expect("kzg should verify");
let kzg_verified_blob = verify_kzg_for_blob(gossip_blob.to_blob(), kzg.as_ref())
.expect("kzg should verify");
kzg_verified_blobs.push(kzg_verified_blob);
let availability = cache
.put_kzg_verified_blob(kzg_verified_blob)
.put_kzg_verified_blobs(root, kzg_verified_blobs.as_slice())
.expect("should put blob");
if blob_index == blobs_expected - 1 {
assert!(matches!(availability, Availability::Available(_)));
} else {
assert!(matches!(availability, Availability::PendingBlobs(_)));
assert!(matches!(availability, Availability::MissingComponents(_)));
assert_eq!(cache.critical.read().in_memory.len(), 1);
}
}
@@ -1126,15 +1201,17 @@ mod test {
"should have expected number of blobs"
);
let root = pending_block.import_data.block_root;
let mut kzg_verified_blobs = vec![];
for gossip_blob in blobs {
let kzg_verified_blob =
verify_kzg_for_blob(gossip_blob, kzg.as_ref()).expect("kzg should verify");
let kzg_verified_blob = verify_kzg_for_blob(gossip_blob.to_blob(), kzg.as_ref())
.expect("kzg should verify");
kzg_verified_blobs.push(kzg_verified_blob);
let availability = cache
.put_kzg_verified_blob(kzg_verified_blob)
.put_kzg_verified_blobs(root, kzg_verified_blobs.as_slice())
.expect("should put blob");
assert_eq!(
availability,
Availability::PendingBlock(root),
Availability::MissingComponents(root),
"should be pending block"
);
assert_eq!(cache.critical.read().in_memory.len(), 1);
@@ -1270,11 +1347,13 @@ mod test {
let blobs_0 = pending_blobs.pop_front().expect("should have blobs");
let expected_blobs = blobs_0.len();
let mut kzg_verified_blobs = vec![];
for (blob_index, gossip_blob) in blobs_0.into_iter().enumerate() {
let kzg_verified_blob =
verify_kzg_for_blob(gossip_blob, kzg.as_ref()).expect("kzg should verify");
let kzg_verified_blob = verify_kzg_for_blob(gossip_blob.to_blob(), kzg.as_ref())
.expect("kzg should verify");
kzg_verified_blobs.push(kzg_verified_blob);
let availability = cache
.put_kzg_verified_blob(kzg_verified_blob)
.put_kzg_verified_blobs(roots[0], kzg_verified_blobs.as_slice())
.expect("should put blob");
if blob_index == expected_blobs - 1 {
assert!(matches!(availability, Availability::Available(_)));
@@ -1284,7 +1363,7 @@ mod test {
cache.critical.read().in_memory.peek(&roots[0]).is_some(),
"first block should be in memory"
);
assert!(matches!(availability, Availability::PendingBlobs(_)));
assert!(matches!(availability, Availability::MissingComponents(_)));
}
}
assert_eq!(
@@ -1360,13 +1439,17 @@ mod test {
for _ in 0..(n_epochs * capacity) {
let pending_block = pending_blocks.pop_front().expect("should have block");
let mut pending_block_blobs = pending_blobs.pop_front().expect("should have blobs");
let block_root = pending_block.block.as_block().canonical_root();
let expected_blobs = pending_block.num_blobs_expected();
if expected_blobs > 1 {
// might as well add a blob too
let mut pending_blobs = pending_blobs.pop_front().expect("should have blobs");
let one_blob = pending_blobs.pop().expect("should have at least one blob");
let kzg_verified_blob =
verify_kzg_for_blob(one_blob, kzg.as_ref()).expect("kzg should verify");
let one_blob = pending_block_blobs
.pop()
.expect("should have at least one blob");
let kzg_verified_blob = verify_kzg_for_blob(one_blob.to_blob(), kzg.as_ref())
.expect("kzg should verify");
let kzg_verified_blobs = vec![kzg_verified_blob];
// generate random boolean
let block_first = (rand::random::<usize>() % 2) == 0;
if block_first {
@@ -1374,43 +1457,41 @@ mod test {
.put_pending_executed_block(pending_block)
.expect("should put block");
assert!(
matches!(availability, Availability::PendingBlobs(_)),
matches!(availability, Availability::MissingComponents(_)),
"should have pending blobs"
);
let availability = cache
.put_kzg_verified_blob(kzg_verified_blob)
.put_kzg_verified_blobs(block_root, kzg_verified_blobs.as_slice())
.expect("should put blob");
assert!(
matches!(availability, Availability::PendingBlobs(_)),
matches!(availability, Availability::MissingComponents(_)),
"availabilty should be pending blobs: {:?}",
availability
);
} else {
let availability = cache
.put_kzg_verified_blob(kzg_verified_blob)
.put_kzg_verified_blobs(block_root, kzg_verified_blobs.as_slice())
.expect("should put blob");
let root = pending_block.block.as_block().canonical_root();
assert_eq!(
availability,
Availability::PendingBlock(root),
Availability::MissingComponents(root),
"should be pending block"
);
let availability = cache
.put_pending_executed_block(pending_block)
.expect("should put block");
assert!(
matches!(availability, Availability::PendingBlobs(_)),
matches!(availability, Availability::MissingComponents(_)),
"should have pending blobs"
);
}
} else {
// still need to pop front so the blob count is correct
pending_blobs.pop_front().expect("should have blobs");
let availability = cache
.put_pending_executed_block(pending_block)
.expect("should put block");
assert!(
matches!(availability, Availability::PendingBlobs(_)),
matches!(availability, Availability::MissingComponents(_)),
"should be pending blobs"
);
}
@@ -1511,63 +1592,63 @@ mod test {
let mut remaining_blobs = HashMap::new();
for _ in 0..(n_epochs * capacity) {
let pending_block = pending_blocks.pop_front().expect("should have block");
let mut pending_block_blobs = pending_blobs.pop_front().expect("should have blobs");
let block_root = pending_block.block.as_block().canonical_root();
let expected_blobs = pending_block.num_blobs_expected();
if expected_blobs > 1 {
// might as well add a blob too
let mut pending_blobs = pending_blobs.pop_front().expect("should have blobs");
let one_blob = pending_blobs.pop().expect("should have at least one blob");
let kzg_verified_blob =
verify_kzg_for_blob(one_blob, kzg.as_ref()).expect("kzg should verify");
let one_blob = pending_block_blobs
.pop()
.expect("should have at least one blob");
let kzg_verified_blob = verify_kzg_for_blob(one_blob.to_blob(), kzg.as_ref())
.expect("kzg should verify");
let kzg_verified_blobs = vec![kzg_verified_blob];
// generate random boolean
let block_first = (rand::random::<usize>() % 2) == 0;
remaining_blobs.insert(block_root, pending_blobs);
if block_first {
let availability = cache
.put_pending_executed_block(pending_block)
.expect("should put block");
assert!(
matches!(availability, Availability::PendingBlobs(_)),
matches!(availability, Availability::MissingComponents(_)),
"should have pending blobs"
);
let availability = cache
.put_kzg_verified_blob(kzg_verified_blob)
.put_kzg_verified_blobs(block_root, kzg_verified_blobs.as_slice())
.expect("should put blob");
assert!(
matches!(availability, Availability::PendingBlobs(_)),
matches!(availability, Availability::MissingComponents(_)),
"availabilty should be pending blobs: {:?}",
availability
);
} else {
let availability = cache
.put_kzg_verified_blob(kzg_verified_blob)
.put_kzg_verified_blobs(block_root, kzg_verified_blobs.as_slice())
.expect("should put blob");
let root = pending_block.block.as_block().canonical_root();
assert_eq!(
availability,
Availability::PendingBlock(root),
Availability::MissingComponents(root),
"should be pending block"
);
let availability = cache
.put_pending_executed_block(pending_block)
.expect("should put block");
assert!(
matches!(availability, Availability::PendingBlobs(_)),
matches!(availability, Availability::MissingComponents(_)),
"should have pending blobs"
);
}
} else {
// still need to pop front so the blob count is correct
let pending_blobs = pending_blobs.pop_front().expect("should have blobs");
remaining_blobs.insert(block_root, pending_blobs);
let availability = cache
.put_pending_executed_block(pending_block)
.expect("should put block");
assert!(
matches!(availability, Availability::PendingBlobs(_)),
matches!(availability, Availability::MissingComponents(_)),
"should be pending blobs"
);
}
remaining_blobs.insert(block_root, pending_block_blobs);
}
// now we should have a full cache spanning multiple epochs
@@ -1626,18 +1707,20 @@ mod test {
);
// now lets insert the remaining blobs until the cache is empty
for (_, blobs) in remaining_blobs {
for (root, blobs) in remaining_blobs {
let additional_blobs = blobs.len();
let mut kzg_verified_blobs = vec![];
for (i, gossip_blob) in blobs.into_iter().enumerate() {
let kzg_verified_blob =
verify_kzg_for_blob(gossip_blob, kzg.as_ref()).expect("kzg should verify");
let kzg_verified_blob = verify_kzg_for_blob(gossip_blob.to_blob(), kzg.as_ref())
.expect("kzg should verify");
kzg_verified_blobs.push(kzg_verified_blob);
let availability = recovered_cache
.put_kzg_verified_blob(kzg_verified_blob)
.put_kzg_verified_blobs(root, kzg_verified_blobs.as_slice())
.expect("should put blob");
if i == additional_blobs - 1 {
assert!(matches!(availability, Availability::Available(_)))
} else {
assert!(matches!(availability, Availability::PendingBlobs(_)));
assert!(matches!(availability, Availability::MissingComponents(_)));
}
}
}
beacon_node/beacon_chain/src/lib.rs
+3 -1
View File
@@ -69,7 +69,9 @@ pub use self::historical_blocks::HistoricalBlockError;
pub use attestation_verification::Error as AttestationError;
pub use beacon_fork_choice_store::{BeaconForkChoiceStore, Error as ForkChoiceStoreError};
pub use block_verification::{
get_block_root, BlockError, ExecutedBlock, ExecutionPayloadError, GossipVerifiedBlock,
get_block_root, AvailabilityPendingExecutedBlock, BlockError, ExecutedBlock,
ExecutionPayloadError, GossipVerifiedBlock, IntoExecutionPendingBlock,
PayloadVerificationOutcome, PayloadVerificationStatus,
};
pub use canonical_head::{CachedHead, CanonicalHead, CanonicalHeadRwLock};
pub use eth1_chain::{Eth1Chain, Eth1ChainBackend};
beacon_node/beacon_chain/src/observed_blob_sidecars.rs
+1 -1
View File
@@ -379,7 +379,7 @@ mod tests {
// Try adding an out of bounds index
let invalid_index = E::max_blobs_per_block() as u64;
let sidecar_d = get_blob_sidecar(0, block_root_a, 4);
let sidecar_d = get_blob_sidecar(0, block_root_a, invalid_index);
assert_eq!(
cache.observe_sidecar(&sidecar_d),
Err(Error::InvalidBlobIndex(invalid_index)),
beacon_node/beacon_chain/src/test_utils.rs
+1 -1
View File
@@ -63,7 +63,7 @@ use types::{typenum::U4294967296, *};
// 4th September 2019
pub const HARNESS_GENESIS_TIME: u64 = 1_567_552_690;
// Environment variable to read if `fork_from_env` feature is enabled.
const FORK_NAME_ENV_VAR: &str = "FORK_NAME";
pub const FORK_NAME_ENV_VAR: &str = "FORK_NAME";
// Default target aggregators to set during testing, this ensures an aggregator at each slot.
//
beacon_node/beacon_chain/tests/attestation_production.rs
+12 -7
View File
@@ -133,10 +133,13 @@ async fn produces_attestations() {
assert_eq!(data.target.root, target_root, "bad target root");
let block_wrapper: BlockWrapper<MainnetEthSpec> = Arc::new(block.clone()).into();
let available_block = chain
let beacon_chain::blob_verification::MaybeAvailableBlock::Available(available_block) = chain
.data_availability_checker
.try_check_availability(block_wrapper)
.unwrap();
.check_availability(block_wrapper)
.unwrap()
else {
panic!("block should be available")
};
let early_attestation = {
let proto_block = chain
@@ -200,11 +203,13 @@ async fn early_attester_cache_old_request() {
.unwrap();
let block_wrapper: BlockWrapper<MainnetEthSpec> = head.beacon_block.clone().into();
let available_block = harness
.chain
let beacon_chain::blob_verification::MaybeAvailableBlock::Available(available_block) = harness.chain
.data_availability_checker
.try_check_availability(block_wrapper)
.unwrap();
.check_availability(block_wrapper)
.unwrap()
else {
panic!("block should be available")
};
harness
.chain