cerc-io/lighthouse
11
0
Fork 0
Code Issues 1 Pull Requests Packages Projects Releases Wiki Activity

Devnet6 (#4404)

Browse Source 
* some blob reprocessing work

* remove ForceBlockLookup

* reorder enum match arms in sync manager

* a lot more reprocessing work

* impl logic for triggerng blob lookups along with block lookups

* deal with rpc blobs in groups per block in the da checker. don't cache missing blob ids in the da checker.

* make single block lookup generic

* more work

* add delayed processing logic and combine some requests

* start fixing some compile errors

* fix compilation in main block lookup mod

* much work

* get things compiling

* parent blob lookups

* fix compile

* revert red/stevie changes

* fix up sync manager delay message logic

* add peer usefulness enum

* should remove lookup refactor

* consolidate retry error handling

* improve peer scoring during certain failures in parent lookups

* improve retry code

* drop parent lookup if either req has a peer disconnect during download

* refactor single block processed method

* processing peer refactor

* smol bugfix

* fix some todos

* fix lints

* fix lints

* fix compile in lookup tests

* fix lints

* fix lints

* fix existing block lookup tests

* renamings

* fix after merge

* cargo fmt

* compilation fix in beacon chain tests

* fix

* refactor lookup tests to work with multiple forks and response types

* make tests into macros

* wrap availability check error

* fix compile after merge

* add random blobs

* start fixing up lookup verify error handling

* some bug fixes and the start of deneb only tests

* make tests work for all forks

* track information about peer source

* error refactoring

* improve peer scoring

* fix test compilation

* make sure blobs are sent for processing after stream termination, delete copied tests

* add some tests and fix a bug

* smol bugfixes and moar tests

* add tests and fix some things

* compile after merge

* lots of refactoring

* retry on invalid block/blob

* merge unknown parent messages before current slot lookup

* get tests compiling

* penalize blob peer on invalid blobs

* Check disk on in-memory cache miss

* Update beacon_node/beacon_chain/src/data_availability_checker/overflow_lru_cache.rs

* Update beacon_node/network/src/sync/network_context.rs

Co-authored-by: Divma <26765164+divagant-martian@users.noreply.github.com>

* fix bug in matching blocks and blobs in range sync

* pr feedback

* fix conflicts

* upgrade logs from warn to crit when we receive incorrect response in range

* synced_and_connected_within_tolerance -> should_search_for_block

* remove todo

* add data gas used and update excess data gas to u64

* Fix Broken Overflow Tests

* payload verification with commitments

* fix merge conflicts

* restore payload file

* Restore payload file

* remove todo

* add max blob commitments per block

* c-kzg lib update

* Fix ef tests

* Abstract over minimal/mainnet spec in kzg crate

* Start integrating new KZG

* checkpoint sync without alignment

* checkpoint sync without alignment

* add import

* add import

* query for checkpoint state by slot rather than state root (teku doesn't serve by state root)

* query for checkpoint state by slot rather than state root (teku doesn't serve by state root)

* loosen check

* get state first and query by most recent block root

* Revert "loosen check"

This reverts commit 069d13dd63aa794a3505db9f17bd1a6b73f0be81.

* get state first and query by most recent block root

* merge max blobs change

* simplify delay logic

* rename unknown parent sync message variants

* rename parameter, block_slot -> slot

* add some docs to the lookup module

* use interval instead of sleep

* drop request if blocks and blobs requests both return `None` for `Id`

* clean up `find_single_lookup` logic

* add lookup source enum

* clean up `find_single_lookup` logic

* add docs to find_single_lookup_request

* move LookupSource our of param where unnecessary

* remove unnecessary todo

* query for block by `state.latest_block_header.slot`

* fix lint

* fix merge transition ef tests

* fix test

* fix test

* fix observed  blob sidecars test

* Add some metrics (#33)

* fix protocol limits for blobs by root

* Update Engine API for 1:1 Structure Method

* make beacon chain tests to fix devnet 6 changes

* get ckzg working and fix some tests

* fix remaining tests

* fix lints

* Fix KZG linking issues

* remove unused dep

* lockfile

* test fixes

* remove dbgs

* remove unwrap

* cleanup tx generator

* small fixes

* fixing fixes

* more self reivew

* more self review

* refactor genesis header initialization

* refactor mock el instantiations

* fix compile

* fix network test, make sure they run for each fork

* pr feedback

* fix last test (hopefully)

---------

Co-authored-by: Pawan Dhananjay <pawandhananjay@gmail.com>
Co-authored-by: Mark Mackey <mark@sigmaprime.io>
Co-authored-by: Divma <26765164+divagant-martian@users.noreply.github.com>
Co-authored-by: Michael Sproul <michael@sigmaprime.io>
This commit is contained in:
realbigsean
2023-06-29 15:35:43 -04:00
committed by GitHub
co-authored by Divma Pawan Dhananjay Mark Mackey Michael Sproul
parent 4c9fcf1e83
commit adbb62f7f3
69 changed files with 2114 additions and 1338 deletions
Download Patch File Download Diff File Expand all files Collapse all files
beacon_node/beacon_chain/src/beacon_chain.rs
+13 -34
View File
@@ -114,9 +114,8 @@ use store::{
use task_executor::{ShutdownReason, TaskExecutor};
use tokio_stream::Stream;
use tree_hash::TreeHash;
use types::beacon_block_body::KzgCommitments;
use types::beacon_state::CloneConfig;
use types::blob_sidecar::{BlobSidecarList, Blobs};
use types::blob_sidecar::BlobSidecarList;
use types::consts::deneb::MIN_EPOCHS_FOR_BLOB_SIDECARS_REQUESTS;
use types::*;
@@ -466,7 +465,7 @@ pub struct BeaconChain<T: BeaconChainTypes> {
pub genesis_backfill_slot: Slot,
pub proposal_blob_cache: BlobCache<T::EthSpec>,
pub data_availability_checker: Arc<DataAvailabilityChecker<T>>,
pub kzg: Option<Arc<Kzg>>,
pub kzg: Option<Arc<Kzg<<T::EthSpec as EthSpec>::Kzg>>>,
}
type BeaconBlockAndState<T, Payload> = (BeaconBlock<T, Payload>, BeaconState<T>);
@@ -2914,7 +2913,14 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
) -> Result<AvailabilityProcessingStatus, BlockError<T::EthSpec>> {
let availability = cache_fn(self.clone())?;
match availability {
Availability::Available(block) => self.import_available_block(block).await,
Availability::Available(block) => {
// This is the time since start of the slot where all the components of the block have become available
let delay =
get_slot_delay_ms(timestamp_now(), block.block.slot(), &self.slot_clock);
metrics::observe_duration(&metrics::BLOCK_AVAILABILITY_DELAY, delay);
// Block is fully available, import into fork choice
self.import_available_block(block).await
}
Availability::MissingComponents(block_root) => Ok(
AvailabilityProcessingStatus::MissingComponents(slot, block_root),
),
@@ -4903,7 +4909,7 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
//FIXME(sean)
// - add a new timer for processing here
if let Some(blobs) = blobs_opt {
if let (Some(blobs), Some(proofs)) = (blobs_opt, proofs_opt) {
let kzg = self
.kzg
.as_ref()
@@ -4924,14 +4930,10 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
)));
}
let kzg_proofs = if let Some(proofs) = proofs_opt {
Vec::from(proofs)
} else {
Self::compute_blob_kzg_proofs(kzg, &blobs, expected_kzg_commitments, slot)?
};
let kzg_proofs = Vec::from(proofs);
kzg_utils::validate_blobs::<T::EthSpec>(
kzg,
kzg.as_ref(),
expected_kzg_commitments,
&blobs,
&kzg_proofs,
@@ -4982,29 +4984,6 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
Ok((block, state))
}
fn compute_blob_kzg_proofs(
kzg: &Arc<Kzg>,
blobs: &Blobs<T::EthSpec>,
expected_kzg_commitments: &KzgCommitments<T::EthSpec>,
slot: Slot,
) -> Result<Vec<KzgProof>, BlockProductionError> {
blobs
.iter()
.enumerate()
.map(|(blob_index, blob)| {
let kzg_commitment = expected_kzg_commitments.get(blob_index).ok_or(
BlockProductionError::MissingKzgCommitment(format!(
"Missing KZG commitment for slot {} blob index {}",
slot, blob_index
)),
)?;
kzg_utils::compute_blob_kzg_proof::<T::EthSpec>(kzg, blob, *kzg_commitment)
.map_err(BlockProductionError::KzgError)
})
.collect::<Result<Vec<KzgProof>, BlockProductionError>>()
}
/// This method must be called whenever an execution engine indicates that a payload is
/// invalid.
///
beacon_node/beacon_chain/src/blob_verification.rs
+75 -28
View File
@@ -4,8 +4,8 @@ use state_processing::state_advance::partial_state_advance;
use std::sync::Arc;
use crate::beacon_chain::{
BeaconChain, BeaconChainTypes, MAXIMUM_GOSSIP_CLOCK_DISPARITY,
VALIDATOR_PUBKEY_CACHE_LOCK_TIMEOUT,
BeaconChain, BeaconChainTypes, BLOCK_PROCESSING_CACHE_LOCK_TIMEOUT,
MAXIMUM_GOSSIP_CLOCK_DISPARITY, VALIDATOR_PUBKEY_CACHE_LOCK_TIMEOUT,
};
use crate::data_availability_checker::{
AvailabilityCheckError, AvailabilityPendingBlock, AvailableBlock,
@@ -20,9 +20,9 @@ use ssz_types::FixedVector;
use std::borrow::Cow;
use types::blob_sidecar::{BlobIdentifier, FixedBlobSidecarList};
use types::{
BeaconBlockRef, BeaconState, BeaconStateError, BlobSidecar, ChainSpec, CloneConfig, Epoch,
EthSpec, FullPayload, Hash256, KzgCommitment, RelativeEpoch, SignedBeaconBlock,
SignedBeaconBlockHeader, SignedBlobSidecar, Slot,
BeaconBlockRef, BeaconState, BeaconStateError, BlobSidecar, BlobSidecarList, ChainSpec,
CloneConfig, Epoch, EthSpec, FullPayload, Hash256, KzgCommitment, RelativeEpoch,
SignedBeaconBlock, SignedBeaconBlockHeader, SignedBlobSidecar, Slot,
};
#[derive(Debug)]
@@ -240,36 +240,72 @@ pub fn validate_blob_sidecar_for_gossip<T: BeaconChainTypes>(
"block_root" => %block_root,
"index" => %blob_index,
);
// The cached head state is in the same epoch as the blob or the state has already been
// advanced to the blob's epoch
let snapshot = &chain.canonical_head.cached_head().snapshot;
if snapshot.beacon_state.current_epoch() == blob_slot.epoch(T::EthSpec::slots_per_epoch()) {
(
snapshot
.beacon_state
.get_beacon_proposer_index(blob_slot, &chain.spec)?,
snapshot.beacon_state.fork(),
)
if let Some(mut snapshot) = chain
.snapshot_cache
.try_read_for(BLOCK_PROCESSING_CACHE_LOCK_TIMEOUT)
.and_then(|snapshot_cache| {
snapshot_cache.get_cloned(block_parent_root, CloneConfig::committee_caches_only())
})
{
if snapshot.beacon_state.slot() == blob_slot {
debug!(
chain.log,
"Cloning snapshot cache state for blob verification";
"block_root" => %block_root,
"index" => %blob_index,
);
(
snapshot
.beacon_state
.get_beacon_proposer_index(blob_slot, &chain.spec)?,
snapshot.beacon_state.fork(),
)
} else {
debug!(
chain.log,
"Cloning and advancing snapshot cache state for blob verification";
"block_root" => %block_root,
"index" => %blob_index,
);
let state = cheap_state_advance_to_obtain_committees(
&mut snapshot.beacon_state,
Some(snapshot.beacon_block_root),
blob_slot,
&chain.spec,
)?;
(
state.get_beacon_proposer_index(blob_slot, &chain.spec)?,
state.fork(),
)
}
}
// Need to advance the state to get the proposer index
else {
// Reaching this condition too often might be an issue since we could theoretically have
// 5 threads (4 blob indices + 1 block) cloning the state.
// We shouldn't be seeing this condition a lot because we try to advance the state
// 3 seconds before the start of a slot. However, if this becomes an issue during testing, we should
// consider sending a blob for reprocessing to reduce the number of state clones.
warn!(
chain.log,
"Cached head not advanced for blob verification";
"Snapshot cache miss for blob verification";
"block_root" => %block_root,
"index" => %blob_index,
"action" => "contact the devs if you see this msg too often"
);
// The state produced is only valid for determining proposer/attester shuffling indices.
let mut cloned_state = snapshot.clone_with(CloneConfig::committee_caches_only());
let parent_block = chain
.get_blinded_block(&block_parent_root)
.map_err(BlobError::BeaconChainError)?
.ok_or_else(|| {
BlobError::from(BeaconChainError::MissingBeaconBlock(block_parent_root))
})?;
let mut parent_state = chain
.get_state(&parent_block.state_root(), Some(parent_block.slot()))?
.ok_or_else(|| {
BeaconChainError::DBInconsistent(format!(
"Missing state {:?}",
parent_block.state_root()
))
})?;
let state = cheap_state_advance_to_obtain_committees(
&mut cloned_state.beacon_state,
None,
&mut parent_state,
Some(parent_block.state_root()),
blob_slot,
&chain.spec,
)?;
@@ -449,8 +485,9 @@ impl<T: EthSpec> KzgVerifiedBlob<T> {
/// Returns an error if the kzg verification check fails.
pub fn verify_kzg_for_blob<T: EthSpec>(
blob: Arc<BlobSidecar<T>>,
kzg: &Kzg,
kzg: &Kzg<T::Kzg>,
) -> Result<KzgVerifiedBlob<T>, AvailabilityCheckError> {
let _timer = crate::metrics::start_timer(&crate::metrics::KZG_VERIFICATION_SINGLE_TIMES);
//TODO(sean) remove clone
if validate_blob::<T>(kzg, blob.blob.clone(), blob.kzg_commitment, blob.kzg_proof)
.map_err(AvailabilityCheckError::Kzg)?
@@ -468,8 +505,9 @@ pub fn verify_kzg_for_blob<T: EthSpec>(
/// in a loop since this function kzg verifies a list of blobs more efficiently.
pub fn verify_kzg_for_blob_list<T: EthSpec>(
blob_list: Vec<Arc<BlobSidecar<T>>>,
kzg: &Kzg,
kzg: &Kzg<T::Kzg>,
) -> Result<KzgVerifiedBlobList<T>, AvailabilityCheckError> {
let _timer = crate::metrics::start_timer(&crate::metrics::KZG_VERIFICATION_BATCH_TIMES);
let (blobs, (commitments, proofs)): (Vec<_>, (Vec<_>, Vec<_>)) = blob_list
.clone()
.into_iter()
@@ -612,6 +650,15 @@ pub enum BlockWrapper<E: EthSpec> {
}
impl<E: EthSpec> BlockWrapper<E> {
pub fn new(block: Arc<SignedBeaconBlock<E>>, blobs: Option<BlobSidecarList<E>>) -> Self {
match blobs {
Some(blobs) => {
let blobs = FixedVector::from(blobs.into_iter().map(Some).collect::<Vec<_>>());
BlockWrapper::BlockAndBlobs(block, blobs)
}
None => BlockWrapper::Block(block),
}
}
pub fn deconstruct(self) -> (Arc<SignedBeaconBlock<E>>, Option<FixedBlobSidecarList<E>>) {
match self {
BlockWrapper::Block(block) => (block, None),
beacon_node/beacon_chain/src/data_availability_checker.rs
+14 -30
View File
@@ -11,7 +11,6 @@ use kzg::Kzg;
use slog::{debug, error};
use slot_clock::SlotClock;
use ssz_types::{Error, FixedVector, VariableList};
use state_processing::per_block_processing::deneb::deneb::verify_kzg_commitments_against_transactions;
use std::collections::HashSet;
use std::sync::Arc;
use strum::IntoStaticStr;
@@ -21,7 +20,7 @@ use types::blob_sidecar::{BlobIdentifier, BlobSidecar, FixedBlobSidecarList};
use types::consts::deneb::MIN_EPOCHS_FOR_BLOB_SIDECARS_REQUESTS;
use types::ssz_tagged_signed_beacon_block;
use types::{
BeaconBlockRef, BlobSidecarList, ChainSpec, Epoch, EthSpec, ExecPayload, FullPayload, Hash256,
BeaconBlockRef, BlobSidecarList, ChainSpec, Epoch, EthSpec, FullPayload, Hash256,
SignedBeaconBlock, SignedBeaconBlockHeader, Slot,
};
@@ -85,7 +84,7 @@ impl From<ssz::DecodeError> for AvailabilityCheckError {
pub struct DataAvailabilityChecker<T: BeaconChainTypes> {
availability_cache: Arc<OverflowLRUCache<T>>,
slot_clock: T::SlotClock,
kzg: Option<Arc<Kzg>>,
kzg: Option<Arc<Kzg<<T::EthSpec as EthSpec>::Kzg>>>,
spec: ChainSpec,
}
@@ -114,7 +113,7 @@ impl<T: EthSpec> Availability<T> {
impl<T: BeaconChainTypes> DataAvailabilityChecker<T> {
pub fn new(
slot_clock: T::SlotClock,
kzg: Option<Arc<Kzg>>,
kzg: Option<Arc<Kzg<<T::EthSpec as EthSpec>::Kzg>>>,
store: BeaconStore<T>,
spec: ChainSpec,
) -> Result<Self, AvailabilityCheckError> {
@@ -296,35 +295,20 @@ impl<T: BeaconChainTypes> DataAvailabilityChecker<T> {
&self,
block: &Arc<SignedBeaconBlock<T::EthSpec, FullPayload<T::EthSpec>>>,
) -> Result<BlobRequirements, AvailabilityCheckError> {
let verified_blobs = if let (Ok(block_kzg_commitments), Ok(payload)) = (
block.message().body().blob_kzg_commitments(),
block.message().body().execution_payload(),
) {
if let Some(transactions) = payload.transactions() {
let verified = verify_kzg_commitments_against_transactions::<T::EthSpec>(
transactions,
block_kzg_commitments,
)
.map_err(|e| AvailabilityCheckError::TxKzgCommitmentMismatch(format!("{e:?}")))?;
if !verified {
return Err(AvailabilityCheckError::TxKzgCommitmentMismatch(
"a commitment and version didn't match".to_string(),
));
}
}
if self.da_check_required(block.epoch()) {
if block_kzg_commitments.is_empty() {
BlobRequirements::EmptyBlobs
let verified_blobs =
if let Ok(block_kzg_commitments) = block.message().body().blob_kzg_commitments() {
if self.da_check_required(block.epoch()) {
if block_kzg_commitments.is_empty() {
BlobRequirements::EmptyBlobs
} else {
BlobRequirements::Required
}
} else {
BlobRequirements::Required
BlobRequirements::NotRequired
}
} else {
BlobRequirements::NotRequired
}
} else {
BlobRequirements::PreDeneb
};
BlobRequirements::PreDeneb
};
Ok(verified_blobs)
}
beacon_node/beacon_chain/src/execution_payload.rs
+13 -4
View File
@@ -18,7 +18,7 @@ use proto_array::{Block as ProtoBlock, ExecutionStatus};
use slog::{debug, warn};
use slot_clock::SlotClock;
use state_processing::per_block_processing::{
compute_timestamp_at_slot, get_expected_withdrawals, is_execution_enabled,
self, compute_timestamp_at_slot, get_expected_withdrawals, is_execution_enabled,
is_merge_transition_complete, partially_verify_execution_payload,
};
use std::sync::Arc;
@@ -68,15 +68,16 @@ impl<T: BeaconChainTypes> PayloadNotifier<T> {
// the block as optimistically imported. This is particularly relevant in the case
// where we do not send the block to the EL at all.
let block_message = block.message();
let payload = block_message.execution_payload()?;
partially_verify_execution_payload::<_, FullPayload<_>>(
state,
block.slot(),
payload,
block_message.body(),
&chain.spec,
)
.map_err(BlockError::PerBlockProcessingError)?;
let payload = block_message.execution_payload()?;
match notify_execution_layer {
NotifyExecutionLayer::No if chain.config.optimistic_finalized_sync => {
// Verify the block hash here in Lighthouse and immediately mark the block as
@@ -139,6 +140,14 @@ async fn notify_new_payload<'a, T: BeaconChainTypes>(
block: BeaconBlockRef<'a, T::EthSpec>,
) -> Result<PayloadVerificationStatus, BlockError<T::EthSpec>> {
let execution_payload = block.execution_payload()?;
let versioned_hashes = block.body().blob_kzg_commitments().ok().map(|commitments| {
commitments
.into_iter()
.map(|commitment| {
per_block_processing::deneb::deneb::kzg_commitment_to_versioned_hash(commitment)
})
.collect::<Vec<_>>()
});
let execution_layer = chain
.execution_layer
@@ -146,7 +155,7 @@ async fn notify_new_payload<'a, T: BeaconChainTypes>(
.ok_or(ExecutionPayloadError::NoExecutionConnection)?;
let new_payload_response = execution_layer
.notify_new_payload(&execution_payload.into())
.notify_new_payload(&execution_payload.into(), versioned_hashes)
.await;
match new_payload_response {
beacon_node/beacon_chain/src/kzg_utils.rs
+29 -17
View File
@@ -1,24 +1,23 @@
use kzg::{Error as KzgError, Kzg, BYTES_PER_BLOB};
use kzg::{Error as KzgError, Kzg, KzgPreset};
use types::{Blob, EthSpec, Hash256, KzgCommitment, KzgProof};
/// Converts a blob ssz List object to an array to be used with the kzg
/// crypto library.
fn ssz_blob_to_crypto_blob<T: EthSpec>(blob: Blob<T>) -> kzg::Blob {
let blob_vec: Vec<u8> = blob.into();
let mut arr = [0; BYTES_PER_BLOB];
arr.copy_from_slice(&blob_vec);
arr.into()
fn ssz_blob_to_crypto_blob<T: EthSpec>(
blob: Blob<T>,
) -> Result<<<T as EthSpec>::Kzg as KzgPreset>::Blob, KzgError> {
T::blob_from_bytes(blob.to_vec().as_slice())
}
/// Validate a single blob-commitment-proof triplet from a `BlobSidecar`.
pub fn validate_blob<T: EthSpec>(
kzg: &Kzg,
kzg: &Kzg<T::Kzg>,
blob: Blob<T>,
kzg_commitment: KzgCommitment,
kzg_proof: KzgProof,
) -> Result<bool, KzgError> {
kzg.verify_blob_kzg_proof(
ssz_blob_to_crypto_blob::<T>(blob),
ssz_blob_to_crypto_blob::<T>(blob)?,
kzg_commitment,
kzg_proof,
)
@@ -26,51 +25,64 @@ pub fn validate_blob<T: EthSpec>(
/// Validate a batch of blob-commitment-proof triplets from multiple `BlobSidecars`.
pub fn validate_blobs<T: EthSpec>(
kzg: &Kzg,
kzg: &Kzg<T::Kzg>,
expected_kzg_commitments: &[KzgCommitment],
blobs: &[Blob<T>],
kzg_proofs: &[KzgProof],
) -> Result<bool, KzgError> {
// TODO(sean) batch verification fails with a single element, it's unclear to me why
if blobs.len() == 1 && kzg_proofs.len() == 1 && expected_kzg_commitments.len() == 1 {
if let (Some(blob), Some(kzg_proof), Some(kzg_commitment)) = (
blobs.get(0),
kzg_proofs.get(0),
expected_kzg_commitments.get(0),
) {
return validate_blob::<T>(kzg, blob.clone(), *kzg_commitment, *kzg_proof);
} else {
return Ok(false);
}
}
let blobs = blobs
.iter()
.map(|blob| ssz_blob_to_crypto_blob::<T>(blob.clone())) // Avoid this clone
.collect::<Vec<_>>();
.collect::<Result<Vec<_>, KzgError>>()?;
kzg.verify_blob_kzg_proof_batch(&blobs, expected_kzg_commitments, kzg_proofs)
}
/// Compute the kzg proof given an ssz blob and its kzg commitment.
pub fn compute_blob_kzg_proof<T: EthSpec>(
kzg: &Kzg,
kzg: &Kzg<T::Kzg>,
blob: &Blob<T>,
kzg_commitment: KzgCommitment,
) -> Result<KzgProof, KzgError> {
// Avoid this blob clone
kzg.compute_blob_kzg_proof(ssz_blob_to_crypto_blob::<T>(blob.clone()), kzg_commitment)
kzg.compute_blob_kzg_proof(ssz_blob_to_crypto_blob::<T>(blob.clone())?, kzg_commitment)
}
/// Compute the kzg commitment for a given blob.
pub fn blob_to_kzg_commitment<T: EthSpec>(
kzg: &Kzg,
kzg: &Kzg<T::Kzg>,
blob: Blob<T>,
) -> Result<KzgCommitment, KzgError> {
kzg.blob_to_kzg_commitment(ssz_blob_to_crypto_blob::<T>(blob))
kzg.blob_to_kzg_commitment(ssz_blob_to_crypto_blob::<T>(blob)?)
}
/// Compute the kzg proof for a given blob and an evaluation point z.
pub fn compute_kzg_proof<T: EthSpec>(
kzg: &Kzg,
kzg: &Kzg<T::Kzg>,
blob: Blob<T>,
z: Hash256,
) -> Result<(KzgProof, Hash256), KzgError> {
let z = z.0.into();
kzg.compute_kzg_proof(ssz_blob_to_crypto_blob::<T>(blob), z)
kzg.compute_kzg_proof(ssz_blob_to_crypto_blob::<T>(blob)?, z)
.map(|(proof, z)| (proof, Hash256::from_slice(&z.to_vec())))
}
/// Verify a `kzg_proof` for a `kzg_commitment` that evaluating a polynomial at `z` results in `y`
pub fn verify_kzg_proof<T: EthSpec>(
kzg: &Kzg,
kzg: &Kzg<T::Kzg>,
kzg_commitment: KzgCommitment,
kzg_proof: KzgProof,
z: Hash256,
beacon_node/beacon_chain/src/metrics.rs
+18
View File
@@ -1027,6 +1027,24 @@ lazy_static! {
"beacon_aggregated_attestation_subsets_total",
"Count of new aggregated attestations that are subsets of already known aggregates"
);
/*
* Kzg related metrics
*/
pub static ref KZG_VERIFICATION_SINGLE_TIMES: Result<Histogram> =
try_create_histogram("kzg_verification_single_seconds", "Runtime of single kzg verification");
pub static ref KZG_VERIFICATION_BATCH_TIMES: Result<Histogram> =
try_create_histogram("kzg_verification_batch_seconds", "Runtime of batched kzg verification");
/*
* Availability related metrics
*/
pub static ref BLOCK_AVAILABILITY_DELAY: Result<Histogram> = try_create_histogram_with_buckets(
"block_availability_delay",
"Duration between start of the slot and the time at which all components of the block are available.",
// Create a custom bucket list for greater granularity in block delay
Ok(vec![0.1, 0.2, 0.3,0.4,0.5,0.75,1.0,1.25,1.5,1.75,2.0,2.5,3.0,3.5,4.0,5.0,6.0,7.0,8.0,9.0,10.0,15.0,20.0])
);
}
/// Scrape the `beacon_chain` for metrics that are not constantly updated (e.g., the present slot,
beacon_node/beacon_chain/src/test_utils.rs
+97 -50
View File
@@ -16,6 +16,7 @@ use crate::{
};
use bls::get_withdrawal_credentials;
use eth2::types::BlockContentsTuple;
use execution_layer::test_utils::generate_genesis_header;
use execution_layer::{
auth::JwtKey,
test_utils::{
@@ -30,8 +31,8 @@ use int_to_bytes::int_to_bytes32;
use kzg::{Kzg, TrustedSetup};
use merkle_proof::MerkleTree;
use operation_pool::ReceivedPreCapella;
use parking_lot::Mutex;
use parking_lot::RwLockWriteGuard;
use parking_lot::{Mutex, RwLock};
use rand::rngs::StdRng;
use rand::Rng;
use rand::SeedableRng;
@@ -53,6 +54,7 @@ use std::str::FromStr;
use std::sync::Arc;
use std::time::Duration;
use store::{config::StoreConfig, HotColdDB, ItemStore, LevelDB, MemoryStore};
use task_executor::TaskExecutor;
use task_executor::{test_utils::TestRuntime, ShutdownReason};
use tree_hash::TreeHash;
use types::sync_selection_proof::SyncSelectionProof;
@@ -195,11 +197,12 @@ impl<E: EthSpec> Builder<EphemeralHarnessType<E>> {
.unwrap(),
);
let mutator = move |builder: BeaconChainBuilder<_>| {
let header = generate_genesis_header::<E>(builder.get_spec(), false);
let genesis_state = interop_genesis_state_with_eth1::<E>(
&validator_keypairs,
HARNESS_GENESIS_TIME,
Hash256::from_slice(DEFAULT_ETH1_BLOCK_HASH),
None,
header,
builder.get_spec(),
)
.expect("should generate interop state");
@@ -256,11 +259,12 @@ impl<E: EthSpec> Builder<DiskHarnessType<E>> {
.expect("cannot build without validator keypairs");
let mutator = move |builder: BeaconChainBuilder<_>| {
let header = generate_genesis_header::<E>(builder.get_spec(), false);
let genesis_state = interop_genesis_state_with_eth1::<E>(
&validator_keypairs,
HARNESS_GENESIS_TIME,
Hash256::from_slice(DEFAULT_ETH1_BLOCK_HASH),
None,
header,
builder.get_spec(),
)
.expect("should generate interop state");
@@ -392,7 +396,7 @@ where
self
}
pub fn execution_layer(mut self, urls: &[&str]) -> Self {
pub fn execution_layer_from_urls(mut self, urls: &[&str]) -> Self {
assert!(
self.execution_layer.is_none(),
"execution layer already defined"
@@ -421,6 +425,11 @@ where
self
}
pub fn execution_layer(mut self, el: Option<ExecutionLayer<E>>) -> Self {
self.execution_layer = el;
self
}
pub fn recalculate_fork_times_with_genesis(mut self, genesis_time: u64) -> Self {
let mock = self
.mock_execution_layer
@@ -434,7 +443,7 @@ where
spec.capella_fork_epoch.map(|epoch| {
genesis_time + spec.seconds_per_slot * E::slots_per_epoch() * epoch.as_u64()
});
mock.server.execution_block_generator().deneb_time = spec.deneb_fork_epoch.map(|epoch| {
mock.server.execution_block_generator().cancun_time = spec.deneb_fork_epoch.map(|epoch| {
genesis_time + spec.seconds_per_slot * E::slots_per_epoch() * epoch.as_u64()
});
@@ -442,30 +451,11 @@ where
}
pub fn mock_execution_layer(mut self) -> Self {
let spec = self.spec.clone().expect("cannot build without spec");
let shanghai_time = spec.capella_fork_epoch.map(|epoch| {
HARNESS_GENESIS_TIME + spec.seconds_per_slot * E::slots_per_epoch() * epoch.as_u64()
});
let deneb_time = spec.deneb_fork_epoch.map(|epoch| {
HARNESS_GENESIS_TIME + spec.seconds_per_slot * E::slots_per_epoch() * epoch.as_u64()
});
let trusted_setup: TrustedSetup =
serde_json::from_reader(eth2_network_config::TRUSTED_SETUP)
.map_err(|e| format!("Unable to read trusted setup file: {}", e))
.expect("should have trusted setup");
let kzg = Kzg::new_from_trusted_setup(trusted_setup).expect("should create kzg");
let mock = MockExecutionLayer::new(
let mock = mock_execution_layer_from_parts::<E>(
self.spec.as_ref().expect("cannot build without spec"),
self.runtime.task_executor.clone(),
DEFAULT_TERMINAL_BLOCK,
shanghai_time,
deneb_time,
None,
Some(JwtKey::from_slice(&DEFAULT_JWT_SECRET).unwrap()),
spec,
None,
Some(kzg),
);
self.execution_layer = Some(mock.el.clone());
self.mock_execution_layer = Some(mock);
@@ -480,29 +470,12 @@ where
// Get a random unused port
let port = unused_port::unused_tcp4_port().unwrap();
let builder_url = SensitiveUrl::parse(format!("http://127.0.0.1:{port}").as_str()).unwrap();
let spec = self.spec.clone().expect("cannot build without spec");
let shanghai_time = spec.capella_fork_epoch.map(|epoch| {
HARNESS_GENESIS_TIME + spec.seconds_per_slot * E::slots_per_epoch() * epoch.as_u64()
});
let deneb_time = spec.deneb_fork_epoch.map(|epoch| {
HARNESS_GENESIS_TIME + spec.seconds_per_slot * E::slots_per_epoch() * epoch.as_u64()
});
let trusted_setup: TrustedSetup =
serde_json::from_reader(eth2_network_config::TRUSTED_SETUP)
.map_err(|e| format!("Unable to read trusted setup file: {}", e))
.expect("should have trusted setup");
let kzg = Kzg::new_from_trusted_setup(trusted_setup).expect("should create kzg");
let mock_el = MockExecutionLayer::new(
let spec = self.spec.as_ref().expect("cannot build without spec");
let mock_el = mock_execution_layer_from_parts::<E>(
spec,
self.runtime.task_executor.clone(),
DEFAULT_TERMINAL_BLOCK,
shanghai_time,
deneb_time,
builder_threshold,
Some(JwtKey::from_slice(&DEFAULT_JWT_SECRET).unwrap()),
spec.clone(),
Some(builder_url.clone()),
Some(kzg),
builder_threshold,
)
.move_to_terminal_block();
@@ -512,7 +485,7 @@ where
mock_el_url,
builder_url,
beacon_url,
spec,
spec.clone(),
self.runtime.task_executor.clone(),
));
self.execution_layer = Some(mock_el.el.clone());
@@ -547,7 +520,7 @@ where
.validator_keypairs
.expect("cannot build without validator keypairs");
let trusted_setup: TrustedSetup =
serde_json::from_reader(eth2_network_config::TRUSTED_SETUP)
serde_json::from_reader(eth2_network_config::get_trusted_setup::<E::Kzg>())
.map_err(|e| format!("Unable to read trusted setup file: {}", e))
.unwrap();
@@ -603,11 +576,44 @@ where
runtime: self.runtime,
mock_execution_layer: self.mock_execution_layer,
mock_builder: self.mock_builder.map(Arc::new),
blob_signature_cache: <_>::default(),
rng: make_rng(),
}
}
}
pub fn mock_execution_layer_from_parts<T: EthSpec>(
spec: &ChainSpec,
task_executor: TaskExecutor,
builder_url: Option<SensitiveUrl>,
builder_threshold: Option<u128>,
) -> MockExecutionLayer<T> {
let shanghai_time = spec.capella_fork_epoch.map(|epoch| {
HARNESS_GENESIS_TIME + spec.seconds_per_slot * T::slots_per_epoch() * epoch.as_u64()
});
let cancun_time = spec.deneb_fork_epoch.map(|epoch| {
HARNESS_GENESIS_TIME + spec.seconds_per_slot * T::slots_per_epoch() * epoch.as_u64()
});
let trusted_setup: TrustedSetup =
serde_json::from_reader(eth2_network_config::get_trusted_setup::<T::Kzg>())
.map_err(|e| format!("Unable to read trusted setup file: {}", e))
.expect("should have trusted setup");
let kzg = Kzg::new_from_trusted_setup(trusted_setup).expect("should create kzg");
MockExecutionLayer::new(
task_executor,
DEFAULT_TERMINAL_BLOCK,
shanghai_time,
cancun_time,
builder_threshold,
Some(JwtKey::from_slice(&DEFAULT_JWT_SECRET).unwrap()),
spec.clone(),
builder_url,
Some(kzg),
)
}
/// A testing harness which can instantiate a `BeaconChain` and populate it with blocks and
/// attestations.
///
@@ -629,9 +635,29 @@ pub struct BeaconChainHarness<T: BeaconChainTypes> {
pub mock_execution_layer: Option<MockExecutionLayer<T::EthSpec>>,
pub mock_builder: Option<Arc<TestingBuilder<T::EthSpec>>>,
/// Cache for blob signature because we don't need them for import, but we do need them
/// to test gossip validation. We always make them during block production but drop them
/// before storing them in the db.
pub blob_signature_cache: Arc<RwLock<HashMap<BlobSignatureKey, Signature>>>,
pub rng: Mutex<StdRng>,
}
#[derive(Clone, Debug, Hash, Eq, PartialEq)]
pub struct BlobSignatureKey {
block_root: Hash256,
blob_index: u64,
}
impl BlobSignatureKey {
pub fn new(block_root: Hash256, blob_index: u64) -> Self {
Self {
block_root,
blob_index,
}
}
}
pub type CommitteeAttestations<E> = Vec<(Attestation<E>, SubnetId)>;
pub type HarnessAttestations<E> =
Vec<(CommitteeAttestations<E>, Option<SignedAggregateAndProof<E>>)>;
@@ -663,6 +689,20 @@ where
.execution_block_generator()
}
pub fn get_head_block(&self) -> BlockWrapper<E> {
let block = self.chain.head_beacon_block();
let block_root = block.canonical_root();
let blobs = self.chain.get_blobs(&block_root).unwrap();
BlockWrapper::new(block, blobs)
}
pub fn get_full_block(&self, block_root: &Hash256) -> BlockWrapper<E> {
let block = self.chain.get_blinded_block(block_root).unwrap().unwrap();
let full_block = self.chain.store.make_full_block(block_root, block).unwrap();
let blobs = self.chain.get_blobs(block_root).unwrap();
BlockWrapper::new(Arc::new(full_block), blobs)
}
pub fn get_all_validators(&self) -> Vec<usize> {
(0..self.validator_keypairs.len()).collect()
}
@@ -824,7 +864,7 @@ where
.proposal_blob_cache
.pop(&signed_block.canonical_root())
{
let signed_blobs = Vec::from(blobs)
let signed_blobs: SignedBlobSidecarList<E> = Vec::from(blobs)
.into_iter()
.map(|blob| {
blob.sign(
@@ -836,6 +876,13 @@ where
})
.collect::<Vec<_>>()
.into();
let mut guard = self.blob_signature_cache.write();
for blob in &signed_blobs {
guard.insert(
BlobSignatureKey::new(blob.message.block_root, blob.message.index),
blob.signature.clone(),
);
}
(signed_block, Some(signed_blobs))
} else {
(signed_block, None)
beacon_node/beacon_chain/tests/attestation_production.rs
+9 -2
View File
@@ -68,6 +68,7 @@ async fn produces_attestations() {
.store
.make_full_block(&block_root, blinded_block)
.unwrap();
let blobs = chain.get_blobs(&block_root).unwrap();
let epoch_boundary_slot = state
.current_epoch()
@@ -132,7 +133,8 @@ async fn produces_attestations() {
assert_eq!(data.target.epoch, state.current_epoch(), "bad target epoch");
assert_eq!(data.target.root, target_root, "bad target root");
let block_wrapper: BlockWrapper<MainnetEthSpec> = Arc::new(block.clone()).into();
let block_wrapper =
BlockWrapper::<MainnetEthSpec>::new(Arc::new(block.clone()), blobs.clone());
let beacon_chain::blob_verification::MaybeAvailableBlock::Available(available_block) = chain
.data_availability_checker
.check_availability(block_wrapper)
@@ -202,7 +204,12 @@ async fn early_attester_cache_old_request() {
.get_block(&head.beacon_block_root)
.unwrap();
let block_wrapper: BlockWrapper<MainnetEthSpec> = head.beacon_block.clone().into();
let head_blobs = harness
.chain
.get_blobs(&head.beacon_block_root)
.expect("should get blobs");
let block_wrapper = BlockWrapper::<MainnetEthSpec>::new(head.beacon_block.clone(), head_blobs);
let beacon_chain::blob_verification::MaybeAvailableBlock::Available(available_block) = harness.chain
.data_availability_checker
.check_availability(block_wrapper)
beacon_node/beacon_chain/tests/attestation_verification.rs
-1
View File
@@ -797,7 +797,6 @@ async fn unaggregated_gossip_verification() {
a.data.target.epoch = early_slot.epoch(E::slots_per_epoch());
},
|tester, err| {
dbg!(&err);
assert!(matches!(
err,
AttnError::PastSlot {
beacon_node/beacon_chain/tests/block_verification.rs
+183 -48
View File
@@ -1,6 +1,7 @@
#![cfg(not(debug_assertions))]
use beacon_chain::blob_verification::BlockWrapper;
use beacon_chain::test_utils::BlobSignatureKey;
use beacon_chain::{
blob_verification::AsBlock,
test_utils::{AttestationStrategy, BeaconChainHarness, BlockStrategy, EphemeralHarnessType},
@@ -36,7 +37,7 @@ lazy_static! {
static ref KEYPAIRS: Vec<Keypair> = types::test_utils::generate_deterministic_keypairs(VALIDATOR_COUNT);
}
async fn get_chain_segment() -> Vec<BeaconSnapshot<E>> {
async fn get_chain_segment() -> (Vec<BeaconSnapshot<E>>, Vec<Option<BlobSidecarList<E>>>) {
let harness = get_harness(VALIDATOR_COUNT);
harness
@@ -48,6 +49,7 @@ async fn get_chain_segment() -> Vec<BeaconSnapshot<E>> {
.await;
let mut segment = Vec::with_capacity(CHAIN_SEGMENT_LENGTH);
let mut segment_blobs = Vec::with_capacity(CHAIN_SEGMENT_LENGTH);
for snapshot in harness
.chain
.chain_dump()
@@ -66,8 +68,76 @@ async fn get_chain_segment() -> Vec<BeaconSnapshot<E>> {
beacon_block: Arc::new(full_block),
beacon_state: snapshot.beacon_state,
});
segment_blobs.push(
harness
.chain
.get_blobs(&snapshot.beacon_block_root)
.unwrap(),
)
}
segment
(segment, segment_blobs)
}
async fn get_chain_segment_with_signed_blobs() -> (
Vec<BeaconSnapshot<E>>,
Vec<Option<VariableList<SignedBlobSidecar<E>, <E as EthSpec>::MaxBlobsPerBlock>>>,
) {
let harness = get_harness(VALIDATOR_COUNT);
harness
.extend_chain(
CHAIN_SEGMENT_LENGTH,
BlockStrategy::OnCanonicalHead,
AttestationStrategy::AllValidators,
)
.await;
let mut segment = Vec::with_capacity(CHAIN_SEGMENT_LENGTH);
let mut segment_blobs = Vec::with_capacity(CHAIN_SEGMENT_LENGTH);
for snapshot in harness
.chain
.chain_dump()
.expect("should dump chain")
.into_iter()
.skip(1)
{
let full_block = harness
.chain
.get_block(&snapshot.beacon_block_root)
.await
.unwrap()
.unwrap();
segment.push(BeaconSnapshot {
beacon_block_root: snapshot.beacon_block_root,
beacon_block: Arc::new(full_block),
beacon_state: snapshot.beacon_state,
});
let signed_blobs = harness
.chain
.get_blobs(&snapshot.beacon_block_root)
.unwrap()
.map(|blobs| {
let blobs = blobs
.into_iter()
.map(|blob| {
let block_root = blob.block_root;
let blob_index = blob.index;
SignedBlobSidecar {
message: blob,
signature: harness
.blob_signature_cache
.read()
.get(&BlobSignatureKey::new(block_root, blob_index))
.unwrap()
.clone(),
}
})
.collect::<Vec<_>>();
VariableList::from(blobs)
});
segment_blobs.push(signed_blobs)
}
(segment, segment_blobs)
}
fn get_harness(validator_count: usize) -> BeaconChainHarness<EphemeralHarnessType<E>> {
@@ -83,10 +153,14 @@ fn get_harness(validator_count: usize) -> BeaconChainHarness<EphemeralHarnessTyp
harness
}
fn chain_segment_blocks(chain_segment: &[BeaconSnapshot<E>]) -> Vec<Arc<SignedBeaconBlock<E>>> {
fn chain_segment_blocks(
chain_segment: &[BeaconSnapshot<E>],
blobs: &[Option<BlobSidecarList<E>>],
) -> Vec<BlockWrapper<E>> {
chain_segment
.iter()
.map(|snapshot| snapshot.beacon_block.clone().into())
.zip(blobs.into_iter())
.map(|(snapshot, blobs)| BlockWrapper::new(snapshot.beacon_block.clone(), blobs.clone()))
.collect()
}
@@ -142,8 +216,8 @@ fn update_parent_roots(snapshots: &mut [BeaconSnapshot<E>]) {
#[tokio::test]
async fn chain_segment_full_segment() {
let harness = get_harness(VALIDATOR_COUNT);
let chain_segment = get_chain_segment().await;
let blocks: Vec<BlockWrapper<E>> = chain_segment_blocks(&chain_segment)
let (chain_segment, chain_segment_blobs) = get_chain_segment().await;
let blocks: Vec<BlockWrapper<E>> = chain_segment_blocks(&chain_segment, &chain_segment_blobs)
.into_iter()
.map(|block| block.into())
.collect();
@@ -181,11 +255,12 @@ async fn chain_segment_full_segment() {
async fn chain_segment_varying_chunk_size() {
for chunk_size in &[1, 2, 3, 5, 31, 32, 33, 42] {
let harness = get_harness(VALIDATOR_COUNT);
let chain_segment = get_chain_segment().await;
let blocks: Vec<BlockWrapper<E>> = chain_segment_blocks(&chain_segment)
.into_iter()
.map(|block| block.into())
.collect();
let (chain_segment, chain_segment_blobs) = get_chain_segment().await;
let blocks: Vec<BlockWrapper<E>> =
chain_segment_blocks(&chain_segment, &chain_segment_blobs)
.into_iter()
.map(|block| block.into())
.collect();
harness
.chain
@@ -214,7 +289,7 @@ async fn chain_segment_varying_chunk_size() {
#[tokio::test]
async fn chain_segment_non_linear_parent_roots() {
let harness = get_harness(VALIDATOR_COUNT);
let chain_segment = get_chain_segment().await;
let (chain_segment, chain_segment_blobs) = get_chain_segment().await;
harness
.chain
@@ -224,10 +299,11 @@ async fn chain_segment_non_linear_parent_roots() {
/*
* Test with a block removed.
*/
let mut blocks: Vec<BlockWrapper<E>> = chain_segment_blocks(&chain_segment)
.into_iter()
.map(|block| block.into())
.collect();
let mut blocks: Vec<BlockWrapper<E>> =
chain_segment_blocks(&chain_segment, &chain_segment_blobs)
.into_iter()
.map(|block| block.into())
.collect();
blocks.remove(2);
assert!(
@@ -245,10 +321,11 @@ async fn chain_segment_non_linear_parent_roots() {
/*
* Test with a modified parent root.
*/
let mut blocks: Vec<BlockWrapper<E>> = chain_segment_blocks(&chain_segment)
.into_iter()
.map(|block| block.into())
.collect();
let mut blocks: Vec<BlockWrapper<E>> =
chain_segment_blocks(&chain_segment, &chain_segment_blobs)
.into_iter()
.map(|block| block.into())
.collect();
let (mut block, signature) = blocks[3].as_block().clone().deconstruct();
*block.parent_root_mut() = Hash256::zero();
@@ -270,7 +347,7 @@ async fn chain_segment_non_linear_parent_roots() {
#[tokio::test]
async fn chain_segment_non_linear_slots() {
let harness = get_harness(VALIDATOR_COUNT);
let chain_segment = get_chain_segment().await;
let (chain_segment, chain_segment_blobs) = get_chain_segment().await;
harness
.chain
.slot_clock
@@ -280,10 +357,11 @@ async fn chain_segment_non_linear_slots() {
* Test where a child is lower than the parent.
*/
let mut blocks: Vec<BlockWrapper<E>> = chain_segment_blocks(&chain_segment)
.into_iter()
.map(|block| block.into())
.collect();
let mut blocks: Vec<BlockWrapper<E>> =
chain_segment_blocks(&chain_segment, &chain_segment_blobs)
.into_iter()
.map(|block| block.into())
.collect();
let (mut block, signature) = blocks[3].as_block().clone().deconstruct();
*block.slot_mut() = Slot::new(0);
blocks[3] = Arc::new(SignedBeaconBlock::from_block(block, signature)).into();
@@ -304,10 +382,11 @@ async fn chain_segment_non_linear_slots() {
* Test where a child is equal to the parent.
*/
let mut blocks: Vec<BlockWrapper<E>> = chain_segment_blocks(&chain_segment)
.into_iter()
.map(|block| block.into())
.collect();
let mut blocks: Vec<BlockWrapper<E>> =
chain_segment_blocks(&chain_segment, &chain_segment_blobs)
.into_iter()
.map(|block| block.into())
.collect();
let (mut block, signature) = blocks[3].as_block().clone().deconstruct();
*block.slot_mut() = blocks[2].slot();
blocks[3] = Arc::new(SignedBeaconBlock::from_block(block, signature)).into();
@@ -327,6 +406,7 @@ async fn chain_segment_non_linear_slots() {
async fn assert_invalid_signature(
chain_segment: &[BeaconSnapshot<E>],
chain_segment_blobs: &[Option<BlobSidecarList<E>>],
harness: &BeaconChainHarness<EphemeralHarnessType<E>>,
block_index: usize,
snapshots: &[BeaconSnapshot<E>],
@@ -334,7 +414,8 @@ async fn assert_invalid_signature(
) {
let blocks: Vec<BlockWrapper<E>> = snapshots
.iter()
.map(|snapshot| snapshot.beacon_block.clone().into())
.zip(chain_segment_blobs.iter())
.map(|(snapshot, blobs)| BlockWrapper::new(snapshot.beacon_block.clone(), blobs.clone()))
.collect();
// Ensure the block will be rejected if imported in a chain segment.
@@ -358,7 +439,8 @@ async fn assert_invalid_signature(
let ancestor_blocks = chain_segment
.iter()
.take(block_index)
.map(|snapshot| snapshot.beacon_block.clone().into())
.zip(chain_segment_blobs.iter())
.map(|(snapshot, blobs)| BlockWrapper::new(snapshot.beacon_block.clone(), blobs.clone()))
.collect();
// We don't care if this fails, we just call this to ensure that all prior blocks have been
// imported prior to this test.
@@ -372,7 +454,10 @@ async fn assert_invalid_signature(
.chain
.process_block(
snapshots[block_index].beacon_block.canonical_root(),
snapshots[block_index].beacon_block.clone(),
BlockWrapper::new(
snapshots[block_index].beacon_block.clone(),
chain_segment_blobs[block_index].clone(),
),
NotifyExecutionLayer::Yes,
)
.await;
@@ -403,7 +488,7 @@ async fn get_invalid_sigs_harness(
}
#[tokio::test]
async fn invalid_signature_gossip_block() {
let chain_segment = get_chain_segment().await;
let (chain_segment, chain_segment_blobs) = get_chain_segment().await;
for &block_index in BLOCK_INDICES {
// Ensure the block will be rejected if imported on its own (without gossip checking).
let harness = get_invalid_sigs_harness(&chain_segment).await;
@@ -421,7 +506,10 @@ async fn invalid_signature_gossip_block() {
let ancestor_blocks = chain_segment
.iter()
.take(block_index)
.map(|snapshot| snapshot.beacon_block.clone().into())
.zip(chain_segment_blobs.iter())
.map(|(snapshot, blobs)| {
BlockWrapper::new(snapshot.beacon_block.clone(), blobs.clone())
})
.collect();
harness
.chain
@@ -449,7 +537,7 @@ async fn invalid_signature_gossip_block() {
#[tokio::test]
async fn invalid_signature_block_proposal() {
let chain_segment = get_chain_segment().await;
let (chain_segment, chain_segment_blobs) = get_chain_segment().await;
for &block_index in BLOCK_INDICES {
let harness = get_invalid_sigs_harness(&chain_segment).await;
let mut snapshots = chain_segment.clone();
@@ -464,7 +552,10 @@ async fn invalid_signature_block_proposal() {
));
let blocks: Vec<BlockWrapper<E>> = snapshots
.iter()
.map(|snapshot| snapshot.beacon_block.clone().into())
.zip(chain_segment_blobs.iter())
.map(|(snapshot, blobs)| {
BlockWrapper::new(snapshot.beacon_block.clone(), blobs.clone())
})
.collect::<Vec<_>>();
// Ensure the block will be rejected if imported in a chain segment.
assert!(
@@ -483,7 +574,7 @@ async fn invalid_signature_block_proposal() {
#[tokio::test]
async fn invalid_signature_randao_reveal() {
let chain_segment = get_chain_segment().await;
let (chain_segment, chain_segment_blobs) = get_chain_segment().await;
for &block_index in BLOCK_INDICES {
let harness = get_invalid_sigs_harness(&chain_segment).await;
let mut snapshots = chain_segment.clone();
@@ -497,13 +588,21 @@ async fn invalid_signature_randao_reveal() {
Arc::new(SignedBeaconBlock::from_block(block, signature));
update_parent_roots(&mut snapshots);
update_proposal_signatures(&mut snapshots, &harness);
assert_invalid_signature(&chain_segment, &harness, block_index, &snapshots, "randao").await;
assert_invalid_signature(
&chain_segment,
&chain_segment_blobs,
&harness,
block_index,
&snapshots,
"randao",
)
.await;
}
}
#[tokio::test]
async fn invalid_signature_proposer_slashing() {
let chain_segment = get_chain_segment().await;
let (chain_segment, chain_segment_blobs) = get_chain_segment().await;
for &block_index in BLOCK_INDICES {
let harness = get_invalid_sigs_harness(&chain_segment).await;
let mut snapshots = chain_segment.clone();
@@ -533,6 +632,7 @@ async fn invalid_signature_proposer_slashing() {
update_proposal_signatures(&mut snapshots, &harness);
assert_invalid_signature(
&chain_segment,
&chain_segment_blobs,
&harness,
block_index,
&snapshots,
@@ -544,7 +644,7 @@ async fn invalid_signature_proposer_slashing() {
#[tokio::test]
async fn invalid_signature_attester_slashing() {
let chain_segment = get_chain_segment().await;
let (chain_segment, chain_segment_blobs) = get_chain_segment().await;
for &block_index in BLOCK_INDICES {
let harness = get_invalid_sigs_harness(&chain_segment).await;
let mut snapshots = chain_segment.clone();
@@ -585,6 +685,7 @@ async fn invalid_signature_attester_slashing() {
update_proposal_signatures(&mut snapshots, &harness);
assert_invalid_signature(
&chain_segment,
&chain_segment_blobs,
&harness,
block_index,
&snapshots,
@@ -596,7 +697,7 @@ async fn invalid_signature_attester_slashing() {
#[tokio::test]
async fn invalid_signature_attestation() {
let chain_segment = get_chain_segment().await;
let (chain_segment, chain_segment_blobs) = get_chain_segment().await;
let mut checked_attestation = false;
for &block_index in BLOCK_INDICES {
@@ -615,6 +716,7 @@ async fn invalid_signature_attestation() {
update_proposal_signatures(&mut snapshots, &harness);
assert_invalid_signature(
&chain_segment,
&chain_segment_blobs,
&harness,
block_index,
&snapshots,
@@ -633,7 +735,7 @@ async fn invalid_signature_attestation() {
#[tokio::test]
async fn invalid_signature_deposit() {
let chain_segment = get_chain_segment().await;
let (chain_segment, chain_segment_blobs) = get_chain_segment().await;
for &block_index in BLOCK_INDICES {
// Note: an invalid deposit signature is permitted!
let harness = get_invalid_sigs_harness(&chain_segment).await;
@@ -663,7 +765,10 @@ async fn invalid_signature_deposit() {
update_proposal_signatures(&mut snapshots, &harness);
let blocks: Vec<BlockWrapper<E>> = snapshots
.iter()
.map(|snapshot| snapshot.beacon_block.clone().into())
.zip(chain_segment_blobs.iter())
.map(|(snapshot, blobs)| {
BlockWrapper::new(snapshot.beacon_block.clone(), blobs.clone())
})
.collect();
assert!(
!matches!(
@@ -681,7 +786,7 @@ async fn invalid_signature_deposit() {
#[tokio::test]
async fn invalid_signature_exit() {
let chain_segment = get_chain_segment().await;
let (chain_segment, chain_segment_blobs) = get_chain_segment().await;
for &block_index in BLOCK_INDICES {
let harness = get_invalid_sigs_harness(&chain_segment).await;
let mut snapshots = chain_segment.clone();
@@ -708,6 +813,7 @@ async fn invalid_signature_exit() {
update_proposal_signatures(&mut snapshots, &harness);
assert_invalid_signature(
&chain_segment,
&chain_segment_blobs,
&harness,
block_index,
&snapshots,
@@ -727,7 +833,7 @@ fn unwrap_err<T, E>(result: Result<T, E>) -> E {
#[tokio::test]
async fn block_gossip_verification() {
let harness = get_harness(VALIDATOR_COUNT);
let chain_segment = get_chain_segment().await;
let (chain_segment, chain_segment_blobs) = get_chain_segment_with_signed_blobs().await;
let block_index = CHAIN_SEGMENT_LENGTH - 2;
@@ -737,7 +843,10 @@ async fn block_gossip_verification() {
.set_slot(chain_segment[block_index].beacon_block.slot().as_u64());
// Import the ancestors prior to the block we're testing.
for snapshot in &chain_segment[0..block_index] {
for (snapshot, blobs_opt) in chain_segment[0..block_index]
.iter()
.zip(chain_segment_blobs.iter())
{
let gossip_verified = harness
.chain
.verify_block_for_gossip(snapshot.beacon_block.clone().into())
@@ -753,6 +862,21 @@ async fn block_gossip_verification() {
)
.await
.expect("should import valid gossip verified block");
if let Some(blobs) = blobs_opt {
for blob in blobs {
let blob_index = blob.message.index;
let gossip_verified = harness
.chain
.verify_blob_sidecar_for_gossip(blob.clone(), blob_index)
.expect("should obtain gossip verified blob");
harness
.chain
.process_blob(gossip_verified)
.await
.expect("should import valid gossip verified blob");
}
}
}
// Recompute the head to ensure we cache the latest view of fork choice.
@@ -1010,14 +1134,25 @@ async fn verify_block_for_gossip_slashing_detection() {
harness.advance_slot();
let state = harness.get_current_state();
let ((block1, _), _) = harness.make_block(state.clone(), Slot::new(1)).await;
let ((block2, _), _) = harness.make_block(state, Slot::new(1)).await;
let ((block1, blobs1), _) = harness.make_block(state.clone(), Slot::new(1)).await;
let ((block2, _blobs2), _) = harness.make_block(state, Slot::new(1)).await;
let verified_block = harness
.chain
.verify_block_for_gossip(Arc::new(block1).into())
.await
.unwrap();
if let Some(blobs) = blobs1 {
for blob in blobs {
let blob_index = blob.message.index;
let verified_blob = harness
.chain
.verify_blob_sidecar_for_gossip(blob, blob_index)
.unwrap();
harness.chain.process_blob(verified_blob).await.unwrap();
}
}
harness
.chain
.process_block(
beacon_node/beacon_chain/tests/payload_invalidation.rs
+13 -10
View File
@@ -167,7 +167,7 @@ impl InvalidPayloadRig {
async fn build_blocks(&mut self, num_blocks: u64, is_valid: Payload) -> Vec<Hash256> {
let mut roots = Vec::with_capacity(num_blocks as usize);
for _ in 0..num_blocks {
roots.push(self.import_block(is_valid.clone()).await);
roots.push(self.import_block(is_valid).await);
}
roots
}
@@ -815,13 +815,16 @@ async fn switches_heads() {
})
.await;
// The fork block should become the head.
assert_eq!(rig.harness.head_block_root(), fork_block_root);
// NOTE: The `import_block` method above will cause the `ExecutionStatus` of the
// `fork_block_root`'s payload to switch from `Optimistic` to `Invalid`. This means it *won't*
// be set as head, it's parent block will instead. This is an issue with the mock EL and/or
// the payload invalidation rig.
assert_eq!(rig.harness.head_block_root(), fork_parent_root);
// The fork block has not yet been validated.
assert!(rig
.execution_status(fork_block_root)
.is_strictly_optimistic());
.is_optimistic_or_invalid());
for root in blocks {
let slot = rig
@@ -1420,13 +1423,13 @@ async fn build_optimistic_chain(
.server
.all_get_block_by_hash_requests_return_natural_value();
return rig;
rig
}
#[tokio::test]
async fn optimistic_transition_block_valid_unfinalized() {
let ttd = 42;
let num_blocks = 16 as usize;
let num_blocks = 16_usize;
let rig = build_optimistic_chain(ttd, ttd, num_blocks).await;
let post_transition_block_root = rig
@@ -1480,7 +1483,7 @@ async fn optimistic_transition_block_valid_unfinalized() {
#[tokio::test]
async fn optimistic_transition_block_valid_finalized() {
let ttd = 42;
let num_blocks = 130 as usize;
let num_blocks = 130_usize;
let rig = build_optimistic_chain(ttd, ttd, num_blocks).await;
let post_transition_block_root = rig
@@ -1535,7 +1538,7 @@ async fn optimistic_transition_block_valid_finalized() {
async fn optimistic_transition_block_invalid_unfinalized() {
let block_ttd = 42;
let rig_ttd = 1337;
let num_blocks = 22 as usize;
let num_blocks = 22_usize;
let rig = build_optimistic_chain(block_ttd, rig_ttd, num_blocks).await;
let post_transition_block_root = rig
@@ -1611,7 +1614,7 @@ async fn optimistic_transition_block_invalid_unfinalized() {
async fn optimistic_transition_block_invalid_unfinalized_syncing_ee() {
let block_ttd = 42;
let rig_ttd = 1337;
let num_blocks = 22 as usize;
let num_blocks = 22_usize;
let rig = build_optimistic_chain(block_ttd, rig_ttd, num_blocks).await;
let post_transition_block_root = rig
@@ -1724,7 +1727,7 @@ async fn optimistic_transition_block_invalid_unfinalized_syncing_ee() {
async fn optimistic_transition_block_invalid_finalized() {
let block_ttd = 42;
let rig_ttd = 1337;
let num_blocks = 130 as usize;
let num_blocks = 130_usize;
let rig = build_optimistic_chain(block_ttd, rig_ttd, num_blocks).await;
let post_transition_block_root = rig
beacon_node/beacon_chain/tests/store_tests.rs
+49 -33
View File
@@ -1,10 +1,12 @@
#![cfg(not(debug_assertions))]
use beacon_chain::attestation_verification::Error as AttnError;
use beacon_chain::blob_verification::BlockWrapper;
use beacon_chain::builder::BeaconChainBuilder;
use beacon_chain::schema_change::migrate_schema;
use beacon_chain::test_utils::{
test_spec, AttestationStrategy, BeaconChainHarness, BlockStrategy, DiskHarnessType,
mock_execution_layer_from_parts, test_spec, AttestationStrategy, BeaconChainHarness,
BlockStrategy, DiskHarnessType,
};
use beacon_chain::validator_monitor::DEFAULT_INDIVIDUAL_TRACKING_THRESHOLD;
use beacon_chain::{
@@ -12,7 +14,7 @@ use beacon_chain::{
migrate::MigratorConfig, BeaconChain, BeaconChainError, BeaconChainTypes, BeaconSnapshot,
ChainConfig, NotifyExecutionLayer, ServerSentEventHandler, WhenSlotSkipped,
};
use eth2_network_config::TRUSTED_SETUP;
use eth2_network_config::get_trusted_setup;
use kzg::TrustedSetup;
use lazy_static::lazy_static;
use logging::test_logger;
@@ -2113,35 +2115,44 @@ async fn weak_subjectivity_sync() {
let store = get_store(&temp2);
let spec = test_spec::<E>();
let seconds_per_slot = spec.seconds_per_slot;
let trusted_setup: TrustedSetup = serde_json::from_reader(TRUSTED_SETUP)
.map_err(|e| println!("Unable to read trusted setup file: {}", e))
.unwrap();
let trusted_setup: TrustedSetup =
serde_json::from_reader(get_trusted_setup::<<E as EthSpec>::Kzg>())
.map_err(|e| println!("Unable to read trusted setup file: {}", e))
.unwrap();
let mock = mock_execution_layer_from_parts(
&harness.spec,
harness.runtime.task_executor.clone(),
None,
None,
);
// Initialise a new beacon chain from the finalized checkpoint
let beacon_chain = Arc::new(
BeaconChainBuilder::<DiskHarnessType<E>>::new(MinimalEthSpec)
.store(store.clone())
.custom_spec(test_spec::<E>())
.task_executor(harness.chain.task_executor.clone())
.weak_subjectivity_state(wss_state, wss_block.clone(), genesis_state)
.unwrap()
.logger(log.clone())
.store_migrator_config(MigratorConfig::default().blocking())
.dummy_eth1_backend()
.expect("should build dummy backend")
.testing_slot_clock(Duration::from_secs(seconds_per_slot))
.expect("should configure testing slot clock")
.shutdown_sender(shutdown_tx)
.chain_config(ChainConfig::default())
.event_handler(Some(ServerSentEventHandler::new_with_capacity(
log.clone(),
1,
)))
.monitor_validators(true, vec![], DEFAULT_INDIVIDUAL_TRACKING_THRESHOLD, log)
.trusted_setup(trusted_setup)
.build()
.expect("should build"),
);
let beacon_chain = BeaconChainBuilder::<DiskHarnessType<E>>::new(MinimalEthSpec)
.store(store.clone())
.custom_spec(test_spec::<E>())
.task_executor(harness.chain.task_executor.clone())
.weak_subjectivity_state(wss_state, wss_block.clone(), genesis_state)
.unwrap()
.logger(log.clone())
.store_migrator_config(MigratorConfig::default().blocking())
.dummy_eth1_backend()
.expect("should build dummy backend")
.testing_slot_clock(Duration::from_secs(seconds_per_slot))
.expect("should configure testing slot clock")
.shutdown_sender(shutdown_tx)
.chain_config(ChainConfig::default())
.event_handler(Some(ServerSentEventHandler::new_with_capacity(
log.clone(),
1,
)))
.execution_layer(Some(mock.el))
.monitor_validators(true, vec![], DEFAULT_INDIVIDUAL_TRACKING_THRESHOLD, log)
.trusted_setup(trusted_setup)
.build()
.expect("should build");
let beacon_chain = Arc::new(beacon_chain);
// Apply blocks forward to reach head.
let chain_dump = harness.chain.chain_dump().unwrap();
@@ -2150,12 +2161,14 @@ async fn weak_subjectivity_sync() {
assert_eq!(new_blocks[0].beacon_block.slot(), wss_slot + 1);
for snapshot in new_blocks {
let block_root = snapshot.beacon_block_root;
let full_block = harness
.chain
.get_block(&snapshot.beacon_block_root)
.await
.unwrap()
.unwrap();
let blobs = harness.chain.get_blobs(&block_root).expect("blobs");
let slot = full_block.slot();
let state_root = full_block.state_root();
@@ -2163,7 +2176,7 @@ async fn weak_subjectivity_sync() {
beacon_chain
.process_block(
full_block.canonical_root(),
Arc::new(full_block),
BlockWrapper::new(Arc::new(full_block), blobs),
NotifyExecutionLayer::Yes,
)
.await
@@ -2210,16 +2223,19 @@ async fn weak_subjectivity_sync() {
let mut available_blocks = vec![];
for blinded in historical_blocks {
let block_root = blinded.canonical_root();
let full_block = harness
.chain
.get_block(&blinded.canonical_root())
.get_block(&block_root)
.await
.expect("should get block")
.expect("should get block");
let blobs = harness.chain.get_blobs(&block_root).expect("blobs");
if let MaybeAvailableBlock::Available(block) = harness
.chain
.data_availability_checker
.check_availability(full_block.into())
.check_availability(BlockWrapper::new(Arc::new(full_block), blobs))
.expect("should check availability")
{
available_blocks.push(block);
@@ -2338,7 +2354,7 @@ async fn finalizes_after_resuming_from_db() {
.default_spec()
.keypairs(KEYPAIRS[0..validator_count].to_vec())
.resumed_disk_store(store)
.mock_execution_layer()
.execution_layer(original_chain.execution_layer.clone())
.build();
assert_chains_pretty_much_the_same(&original_chain, &resumed_harness.chain);
beacon_node/beacon_chain/tests/tests.rs
+1 -1
View File
@@ -684,7 +684,7 @@ async fn run_skip_slot_test(skip_slots: u64) {
.chain
.process_block(
harness_a.chain.head_snapshot().beacon_block_root,
harness_a.chain.head_snapshot().beacon_block.clone(),
harness_a.get_head_block(),
NotifyExecutionLayer::Yes,
)
.await