015ab7d0a7
## Issue Addressed Closes #2052 ## Proposed Changes - Refactor the attester/proposer duties endpoints in the BN - Performance improvements - Fixes some potential inconsistencies with the dependent root fields. - Removes `http_api::beacon_proposer_cache` and just uses the one on the `BeaconChain` instead. - Move the code for the proposer/attester duties endpoints into separate files, for readability. - Refactor the `DutiesService` in the VC - Required to reduce the delay on broadcasting new blocks. - Gets rid of the `ValidatorDuty` shim struct that came about when we adopted the standard API. - Separate block/attestation duty tasks so that they don't block each other when one is slow. - In the VC, use `PublicKeyBytes` to represent validators instead of `PublicKey`. `PublicKey` is a legit crypto object whilst `PublicKeyBytes` is just a byte-array, it's much faster to clone/hash `PublicKeyBytes` and this change has had a significant impact on runtimes. - Unfortunately this has created lots of dust changes. - In the BN, store `PublicKeyBytes` in the `beacon_proposer_cache` and allow access to them. The HTTP API always sends `PublicKeyBytes` over the wire and the conversion from `PublicKey` -> `PublickeyBytes` is non-trivial, especially when queries have 100s/1000s of validators (like Pyrmont). - Add the `state_processing::state_advance` mod which dedups a lot of the "apply `n` skip slots to the state" code. - This also fixes a bug with some functions which were failing to include a state root as per [this comment](072695284f/consensus/state_processing/src/state_advance.rs (L69-L74)
). I couldn't find any instance of this bug that resulted in anything more severe than keying a shuffling cache by the wrong block root. - Swap the VC block service to use `mpsc` from `tokio` instead of `futures`. This is consistent with the rest of the code base. ~~This PR *reduces* the size of the codebase 🎉~~ It *used* to reduce the size of the code base before I added more comments. ## Observations on Prymont - Proposer duties times down from peaks of 450ms to consistent <1ms. - Current epoch attester duties times down from >1s peaks to a consistent 20-30ms. - Block production down from +600ms to 100-200ms. ## Additional Info - ~~Blocked on #2241~~ - ~~Blocked on #2234~~ ## TODO - [x] ~~Refactor this into some smaller PRs?~~ Leaving this as-is for now. - [x] Address `per_slot_processing` roots. - [x] Investigate slow next epoch times. Not getting added to cache on block processing? - [x] Consider [this](072695284f/beacon_node/store/src/hot_cold_store.rs (L811-L812)
) in the scenario of replacing the state roots Co-authored-by: pawan <pawandhananjay@gmail.com> Co-authored-by: Michael Sproul <michael@sigmaprime.io>
148 lines
4.3 KiB
Rust
148 lines
4.3 KiB
Rust
use crate::*;
|
|
use tempfile::{tempdir, TempDir};
|
|
use types::{
|
|
test_utils::generate_deterministic_keypair, AttestationData, BeaconBlockHeader, Hash256,
|
|
PublicKeyBytes,
|
|
};
|
|
|
|
pub const DEFAULT_VALIDATOR_INDEX: usize = 0;
|
|
pub const DEFAULT_DOMAIN: Hash256 = Hash256::zero();
|
|
pub const DEFAULT_GENESIS_VALIDATORS_ROOT: Hash256 = Hash256::zero();
|
|
|
|
pub fn pubkey(index: usize) -> PublicKeyBytes {
|
|
generate_deterministic_keypair(index).pk.compress()
|
|
}
|
|
|
|
pub struct Test<T> {
|
|
pubkey: PublicKeyBytes,
|
|
data: T,
|
|
domain: Hash256,
|
|
expected: Result<Safe, NotSafe>,
|
|
}
|
|
|
|
impl<T> Test<T> {
|
|
pub fn single(data: T) -> Self {
|
|
Self::with_pubkey(pubkey(DEFAULT_VALIDATOR_INDEX), data)
|
|
}
|
|
|
|
pub fn with_pubkey(pubkey: PublicKeyBytes, data: T) -> Self {
|
|
Self {
|
|
pubkey,
|
|
data,
|
|
domain: DEFAULT_DOMAIN,
|
|
expected: Ok(Safe::Valid),
|
|
}
|
|
}
|
|
|
|
pub fn with_domain(mut self, domain: Hash256) -> Self {
|
|
self.domain = domain;
|
|
self
|
|
}
|
|
|
|
pub fn expect_result(mut self, result: Result<Safe, NotSafe>) -> Self {
|
|
self.expected = result;
|
|
self
|
|
}
|
|
|
|
pub fn expect_invalid_att(self, error: InvalidAttestation) -> Self {
|
|
self.expect_result(Err(NotSafe::InvalidAttestation(error)))
|
|
}
|
|
|
|
pub fn expect_invalid_block(self, error: InvalidBlock) -> Self {
|
|
self.expect_result(Err(NotSafe::InvalidBlock(error)))
|
|
}
|
|
|
|
pub fn expect_same_data(self) -> Self {
|
|
self.expect_result(Ok(Safe::SameData))
|
|
}
|
|
}
|
|
|
|
pub struct StreamTest<T> {
|
|
/// Validators to register.
|
|
pub registered_validators: Vec<PublicKeyBytes>,
|
|
/// Vector of cases and the value expected when calling `check_and_insert_X`.
|
|
pub cases: Vec<Test<T>>,
|
|
}
|
|
|
|
impl<T> Default for StreamTest<T> {
|
|
fn default() -> Self {
|
|
Self {
|
|
registered_validators: vec![pubkey(DEFAULT_VALIDATOR_INDEX)],
|
|
cases: vec![],
|
|
}
|
|
}
|
|
}
|
|
|
|
impl<T> StreamTest<T> {
|
|
/// The number of test cases that are expected to pass processing successfully.
|
|
fn num_expected_successes(&self) -> usize {
|
|
self.cases
|
|
.iter()
|
|
.filter(|case| case.expected.is_ok())
|
|
.count()
|
|
}
|
|
}
|
|
|
|
impl StreamTest<AttestationData> {
|
|
pub fn run(&self) {
|
|
let dir = tempdir().unwrap();
|
|
let slashing_db_file = dir.path().join("slashing_protection.sqlite");
|
|
let slashing_db = SlashingDatabase::create(&slashing_db_file).unwrap();
|
|
|
|
for pubkey in &self.registered_validators {
|
|
slashing_db.register_validator(*pubkey).unwrap();
|
|
}
|
|
|
|
for (i, test) in self.cases.iter().enumerate() {
|
|
assert_eq!(
|
|
slashing_db.check_and_insert_attestation(&test.pubkey, &test.data, test.domain),
|
|
test.expected,
|
|
"attestation {} not processed as expected",
|
|
i
|
|
);
|
|
}
|
|
|
|
roundtrip_database(&dir, &slashing_db, self.num_expected_successes() == 0);
|
|
}
|
|
}
|
|
|
|
impl StreamTest<BeaconBlockHeader> {
|
|
pub fn run(&self) {
|
|
let dir = tempdir().unwrap();
|
|
let slashing_db_file = dir.path().join("slashing_protection.sqlite");
|
|
let slashing_db = SlashingDatabase::create(&slashing_db_file).unwrap();
|
|
|
|
for pubkey in &self.registered_validators {
|
|
slashing_db.register_validator(*pubkey).unwrap();
|
|
}
|
|
|
|
for (i, test) in self.cases.iter().enumerate() {
|
|
assert_eq!(
|
|
slashing_db.check_and_insert_block_proposal(&test.pubkey, &test.data, test.domain),
|
|
test.expected,
|
|
"attestation {} not processed as expected",
|
|
i
|
|
);
|
|
}
|
|
|
|
roundtrip_database(&dir, &slashing_db, self.num_expected_successes() == 0);
|
|
}
|
|
}
|
|
|
|
fn roundtrip_database(dir: &TempDir, db: &SlashingDatabase, is_empty: bool) {
|
|
let exported = db
|
|
.export_interchange_info(DEFAULT_GENESIS_VALIDATORS_ROOT)
|
|
.unwrap();
|
|
let new_db =
|
|
SlashingDatabase::create(&dir.path().join("roundtrip_slashing_protection.sqlite")).unwrap();
|
|
new_db
|
|
.import_interchange_info(exported.clone(), DEFAULT_GENESIS_VALIDATORS_ROOT)
|
|
.unwrap();
|
|
let reexported = new_db
|
|
.export_interchange_info(DEFAULT_GENESIS_VALIDATORS_ROOT)
|
|
.unwrap();
|
|
|
|
assert_eq!(exported, reexported);
|
|
assert_eq!(is_empty, exported.is_empty());
|
|
}
|