cerc-io/lighthouse
16
0
Fork 0
Code Issues 1 Pull Requests Packages Projects Releases Wiki Activity

Re-merge updated block processing to v0.2.0 (#962)

Browse Source 
* Start updating types

* WIP

* Signature hacking

* Existing EF tests passing with fake_crypto

* Updates

* Delete outdated API spec

* The refactor continues

* It compiles

* WIP test fixes

* All release tests passing bar genesis state parsing

* Update and test YamlConfig

* Update to spec v0.10 compatible BLS

* Updates to BLS EF tests

* Add EF test for AggregateVerify

And delete unused hash2curve tests for uncompressed points

* Update EF tests to v0.10.1

* Use optional block root correctly in block proc

* Use genesis fork in deposit domain. All tests pass

* Cargo fmt

* Fast aggregate verify test

* Update REST API docs

* Cargo fmt

* Fix unused import

* Bump spec tags to v0.10.1

* Add `seconds_per_eth1_block` to chainspec

* Update to timestamp based eth1 voting scheme

* Return None from `get_votes_to_consider` if block cache is empty

* Handle overflows in `is_candidate_block`

* Revert to failing tests

* Fix eth1 data sets test

* Choose default vote according to spec

* Fix collect_valid_votes tests

* Fix `get_votes_to_consider` to choose all eligible blocks

* Uncomment winning_vote tests

* Add comments; remove unused code

* Reduce seconds_per_eth1_block for simulation

* Addressed review comments

* Add test for default vote case

* Fix logs

* Remove unused functions

* Meter default eth1 votes

* Fix comments

* Address review comments; remove unused dependency

* Add first attempt at attestation proc. re-write

* Add version 2 of attestation processing

* Minor fixes

* Add validator pubkey cache

* Make get_indexed_attestation take a committee

* Link signature processing into new attn verification

* First working version

* Ensure pubkey cache is updated

* Add more metrics, slight optimizations

* Clone committee cache during attestation processing

* Update shuffling cache during block processing

* Remove old commented-out code

* Fix shuffling cache insert bug

* Used indexed attestation in fork choice

* Restructure attn processing, add metrics

* Add more detailed metrics

* Tidy, fix failing tests

* Fix failing tests, tidy

* Disable/delete two outdated tests

* Add new Pubkeys struct to signature_sets

* Refactor with functional approach

* Update beacon chain

* Remove decompressed member from pubkey bytes

* Add hashmap for indices lookup

* Add state cache, remove store cache

* Only build the head committee cache

* Change `get_attesting_indices` to use Vec

* Fix failing test

* Tidy

* Add pubkey cache persistence file

* Add more comments

* Integrate persistence file into builder

* Add pubkey cache tests

* Add data_dir to beacon chain builder

* Remove Option in pubkey cache persistence file

* Ensure consistency between datadir/data_dir

* Fix failing network test

* Tidy

* Fix todos

* Improve tests

* Fix compile error

* Fix compile error from merge

* Split up block processing metrics

* Tidy

* Refactor get_pubkey_from_state

* Remove commented-out code

* Rename state_cache -> checkpoint_cache

* Rename Checkpoint -> Snapshot

* Tidy, add comments

* Tidy up find_head function

* Change some checkpoint -> snapshot

* Add tests

* Expose max_len

* Remove dead code

* Tidy

* Fix bug

* Add sync-speed metric

* Add first attempt at VerifiableBlock

* Start integrating into beacon chain

* Integrate VerifiableBlock

* Rename VerifableBlock -> PartialBlockVerification

* Add start of typed methods

* Add progress

* Add further progress

* Rename structs

* Add full block verification to block_processing.rs

* Further beacon chain integration

* Update checks for gossip

* Add todo

* Start adding segement verification

* Add passing chain segement test

* Initial integration with batch sync

* Minor changes

* Tidy, add more error checking

* Start adding chain_segment tests

* Finish invalid signature tests

* Include single and gossip verified blocks in tests

* Add gossip verification tests

* Start adding docs

* Finish adding comments to block_processing.rs

* Rename block_processing.rs -> block_verification

* Start removing old block processing code

* Fixes beacon_chain compilation

* Fix project-wide compile errors

* Remove old code

* Fix bug with beacon proposer index

* Fix shim for BlockProcessingError

* Only process one epoch at a time

* Fix loop in chain segment processing

* Add caching for state.eth1_data_votes

* Add BeaconChain::validator_pubkey

* Revert "Add caching for state.eth1_data_votes"

This reverts commit cd73dcd6434fb8d8e6bf30c5356355598ea7b78e.

* Add sync-speed metric (#898)

* Add PH & MS slot clock changes

* Account for genesis time

* Use checked mul

* Account for genesis slot

* Change API

* Allow for clock disparity

* Refactor "duration to..." functions

* Ensure errors are returned during batch processing

Co-authored-by: Michael Sproul <michael@sigmaprime.io>
Co-authored-by: Michael Sproul <micsproul@gmail.com>
Co-authored-by: pawan <pawandhananjay@gmail.com>
Co-authored-by: Paul Hauner <paul@paulhauner.com>
This commit is contained in:
Age Manning 2020-03-26 17:34:06 +11:00
parent 3606f0447d
commit bb065e3d00
No known key found for this signature in database
GPG Key ID: 05EED64B79E06A93
5 changed files with 72 additions and 78 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

113
Cargo.lock generated
View File

@ -86,7 +86,7 @@ dependencies = [
[[package]]
name = "amcl"
version = "0.2.0"
source = "git+https://github.com/sigp/milagro_bls?branch=eth2.0-v0.10#9f946ca454ad1da4cd1344bd632b7306704bc904"
source = "git+https://github.com/sigp/milagro_bls?branch=eth2.0-v0.10#38c6c33925b24c9319a1febfb621ff9bbf6d49f7"
dependencies = [
"hex 0.3.2",
"lazy_static",
@ -144,7 +144,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0d0864d84b8e07b145449be9a8537db86bf9de5ce03b913214694643b4743502"
dependencies = [
"quote 1.0.3",
"syn 1.0.16",
"syn 1.0.17",
]
[[package]]
@ -172,9 +172,9 @@ checksum = "f8aac770f1885fd7e387acedd76065302551364496e46b3dd00860b2f8359b9d"
[[package]]
name = "backtrace"
version = "0.3.45"
version = "0.3.46"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ad235dabf00f36301792cfe82499880ba54c6486be094d1047b02bacb67c14e8"
checksum = "b1e692897359247cc6bb902933361652380af0f1b7651ae5c5013407f30e109e"
dependencies = [
"backtrace-sys",
"cfg-if",
@ -184,9 +184,9 @@ dependencies = [
[[package]]
name = "backtrace-sys"
version = "0.1.34"
version = "0.1.35"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ca797db0057bae1a7aa2eef3283a874695455cecf08a43bfb8507ee0ebc1ed69"
checksum = "7de8aba10a69c8e8d7622c5710229485ec32e9d55fdad160ea559c086fdcd118"
dependencies = [
"cc",
"libc",
@ -430,9 +430,9 @@ dependencies = [
[[package]]
name = "bumpalo"
version = "3.2.0"
version = "3.2.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1f359dc14ff8911330a51ef78022d376f25ed00248912803b58f00cb1c27f742"
checksum = "12ae9db68ad7fac5fe51304d20f016c911539251075a214f8e663babefa35187"
[[package]]
name = "byte-slice-cast"
@ -757,20 +757,11 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "69323bff1fb41c635347b8ead484a5ca6c3f11914d784170b158d8449ab07f8e"
dependencies = [
"cfg-if",
"crossbeam-channel 0.4.2",
"crossbeam-channel",
"crossbeam-deque",
"crossbeam-epoch",
"crossbeam-queue",
"crossbeam-utils 0.7.2",
]
[[package]]
name = "crossbeam-channel"
version = "0.3.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c8ec7fcd21571dc78f96cc96243cab8d8f035247c3efd16c687be154c3fa9efa"
dependencies = [
"crossbeam-utils 0.6.6",
"crossbeam-utils",
]
[[package]]
@ -779,7 +770,7 @@ version = "0.4.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "cced8691919c02aac3cb0a1bc2e9b73d89e832bf9a06fc579d4e71b68a2da061"
dependencies = [
"crossbeam-utils 0.7.2",
"crossbeam-utils",
"maybe-uninit",
]
@ -790,7 +781,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9f02af974daeee82218205558e51ec8768b48cf524bd01d550abe5573a608285"
dependencies = [
"crossbeam-epoch",
"crossbeam-utils 0.7.2",
"crossbeam-utils",
"maybe-uninit",
]
@ -802,7 +793,7 @@ checksum = "058ed274caafc1f60c4997b5fc07bf7dc7cca454af7c6e81edffe5f33f70dace"
dependencies = [
"autocfg 1.0.0",
"cfg-if",
"crossbeam-utils 0.7.2",
"crossbeam-utils",
"lazy_static",
"maybe-uninit",
"memoffset",
@ -816,17 +807,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c695eeca1e7173472a32221542ae469b3e9aac3a4fc81f7696bcad82029493db"
dependencies = [
"cfg-if",
"crossbeam-utils 0.7.2",
]
[[package]]
name = "crossbeam-utils"
version = "0.6.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "04973fa96e96579258a5091af6003abde64af786b860f18622b82e026cca60e6"
dependencies = [
"cfg-if",
"lazy_static",
"crossbeam-utils",
]
[[package]]
@ -1427,7 +1408,7 @@ checksum = "030a733c8287d6213886dd487564ff5c8f6aae10278b3588ed177f9d18f8d231"
dependencies = [
"proc-macro2 1.0.9",
"quote 1.0.3",
"syn 1.0.16",
"syn 1.0.17",
"synstructure",
]
@ -2136,7 +2117,7 @@ version = "0.13.0"
source = "git+https://github.com/SigP/rust-libp2p?rev=44d7a9c9cd7be74109817bcabe74b991d5bd0fee#44d7a9c9cd7be74109817bcabe74b991d5bd0fee"
dependencies = [
"quote 1.0.3",
"syn 1.0.16",
"syn 1.0.17",
]
[[package]]
@ -2674,13 +2655,12 @@ dependencies = [
[[package]]
name = "milagro_bls"
version = "1.0.1"
source = "git+https://github.com/sigp/milagro_bls?branch=eth2.0-v0.10#9f946ca454ad1da4cd1344bd632b7306704bc904"
source = "git+https://github.com/sigp/milagro_bls?branch=eth2.0-v0.10#38c6c33925b24c9319a1febfb621ff9bbf6d49f7"
dependencies = [
"amcl",
"hex 0.4.2",
"lazy_static",
"rand 0.7.3",
"yaml-rust",
"zeroize 1.1.0",
]
@ -3218,14 +3198,9 @@ dependencies = [
[[package]]
name = "proc-macro-hack"
version = "0.5.12"
version = "0.5.14"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f918f2b601f93baa836c1c2945faef682ba5b6d4828ecb45eeb7cc3c71b811b4"
dependencies = [
"proc-macro2 1.0.9",
"quote 1.0.3",
"syn 1.0.16",
]
checksum = "fcfdefadc3d57ca21cf17990a28ef4c0f7c61383a28cb7604cf4a18e6ede1420"
[[package]]
name = "proc-macro2"
@ -3550,7 +3525,7 @@ checksum = "08a89b46efaf957e52b18062fb2f4660f8b8a4dde1807ca002690868ef2c85a9"
dependencies = [
"crossbeam-deque",
"crossbeam-queue",
"crossbeam-utils 0.7.2",
"crossbeam-utils",
"lazy_static",
"num_cpus",
]
@ -3583,9 +3558,9 @@ dependencies = [
[[package]]
name = "regex"
version = "1.3.5"
version = "1.3.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8900ebc1363efa7ea1c399ccc32daed870b4002651e0bed86e72d501ebbe0048"
checksum = "7f6946991529684867e47d86474e3a6d0c0ab9b82d5821e314b1ede31fa3a4b3"
dependencies = [
"aho-corasick",
"memchr",
@ -3761,7 +3736,7 @@ dependencies = [
"base64 0.11.0",
"blake2b_simd",
"constant_time_eq",
"crossbeam-utils 0.7.2",
"crossbeam-utils",
]
[[package]]
@ -3831,9 +3806,9 @@ dependencies = [
[[package]]
name = "schannel"
version = "0.1.17"
version = "0.1.18"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "507a9e6e8ffe0a4e0ebb9a10293e62fdf7657c06f1b8bb07a8fcf697d2abf295"
checksum = "039c25b130bd8c1321ee2d7de7fde2659fa9c2744e4bb29711cfc852ea53cd19"
dependencies = [
"lazy_static",
"winapi 0.3.8",
@ -3918,22 +3893,22 @@ checksum = "a0eddf2e8f50ced781f288c19f18621fa72a3779e3cb58dbf23b07469b0abeb4"
[[package]]
name = "serde"
version = "1.0.104"
version = "1.0.105"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "414115f25f818d7dfccec8ee535d76949ae78584fc4f79a6f45a904bf8ab4449"
checksum = "e707fbbf255b8fc8c3b99abb91e7257a622caeb20a9818cbadbeeede4e0932ff"
dependencies = [
"serde_derive",
]
[[package]]
name = "serde_derive"
version = "1.0.104"
version = "1.0.105"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "128f9e303a5a29922045a830221b8f78ec74a5f544944f3d5984f8ec3895ef64"
checksum = "ac5d00fc561ba2724df6758a17de23df5914f20e41cb00f94d5b7ae42fffaff8"
dependencies = [
"proc-macro2 1.0.9",
"quote 1.0.3",
"syn 1.0.16",
"syn 1.0.17",
]
[[package]]
@ -3963,7 +3938,7 @@ checksum = "cd02c7587ec314570041b2754829f84d873ced14a96d1fd1823531e11db40573"
dependencies = [
"proc-macro2 1.0.9",
"quote 1.0.3",
"syn 1.0.16",
"syn 1.0.17",
]
[[package]]
@ -4081,11 +4056,11 @@ checksum = "1cc9c640a4adbfbcc11ffb95efe5aa7af7309e002adab54b185507dbf2377b99"
[[package]]
name = "slog-async"
version = "2.4.0"
version = "2.5.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "78ca925b180da88ccc595cbe4a3d378d79cb49fe5906c2cbc2488eaf700913ee"
checksum = "51b3336ce47ce2f96673499fc07eb85e3472727b9a7a2959964b002c2ce8fbbb"
dependencies = [
"crossbeam-channel 0.3.9",
"crossbeam-channel",
"slog",
"take_mut",
"thread_local",
@ -4375,9 +4350,9 @@ dependencies = [
[[package]]
name = "syn"
version = "1.0.16"
version = "1.0.17"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "123bd9499cfb380418d509322d7a6d52e5315f064fe4b3ad18a53d6b92c07859"
checksum = "0df0eb663f387145cab623dea85b09c2c5b4b0aef44e945d928e682fce71bb03"
dependencies = [
"proc-macro2 1.0.9",
"quote 1.0.3",
@ -4392,7 +4367,7 @@ checksum = "67656ea1dc1b41b1451851562ea232ec2e5a80242139f7e679ceccfb5d61f545"
dependencies = [
"proc-macro2 1.0.9",
"quote 1.0.3",
"syn 1.0.16",
"syn 1.0.17",
"unicode-xid 0.2.0",
]
@ -4613,7 +4588,7 @@ version = "0.1.10"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "fb2d1b8f4548dbf5e1f7818512e9c406860678f29c300cdf0ebac72d1a3a1671"
dependencies = [
"crossbeam-utils 0.7.2",
"crossbeam-utils",
"futures",
]
@ -4657,7 +4632,7 @@ version = "0.1.12"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "09bc590ec4ba8ba87652da2068d150dcada2cfa2e07faae270a5e0409aa51351"
dependencies = [
"crossbeam-utils 0.7.2",
"crossbeam-utils",
"futures",
"lazy_static",
"log 0.4.8",
@ -4716,7 +4691,7 @@ checksum = "df720b6581784c118f0eb4310796b12b1d242a7eb95f716a8367855325c25f89"
dependencies = [
"crossbeam-deque",
"crossbeam-queue",
"crossbeam-utils 0.7.2",
"crossbeam-utils",
"futures",
"lazy_static",
"log 0.4.8",
@ -4741,7 +4716,7 @@ version = "0.2.13"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "93044f2d313c95ff1cb7809ce9a7a05735b012288a888b62d4434fd58c94f296"
dependencies = [
"crossbeam-utils 0.7.2",
"crossbeam-utils",
"futures",
"slab 0.4.2",
"tokio-executor",
@ -4833,7 +4808,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "edcf0b9b2caa5f4804ef77aeee1b929629853d806117c48258f402b69737e65c"
dependencies = [
"quote 1.0.3",
"syn 1.0.16",
"syn 1.0.17",
]
[[package]]
@ -5189,7 +5164,7 @@ dependencies = [
"log 0.4.8",
"proc-macro2 1.0.9",
"quote 1.0.3",
"syn 1.0.16",
"syn 1.0.17",
"wasm-bindgen-shared",
]
@ -5236,7 +5211,7 @@ checksum = "25bda4168030a6412ea8a047e27238cadf56f0e53516e1e83fec0a8b7c786f6d"
dependencies = [
"proc-macro2 1.0.9",
"quote 1.0.3",
"syn 1.0.16",
"syn 1.0.17",
"wasm-bindgen-backend",
"wasm-bindgen-shared",
]

5
beacon_node/beacon_chain/src/beacon_chain.rs
View File

@ -80,6 +80,11 @@ pub enum AttestationType {
Aggregated,
}
/// The accepted clock drift for nodes gossiping blocks and attestations (spec v0.11.0). See:
///
/// https://github.com/ethereum/eth2.0-specs/blob/v0.11.0/specs/phase0/p2p-interface.md#configuration
pub const MAXIMUM_GOSSIP_CLOCK_DISPARITY: Duration = Duration::from_millis(500);
#[derive(Debug, PartialEq)]
pub enum AttestationProcessingOutcome {
Processed,

19
beacon_node/beacon_chain/src/block_verification.rs
View File

@ -42,10 +42,14 @@
//! ```
use crate::validator_pubkey_cache::ValidatorPubkeyCache;
use crate::{
beacon_chain::{BLOCK_PROCESSING_CACHE_LOCK_TIMEOUT, VALIDATOR_PUBKEY_CACHE_LOCK_TIMEOUT},
beacon_chain::{
BLOCK_PROCESSING_CACHE_LOCK_TIMEOUT, MAXIMUM_GOSSIP_CLOCK_DISPARITY,
VALIDATOR_PUBKEY_CACHE_LOCK_TIMEOUT,
},
metrics, BeaconChain, BeaconChainError, BeaconChainTypes, BeaconSnapshot,
};
use parking_lot::RwLockReadGuard;
use slot_clock::SlotClock;
use state_processing::{
block_signature_verifier::{
BlockSignatureVerifier, Error as BlockSignatureVerifierError, G1Point,
@ -259,19 +263,18 @@ impl<T: BeaconChainTypes> GossipVerifiedBlock<T> {
chain: &BeaconChain<T>,
) -> Result<Self, BlockError> {
// Do not gossip or process blocks from future slots.
//
// TODO: adjust this to allow for clock disparity tolerance.
let present_slot = chain.slot()?;
if block.slot() > present_slot {
let present_slot_with_tolerance = chain
.slot_clock
.now_with_future_tolerance(MAXIMUM_GOSSIP_CLOCK_DISPARITY)
.ok_or_else(|| BeaconChainError::UnableToReadSlot)?;
if block.slot() > present_slot_with_tolerance {
return Err(BlockError::FutureSlot {
present_slot,
present_slot: present_slot_with_tolerance,
block_slot: block.slot(),
});
}
// Do not gossip a block from a finalized slot.
//
// TODO: adjust this to allow for clock disparity tolerance.
check_block_against_finalized_slot(&block.message, chain)?;
// TODO: add check for the `(block.proposer_index, block.slot)` tuple once we have v0.11.0

2
beacon_node/beacon_chain/src/errors.rs
View File

@ -72,8 +72,8 @@ easy_from_to!(SlotProcessingError, BeaconChainError);
easy_from_to!(AttestationValidationError, BeaconChainError);
easy_from_to!(SszTypesError, BeaconChainError);
easy_from_to!(OpPoolError, BeaconChainError);
easy_from_to!(BlockSignatureVerifierError, BeaconChainError);
easy_from_to!(NaiveAggregationError, BeaconChainError);
easy_from_to!(BlockSignatureVerifierError, BeaconChainError);
#[derive(Debug, PartialEq)]
pub enum BlockProductionError {

11
beacon_node/network/src/sync/range_sync/batch_processing.rs
View File

@ -68,6 +68,8 @@ fn process_batch<T: BeaconChainTypes>(
log, "Parent block is unknown";
"parent_root" => format!("{}", parent),
);
return Err(format!("Block has an unknown parent: {}", parent));
}
Err(BlockError::BlockIsAlreadyKnown) => {
// this block is already known to us, move to the next
@ -97,6 +99,11 @@ fn process_batch<T: BeaconChainTypes>(
"FUTURE_SLOT_TOLERANCE" => FUTURE_SLOT_TOLERANCE,
);
}
return Err(format!(
"Block with slot {} is higher than the current slot {}",
block_slot, present_slot
));
}
Err(BlockError::WouldRevertFinalizedSlot { .. }) => {
debug!(
@ -115,6 +122,8 @@ fn process_batch<T: BeaconChainTypes>(
"msg" => "unexpected condition in processing block.",
"outcome" => format!("{:?}", e)
);
return Err(format!("Internal error whilst processing block: {:?}", e));
}
other => {
warn!(
@ -122,6 +131,8 @@ fn process_batch<T: BeaconChainTypes>(
"msg" => "peer sent invalid block",
"outcome" => format!("{:?}", other),
);
return Err(format!("Peer sent invalid block. Reason: {:?}", other));
}
}
} else {