2019-11-27 01:47:46 +00:00
|
|
|
#![cfg(test)]
|
Rename eth2_libp2p to lighthouse_network (#2702)
## Description
The `eth2_libp2p` crate was originally named and designed to incorporate a simple libp2p integration into lighthouse. Since its origins the crates purpose has expanded dramatically. It now houses a lot more sophistication that is specific to lighthouse and no longer just a libp2p integration.
As of this writing it currently houses the following high-level lighthouse-specific logic:
- Lighthouse's implementation of the eth2 RPC protocol and specific encodings/decodings
- Integration and handling of ENRs with respect to libp2p and eth2
- Lighthouse's discovery logic, its integration with discv5 and logic about searching and handling peers.
- Lighthouse's peer manager - This is a large module handling various aspects of Lighthouse's network, such as peer scoring, handling pings and metadata, connection maintenance and recording, etc.
- Lighthouse's peer database - This is a collection of information stored for each individual peer which is specific to lighthouse. We store connection state, sync state, last seen ips and scores etc. The data stored for each peer is designed for various elements of the lighthouse code base such as syncing and the http api.
- Gossipsub scoring - This stores a collection of gossipsub 1.1 scoring mechanisms that are continuously analyssed and updated based on the ethereum 2 networks and how Lighthouse performs on these networks.
- Lighthouse specific types for managing gossipsub topics, sync status and ENR fields
- Lighthouse's network HTTP API metrics - A collection of metrics for lighthouse network monitoring
- Lighthouse's custom configuration of all networking protocols, RPC, gossipsub, discovery, identify and libp2p.
Therefore it makes sense to rename the crate to be more akin to its current purposes, simply that it manages the majority of Lighthouse's network stack. This PR renames this crate to `lighthouse_network`
Co-authored-by: Paul Hauner <paul@paulhauner.com>
2021-10-19 00:30:39 +00:00
|
|
|
use lighthouse_network::rpc::methods::*;
|
2021-11-09 16:42:02 +00:00
|
|
|
use lighthouse_network::{
|
2021-12-02 02:00:39 +00:00
|
|
|
rpc::max_rpc_size, BehaviourEvent, Libp2pEvent, ReportSource, Request, Response,
|
2021-11-09 16:42:02 +00:00
|
|
|
};
|
2020-05-17 11:16:48 +00:00
|
|
|
use slog::{debug, warn, Level};
|
2021-11-09 16:42:02 +00:00
|
|
|
use ssz::Encode;
|
2020-06-12 00:04:50 +00:00
|
|
|
use ssz_types::VariableList;
|
2020-11-28 05:30:57 +00:00
|
|
|
use std::sync::Arc;
|
2019-11-27 01:47:46 +00:00
|
|
|
use std::time::Duration;
|
2020-11-28 05:30:57 +00:00
|
|
|
use tokio::runtime::Runtime;
|
|
|
|
use tokio::time::sleep;
|
Initial work towards v0.2.0 (#924)
* Remove ping protocol
* Initial renaming of network services
* Correct rebasing relative to latest master
* Start updating types
* Adds HashMapDelay struct to utils
* Initial network restructure
* Network restructure. Adds new types for v0.2.0
* Removes build artefacts
* Shift validation to beacon chain
* Temporarily remove gossip validation
This is to be updated to match current optimisation efforts.
* Adds AggregateAndProof
* Begin rebuilding pubsub encoding/decoding
* Signature hacking
* Shift gossipsup decoding into eth2_libp2p
* Existing EF tests passing with fake_crypto
* Shifts block encoding/decoding into RPC
* Delete outdated API spec
* All release tests passing bar genesis state parsing
* Update and test YamlConfig
* Update to spec v0.10 compatible BLS
* Updates to BLS EF tests
* Add EF test for AggregateVerify
And delete unused hash2curve tests for uncompressed points
* Update EF tests to v0.10.1
* Use optional block root correctly in block proc
* Use genesis fork in deposit domain. All tests pass
* Fast aggregate verify test
* Update REST API docs
* Fix unused import
* Bump spec tags to v0.10.1
* Add `seconds_per_eth1_block` to chainspec
* Update to timestamp based eth1 voting scheme
* Return None from `get_votes_to_consider` if block cache is empty
* Handle overflows in `is_candidate_block`
* Revert to failing tests
* Fix eth1 data sets test
* Choose default vote according to spec
* Fix collect_valid_votes tests
* Fix `get_votes_to_consider` to choose all eligible blocks
* Uncomment winning_vote tests
* Add comments; remove unused code
* Reduce seconds_per_eth1_block for simulation
* Addressed review comments
* Add test for default vote case
* Fix logs
* Remove unused functions
* Meter default eth1 votes
* Fix comments
* Progress on attestation service
* Address review comments; remove unused dependency
* Initial work on removing libp2p lock
* Add LRU caches to store (rollup)
* Update attestation validation for DB changes (WIP)
* Initial version of should_forward_block
* Scaffold
* Progress on attestation validation
Also, consolidate prod+testing slot clocks so that they share much
of the same implementation and can both handle sub-slot time changes.
* Removes lock from libp2p service
* Completed network lock removal
* Finish(?) attestation processing
* Correct network termination future
* Add slot check to block check
* Correct fmt issues
* Remove Drop implementation for network service
* Add first attempt at attestation proc. re-write
* Add version 2 of attestation processing
* Minor fixes
* Add validator pubkey cache
* Make get_indexed_attestation take a committee
* Link signature processing into new attn verification
* First working version
* Ensure pubkey cache is updated
* Add more metrics, slight optimizations
* Clone committee cache during attestation processing
* Update shuffling cache during block processing
* Remove old commented-out code
* Fix shuffling cache insert bug
* Used indexed attestation in fork choice
* Restructure attn processing, add metrics
* Add more detailed metrics
* Tidy, fix failing tests
* Fix failing tests, tidy
* Address reviewers suggestions
* Disable/delete two outdated tests
* Modification of validator for subscriptions
* Add slot signing to validator client
* Further progress on validation subscription
* Adds necessary validator subscription functionality
* Add new Pubkeys struct to signature_sets
* Refactor with functional approach
* Update beacon chain
* Clean up validator <-> beacon node http types
* Add aggregator status to ValidatorDuty
* Impl Clone for manual slot clock
* Fix minor errors
* Further progress validator client subscription
* Initial subscription and aggregation handling
* Remove decompressed member from pubkey bytes
* Progress to modifying val client for attestation aggregation
* First draft of validator client upgrade for aggregate attestations
* Add hashmap for indices lookup
* Add state cache, remove store cache
* Only build the head committee cache
* Removes lock on a network channel
* Partially implement beacon node subscription http api
* Correct compilation issues
* Change `get_attesting_indices` to use Vec
* Fix failing test
* Partial implementation of timer
* Adds timer, removes exit_future, http api to op pool
* Partial multiple aggregate attestation handling
* Permits bulk messages accross gossipsub network channel
* Correct compile issues
* Improve gosispsub messaging and correct rest api helpers
* Added global gossipsub subscriptions
* Update validator subscriptions data structs
* Tidy
* Re-structure validator subscriptions
* Initial handling of subscriptions
* Re-structure network service
* Add pubkey cache persistence file
* Add more comments
* Integrate persistence file into builder
* Add pubkey cache tests
* Add HashSetDelay and introduce into attestation service
* Handles validator subscriptions
* Add data_dir to beacon chain builder
* Remove Option in pubkey cache persistence file
* Ensure consistency between datadir/data_dir
* Fix failing network test
* Peer subnet discovery gets queued for future subscriptions
* Reorganise attestation service functions
* Initial wiring of attestation service
* First draft of attestation service timing logic
* Correct minor typos
* Tidy
* Fix todos
* Improve tests
* Add PeerInfo to connected peers mapping
* Fix compile error
* Fix compile error from merge
* Split up block processing metrics
* Tidy
* Refactor get_pubkey_from_state
* Remove commented-out code
* Rename state_cache -> checkpoint_cache
* Rename Checkpoint -> Snapshot
* Tidy, add comments
* Tidy up find_head function
* Change some checkpoint -> snapshot
* Add tests
* Expose max_len
* Remove dead code
* Tidy
* Fix bug
* Add sync-speed metric
* Add first attempt at VerifiableBlock
* Start integrating into beacon chain
* Integrate VerifiableBlock
* Rename VerifableBlock -> PartialBlockVerification
* Add start of typed methods
* Add progress
* Add further progress
* Rename structs
* Add full block verification to block_processing.rs
* Further beacon chain integration
* Update checks for gossip
* Add todo
* Start adding segement verification
* Add passing chain segement test
* Initial integration with batch sync
* Minor changes
* Tidy, add more error checking
* Start adding chain_segment tests
* Finish invalid signature tests
* Include single and gossip verified blocks in tests
* Add gossip verification tests
* Start adding docs
* Finish adding comments to block_processing.rs
* Rename block_processing.rs -> block_verification
* Start removing old block processing code
* Fixes beacon_chain compilation
* Fix project-wide compile errors
* Remove old code
* Correct code to pass all tests
* Fix bug with beacon proposer index
* Fix shim for BlockProcessingError
* Only process one epoch at a time
* Fix loop in chain segment processing
* Correct tests from master merge
* Add caching for state.eth1_data_votes
* Add BeaconChain::validator_pubkey
* Revert "Add caching for state.eth1_data_votes"
This reverts commit cd73dcd6434fb8d8e6bf30c5356355598ea7b78e.
Co-authored-by: Grant Wuerker <gwuerker@gmail.com>
Co-authored-by: Michael Sproul <michael@sigmaprime.io>
Co-authored-by: Michael Sproul <micsproul@gmail.com>
Co-authored-by: pawan <pawandhananjay@gmail.com>
Co-authored-by: Paul Hauner <paul@paulhauner.com>
2020-03-17 06:24:44 +00:00
|
|
|
use types::{
|
2021-12-02 02:00:39 +00:00
|
|
|
BeaconBlock, BeaconBlockAltair, BeaconBlockBase, BeaconBlockMerge, Epoch, EthSpec, ForkContext,
|
|
|
|
Hash256, MinimalEthSpec, Signature, SignedBeaconBlock, Slot,
|
Initial work towards v0.2.0 (#924)
* Remove ping protocol
* Initial renaming of network services
* Correct rebasing relative to latest master
* Start updating types
* Adds HashMapDelay struct to utils
* Initial network restructure
* Network restructure. Adds new types for v0.2.0
* Removes build artefacts
* Shift validation to beacon chain
* Temporarily remove gossip validation
This is to be updated to match current optimisation efforts.
* Adds AggregateAndProof
* Begin rebuilding pubsub encoding/decoding
* Signature hacking
* Shift gossipsup decoding into eth2_libp2p
* Existing EF tests passing with fake_crypto
* Shifts block encoding/decoding into RPC
* Delete outdated API spec
* All release tests passing bar genesis state parsing
* Update and test YamlConfig
* Update to spec v0.10 compatible BLS
* Updates to BLS EF tests
* Add EF test for AggregateVerify
And delete unused hash2curve tests for uncompressed points
* Update EF tests to v0.10.1
* Use optional block root correctly in block proc
* Use genesis fork in deposit domain. All tests pass
* Fast aggregate verify test
* Update REST API docs
* Fix unused import
* Bump spec tags to v0.10.1
* Add `seconds_per_eth1_block` to chainspec
* Update to timestamp based eth1 voting scheme
* Return None from `get_votes_to_consider` if block cache is empty
* Handle overflows in `is_candidate_block`
* Revert to failing tests
* Fix eth1 data sets test
* Choose default vote according to spec
* Fix collect_valid_votes tests
* Fix `get_votes_to_consider` to choose all eligible blocks
* Uncomment winning_vote tests
* Add comments; remove unused code
* Reduce seconds_per_eth1_block for simulation
* Addressed review comments
* Add test for default vote case
* Fix logs
* Remove unused functions
* Meter default eth1 votes
* Fix comments
* Progress on attestation service
* Address review comments; remove unused dependency
* Initial work on removing libp2p lock
* Add LRU caches to store (rollup)
* Update attestation validation for DB changes (WIP)
* Initial version of should_forward_block
* Scaffold
* Progress on attestation validation
Also, consolidate prod+testing slot clocks so that they share much
of the same implementation and can both handle sub-slot time changes.
* Removes lock from libp2p service
* Completed network lock removal
* Finish(?) attestation processing
* Correct network termination future
* Add slot check to block check
* Correct fmt issues
* Remove Drop implementation for network service
* Add first attempt at attestation proc. re-write
* Add version 2 of attestation processing
* Minor fixes
* Add validator pubkey cache
* Make get_indexed_attestation take a committee
* Link signature processing into new attn verification
* First working version
* Ensure pubkey cache is updated
* Add more metrics, slight optimizations
* Clone committee cache during attestation processing
* Update shuffling cache during block processing
* Remove old commented-out code
* Fix shuffling cache insert bug
* Used indexed attestation in fork choice
* Restructure attn processing, add metrics
* Add more detailed metrics
* Tidy, fix failing tests
* Fix failing tests, tidy
* Address reviewers suggestions
* Disable/delete two outdated tests
* Modification of validator for subscriptions
* Add slot signing to validator client
* Further progress on validation subscription
* Adds necessary validator subscription functionality
* Add new Pubkeys struct to signature_sets
* Refactor with functional approach
* Update beacon chain
* Clean up validator <-> beacon node http types
* Add aggregator status to ValidatorDuty
* Impl Clone for manual slot clock
* Fix minor errors
* Further progress validator client subscription
* Initial subscription and aggregation handling
* Remove decompressed member from pubkey bytes
* Progress to modifying val client for attestation aggregation
* First draft of validator client upgrade for aggregate attestations
* Add hashmap for indices lookup
* Add state cache, remove store cache
* Only build the head committee cache
* Removes lock on a network channel
* Partially implement beacon node subscription http api
* Correct compilation issues
* Change `get_attesting_indices` to use Vec
* Fix failing test
* Partial implementation of timer
* Adds timer, removes exit_future, http api to op pool
* Partial multiple aggregate attestation handling
* Permits bulk messages accross gossipsub network channel
* Correct compile issues
* Improve gosispsub messaging and correct rest api helpers
* Added global gossipsub subscriptions
* Update validator subscriptions data structs
* Tidy
* Re-structure validator subscriptions
* Initial handling of subscriptions
* Re-structure network service
* Add pubkey cache persistence file
* Add more comments
* Integrate persistence file into builder
* Add pubkey cache tests
* Add HashSetDelay and introduce into attestation service
* Handles validator subscriptions
* Add data_dir to beacon chain builder
* Remove Option in pubkey cache persistence file
* Ensure consistency between datadir/data_dir
* Fix failing network test
* Peer subnet discovery gets queued for future subscriptions
* Reorganise attestation service functions
* Initial wiring of attestation service
* First draft of attestation service timing logic
* Correct minor typos
* Tidy
* Fix todos
* Improve tests
* Add PeerInfo to connected peers mapping
* Fix compile error
* Fix compile error from merge
* Split up block processing metrics
* Tidy
* Refactor get_pubkey_from_state
* Remove commented-out code
* Rename state_cache -> checkpoint_cache
* Rename Checkpoint -> Snapshot
* Tidy, add comments
* Tidy up find_head function
* Change some checkpoint -> snapshot
* Add tests
* Expose max_len
* Remove dead code
* Tidy
* Fix bug
* Add sync-speed metric
* Add first attempt at VerifiableBlock
* Start integrating into beacon chain
* Integrate VerifiableBlock
* Rename VerifableBlock -> PartialBlockVerification
* Add start of typed methods
* Add progress
* Add further progress
* Rename structs
* Add full block verification to block_processing.rs
* Further beacon chain integration
* Update checks for gossip
* Add todo
* Start adding segement verification
* Add passing chain segement test
* Initial integration with batch sync
* Minor changes
* Tidy, add more error checking
* Start adding chain_segment tests
* Finish invalid signature tests
* Include single and gossip verified blocks in tests
* Add gossip verification tests
* Start adding docs
* Finish adding comments to block_processing.rs
* Rename block_processing.rs -> block_verification
* Start removing old block processing code
* Fixes beacon_chain compilation
* Fix project-wide compile errors
* Remove old code
* Correct code to pass all tests
* Fix bug with beacon proposer index
* Fix shim for BlockProcessingError
* Only process one epoch at a time
* Fix loop in chain segment processing
* Correct tests from master merge
* Add caching for state.eth1_data_votes
* Add BeaconChain::validator_pubkey
* Revert "Add caching for state.eth1_data_votes"
This reverts commit cd73dcd6434fb8d8e6bf30c5356355598ea7b78e.
Co-authored-by: Grant Wuerker <gwuerker@gmail.com>
Co-authored-by: Michael Sproul <michael@sigmaprime.io>
Co-authored-by: Michael Sproul <micsproul@gmail.com>
Co-authored-by: pawan <pawandhananjay@gmail.com>
Co-authored-by: Paul Hauner <paul@paulhauner.com>
2020-03-17 06:24:44 +00:00
|
|
|
};
|
2019-11-27 01:47:46 +00:00
|
|
|
|
|
|
|
mod common;
|
|
|
|
|
Initial work towards v0.2.0 (#924)
* Remove ping protocol
* Initial renaming of network services
* Correct rebasing relative to latest master
* Start updating types
* Adds HashMapDelay struct to utils
* Initial network restructure
* Network restructure. Adds new types for v0.2.0
* Removes build artefacts
* Shift validation to beacon chain
* Temporarily remove gossip validation
This is to be updated to match current optimisation efforts.
* Adds AggregateAndProof
* Begin rebuilding pubsub encoding/decoding
* Signature hacking
* Shift gossipsup decoding into eth2_libp2p
* Existing EF tests passing with fake_crypto
* Shifts block encoding/decoding into RPC
* Delete outdated API spec
* All release tests passing bar genesis state parsing
* Update and test YamlConfig
* Update to spec v0.10 compatible BLS
* Updates to BLS EF tests
* Add EF test for AggregateVerify
And delete unused hash2curve tests for uncompressed points
* Update EF tests to v0.10.1
* Use optional block root correctly in block proc
* Use genesis fork in deposit domain. All tests pass
* Fast aggregate verify test
* Update REST API docs
* Fix unused import
* Bump spec tags to v0.10.1
* Add `seconds_per_eth1_block` to chainspec
* Update to timestamp based eth1 voting scheme
* Return None from `get_votes_to_consider` if block cache is empty
* Handle overflows in `is_candidate_block`
* Revert to failing tests
* Fix eth1 data sets test
* Choose default vote according to spec
* Fix collect_valid_votes tests
* Fix `get_votes_to_consider` to choose all eligible blocks
* Uncomment winning_vote tests
* Add comments; remove unused code
* Reduce seconds_per_eth1_block for simulation
* Addressed review comments
* Add test for default vote case
* Fix logs
* Remove unused functions
* Meter default eth1 votes
* Fix comments
* Progress on attestation service
* Address review comments; remove unused dependency
* Initial work on removing libp2p lock
* Add LRU caches to store (rollup)
* Update attestation validation for DB changes (WIP)
* Initial version of should_forward_block
* Scaffold
* Progress on attestation validation
Also, consolidate prod+testing slot clocks so that they share much
of the same implementation and can both handle sub-slot time changes.
* Removes lock from libp2p service
* Completed network lock removal
* Finish(?) attestation processing
* Correct network termination future
* Add slot check to block check
* Correct fmt issues
* Remove Drop implementation for network service
* Add first attempt at attestation proc. re-write
* Add version 2 of attestation processing
* Minor fixes
* Add validator pubkey cache
* Make get_indexed_attestation take a committee
* Link signature processing into new attn verification
* First working version
* Ensure pubkey cache is updated
* Add more metrics, slight optimizations
* Clone committee cache during attestation processing
* Update shuffling cache during block processing
* Remove old commented-out code
* Fix shuffling cache insert bug
* Used indexed attestation in fork choice
* Restructure attn processing, add metrics
* Add more detailed metrics
* Tidy, fix failing tests
* Fix failing tests, tidy
* Address reviewers suggestions
* Disable/delete two outdated tests
* Modification of validator for subscriptions
* Add slot signing to validator client
* Further progress on validation subscription
* Adds necessary validator subscription functionality
* Add new Pubkeys struct to signature_sets
* Refactor with functional approach
* Update beacon chain
* Clean up validator <-> beacon node http types
* Add aggregator status to ValidatorDuty
* Impl Clone for manual slot clock
* Fix minor errors
* Further progress validator client subscription
* Initial subscription and aggregation handling
* Remove decompressed member from pubkey bytes
* Progress to modifying val client for attestation aggregation
* First draft of validator client upgrade for aggregate attestations
* Add hashmap for indices lookup
* Add state cache, remove store cache
* Only build the head committee cache
* Removes lock on a network channel
* Partially implement beacon node subscription http api
* Correct compilation issues
* Change `get_attesting_indices` to use Vec
* Fix failing test
* Partial implementation of timer
* Adds timer, removes exit_future, http api to op pool
* Partial multiple aggregate attestation handling
* Permits bulk messages accross gossipsub network channel
* Correct compile issues
* Improve gosispsub messaging and correct rest api helpers
* Added global gossipsub subscriptions
* Update validator subscriptions data structs
* Tidy
* Re-structure validator subscriptions
* Initial handling of subscriptions
* Re-structure network service
* Add pubkey cache persistence file
* Add more comments
* Integrate persistence file into builder
* Add pubkey cache tests
* Add HashSetDelay and introduce into attestation service
* Handles validator subscriptions
* Add data_dir to beacon chain builder
* Remove Option in pubkey cache persistence file
* Ensure consistency between datadir/data_dir
* Fix failing network test
* Peer subnet discovery gets queued for future subscriptions
* Reorganise attestation service functions
* Initial wiring of attestation service
* First draft of attestation service timing logic
* Correct minor typos
* Tidy
* Fix todos
* Improve tests
* Add PeerInfo to connected peers mapping
* Fix compile error
* Fix compile error from merge
* Split up block processing metrics
* Tidy
* Refactor get_pubkey_from_state
* Remove commented-out code
* Rename state_cache -> checkpoint_cache
* Rename Checkpoint -> Snapshot
* Tidy, add comments
* Tidy up find_head function
* Change some checkpoint -> snapshot
* Add tests
* Expose max_len
* Remove dead code
* Tidy
* Fix bug
* Add sync-speed metric
* Add first attempt at VerifiableBlock
* Start integrating into beacon chain
* Integrate VerifiableBlock
* Rename VerifableBlock -> PartialBlockVerification
* Add start of typed methods
* Add progress
* Add further progress
* Rename structs
* Add full block verification to block_processing.rs
* Further beacon chain integration
* Update checks for gossip
* Add todo
* Start adding segement verification
* Add passing chain segement test
* Initial integration with batch sync
* Minor changes
* Tidy, add more error checking
* Start adding chain_segment tests
* Finish invalid signature tests
* Include single and gossip verified blocks in tests
* Add gossip verification tests
* Start adding docs
* Finish adding comments to block_processing.rs
* Rename block_processing.rs -> block_verification
* Start removing old block processing code
* Fixes beacon_chain compilation
* Fix project-wide compile errors
* Remove old code
* Correct code to pass all tests
* Fix bug with beacon proposer index
* Fix shim for BlockProcessingError
* Only process one epoch at a time
* Fix loop in chain segment processing
* Correct tests from master merge
* Add caching for state.eth1_data_votes
* Add BeaconChain::validator_pubkey
* Revert "Add caching for state.eth1_data_votes"
This reverts commit cd73dcd6434fb8d8e6bf30c5356355598ea7b78e.
Co-authored-by: Grant Wuerker <gwuerker@gmail.com>
Co-authored-by: Michael Sproul <michael@sigmaprime.io>
Co-authored-by: Michael Sproul <micsproul@gmail.com>
Co-authored-by: pawan <pawandhananjay@gmail.com>
Co-authored-by: Paul Hauner <paul@paulhauner.com>
2020-03-17 06:24:44 +00:00
|
|
|
type E = MinimalEthSpec;
|
|
|
|
|
2021-12-02 02:00:39 +00:00
|
|
|
/// Merge block with length < max_rpc_size.
|
|
|
|
fn merge_block_small(fork_context: &ForkContext) -> BeaconBlock<E> {
|
2021-11-09 16:42:02 +00:00
|
|
|
let mut block = BeaconBlockMerge::empty(&E::default_spec());
|
|
|
|
let tx = VariableList::from(vec![0; 1024]);
|
|
|
|
let txs = VariableList::from(std::iter::repeat(tx).take(100).collect::<Vec<_>>());
|
|
|
|
|
|
|
|
block.body.execution_payload.transactions = txs;
|
|
|
|
|
|
|
|
let block = BeaconBlock::Merge(block);
|
2021-12-02 02:00:39 +00:00
|
|
|
assert!(block.ssz_bytes_len() <= max_rpc_size(fork_context));
|
2021-11-09 16:42:02 +00:00
|
|
|
block
|
|
|
|
}
|
|
|
|
|
|
|
|
/// Merge block with length > MAX_RPC_SIZE.
|
|
|
|
/// The max limit for a merge block is in the order of ~16GiB which wouldn't fit in memory.
|
|
|
|
/// Hence, we generate a merge block just greater than `MAX_RPC_SIZE` to test rejection on the rpc layer.
|
2021-12-02 02:00:39 +00:00
|
|
|
fn merge_block_large(fork_context: &ForkContext) -> BeaconBlock<E> {
|
2021-11-09 16:42:02 +00:00
|
|
|
let mut block = BeaconBlockMerge::empty(&E::default_spec());
|
|
|
|
let tx = VariableList::from(vec![0; 1024]);
|
|
|
|
let txs = VariableList::from(std::iter::repeat(tx).take(100000).collect::<Vec<_>>());
|
|
|
|
|
|
|
|
block.body.execution_payload.transactions = txs;
|
|
|
|
|
|
|
|
let block = BeaconBlock::Merge(block);
|
2021-12-02 02:00:39 +00:00
|
|
|
assert!(block.ssz_bytes_len() > max_rpc_size(fork_context));
|
2021-11-09 16:42:02 +00:00
|
|
|
block
|
|
|
|
}
|
|
|
|
|
2019-11-27 01:47:46 +00:00
|
|
|
// Tests the STATUS RPC message
|
2020-11-28 05:30:57 +00:00
|
|
|
#[test]
|
2021-01-28 23:31:06 +00:00
|
|
|
#[allow(clippy::single_match)]
|
2020-11-28 05:30:57 +00:00
|
|
|
fn test_status_rpc() {
|
2019-11-27 01:47:46 +00:00
|
|
|
// set up the logging. The level and enabled logging or not
|
2020-05-17 11:16:48 +00:00
|
|
|
let log_level = Level::Debug;
|
2019-11-27 01:47:46 +00:00
|
|
|
let enable_logging = false;
|
|
|
|
|
2020-11-28 05:30:57 +00:00
|
|
|
let rt = Arc::new(Runtime::new().unwrap());
|
|
|
|
|
2019-11-27 01:47:46 +00:00
|
|
|
let log = common::build_log(log_level, enable_logging);
|
|
|
|
|
2020-11-28 05:30:57 +00:00
|
|
|
rt.block_on(async {
|
|
|
|
// get sender/receiver
|
|
|
|
let (mut sender, mut receiver) = common::build_node_pair(Arc::downgrade(&rt), &log).await;
|
|
|
|
|
|
|
|
// Dummy STATUS RPC message
|
|
|
|
let rpc_request = Request::Status(StatusMessage {
|
|
|
|
fork_digest: [0; 4],
|
|
|
|
finalized_root: Hash256::from_low_u64_be(0),
|
|
|
|
finalized_epoch: Epoch::new(1),
|
|
|
|
head_root: Hash256::from_low_u64_be(0),
|
|
|
|
head_slot: Slot::new(1),
|
|
|
|
});
|
|
|
|
|
|
|
|
// Dummy STATUS RPC message
|
|
|
|
let rpc_response = Response::Status(StatusMessage {
|
|
|
|
fork_digest: [0; 4],
|
|
|
|
finalized_root: Hash256::from_low_u64_be(0),
|
|
|
|
finalized_epoch: Epoch::new(1),
|
|
|
|
head_root: Hash256::from_low_u64_be(0),
|
|
|
|
head_slot: Slot::new(1),
|
|
|
|
});
|
|
|
|
|
|
|
|
// build the sender future
|
|
|
|
let sender_future = async {
|
|
|
|
loop {
|
|
|
|
match sender.next_event().await {
|
2021-07-13 00:48:33 +00:00
|
|
|
Libp2pEvent::Behaviour(BehaviourEvent::PeerConnectedOutgoing(peer_id)) => {
|
2020-11-28 05:30:57 +00:00
|
|
|
// Send a STATUS message
|
|
|
|
debug!(log, "Sending RPC");
|
2021-06-03 01:11:33 +00:00
|
|
|
sender.swarm.behaviour_mut().send_request(
|
2020-11-28 05:30:57 +00:00
|
|
|
peer_id,
|
|
|
|
RequestId::Sync(10),
|
|
|
|
rpc_request.clone(),
|
|
|
|
);
|
|
|
|
}
|
|
|
|
Libp2pEvent::Behaviour(BehaviourEvent::ResponseReceived {
|
|
|
|
peer_id: _,
|
|
|
|
id: RequestId::Sync(10),
|
|
|
|
response,
|
|
|
|
}) => {
|
|
|
|
// Should receive the RPC response
|
|
|
|
debug!(log, "Sender Received");
|
|
|
|
assert_eq!(response, rpc_response.clone());
|
|
|
|
debug!(log, "Sender Completed");
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
_ => {}
|
2020-06-05 03:07:59 +00:00
|
|
|
}
|
2020-05-17 11:16:48 +00:00
|
|
|
}
|
2020-11-28 05:30:57 +00:00
|
|
|
};
|
|
|
|
|
|
|
|
// build the receiver future
|
|
|
|
let receiver_future = async {
|
|
|
|
loop {
|
|
|
|
match receiver.next_event().await {
|
|
|
|
Libp2pEvent::Behaviour(BehaviourEvent::RequestReceived {
|
|
|
|
peer_id,
|
|
|
|
id,
|
|
|
|
request,
|
|
|
|
}) => {
|
|
|
|
if request == rpc_request {
|
|
|
|
// send the response
|
|
|
|
debug!(log, "Receiver Received");
|
2021-06-03 01:11:33 +00:00
|
|
|
receiver.swarm.behaviour_mut().send_successful_response(
|
2020-11-28 05:30:57 +00:00
|
|
|
peer_id,
|
|
|
|
id,
|
|
|
|
rpc_response.clone(),
|
|
|
|
);
|
|
|
|
}
|
2019-11-27 01:47:46 +00:00
|
|
|
}
|
2020-11-28 05:30:57 +00:00
|
|
|
_ => {} // Ignore other events
|
2020-05-17 11:16:48 +00:00
|
|
|
}
|
2019-11-27 01:47:46 +00:00
|
|
|
}
|
2020-11-28 05:30:57 +00:00
|
|
|
};
|
2019-11-27 01:47:46 +00:00
|
|
|
|
2020-11-28 05:30:57 +00:00
|
|
|
tokio::select! {
|
|
|
|
_ = sender_future => {}
|
|
|
|
_ = receiver_future => {}
|
|
|
|
_ = sleep(Duration::from_secs(30)) => {
|
|
|
|
panic!("Future timed out");
|
|
|
|
}
|
2020-05-17 11:16:48 +00:00
|
|
|
}
|
2020-11-28 05:30:57 +00:00
|
|
|
})
|
2019-11-27 01:47:46 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
// Tests a streamed BlocksByRange RPC Message
|
2020-11-28 05:30:57 +00:00
|
|
|
#[test]
|
2021-01-28 23:31:06 +00:00
|
|
|
#[allow(clippy::single_match)]
|
2020-11-28 05:30:57 +00:00
|
|
|
fn test_blocks_by_range_chunked_rpc() {
|
2019-11-27 01:47:46 +00:00
|
|
|
// set up the logging. The level and enabled logging or not
|
2021-11-09 16:42:02 +00:00
|
|
|
let log_level = Level::Debug;
|
2019-11-27 01:47:46 +00:00
|
|
|
let enable_logging = false;
|
|
|
|
|
2021-11-09 16:42:02 +00:00
|
|
|
let messages_to_send = 6;
|
2019-11-27 01:47:46 +00:00
|
|
|
|
|
|
|
let log = common::build_log(log_level, enable_logging);
|
|
|
|
|
2020-11-28 05:30:57 +00:00
|
|
|
let rt = Arc::new(Runtime::new().unwrap());
|
|
|
|
|
|
|
|
rt.block_on(async {
|
|
|
|
// get sender/receiver
|
|
|
|
let (mut sender, mut receiver) = common::build_node_pair(Arc::downgrade(&rt), &log).await;
|
|
|
|
|
|
|
|
// BlocksByRange Request
|
|
|
|
let rpc_request = Request::BlocksByRange(BlocksByRangeRequest {
|
|
|
|
start_slot: 0,
|
|
|
|
count: messages_to_send,
|
|
|
|
step: 0,
|
|
|
|
});
|
|
|
|
|
|
|
|
let spec = E::default_spec();
|
2021-09-17 01:11:16 +00:00
|
|
|
|
|
|
|
// BlocksByRange Response
|
|
|
|
let full_block = BeaconBlock::Base(BeaconBlockBase::<E>::full(&spec));
|
|
|
|
let signed_full_block = SignedBeaconBlock::from_block(full_block, Signature::empty());
|
|
|
|
let rpc_response_base = Response::BlocksByRange(Some(Box::new(signed_full_block)));
|
|
|
|
|
|
|
|
let full_block = BeaconBlock::Altair(BeaconBlockAltair::<E>::full(&spec));
|
|
|
|
let signed_full_block = SignedBeaconBlock::from_block(full_block, Signature::empty());
|
|
|
|
let rpc_response_altair = Response::BlocksByRange(Some(Box::new(signed_full_block)));
|
2020-11-28 05:30:57 +00:00
|
|
|
|
2021-12-02 02:00:39 +00:00
|
|
|
let full_block = merge_block_small(&common::fork_context());
|
2021-11-09 16:42:02 +00:00
|
|
|
let signed_full_block = SignedBeaconBlock::from_block(full_block, Signature::empty());
|
|
|
|
let rpc_response_merge_small = Response::BlocksByRange(Some(Box::new(signed_full_block)));
|
|
|
|
|
2020-11-28 05:30:57 +00:00
|
|
|
// keep count of the number of messages received
|
|
|
|
let mut messages_received = 0;
|
2021-11-09 16:42:02 +00:00
|
|
|
let request_id = RequestId::Sync(messages_to_send as usize);
|
2020-11-28 05:30:57 +00:00
|
|
|
// build the sender future
|
|
|
|
let sender_future = async {
|
|
|
|
loop {
|
|
|
|
match sender.next_event().await {
|
2021-07-13 00:48:33 +00:00
|
|
|
Libp2pEvent::Behaviour(BehaviourEvent::PeerConnectedOutgoing(peer_id)) => {
|
2020-11-28 05:30:57 +00:00
|
|
|
// Send a STATUS message
|
|
|
|
debug!(log, "Sending RPC");
|
2021-06-03 01:11:33 +00:00
|
|
|
sender.swarm.behaviour_mut().send_request(
|
2020-11-28 05:30:57 +00:00
|
|
|
peer_id,
|
2021-11-09 16:42:02 +00:00
|
|
|
request_id,
|
2020-11-28 05:30:57 +00:00
|
|
|
rpc_request.clone(),
|
|
|
|
);
|
|
|
|
}
|
|
|
|
Libp2pEvent::Behaviour(BehaviourEvent::ResponseReceived {
|
|
|
|
peer_id: _,
|
2021-11-09 16:42:02 +00:00
|
|
|
id: _,
|
2020-11-28 05:30:57 +00:00
|
|
|
response,
|
|
|
|
}) => {
|
|
|
|
warn!(log, "Sender received a response");
|
|
|
|
match response {
|
|
|
|
Response::BlocksByRange(Some(_)) => {
|
2021-11-09 16:42:02 +00:00
|
|
|
if messages_received < 2 {
|
2021-09-17 01:11:16 +00:00
|
|
|
assert_eq!(response, rpc_response_base.clone());
|
2021-11-09 16:42:02 +00:00
|
|
|
} else if messages_received < 4 {
|
2021-09-17 01:11:16 +00:00
|
|
|
assert_eq!(response, rpc_response_altair.clone());
|
2021-11-09 16:42:02 +00:00
|
|
|
} else {
|
|
|
|
assert_eq!(response, rpc_response_merge_small.clone());
|
2021-09-17 01:11:16 +00:00
|
|
|
}
|
2020-11-28 05:30:57 +00:00
|
|
|
messages_received += 1;
|
|
|
|
warn!(log, "Chunk received");
|
|
|
|
}
|
|
|
|
Response::BlocksByRange(None) => {
|
2021-11-09 16:42:02 +00:00
|
|
|
// should be exactly `messages_to_send` messages before terminating
|
2020-11-28 05:30:57 +00:00
|
|
|
assert_eq!(messages_received, messages_to_send);
|
|
|
|
// end the test
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
_ => panic!("Invalid RPC received"),
|
2020-06-05 03:07:59 +00:00
|
|
|
}
|
2019-11-27 01:47:46 +00:00
|
|
|
}
|
2020-11-28 05:30:57 +00:00
|
|
|
_ => {} // Ignore other behaviour events
|
2020-06-05 03:07:59 +00:00
|
|
|
}
|
2020-05-17 11:16:48 +00:00
|
|
|
}
|
2020-11-28 05:30:57 +00:00
|
|
|
};
|
|
|
|
|
|
|
|
// build the receiver future
|
|
|
|
let receiver_future = async {
|
|
|
|
loop {
|
|
|
|
match receiver.next_event().await {
|
|
|
|
Libp2pEvent::Behaviour(BehaviourEvent::RequestReceived {
|
|
|
|
peer_id,
|
|
|
|
id,
|
|
|
|
request,
|
|
|
|
}) => {
|
|
|
|
if request == rpc_request {
|
|
|
|
// send the response
|
|
|
|
warn!(log, "Receiver got request");
|
2021-09-17 01:11:16 +00:00
|
|
|
for i in 0..messages_to_send {
|
2021-11-09 16:42:02 +00:00
|
|
|
// Send first third of responses as base blocks,
|
|
|
|
// second as altair and third as merge.
|
|
|
|
let rpc_response = if i < 2 {
|
2021-09-17 01:11:16 +00:00
|
|
|
rpc_response_base.clone()
|
2021-11-09 16:42:02 +00:00
|
|
|
} else if i < 4 {
|
2021-09-17 01:11:16 +00:00
|
|
|
rpc_response_altair.clone()
|
2021-11-09 16:42:02 +00:00
|
|
|
} else {
|
|
|
|
rpc_response_merge_small.clone()
|
2021-09-17 01:11:16 +00:00
|
|
|
};
|
2021-06-03 01:11:33 +00:00
|
|
|
receiver.swarm.behaviour_mut().send_successful_response(
|
2021-01-28 23:31:06 +00:00
|
|
|
peer_id,
|
2020-11-28 05:30:57 +00:00
|
|
|
id,
|
|
|
|
rpc_response.clone(),
|
|
|
|
);
|
|
|
|
}
|
|
|
|
// send the stream termination
|
2021-06-03 01:11:33 +00:00
|
|
|
receiver.swarm.behaviour_mut().send_successful_response(
|
2020-11-28 05:30:57 +00:00
|
|
|
peer_id,
|
2020-06-05 03:07:59 +00:00
|
|
|
id,
|
2020-11-28 05:30:57 +00:00
|
|
|
Response::BlocksByRange(None),
|
2020-06-05 03:07:59 +00:00
|
|
|
);
|
2019-11-27 01:47:46 +00:00
|
|
|
}
|
|
|
|
}
|
2020-11-28 05:30:57 +00:00
|
|
|
_ => {} // Ignore other events
|
2020-05-17 11:16:48 +00:00
|
|
|
}
|
2019-11-27 01:47:46 +00:00
|
|
|
}
|
2020-11-28 05:30:57 +00:00
|
|
|
};
|
2019-11-27 01:47:46 +00:00
|
|
|
|
2020-11-28 05:30:57 +00:00
|
|
|
tokio::select! {
|
|
|
|
_ = sender_future => {}
|
|
|
|
_ = receiver_future => {}
|
2021-11-09 16:42:02 +00:00
|
|
|
_ = sleep(Duration::from_secs(30)) => {
|
|
|
|
panic!("Future timed out");
|
|
|
|
}
|
|
|
|
}
|
|
|
|
})
|
|
|
|
}
|
|
|
|
|
|
|
|
// Tests rejection of blocks over `MAX_RPC_SIZE`.
|
|
|
|
#[test]
|
|
|
|
#[allow(clippy::single_match)]
|
|
|
|
fn test_blocks_by_range_over_limit() {
|
|
|
|
// set up the logging. The level and enabled logging or not
|
|
|
|
let log_level = Level::Debug;
|
|
|
|
let enable_logging = false;
|
|
|
|
|
|
|
|
let messages_to_send = 5;
|
|
|
|
|
|
|
|
let log = common::build_log(log_level, enable_logging);
|
|
|
|
|
|
|
|
let rt = Arc::new(Runtime::new().unwrap());
|
|
|
|
|
|
|
|
rt.block_on(async {
|
|
|
|
// get sender/receiver
|
|
|
|
let (mut sender, mut receiver) = common::build_node_pair(Arc::downgrade(&rt), &log).await;
|
|
|
|
|
|
|
|
// BlocksByRange Request
|
|
|
|
let rpc_request = Request::BlocksByRange(BlocksByRangeRequest {
|
|
|
|
start_slot: 0,
|
|
|
|
count: messages_to_send,
|
|
|
|
step: 0,
|
|
|
|
});
|
|
|
|
|
|
|
|
// BlocksByRange Response
|
2021-12-02 02:00:39 +00:00
|
|
|
let full_block = merge_block_large(&common::fork_context());
|
2021-11-09 16:42:02 +00:00
|
|
|
let signed_full_block = SignedBeaconBlock::from_block(full_block, Signature::empty());
|
|
|
|
let rpc_response_merge_large = Response::BlocksByRange(Some(Box::new(signed_full_block)));
|
|
|
|
|
|
|
|
let request_id = RequestId::Sync(messages_to_send as usize);
|
|
|
|
// build the sender future
|
|
|
|
let sender_future = async {
|
|
|
|
loop {
|
|
|
|
match sender.next_event().await {
|
|
|
|
Libp2pEvent::Behaviour(BehaviourEvent::PeerConnectedOutgoing(peer_id)) => {
|
|
|
|
// Send a STATUS message
|
|
|
|
debug!(log, "Sending RPC");
|
|
|
|
sender.swarm.behaviour_mut().send_request(
|
|
|
|
peer_id,
|
|
|
|
request_id,
|
|
|
|
rpc_request.clone(),
|
|
|
|
);
|
|
|
|
}
|
|
|
|
// The request will fail because the sender will refuse to send anything > MAX_RPC_SIZE
|
|
|
|
Libp2pEvent::Behaviour(BehaviourEvent::RPCFailed { id, .. }) => {
|
|
|
|
assert_eq!(id, request_id);
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
_ => {} // Ignore other behaviour events
|
|
|
|
}
|
|
|
|
}
|
|
|
|
};
|
|
|
|
|
|
|
|
// build the receiver future
|
|
|
|
let receiver_future = async {
|
|
|
|
loop {
|
|
|
|
match receiver.next_event().await {
|
|
|
|
Libp2pEvent::Behaviour(BehaviourEvent::RequestReceived {
|
|
|
|
peer_id,
|
|
|
|
id,
|
|
|
|
request,
|
|
|
|
}) => {
|
|
|
|
if request == rpc_request {
|
|
|
|
// send the response
|
|
|
|
warn!(log, "Receiver got request");
|
|
|
|
for _ in 0..messages_to_send {
|
|
|
|
let rpc_response = rpc_response_merge_large.clone();
|
|
|
|
receiver.swarm.behaviour_mut().send_successful_response(
|
|
|
|
peer_id,
|
|
|
|
id,
|
|
|
|
rpc_response.clone(),
|
|
|
|
);
|
|
|
|
}
|
|
|
|
// send the stream termination
|
|
|
|
receiver.swarm.behaviour_mut().send_successful_response(
|
|
|
|
peer_id,
|
|
|
|
id,
|
|
|
|
Response::BlocksByRange(None),
|
|
|
|
);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
_ => {} // Ignore other events
|
|
|
|
}
|
|
|
|
}
|
|
|
|
};
|
|
|
|
|
|
|
|
tokio::select! {
|
|
|
|
_ = sender_future => {}
|
|
|
|
_ = receiver_future => {}
|
|
|
|
_ = sleep(Duration::from_secs(30)) => {
|
|
|
|
panic!("Future timed out");
|
2020-11-28 05:30:57 +00:00
|
|
|
}
|
2020-05-17 11:16:48 +00:00
|
|
|
}
|
2020-11-28 05:30:57 +00:00
|
|
|
})
|
2019-11-27 01:47:46 +00:00
|
|
|
}
|
|
|
|
|
2020-05-18 11:41:01 +00:00
|
|
|
// Tests that a streamed BlocksByRange RPC Message terminates when all expected chunks were received
|
2020-11-28 05:30:57 +00:00
|
|
|
#[test]
|
|
|
|
fn test_blocks_by_range_chunked_rpc_terminates_correctly() {
|
2020-05-18 11:41:01 +00:00
|
|
|
// set up the logging. The level and enabled logging or not
|
|
|
|
let log_level = Level::Debug;
|
|
|
|
let enable_logging = false;
|
|
|
|
|
|
|
|
let messages_to_send = 10;
|
|
|
|
let extra_messages_to_send = 10;
|
|
|
|
|
|
|
|
let log = common::build_log(log_level, enable_logging);
|
|
|
|
|
2020-11-28 05:30:57 +00:00
|
|
|
let rt = Arc::new(Runtime::new().unwrap());
|
|
|
|
|
|
|
|
rt.block_on(async {
|
|
|
|
// get sender/receiver
|
|
|
|
let (mut sender, mut receiver) = common::build_node_pair(Arc::downgrade(&rt), &log).await;
|
|
|
|
|
|
|
|
// BlocksByRange Request
|
|
|
|
let rpc_request = Request::BlocksByRange(BlocksByRangeRequest {
|
|
|
|
start_slot: 0,
|
|
|
|
count: messages_to_send,
|
|
|
|
step: 0,
|
|
|
|
});
|
|
|
|
|
|
|
|
// BlocksByRange Response
|
|
|
|
let spec = E::default_spec();
|
|
|
|
let empty_block = BeaconBlock::empty(&spec);
|
2021-07-09 06:15:32 +00:00
|
|
|
let empty_signed = SignedBeaconBlock::from_block(empty_block, Signature::empty());
|
2020-11-28 05:30:57 +00:00
|
|
|
let rpc_response = Response::BlocksByRange(Some(Box::new(empty_signed)));
|
|
|
|
|
|
|
|
// keep count of the number of messages received
|
|
|
|
let mut messages_received: u64 = 0;
|
2021-11-09 16:42:02 +00:00
|
|
|
let request_id = RequestId::Sync(messages_to_send as usize);
|
2020-11-28 05:30:57 +00:00
|
|
|
// build the sender future
|
|
|
|
let sender_future = async {
|
|
|
|
loop {
|
|
|
|
match sender.next_event().await {
|
2021-07-13 00:48:33 +00:00
|
|
|
Libp2pEvent::Behaviour(BehaviourEvent::PeerConnectedOutgoing(peer_id)) => {
|
2020-11-28 05:30:57 +00:00
|
|
|
// Send a STATUS message
|
|
|
|
debug!(log, "Sending RPC");
|
2021-06-03 01:11:33 +00:00
|
|
|
sender.swarm.behaviour_mut().send_request(
|
2020-11-28 05:30:57 +00:00
|
|
|
peer_id,
|
2021-11-09 16:42:02 +00:00
|
|
|
request_id,
|
2020-11-28 05:30:57 +00:00
|
|
|
rpc_request.clone(),
|
|
|
|
);
|
|
|
|
}
|
|
|
|
Libp2pEvent::Behaviour(BehaviourEvent::ResponseReceived {
|
|
|
|
peer_id: _,
|
2021-11-09 16:42:02 +00:00
|
|
|
id: _,
|
2020-11-28 05:30:57 +00:00
|
|
|
response,
|
|
|
|
}) =>
|
|
|
|
// Should receive the RPC response
|
|
|
|
{
|
|
|
|
debug!(log, "Sender received a response");
|
|
|
|
match response {
|
|
|
|
Response::BlocksByRange(Some(_)) => {
|
|
|
|
assert_eq!(response, rpc_response.clone());
|
|
|
|
messages_received += 1;
|
|
|
|
}
|
|
|
|
Response::BlocksByRange(None) => {
|
|
|
|
// should be exactly 10 messages, as requested
|
|
|
|
assert_eq!(messages_received, messages_to_send);
|
|
|
|
}
|
|
|
|
_ => panic!("Invalid RPC received"),
|
|
|
|
}
|
|
|
|
}
|
2020-05-18 11:41:01 +00:00
|
|
|
|
2020-11-28 05:30:57 +00:00
|
|
|
_ => {} // Ignore other behaviour events
|
2020-05-18 11:41:01 +00:00
|
|
|
}
|
2020-11-28 05:30:57 +00:00
|
|
|
}
|
|
|
|
};
|
|
|
|
|
|
|
|
// determine messages to send (PeerId, RequestId). If some, indicates we still need to send
|
|
|
|
// messages
|
|
|
|
let mut message_info = None;
|
|
|
|
// the number of messages we've sent
|
|
|
|
let mut messages_sent = 0;
|
|
|
|
let receiver_future = async {
|
|
|
|
loop {
|
|
|
|
// this future either drives the sending/receiving or times out allowing messages to be
|
|
|
|
// sent in the timeout
|
|
|
|
match futures::future::select(
|
|
|
|
Box::pin(receiver.next_event()),
|
2021-02-10 23:29:49 +00:00
|
|
|
Box::pin(tokio::time::sleep(Duration::from_secs(1))),
|
2020-11-28 05:30:57 +00:00
|
|
|
)
|
|
|
|
.await
|
2020-06-05 03:07:59 +00:00
|
|
|
{
|
2020-11-28 05:30:57 +00:00
|
|
|
futures::future::Either::Left((
|
|
|
|
Libp2pEvent::Behaviour(BehaviourEvent::RequestReceived {
|
|
|
|
peer_id,
|
|
|
|
id,
|
|
|
|
request,
|
|
|
|
}),
|
|
|
|
_,
|
|
|
|
)) => {
|
|
|
|
if request == rpc_request {
|
|
|
|
// send the response
|
|
|
|
warn!(log, "Receiver got request");
|
|
|
|
message_info = Some((peer_id, id));
|
2020-05-18 11:41:01 +00:00
|
|
|
}
|
|
|
|
}
|
2020-11-28 05:30:57 +00:00
|
|
|
futures::future::Either::Right((_, _)) => {} // The timeout hit, send messages if required
|
|
|
|
_ => continue,
|
2020-06-05 03:07:59 +00:00
|
|
|
}
|
|
|
|
|
2020-11-28 05:30:57 +00:00
|
|
|
// if we need to send messages send them here. This will happen after a delay
|
|
|
|
if message_info.is_some() {
|
|
|
|
messages_sent += 1;
|
|
|
|
let (peer_id, stream_id) = message_info.as_ref().unwrap();
|
2021-06-03 01:11:33 +00:00
|
|
|
receiver.swarm.behaviour_mut().send_successful_response(
|
2021-01-28 23:31:06 +00:00
|
|
|
*peer_id,
|
|
|
|
*stream_id,
|
2020-11-28 05:30:57 +00:00
|
|
|
rpc_response.clone(),
|
|
|
|
);
|
|
|
|
debug!(log, "Sending message {}", messages_sent);
|
|
|
|
if messages_sent == messages_to_send + extra_messages_to_send {
|
|
|
|
// stop sending messages
|
|
|
|
return;
|
2020-05-18 11:41:01 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
2020-11-28 05:30:57 +00:00
|
|
|
};
|
2020-05-18 11:41:01 +00:00
|
|
|
|
2020-11-28 05:30:57 +00:00
|
|
|
tokio::select! {
|
|
|
|
_ = sender_future => {}
|
|
|
|
_ = receiver_future => {}
|
|
|
|
_ = sleep(Duration::from_secs(30)) => {
|
|
|
|
panic!("Future timed out");
|
2020-05-18 11:41:01 +00:00
|
|
|
}
|
|
|
|
}
|
2020-11-28 05:30:57 +00:00
|
|
|
})
|
2020-05-18 11:41:01 +00:00
|
|
|
}
|
|
|
|
|
2019-11-27 01:47:46 +00:00
|
|
|
// Tests an empty response to a BlocksByRange RPC Message
|
2020-11-28 05:30:57 +00:00
|
|
|
#[test]
|
2021-01-28 23:31:06 +00:00
|
|
|
#[allow(clippy::single_match)]
|
2020-11-28 05:30:57 +00:00
|
|
|
fn test_blocks_by_range_single_empty_rpc() {
|
2019-11-27 01:47:46 +00:00
|
|
|
// set up the logging. The level and enabled logging or not
|
|
|
|
let log_level = Level::Trace;
|
|
|
|
let enable_logging = false;
|
|
|
|
|
|
|
|
let log = common::build_log(log_level, enable_logging);
|
2020-11-28 05:30:57 +00:00
|
|
|
let rt = Arc::new(Runtime::new().unwrap());
|
|
|
|
|
|
|
|
rt.block_on(async {
|
|
|
|
// get sender/receiver
|
|
|
|
let (mut sender, mut receiver) = common::build_node_pair(Arc::downgrade(&rt), &log).await;
|
|
|
|
|
|
|
|
// BlocksByRange Request
|
|
|
|
let rpc_request = Request::BlocksByRange(BlocksByRangeRequest {
|
|
|
|
start_slot: 0,
|
|
|
|
count: 10,
|
|
|
|
step: 0,
|
|
|
|
});
|
|
|
|
|
|
|
|
// BlocksByRange Response
|
|
|
|
let spec = E::default_spec();
|
|
|
|
let empty_block = BeaconBlock::empty(&spec);
|
2021-07-09 06:15:32 +00:00
|
|
|
let empty_signed = SignedBeaconBlock::from_block(empty_block, Signature::empty());
|
2020-11-28 05:30:57 +00:00
|
|
|
let rpc_response = Response::BlocksByRange(Some(Box::new(empty_signed)));
|
|
|
|
|
|
|
|
let messages_to_send = 1;
|
|
|
|
|
|
|
|
// keep count of the number of messages received
|
|
|
|
let mut messages_received = 0;
|
|
|
|
// build the sender future
|
|
|
|
let sender_future = async {
|
|
|
|
loop {
|
|
|
|
match sender.next_event().await {
|
2021-07-13 00:48:33 +00:00
|
|
|
Libp2pEvent::Behaviour(BehaviourEvent::PeerConnectedOutgoing(peer_id)) => {
|
2020-11-28 05:30:57 +00:00
|
|
|
// Send a STATUS message
|
|
|
|
debug!(log, "Sending RPC");
|
2021-06-03 01:11:33 +00:00
|
|
|
sender.swarm.behaviour_mut().send_request(
|
2020-11-28 05:30:57 +00:00
|
|
|
peer_id,
|
|
|
|
RequestId::Sync(10),
|
|
|
|
rpc_request.clone(),
|
|
|
|
);
|
2020-06-05 03:07:59 +00:00
|
|
|
}
|
2020-11-28 05:30:57 +00:00
|
|
|
Libp2pEvent::Behaviour(BehaviourEvent::ResponseReceived {
|
|
|
|
peer_id: _,
|
|
|
|
id: RequestId::Sync(10),
|
|
|
|
response,
|
|
|
|
}) => match response {
|
|
|
|
Response::BlocksByRange(Some(_)) => {
|
|
|
|
assert_eq!(response, rpc_response.clone());
|
|
|
|
messages_received += 1;
|
|
|
|
warn!(log, "Chunk received");
|
|
|
|
}
|
|
|
|
Response::BlocksByRange(None) => {
|
|
|
|
// should be exactly 10 messages before terminating
|
|
|
|
assert_eq!(messages_received, messages_to_send);
|
|
|
|
// end the test
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
_ => panic!("Invalid RPC received"),
|
|
|
|
},
|
|
|
|
_ => {} // Ignore other behaviour events
|
|
|
|
}
|
2020-05-17 11:16:48 +00:00
|
|
|
}
|
2020-11-28 05:30:57 +00:00
|
|
|
};
|
|
|
|
|
|
|
|
// build the receiver future
|
|
|
|
let receiver_future = async {
|
|
|
|
loop {
|
|
|
|
match receiver.next_event().await {
|
|
|
|
Libp2pEvent::Behaviour(BehaviourEvent::RequestReceived {
|
|
|
|
peer_id,
|
|
|
|
id,
|
|
|
|
request,
|
|
|
|
}) => {
|
|
|
|
if request == rpc_request {
|
|
|
|
// send the response
|
|
|
|
warn!(log, "Receiver got request");
|
|
|
|
|
|
|
|
for _ in 1..=messages_to_send {
|
2021-06-03 01:11:33 +00:00
|
|
|
receiver.swarm.behaviour_mut().send_successful_response(
|
2021-01-28 23:31:06 +00:00
|
|
|
peer_id,
|
2020-11-28 05:30:57 +00:00
|
|
|
id,
|
|
|
|
rpc_response.clone(),
|
|
|
|
);
|
|
|
|
}
|
|
|
|
// send the stream termination
|
2021-06-03 01:11:33 +00:00
|
|
|
receiver.swarm.behaviour_mut().send_successful_response(
|
2020-11-28 05:30:57 +00:00
|
|
|
peer_id,
|
2020-06-05 03:07:59 +00:00
|
|
|
id,
|
2020-11-28 05:30:57 +00:00
|
|
|
Response::BlocksByRange(None),
|
2020-06-05 03:07:59 +00:00
|
|
|
);
|
2020-04-07 15:08:05 +00:00
|
|
|
}
|
2019-11-27 01:47:46 +00:00
|
|
|
}
|
2020-11-28 05:30:57 +00:00
|
|
|
_ => {} // Ignore other events
|
2020-05-17 11:16:48 +00:00
|
|
|
}
|
2020-11-28 05:30:57 +00:00
|
|
|
}
|
|
|
|
};
|
|
|
|
tokio::select! {
|
|
|
|
_ = sender_future => {}
|
|
|
|
_ = receiver_future => {}
|
|
|
|
_ = sleep(Duration::from_secs(20)) => {
|
|
|
|
panic!("Future timed out");
|
2019-11-27 01:47:46 +00:00
|
|
|
}
|
|
|
|
}
|
2020-11-28 05:30:57 +00:00
|
|
|
})
|
Add snappy compression support (#866)
* notes from call
* should_forward function
* should_forward_block cleanup
* some cleanup and notes
* complete draft
* corrections
* some cleanup
* ran cargo fmt
* Revert "ran cargo fmt"
This reverts commit 464a5c4b62401c68fff7d36d3cd8069560a57713.
* ran cargo fmt after updating
* proposer index mods
* fmt
* new way of reading state
* fmt
* fmt
* compiles again
* fmt
* Correct stream timeout delay queue handling
* Correct small typo
* Support BlocksByRange step parameter
* Add initial docs to syncing
* Initial start of naive-attestation-aggregation
* Remove ping protocol
* Initial renaming of network services
* Correct rebasing relative to latest master
* Adds HashMapDelay struct to utils
* Initial network restructure
* Network restructure. Adds new types for v0.2.0
* Removes build artefacts
* Shift validation to beacon chain
* Temporarily remove gossip validation
This is to be updated to match current optimisation efforts.
* Adds AggregateAndProof
* Begin rebuilding pubsub encoding/decoding
* Temp commit
* Shift gossipsup decoding in eth2_libp2p
* Shifts block encoding/decoding into RPC
* Progress on attestation service
* Initial work on removing libp2p lock
* Add LRU caches to store (rollup)
* Update attestation validation for DB changes (WIP)
* Initial version of should_forward_block
* Scaffold
* Progress on attestation validation
Also, consolidate prod+testing slot clocks so that they share much
of the same implementation and can both handle sub-slot time changes.
* Removes lock from libp2p service
* Completed network lock removal
* Finish(?) attestation processing
* Correct network termination future
* Add slot check to block check
* Correct fmt issues
* Remove Drop implementation for network service
* Address reviewers suggestions
* Modification of validator for subscriptions
* Add slot signing to validator client
* Further progress on validation subscription
* Register SSZ snappy protocol messages
* Add initial idea of snappy compressed Codec by replacing the UVI codec with the snap library
* Fix matching against protocol string
* Adds TODOs for implementation work
* Implements check against max packet size before attempting to decode snappy compressed packages
* Add ssz_snappy codec
* Adds necessary validator subscription functionality
* Progress on snappy codec
* Clean up validator <-> beacon node http types
* Add aggregator status to ValidatorDuty
* clear buffer after decoding
* Impl Clone for manual slot clock
* Fix minor errors
* Further progress validator client subscription
* Initial subscription and aggregation handling
* Progress to modifying val client for attestation aggregation
* First draft of validator client upgrade for aggregate attestations
* Trying something
* Length prefix compressed data
* Fix gossipsub tests
* Working snappy frames with compressed length prefix
* Removes lock on a network channel
* Partially implement beacon node subscription http api
* Uncompressed length prefix working
* Cleanup
* Remove Testing request and response
* Return codec from match statement; reduce code duplication
* Fix unsafe unwrap in Outbound decode
* Add length checks
* All encode/decode functions use snappy frame format
* Add a `full` BeaconBlock method
* Add executable to test rpc against other impls
* Remove unused code; minor fixes
* Add PH & MS slot clock changes
* Account for genesis time
* Use checked mul
* Account for genesis slot
* Change API
* Refactor "duration to..." functions
* Re-merge updated block processing to v0.2.0 (#962)
* Start updating types
* WIP
* Signature hacking
* Existing EF tests passing with fake_crypto
* Updates
* Delete outdated API spec
* The refactor continues
* It compiles
* WIP test fixes
* All release tests passing bar genesis state parsing
* Update and test YamlConfig
* Update to spec v0.10 compatible BLS
* Updates to BLS EF tests
* Add EF test for AggregateVerify
And delete unused hash2curve tests for uncompressed points
* Update EF tests to v0.10.1
* Use optional block root correctly in block proc
* Use genesis fork in deposit domain. All tests pass
* Cargo fmt
* Fast aggregate verify test
* Update REST API docs
* Cargo fmt
* Fix unused import
* Bump spec tags to v0.10.1
* Add `seconds_per_eth1_block` to chainspec
* Update to timestamp based eth1 voting scheme
* Return None from `get_votes_to_consider` if block cache is empty
* Handle overflows in `is_candidate_block`
* Revert to failing tests
* Fix eth1 data sets test
* Choose default vote according to spec
* Fix collect_valid_votes tests
* Fix `get_votes_to_consider` to choose all eligible blocks
* Uncomment winning_vote tests
* Add comments; remove unused code
* Reduce seconds_per_eth1_block for simulation
* Addressed review comments
* Add test for default vote case
* Fix logs
* Remove unused functions
* Meter default eth1 votes
* Fix comments
* Address review comments; remove unused dependency
* Add first attempt at attestation proc. re-write
* Add version 2 of attestation processing
* Minor fixes
* Add validator pubkey cache
* Make get_indexed_attestation take a committee
* Link signature processing into new attn verification
* First working version
* Ensure pubkey cache is updated
* Add more metrics, slight optimizations
* Clone committee cache during attestation processing
* Update shuffling cache during block processing
* Remove old commented-out code
* Fix shuffling cache insert bug
* Used indexed attestation in fork choice
* Restructure attn processing, add metrics
* Add more detailed metrics
* Tidy, fix failing tests
* Fix failing tests, tidy
* Disable/delete two outdated tests
* Add new Pubkeys struct to signature_sets
* Refactor with functional approach
* Update beacon chain
* Remove decompressed member from pubkey bytes
* Add hashmap for indices lookup
* Add state cache, remove store cache
* Only build the head committee cache
* Change `get_attesting_indices` to use Vec
* Fix failing test
* Tidy
* Add pubkey cache persistence file
* Add more comments
* Integrate persistence file into builder
* Add pubkey cache tests
* Add data_dir to beacon chain builder
* Remove Option in pubkey cache persistence file
* Ensure consistency between datadir/data_dir
* Fix failing network test
* Tidy
* Fix todos
* Improve tests
* Fix compile error
* Fix compile error from merge
* Split up block processing metrics
* Tidy
* Refactor get_pubkey_from_state
* Remove commented-out code
* Rename state_cache -> checkpoint_cache
* Rename Checkpoint -> Snapshot
* Tidy, add comments
* Tidy up find_head function
* Change some checkpoint -> snapshot
* Add tests
* Expose max_len
* Remove dead code
* Tidy
* Fix bug
* Add sync-speed metric
* Add first attempt at VerifiableBlock
* Start integrating into beacon chain
* Integrate VerifiableBlock
* Rename VerifableBlock -> PartialBlockVerification
* Add start of typed methods
* Add progress
* Add further progress
* Rename structs
* Add full block verification to block_processing.rs
* Further beacon chain integration
* Update checks for gossip
* Add todo
* Start adding segement verification
* Add passing chain segement test
* Initial integration with batch sync
* Minor changes
* Tidy, add more error checking
* Start adding chain_segment tests
* Finish invalid signature tests
* Include single and gossip verified blocks in tests
* Add gossip verification tests
* Start adding docs
* Finish adding comments to block_processing.rs
* Rename block_processing.rs -> block_verification
* Start removing old block processing code
* Fixes beacon_chain compilation
* Fix project-wide compile errors
* Remove old code
* Fix bug with beacon proposer index
* Fix shim for BlockProcessingError
* Only process one epoch at a time
* Fix loop in chain segment processing
* Add caching for state.eth1_data_votes
* Add BeaconChain::validator_pubkey
* Revert "Add caching for state.eth1_data_votes"
This reverts commit cd73dcd6434fb8d8e6bf30c5356355598ea7b78e.
* Add sync-speed metric (#898)
* Add PH & MS slot clock changes
* Account for genesis time
* Use checked mul
* Account for genesis slot
* Change API
* Allow for clock disparity
* Refactor "duration to..." functions
* Ensure errors are returned during batch processing
Co-authored-by: Michael Sproul <michael@sigmaprime.io>
Co-authored-by: Michael Sproul <micsproul@gmail.com>
Co-authored-by: pawan <pawandhananjay@gmail.com>
Co-authored-by: Paul Hauner <paul@paulhauner.com>
* Enr fork (#967)
* Start fixing enr-fork-id
* Fix time-until-next-fork logic
* Remove fork crate
* Update any references to beacon_chain.spec.milliseconds_per_slot to beacon_chain.slot_clock.slot_diration().as_nillis() in the attestation service. (#968)
* Process network attestations (#966)
* Start updating types
* WIP
* Signature hacking
* Existing EF tests passing with fake_crypto
* Updates
* Delete outdated API spec
* The refactor continues
* It compiles
* WIP test fixes
* All release tests passing bar genesis state parsing
* Update and test YamlConfig
* Update to spec v0.10 compatible BLS
* Updates to BLS EF tests
* Add EF test for AggregateVerify
And delete unused hash2curve tests for uncompressed points
* Update EF tests to v0.10.1
* Use optional block root correctly in block proc
* Use genesis fork in deposit domain. All tests pass
* Cargo fmt
* Fast aggregate verify test
* Update REST API docs
* Cargo fmt
* Fix unused import
* Bump spec tags to v0.10.1
* Add `seconds_per_eth1_block` to chainspec
* Update to timestamp based eth1 voting scheme
* Return None from `get_votes_to_consider` if block cache is empty
* Handle overflows in `is_candidate_block`
* Revert to failing tests
* Fix eth1 data sets test
* Choose default vote according to spec
* Fix collect_valid_votes tests
* Fix `get_votes_to_consider` to choose all eligible blocks
* Uncomment winning_vote tests
* Add comments; remove unused code
* Reduce seconds_per_eth1_block for simulation
* Addressed review comments
* Add test for default vote case
* Fix logs
* Remove unused functions
* Meter default eth1 votes
* Fix comments
* Address review comments; remove unused dependency
* Add first attempt at attestation proc. re-write
* Add version 2 of attestation processing
* Minor fixes
* Add validator pubkey cache
* Make get_indexed_attestation take a committee
* Link signature processing into new attn verification
* First working version
* Ensure pubkey cache is updated
* Add more metrics, slight optimizations
* Clone committee cache during attestation processing
* Update shuffling cache during block processing
* Remove old commented-out code
* Fix shuffling cache insert bug
* Used indexed attestation in fork choice
* Restructure attn processing, add metrics
* Add more detailed metrics
* Tidy, fix failing tests
* Fix failing tests, tidy
* Disable/delete two outdated tests
* Add new Pubkeys struct to signature_sets
* Refactor with functional approach
* Update beacon chain
* Remove decompressed member from pubkey bytes
* Add hashmap for indices lookup
* Add state cache, remove store cache
* Only build the head committee cache
* Change `get_attesting_indices` to use Vec
* Fix failing test
* Tidy
* Add pubkey cache persistence file
* Add more comments
* Integrate persistence file into builder
* Add pubkey cache tests
* Add data_dir to beacon chain builder
* Remove Option in pubkey cache persistence file
* Ensure consistency between datadir/data_dir
* Fix failing network test
* Tidy
* Fix todos
* Improve tests
* Fix compile error
* Fix compile error from merge
* Split up block processing metrics
* Tidy
* Refactor get_pubkey_from_state
* Remove commented-out code
* Rename state_cache -> checkpoint_cache
* Rename Checkpoint -> Snapshot
* Tidy, add comments
* Tidy up find_head function
* Change some checkpoint -> snapshot
* Add tests
* Expose max_len
* Remove dead code
* Tidy
* Fix bug
* Add sync-speed metric
* Add first attempt at VerifiableBlock
* Start integrating into beacon chain
* Integrate VerifiableBlock
* Rename VerifableBlock -> PartialBlockVerification
* Add start of typed methods
* Add progress
* Add further progress
* Rename structs
* Add full block verification to block_processing.rs
* Further beacon chain integration
* Update checks for gossip
* Add todo
* Start adding segement verification
* Add passing chain segement test
* Initial integration with batch sync
* Minor changes
* Tidy, add more error checking
* Start adding chain_segment tests
* Finish invalid signature tests
* Include single and gossip verified blocks in tests
* Add gossip verification tests
* Start adding docs
* Finish adding comments to block_processing.rs
* Rename block_processing.rs -> block_verification
* Start removing old block processing code
* Fixes beacon_chain compilation
* Fix project-wide compile errors
* Remove old code
* Fix bug with beacon proposer index
* Fix shim for BlockProcessingError
* Only process one epoch at a time
* Fix loop in chain segment processing
* Add caching for state.eth1_data_votes
* Add BeaconChain::validator_pubkey
* Revert "Add caching for state.eth1_data_votes"
This reverts commit cd73dcd6434fb8d8e6bf30c5356355598ea7b78e.
* Allow for clock disparity
* Ensure errors are returned during batch processing
* Add block gossip verification
* Connect attestation processing to beacon chain
* Optimistically subscribe to subnets on the same slot
Co-authored-by: Michael Sproul <michael@sigmaprime.io>
Co-authored-by: Michael Sproul <micsproul@gmail.com>
Co-authored-by: pawan <pawandhananjay@gmail.com>
Co-authored-by: Paul Hauner <paul@paulhauner.com>
* Update /validator/subscribe (#969)
* Add progress on duties refactor
* Add simple is_aggregator bool to val subscription
* Remove unused function
* Upgrade sim (#972)
* Add progress on duties refactor
* Add simple is_aggregator bool to val subscription
* Add the no-eth1-sim, refactor sim
* Sends discovery for persistent subnets (#973)
* main takes cmdline arguments
* Add test script
* Fix errors
* snappy uses Uvi to encode/decode length prefix
* Add more comments
* Run fmt
Co-authored-by: Grant Wuerker <gwuerker@gmail.com>
Co-authored-by: Age Manning <Age@AgeManning.com>
Co-authored-by: Michael Sproul <micsproul@gmail.com>
Co-authored-by: Michael Sproul <michael@sigmaprime.io>
Co-authored-by: b-m-f <max@ehlers.berlin>
Co-authored-by: Paul Hauner <paul@paulhauner.com>
Co-authored-by: realbigsean <seananderson33@GMAIL.com>
2020-04-16 01:42:51 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
// Tests a streamed, chunked BlocksByRoot RPC Message
|
|
|
|
// The size of the reponse is a full `BeaconBlock`
|
|
|
|
// which is greater than the Snappy frame size. Hence, this test
|
|
|
|
// serves to test the snappy framing format as well.
|
2020-11-28 05:30:57 +00:00
|
|
|
#[test]
|
2021-01-28 23:31:06 +00:00
|
|
|
#[allow(clippy::single_match)]
|
2020-11-28 05:30:57 +00:00
|
|
|
fn test_blocks_by_root_chunked_rpc() {
|
Add snappy compression support (#866)
* notes from call
* should_forward function
* should_forward_block cleanup
* some cleanup and notes
* complete draft
* corrections
* some cleanup
* ran cargo fmt
* Revert "ran cargo fmt"
This reverts commit 464a5c4b62401c68fff7d36d3cd8069560a57713.
* ran cargo fmt after updating
* proposer index mods
* fmt
* new way of reading state
* fmt
* fmt
* compiles again
* fmt
* Correct stream timeout delay queue handling
* Correct small typo
* Support BlocksByRange step parameter
* Add initial docs to syncing
* Initial start of naive-attestation-aggregation
* Remove ping protocol
* Initial renaming of network services
* Correct rebasing relative to latest master
* Adds HashMapDelay struct to utils
* Initial network restructure
* Network restructure. Adds new types for v0.2.0
* Removes build artefacts
* Shift validation to beacon chain
* Temporarily remove gossip validation
This is to be updated to match current optimisation efforts.
* Adds AggregateAndProof
* Begin rebuilding pubsub encoding/decoding
* Temp commit
* Shift gossipsup decoding in eth2_libp2p
* Shifts block encoding/decoding into RPC
* Progress on attestation service
* Initial work on removing libp2p lock
* Add LRU caches to store (rollup)
* Update attestation validation for DB changes (WIP)
* Initial version of should_forward_block
* Scaffold
* Progress on attestation validation
Also, consolidate prod+testing slot clocks so that they share much
of the same implementation and can both handle sub-slot time changes.
* Removes lock from libp2p service
* Completed network lock removal
* Finish(?) attestation processing
* Correct network termination future
* Add slot check to block check
* Correct fmt issues
* Remove Drop implementation for network service
* Address reviewers suggestions
* Modification of validator for subscriptions
* Add slot signing to validator client
* Further progress on validation subscription
* Register SSZ snappy protocol messages
* Add initial idea of snappy compressed Codec by replacing the UVI codec with the snap library
* Fix matching against protocol string
* Adds TODOs for implementation work
* Implements check against max packet size before attempting to decode snappy compressed packages
* Add ssz_snappy codec
* Adds necessary validator subscription functionality
* Progress on snappy codec
* Clean up validator <-> beacon node http types
* Add aggregator status to ValidatorDuty
* clear buffer after decoding
* Impl Clone for manual slot clock
* Fix minor errors
* Further progress validator client subscription
* Initial subscription and aggregation handling
* Progress to modifying val client for attestation aggregation
* First draft of validator client upgrade for aggregate attestations
* Trying something
* Length prefix compressed data
* Fix gossipsub tests
* Working snappy frames with compressed length prefix
* Removes lock on a network channel
* Partially implement beacon node subscription http api
* Uncompressed length prefix working
* Cleanup
* Remove Testing request and response
* Return codec from match statement; reduce code duplication
* Fix unsafe unwrap in Outbound decode
* Add length checks
* All encode/decode functions use snappy frame format
* Add a `full` BeaconBlock method
* Add executable to test rpc against other impls
* Remove unused code; minor fixes
* Add PH & MS slot clock changes
* Account for genesis time
* Use checked mul
* Account for genesis slot
* Change API
* Refactor "duration to..." functions
* Re-merge updated block processing to v0.2.0 (#962)
* Start updating types
* WIP
* Signature hacking
* Existing EF tests passing with fake_crypto
* Updates
* Delete outdated API spec
* The refactor continues
* It compiles
* WIP test fixes
* All release tests passing bar genesis state parsing
* Update and test YamlConfig
* Update to spec v0.10 compatible BLS
* Updates to BLS EF tests
* Add EF test for AggregateVerify
And delete unused hash2curve tests for uncompressed points
* Update EF tests to v0.10.1
* Use optional block root correctly in block proc
* Use genesis fork in deposit domain. All tests pass
* Cargo fmt
* Fast aggregate verify test
* Update REST API docs
* Cargo fmt
* Fix unused import
* Bump spec tags to v0.10.1
* Add `seconds_per_eth1_block` to chainspec
* Update to timestamp based eth1 voting scheme
* Return None from `get_votes_to_consider` if block cache is empty
* Handle overflows in `is_candidate_block`
* Revert to failing tests
* Fix eth1 data sets test
* Choose default vote according to spec
* Fix collect_valid_votes tests
* Fix `get_votes_to_consider` to choose all eligible blocks
* Uncomment winning_vote tests
* Add comments; remove unused code
* Reduce seconds_per_eth1_block for simulation
* Addressed review comments
* Add test for default vote case
* Fix logs
* Remove unused functions
* Meter default eth1 votes
* Fix comments
* Address review comments; remove unused dependency
* Add first attempt at attestation proc. re-write
* Add version 2 of attestation processing
* Minor fixes
* Add validator pubkey cache
* Make get_indexed_attestation take a committee
* Link signature processing into new attn verification
* First working version
* Ensure pubkey cache is updated
* Add more metrics, slight optimizations
* Clone committee cache during attestation processing
* Update shuffling cache during block processing
* Remove old commented-out code
* Fix shuffling cache insert bug
* Used indexed attestation in fork choice
* Restructure attn processing, add metrics
* Add more detailed metrics
* Tidy, fix failing tests
* Fix failing tests, tidy
* Disable/delete two outdated tests
* Add new Pubkeys struct to signature_sets
* Refactor with functional approach
* Update beacon chain
* Remove decompressed member from pubkey bytes
* Add hashmap for indices lookup
* Add state cache, remove store cache
* Only build the head committee cache
* Change `get_attesting_indices` to use Vec
* Fix failing test
* Tidy
* Add pubkey cache persistence file
* Add more comments
* Integrate persistence file into builder
* Add pubkey cache tests
* Add data_dir to beacon chain builder
* Remove Option in pubkey cache persistence file
* Ensure consistency between datadir/data_dir
* Fix failing network test
* Tidy
* Fix todos
* Improve tests
* Fix compile error
* Fix compile error from merge
* Split up block processing metrics
* Tidy
* Refactor get_pubkey_from_state
* Remove commented-out code
* Rename state_cache -> checkpoint_cache
* Rename Checkpoint -> Snapshot
* Tidy, add comments
* Tidy up find_head function
* Change some checkpoint -> snapshot
* Add tests
* Expose max_len
* Remove dead code
* Tidy
* Fix bug
* Add sync-speed metric
* Add first attempt at VerifiableBlock
* Start integrating into beacon chain
* Integrate VerifiableBlock
* Rename VerifableBlock -> PartialBlockVerification
* Add start of typed methods
* Add progress
* Add further progress
* Rename structs
* Add full block verification to block_processing.rs
* Further beacon chain integration
* Update checks for gossip
* Add todo
* Start adding segement verification
* Add passing chain segement test
* Initial integration with batch sync
* Minor changes
* Tidy, add more error checking
* Start adding chain_segment tests
* Finish invalid signature tests
* Include single and gossip verified blocks in tests
* Add gossip verification tests
* Start adding docs
* Finish adding comments to block_processing.rs
* Rename block_processing.rs -> block_verification
* Start removing old block processing code
* Fixes beacon_chain compilation
* Fix project-wide compile errors
* Remove old code
* Fix bug with beacon proposer index
* Fix shim for BlockProcessingError
* Only process one epoch at a time
* Fix loop in chain segment processing
* Add caching for state.eth1_data_votes
* Add BeaconChain::validator_pubkey
* Revert "Add caching for state.eth1_data_votes"
This reverts commit cd73dcd6434fb8d8e6bf30c5356355598ea7b78e.
* Add sync-speed metric (#898)
* Add PH & MS slot clock changes
* Account for genesis time
* Use checked mul
* Account for genesis slot
* Change API
* Allow for clock disparity
* Refactor "duration to..." functions
* Ensure errors are returned during batch processing
Co-authored-by: Michael Sproul <michael@sigmaprime.io>
Co-authored-by: Michael Sproul <micsproul@gmail.com>
Co-authored-by: pawan <pawandhananjay@gmail.com>
Co-authored-by: Paul Hauner <paul@paulhauner.com>
* Enr fork (#967)
* Start fixing enr-fork-id
* Fix time-until-next-fork logic
* Remove fork crate
* Update any references to beacon_chain.spec.milliseconds_per_slot to beacon_chain.slot_clock.slot_diration().as_nillis() in the attestation service. (#968)
* Process network attestations (#966)
* Start updating types
* WIP
* Signature hacking
* Existing EF tests passing with fake_crypto
* Updates
* Delete outdated API spec
* The refactor continues
* It compiles
* WIP test fixes
* All release tests passing bar genesis state parsing
* Update and test YamlConfig
* Update to spec v0.10 compatible BLS
* Updates to BLS EF tests
* Add EF test for AggregateVerify
And delete unused hash2curve tests for uncompressed points
* Update EF tests to v0.10.1
* Use optional block root correctly in block proc
* Use genesis fork in deposit domain. All tests pass
* Cargo fmt
* Fast aggregate verify test
* Update REST API docs
* Cargo fmt
* Fix unused import
* Bump spec tags to v0.10.1
* Add `seconds_per_eth1_block` to chainspec
* Update to timestamp based eth1 voting scheme
* Return None from `get_votes_to_consider` if block cache is empty
* Handle overflows in `is_candidate_block`
* Revert to failing tests
* Fix eth1 data sets test
* Choose default vote according to spec
* Fix collect_valid_votes tests
* Fix `get_votes_to_consider` to choose all eligible blocks
* Uncomment winning_vote tests
* Add comments; remove unused code
* Reduce seconds_per_eth1_block for simulation
* Addressed review comments
* Add test for default vote case
* Fix logs
* Remove unused functions
* Meter default eth1 votes
* Fix comments
* Address review comments; remove unused dependency
* Add first attempt at attestation proc. re-write
* Add version 2 of attestation processing
* Minor fixes
* Add validator pubkey cache
* Make get_indexed_attestation take a committee
* Link signature processing into new attn verification
* First working version
* Ensure pubkey cache is updated
* Add more metrics, slight optimizations
* Clone committee cache during attestation processing
* Update shuffling cache during block processing
* Remove old commented-out code
* Fix shuffling cache insert bug
* Used indexed attestation in fork choice
* Restructure attn processing, add metrics
* Add more detailed metrics
* Tidy, fix failing tests
* Fix failing tests, tidy
* Disable/delete two outdated tests
* Add new Pubkeys struct to signature_sets
* Refactor with functional approach
* Update beacon chain
* Remove decompressed member from pubkey bytes
* Add hashmap for indices lookup
* Add state cache, remove store cache
* Only build the head committee cache
* Change `get_attesting_indices` to use Vec
* Fix failing test
* Tidy
* Add pubkey cache persistence file
* Add more comments
* Integrate persistence file into builder
* Add pubkey cache tests
* Add data_dir to beacon chain builder
* Remove Option in pubkey cache persistence file
* Ensure consistency between datadir/data_dir
* Fix failing network test
* Tidy
* Fix todos
* Improve tests
* Fix compile error
* Fix compile error from merge
* Split up block processing metrics
* Tidy
* Refactor get_pubkey_from_state
* Remove commented-out code
* Rename state_cache -> checkpoint_cache
* Rename Checkpoint -> Snapshot
* Tidy, add comments
* Tidy up find_head function
* Change some checkpoint -> snapshot
* Add tests
* Expose max_len
* Remove dead code
* Tidy
* Fix bug
* Add sync-speed metric
* Add first attempt at VerifiableBlock
* Start integrating into beacon chain
* Integrate VerifiableBlock
* Rename VerifableBlock -> PartialBlockVerification
* Add start of typed methods
* Add progress
* Add further progress
* Rename structs
* Add full block verification to block_processing.rs
* Further beacon chain integration
* Update checks for gossip
* Add todo
* Start adding segement verification
* Add passing chain segement test
* Initial integration with batch sync
* Minor changes
* Tidy, add more error checking
* Start adding chain_segment tests
* Finish invalid signature tests
* Include single and gossip verified blocks in tests
* Add gossip verification tests
* Start adding docs
* Finish adding comments to block_processing.rs
* Rename block_processing.rs -> block_verification
* Start removing old block processing code
* Fixes beacon_chain compilation
* Fix project-wide compile errors
* Remove old code
* Fix bug with beacon proposer index
* Fix shim for BlockProcessingError
* Only process one epoch at a time
* Fix loop in chain segment processing
* Add caching for state.eth1_data_votes
* Add BeaconChain::validator_pubkey
* Revert "Add caching for state.eth1_data_votes"
This reverts commit cd73dcd6434fb8d8e6bf30c5356355598ea7b78e.
* Allow for clock disparity
* Ensure errors are returned during batch processing
* Add block gossip verification
* Connect attestation processing to beacon chain
* Optimistically subscribe to subnets on the same slot
Co-authored-by: Michael Sproul <michael@sigmaprime.io>
Co-authored-by: Michael Sproul <micsproul@gmail.com>
Co-authored-by: pawan <pawandhananjay@gmail.com>
Co-authored-by: Paul Hauner <paul@paulhauner.com>
* Update /validator/subscribe (#969)
* Add progress on duties refactor
* Add simple is_aggregator bool to val subscription
* Remove unused function
* Upgrade sim (#972)
* Add progress on duties refactor
* Add simple is_aggregator bool to val subscription
* Add the no-eth1-sim, refactor sim
* Sends discovery for persistent subnets (#973)
* main takes cmdline arguments
* Add test script
* Fix errors
* snappy uses Uvi to encode/decode length prefix
* Add more comments
* Run fmt
Co-authored-by: Grant Wuerker <gwuerker@gmail.com>
Co-authored-by: Age Manning <Age@AgeManning.com>
Co-authored-by: Michael Sproul <micsproul@gmail.com>
Co-authored-by: Michael Sproul <michael@sigmaprime.io>
Co-authored-by: b-m-f <max@ehlers.berlin>
Co-authored-by: Paul Hauner <paul@paulhauner.com>
Co-authored-by: realbigsean <seananderson33@GMAIL.com>
2020-04-16 01:42:51 +00:00
|
|
|
// set up the logging. The level and enabled logging or not
|
2020-05-17 11:16:48 +00:00
|
|
|
let log_level = Level::Debug;
|
Add snappy compression support (#866)
* notes from call
* should_forward function
* should_forward_block cleanup
* some cleanup and notes
* complete draft
* corrections
* some cleanup
* ran cargo fmt
* Revert "ran cargo fmt"
This reverts commit 464a5c4b62401c68fff7d36d3cd8069560a57713.
* ran cargo fmt after updating
* proposer index mods
* fmt
* new way of reading state
* fmt
* fmt
* compiles again
* fmt
* Correct stream timeout delay queue handling
* Correct small typo
* Support BlocksByRange step parameter
* Add initial docs to syncing
* Initial start of naive-attestation-aggregation
* Remove ping protocol
* Initial renaming of network services
* Correct rebasing relative to latest master
* Adds HashMapDelay struct to utils
* Initial network restructure
* Network restructure. Adds new types for v0.2.0
* Removes build artefacts
* Shift validation to beacon chain
* Temporarily remove gossip validation
This is to be updated to match current optimisation efforts.
* Adds AggregateAndProof
* Begin rebuilding pubsub encoding/decoding
* Temp commit
* Shift gossipsup decoding in eth2_libp2p
* Shifts block encoding/decoding into RPC
* Progress on attestation service
* Initial work on removing libp2p lock
* Add LRU caches to store (rollup)
* Update attestation validation for DB changes (WIP)
* Initial version of should_forward_block
* Scaffold
* Progress on attestation validation
Also, consolidate prod+testing slot clocks so that they share much
of the same implementation and can both handle sub-slot time changes.
* Removes lock from libp2p service
* Completed network lock removal
* Finish(?) attestation processing
* Correct network termination future
* Add slot check to block check
* Correct fmt issues
* Remove Drop implementation for network service
* Address reviewers suggestions
* Modification of validator for subscriptions
* Add slot signing to validator client
* Further progress on validation subscription
* Register SSZ snappy protocol messages
* Add initial idea of snappy compressed Codec by replacing the UVI codec with the snap library
* Fix matching against protocol string
* Adds TODOs for implementation work
* Implements check against max packet size before attempting to decode snappy compressed packages
* Add ssz_snappy codec
* Adds necessary validator subscription functionality
* Progress on snappy codec
* Clean up validator <-> beacon node http types
* Add aggregator status to ValidatorDuty
* clear buffer after decoding
* Impl Clone for manual slot clock
* Fix minor errors
* Further progress validator client subscription
* Initial subscription and aggregation handling
* Progress to modifying val client for attestation aggregation
* First draft of validator client upgrade for aggregate attestations
* Trying something
* Length prefix compressed data
* Fix gossipsub tests
* Working snappy frames with compressed length prefix
* Removes lock on a network channel
* Partially implement beacon node subscription http api
* Uncompressed length prefix working
* Cleanup
* Remove Testing request and response
* Return codec from match statement; reduce code duplication
* Fix unsafe unwrap in Outbound decode
* Add length checks
* All encode/decode functions use snappy frame format
* Add a `full` BeaconBlock method
* Add executable to test rpc against other impls
* Remove unused code; minor fixes
* Add PH & MS slot clock changes
* Account for genesis time
* Use checked mul
* Account for genesis slot
* Change API
* Refactor "duration to..." functions
* Re-merge updated block processing to v0.2.0 (#962)
* Start updating types
* WIP
* Signature hacking
* Existing EF tests passing with fake_crypto
* Updates
* Delete outdated API spec
* The refactor continues
* It compiles
* WIP test fixes
* All release tests passing bar genesis state parsing
* Update and test YamlConfig
* Update to spec v0.10 compatible BLS
* Updates to BLS EF tests
* Add EF test for AggregateVerify
And delete unused hash2curve tests for uncompressed points
* Update EF tests to v0.10.1
* Use optional block root correctly in block proc
* Use genesis fork in deposit domain. All tests pass
* Cargo fmt
* Fast aggregate verify test
* Update REST API docs
* Cargo fmt
* Fix unused import
* Bump spec tags to v0.10.1
* Add `seconds_per_eth1_block` to chainspec
* Update to timestamp based eth1 voting scheme
* Return None from `get_votes_to_consider` if block cache is empty
* Handle overflows in `is_candidate_block`
* Revert to failing tests
* Fix eth1 data sets test
* Choose default vote according to spec
* Fix collect_valid_votes tests
* Fix `get_votes_to_consider` to choose all eligible blocks
* Uncomment winning_vote tests
* Add comments; remove unused code
* Reduce seconds_per_eth1_block for simulation
* Addressed review comments
* Add test for default vote case
* Fix logs
* Remove unused functions
* Meter default eth1 votes
* Fix comments
* Address review comments; remove unused dependency
* Add first attempt at attestation proc. re-write
* Add version 2 of attestation processing
* Minor fixes
* Add validator pubkey cache
* Make get_indexed_attestation take a committee
* Link signature processing into new attn verification
* First working version
* Ensure pubkey cache is updated
* Add more metrics, slight optimizations
* Clone committee cache during attestation processing
* Update shuffling cache during block processing
* Remove old commented-out code
* Fix shuffling cache insert bug
* Used indexed attestation in fork choice
* Restructure attn processing, add metrics
* Add more detailed metrics
* Tidy, fix failing tests
* Fix failing tests, tidy
* Disable/delete two outdated tests
* Add new Pubkeys struct to signature_sets
* Refactor with functional approach
* Update beacon chain
* Remove decompressed member from pubkey bytes
* Add hashmap for indices lookup
* Add state cache, remove store cache
* Only build the head committee cache
* Change `get_attesting_indices` to use Vec
* Fix failing test
* Tidy
* Add pubkey cache persistence file
* Add more comments
* Integrate persistence file into builder
* Add pubkey cache tests
* Add data_dir to beacon chain builder
* Remove Option in pubkey cache persistence file
* Ensure consistency between datadir/data_dir
* Fix failing network test
* Tidy
* Fix todos
* Improve tests
* Fix compile error
* Fix compile error from merge
* Split up block processing metrics
* Tidy
* Refactor get_pubkey_from_state
* Remove commented-out code
* Rename state_cache -> checkpoint_cache
* Rename Checkpoint -> Snapshot
* Tidy, add comments
* Tidy up find_head function
* Change some checkpoint -> snapshot
* Add tests
* Expose max_len
* Remove dead code
* Tidy
* Fix bug
* Add sync-speed metric
* Add first attempt at VerifiableBlock
* Start integrating into beacon chain
* Integrate VerifiableBlock
* Rename VerifableBlock -> PartialBlockVerification
* Add start of typed methods
* Add progress
* Add further progress
* Rename structs
* Add full block verification to block_processing.rs
* Further beacon chain integration
* Update checks for gossip
* Add todo
* Start adding segement verification
* Add passing chain segement test
* Initial integration with batch sync
* Minor changes
* Tidy, add more error checking
* Start adding chain_segment tests
* Finish invalid signature tests
* Include single and gossip verified blocks in tests
* Add gossip verification tests
* Start adding docs
* Finish adding comments to block_processing.rs
* Rename block_processing.rs -> block_verification
* Start removing old block processing code
* Fixes beacon_chain compilation
* Fix project-wide compile errors
* Remove old code
* Fix bug with beacon proposer index
* Fix shim for BlockProcessingError
* Only process one epoch at a time
* Fix loop in chain segment processing
* Add caching for state.eth1_data_votes
* Add BeaconChain::validator_pubkey
* Revert "Add caching for state.eth1_data_votes"
This reverts commit cd73dcd6434fb8d8e6bf30c5356355598ea7b78e.
* Add sync-speed metric (#898)
* Add PH & MS slot clock changes
* Account for genesis time
* Use checked mul
* Account for genesis slot
* Change API
* Allow for clock disparity
* Refactor "duration to..." functions
* Ensure errors are returned during batch processing
Co-authored-by: Michael Sproul <michael@sigmaprime.io>
Co-authored-by: Michael Sproul <micsproul@gmail.com>
Co-authored-by: pawan <pawandhananjay@gmail.com>
Co-authored-by: Paul Hauner <paul@paulhauner.com>
* Enr fork (#967)
* Start fixing enr-fork-id
* Fix time-until-next-fork logic
* Remove fork crate
* Update any references to beacon_chain.spec.milliseconds_per_slot to beacon_chain.slot_clock.slot_diration().as_nillis() in the attestation service. (#968)
* Process network attestations (#966)
* Start updating types
* WIP
* Signature hacking
* Existing EF tests passing with fake_crypto
* Updates
* Delete outdated API spec
* The refactor continues
* It compiles
* WIP test fixes
* All release tests passing bar genesis state parsing
* Update and test YamlConfig
* Update to spec v0.10 compatible BLS
* Updates to BLS EF tests
* Add EF test for AggregateVerify
And delete unused hash2curve tests for uncompressed points
* Update EF tests to v0.10.1
* Use optional block root correctly in block proc
* Use genesis fork in deposit domain. All tests pass
* Cargo fmt
* Fast aggregate verify test
* Update REST API docs
* Cargo fmt
* Fix unused import
* Bump spec tags to v0.10.1
* Add `seconds_per_eth1_block` to chainspec
* Update to timestamp based eth1 voting scheme
* Return None from `get_votes_to_consider` if block cache is empty
* Handle overflows in `is_candidate_block`
* Revert to failing tests
* Fix eth1 data sets test
* Choose default vote according to spec
* Fix collect_valid_votes tests
* Fix `get_votes_to_consider` to choose all eligible blocks
* Uncomment winning_vote tests
* Add comments; remove unused code
* Reduce seconds_per_eth1_block for simulation
* Addressed review comments
* Add test for default vote case
* Fix logs
* Remove unused functions
* Meter default eth1 votes
* Fix comments
* Address review comments; remove unused dependency
* Add first attempt at attestation proc. re-write
* Add version 2 of attestation processing
* Minor fixes
* Add validator pubkey cache
* Make get_indexed_attestation take a committee
* Link signature processing into new attn verification
* First working version
* Ensure pubkey cache is updated
* Add more metrics, slight optimizations
* Clone committee cache during attestation processing
* Update shuffling cache during block processing
* Remove old commented-out code
* Fix shuffling cache insert bug
* Used indexed attestation in fork choice
* Restructure attn processing, add metrics
* Add more detailed metrics
* Tidy, fix failing tests
* Fix failing tests, tidy
* Disable/delete two outdated tests
* Add new Pubkeys struct to signature_sets
* Refactor with functional approach
* Update beacon chain
* Remove decompressed member from pubkey bytes
* Add hashmap for indices lookup
* Add state cache, remove store cache
* Only build the head committee cache
* Change `get_attesting_indices` to use Vec
* Fix failing test
* Tidy
* Add pubkey cache persistence file
* Add more comments
* Integrate persistence file into builder
* Add pubkey cache tests
* Add data_dir to beacon chain builder
* Remove Option in pubkey cache persistence file
* Ensure consistency between datadir/data_dir
* Fix failing network test
* Tidy
* Fix todos
* Improve tests
* Fix compile error
* Fix compile error from merge
* Split up block processing metrics
* Tidy
* Refactor get_pubkey_from_state
* Remove commented-out code
* Rename state_cache -> checkpoint_cache
* Rename Checkpoint -> Snapshot
* Tidy, add comments
* Tidy up find_head function
* Change some checkpoint -> snapshot
* Add tests
* Expose max_len
* Remove dead code
* Tidy
* Fix bug
* Add sync-speed metric
* Add first attempt at VerifiableBlock
* Start integrating into beacon chain
* Integrate VerifiableBlock
* Rename VerifableBlock -> PartialBlockVerification
* Add start of typed methods
* Add progress
* Add further progress
* Rename structs
* Add full block verification to block_processing.rs
* Further beacon chain integration
* Update checks for gossip
* Add todo
* Start adding segement verification
* Add passing chain segement test
* Initial integration with batch sync
* Minor changes
* Tidy, add more error checking
* Start adding chain_segment tests
* Finish invalid signature tests
* Include single and gossip verified blocks in tests
* Add gossip verification tests
* Start adding docs
* Finish adding comments to block_processing.rs
* Rename block_processing.rs -> block_verification
* Start removing old block processing code
* Fixes beacon_chain compilation
* Fix project-wide compile errors
* Remove old code
* Fix bug with beacon proposer index
* Fix shim for BlockProcessingError
* Only process one epoch at a time
* Fix loop in chain segment processing
* Add caching for state.eth1_data_votes
* Add BeaconChain::validator_pubkey
* Revert "Add caching for state.eth1_data_votes"
This reverts commit cd73dcd6434fb8d8e6bf30c5356355598ea7b78e.
* Allow for clock disparity
* Ensure errors are returned during batch processing
* Add block gossip verification
* Connect attestation processing to beacon chain
* Optimistically subscribe to subnets on the same slot
Co-authored-by: Michael Sproul <michael@sigmaprime.io>
Co-authored-by: Michael Sproul <micsproul@gmail.com>
Co-authored-by: pawan <pawandhananjay@gmail.com>
Co-authored-by: Paul Hauner <paul@paulhauner.com>
* Update /validator/subscribe (#969)
* Add progress on duties refactor
* Add simple is_aggregator bool to val subscription
* Remove unused function
* Upgrade sim (#972)
* Add progress on duties refactor
* Add simple is_aggregator bool to val subscription
* Add the no-eth1-sim, refactor sim
* Sends discovery for persistent subnets (#973)
* main takes cmdline arguments
* Add test script
* Fix errors
* snappy uses Uvi to encode/decode length prefix
* Add more comments
* Run fmt
Co-authored-by: Grant Wuerker <gwuerker@gmail.com>
Co-authored-by: Age Manning <Age@AgeManning.com>
Co-authored-by: Michael Sproul <micsproul@gmail.com>
Co-authored-by: Michael Sproul <michael@sigmaprime.io>
Co-authored-by: b-m-f <max@ehlers.berlin>
Co-authored-by: Paul Hauner <paul@paulhauner.com>
Co-authored-by: realbigsean <seananderson33@GMAIL.com>
2020-04-16 01:42:51 +00:00
|
|
|
let enable_logging = false;
|
|
|
|
|
2021-11-09 16:42:02 +00:00
|
|
|
let messages_to_send = 6;
|
Add snappy compression support (#866)
* notes from call
* should_forward function
* should_forward_block cleanup
* some cleanup and notes
* complete draft
* corrections
* some cleanup
* ran cargo fmt
* Revert "ran cargo fmt"
This reverts commit 464a5c4b62401c68fff7d36d3cd8069560a57713.
* ran cargo fmt after updating
* proposer index mods
* fmt
* new way of reading state
* fmt
* fmt
* compiles again
* fmt
* Correct stream timeout delay queue handling
* Correct small typo
* Support BlocksByRange step parameter
* Add initial docs to syncing
* Initial start of naive-attestation-aggregation
* Remove ping protocol
* Initial renaming of network services
* Correct rebasing relative to latest master
* Adds HashMapDelay struct to utils
* Initial network restructure
* Network restructure. Adds new types for v0.2.0
* Removes build artefacts
* Shift validation to beacon chain
* Temporarily remove gossip validation
This is to be updated to match current optimisation efforts.
* Adds AggregateAndProof
* Begin rebuilding pubsub encoding/decoding
* Temp commit
* Shift gossipsup decoding in eth2_libp2p
* Shifts block encoding/decoding into RPC
* Progress on attestation service
* Initial work on removing libp2p lock
* Add LRU caches to store (rollup)
* Update attestation validation for DB changes (WIP)
* Initial version of should_forward_block
* Scaffold
* Progress on attestation validation
Also, consolidate prod+testing slot clocks so that they share much
of the same implementation and can both handle sub-slot time changes.
* Removes lock from libp2p service
* Completed network lock removal
* Finish(?) attestation processing
* Correct network termination future
* Add slot check to block check
* Correct fmt issues
* Remove Drop implementation for network service
* Address reviewers suggestions
* Modification of validator for subscriptions
* Add slot signing to validator client
* Further progress on validation subscription
* Register SSZ snappy protocol messages
* Add initial idea of snappy compressed Codec by replacing the UVI codec with the snap library
* Fix matching against protocol string
* Adds TODOs for implementation work
* Implements check against max packet size before attempting to decode snappy compressed packages
* Add ssz_snappy codec
* Adds necessary validator subscription functionality
* Progress on snappy codec
* Clean up validator <-> beacon node http types
* Add aggregator status to ValidatorDuty
* clear buffer after decoding
* Impl Clone for manual slot clock
* Fix minor errors
* Further progress validator client subscription
* Initial subscription and aggregation handling
* Progress to modifying val client for attestation aggregation
* First draft of validator client upgrade for aggregate attestations
* Trying something
* Length prefix compressed data
* Fix gossipsub tests
* Working snappy frames with compressed length prefix
* Removes lock on a network channel
* Partially implement beacon node subscription http api
* Uncompressed length prefix working
* Cleanup
* Remove Testing request and response
* Return codec from match statement; reduce code duplication
* Fix unsafe unwrap in Outbound decode
* Add length checks
* All encode/decode functions use snappy frame format
* Add a `full` BeaconBlock method
* Add executable to test rpc against other impls
* Remove unused code; minor fixes
* Add PH & MS slot clock changes
* Account for genesis time
* Use checked mul
* Account for genesis slot
* Change API
* Refactor "duration to..." functions
* Re-merge updated block processing to v0.2.0 (#962)
* Start updating types
* WIP
* Signature hacking
* Existing EF tests passing with fake_crypto
* Updates
* Delete outdated API spec
* The refactor continues
* It compiles
* WIP test fixes
* All release tests passing bar genesis state parsing
* Update and test YamlConfig
* Update to spec v0.10 compatible BLS
* Updates to BLS EF tests
* Add EF test for AggregateVerify
And delete unused hash2curve tests for uncompressed points
* Update EF tests to v0.10.1
* Use optional block root correctly in block proc
* Use genesis fork in deposit domain. All tests pass
* Cargo fmt
* Fast aggregate verify test
* Update REST API docs
* Cargo fmt
* Fix unused import
* Bump spec tags to v0.10.1
* Add `seconds_per_eth1_block` to chainspec
* Update to timestamp based eth1 voting scheme
* Return None from `get_votes_to_consider` if block cache is empty
* Handle overflows in `is_candidate_block`
* Revert to failing tests
* Fix eth1 data sets test
* Choose default vote according to spec
* Fix collect_valid_votes tests
* Fix `get_votes_to_consider` to choose all eligible blocks
* Uncomment winning_vote tests
* Add comments; remove unused code
* Reduce seconds_per_eth1_block for simulation
* Addressed review comments
* Add test for default vote case
* Fix logs
* Remove unused functions
* Meter default eth1 votes
* Fix comments
* Address review comments; remove unused dependency
* Add first attempt at attestation proc. re-write
* Add version 2 of attestation processing
* Minor fixes
* Add validator pubkey cache
* Make get_indexed_attestation take a committee
* Link signature processing into new attn verification
* First working version
* Ensure pubkey cache is updated
* Add more metrics, slight optimizations
* Clone committee cache during attestation processing
* Update shuffling cache during block processing
* Remove old commented-out code
* Fix shuffling cache insert bug
* Used indexed attestation in fork choice
* Restructure attn processing, add metrics
* Add more detailed metrics
* Tidy, fix failing tests
* Fix failing tests, tidy
* Disable/delete two outdated tests
* Add new Pubkeys struct to signature_sets
* Refactor with functional approach
* Update beacon chain
* Remove decompressed member from pubkey bytes
* Add hashmap for indices lookup
* Add state cache, remove store cache
* Only build the head committee cache
* Change `get_attesting_indices` to use Vec
* Fix failing test
* Tidy
* Add pubkey cache persistence file
* Add more comments
* Integrate persistence file into builder
* Add pubkey cache tests
* Add data_dir to beacon chain builder
* Remove Option in pubkey cache persistence file
* Ensure consistency between datadir/data_dir
* Fix failing network test
* Tidy
* Fix todos
* Improve tests
* Fix compile error
* Fix compile error from merge
* Split up block processing metrics
* Tidy
* Refactor get_pubkey_from_state
* Remove commented-out code
* Rename state_cache -> checkpoint_cache
* Rename Checkpoint -> Snapshot
* Tidy, add comments
* Tidy up find_head function
* Change some checkpoint -> snapshot
* Add tests
* Expose max_len
* Remove dead code
* Tidy
* Fix bug
* Add sync-speed metric
* Add first attempt at VerifiableBlock
* Start integrating into beacon chain
* Integrate VerifiableBlock
* Rename VerifableBlock -> PartialBlockVerification
* Add start of typed methods
* Add progress
* Add further progress
* Rename structs
* Add full block verification to block_processing.rs
* Further beacon chain integration
* Update checks for gossip
* Add todo
* Start adding segement verification
* Add passing chain segement test
* Initial integration with batch sync
* Minor changes
* Tidy, add more error checking
* Start adding chain_segment tests
* Finish invalid signature tests
* Include single and gossip verified blocks in tests
* Add gossip verification tests
* Start adding docs
* Finish adding comments to block_processing.rs
* Rename block_processing.rs -> block_verification
* Start removing old block processing code
* Fixes beacon_chain compilation
* Fix project-wide compile errors
* Remove old code
* Fix bug with beacon proposer index
* Fix shim for BlockProcessingError
* Only process one epoch at a time
* Fix loop in chain segment processing
* Add caching for state.eth1_data_votes
* Add BeaconChain::validator_pubkey
* Revert "Add caching for state.eth1_data_votes"
This reverts commit cd73dcd6434fb8d8e6bf30c5356355598ea7b78e.
* Add sync-speed metric (#898)
* Add PH & MS slot clock changes
* Account for genesis time
* Use checked mul
* Account for genesis slot
* Change API
* Allow for clock disparity
* Refactor "duration to..." functions
* Ensure errors are returned during batch processing
Co-authored-by: Michael Sproul <michael@sigmaprime.io>
Co-authored-by: Michael Sproul <micsproul@gmail.com>
Co-authored-by: pawan <pawandhananjay@gmail.com>
Co-authored-by: Paul Hauner <paul@paulhauner.com>
* Enr fork (#967)
* Start fixing enr-fork-id
* Fix time-until-next-fork logic
* Remove fork crate
* Update any references to beacon_chain.spec.milliseconds_per_slot to beacon_chain.slot_clock.slot_diration().as_nillis() in the attestation service. (#968)
* Process network attestations (#966)
* Start updating types
* WIP
* Signature hacking
* Existing EF tests passing with fake_crypto
* Updates
* Delete outdated API spec
* The refactor continues
* It compiles
* WIP test fixes
* All release tests passing bar genesis state parsing
* Update and test YamlConfig
* Update to spec v0.10 compatible BLS
* Updates to BLS EF tests
* Add EF test for AggregateVerify
And delete unused hash2curve tests for uncompressed points
* Update EF tests to v0.10.1
* Use optional block root correctly in block proc
* Use genesis fork in deposit domain. All tests pass
* Cargo fmt
* Fast aggregate verify test
* Update REST API docs
* Cargo fmt
* Fix unused import
* Bump spec tags to v0.10.1
* Add `seconds_per_eth1_block` to chainspec
* Update to timestamp based eth1 voting scheme
* Return None from `get_votes_to_consider` if block cache is empty
* Handle overflows in `is_candidate_block`
* Revert to failing tests
* Fix eth1 data sets test
* Choose default vote according to spec
* Fix collect_valid_votes tests
* Fix `get_votes_to_consider` to choose all eligible blocks
* Uncomment winning_vote tests
* Add comments; remove unused code
* Reduce seconds_per_eth1_block for simulation
* Addressed review comments
* Add test for default vote case
* Fix logs
* Remove unused functions
* Meter default eth1 votes
* Fix comments
* Address review comments; remove unused dependency
* Add first attempt at attestation proc. re-write
* Add version 2 of attestation processing
* Minor fixes
* Add validator pubkey cache
* Make get_indexed_attestation take a committee
* Link signature processing into new attn verification
* First working version
* Ensure pubkey cache is updated
* Add more metrics, slight optimizations
* Clone committee cache during attestation processing
* Update shuffling cache during block processing
* Remove old commented-out code
* Fix shuffling cache insert bug
* Used indexed attestation in fork choice
* Restructure attn processing, add metrics
* Add more detailed metrics
* Tidy, fix failing tests
* Fix failing tests, tidy
* Disable/delete two outdated tests
* Add new Pubkeys struct to signature_sets
* Refactor with functional approach
* Update beacon chain
* Remove decompressed member from pubkey bytes
* Add hashmap for indices lookup
* Add state cache, remove store cache
* Only build the head committee cache
* Change `get_attesting_indices` to use Vec
* Fix failing test
* Tidy
* Add pubkey cache persistence file
* Add more comments
* Integrate persistence file into builder
* Add pubkey cache tests
* Add data_dir to beacon chain builder
* Remove Option in pubkey cache persistence file
* Ensure consistency between datadir/data_dir
* Fix failing network test
* Tidy
* Fix todos
* Improve tests
* Fix compile error
* Fix compile error from merge
* Split up block processing metrics
* Tidy
* Refactor get_pubkey_from_state
* Remove commented-out code
* Rename state_cache -> checkpoint_cache
* Rename Checkpoint -> Snapshot
* Tidy, add comments
* Tidy up find_head function
* Change some checkpoint -> snapshot
* Add tests
* Expose max_len
* Remove dead code
* Tidy
* Fix bug
* Add sync-speed metric
* Add first attempt at VerifiableBlock
* Start integrating into beacon chain
* Integrate VerifiableBlock
* Rename VerifableBlock -> PartialBlockVerification
* Add start of typed methods
* Add progress
* Add further progress
* Rename structs
* Add full block verification to block_processing.rs
* Further beacon chain integration
* Update checks for gossip
* Add todo
* Start adding segement verification
* Add passing chain segement test
* Initial integration with batch sync
* Minor changes
* Tidy, add more error checking
* Start adding chain_segment tests
* Finish invalid signature tests
* Include single and gossip verified blocks in tests
* Add gossip verification tests
* Start adding docs
* Finish adding comments to block_processing.rs
* Rename block_processing.rs -> block_verification
* Start removing old block processing code
* Fixes beacon_chain compilation
* Fix project-wide compile errors
* Remove old code
* Fix bug with beacon proposer index
* Fix shim for BlockProcessingError
* Only process one epoch at a time
* Fix loop in chain segment processing
* Add caching for state.eth1_data_votes
* Add BeaconChain::validator_pubkey
* Revert "Add caching for state.eth1_data_votes"
This reverts commit cd73dcd6434fb8d8e6bf30c5356355598ea7b78e.
* Allow for clock disparity
* Ensure errors are returned during batch processing
* Add block gossip verification
* Connect attestation processing to beacon chain
* Optimistically subscribe to subnets on the same slot
Co-authored-by: Michael Sproul <michael@sigmaprime.io>
Co-authored-by: Michael Sproul <micsproul@gmail.com>
Co-authored-by: pawan <pawandhananjay@gmail.com>
Co-authored-by: Paul Hauner <paul@paulhauner.com>
* Update /validator/subscribe (#969)
* Add progress on duties refactor
* Add simple is_aggregator bool to val subscription
* Remove unused function
* Upgrade sim (#972)
* Add progress on duties refactor
* Add simple is_aggregator bool to val subscription
* Add the no-eth1-sim, refactor sim
* Sends discovery for persistent subnets (#973)
* main takes cmdline arguments
* Add test script
* Fix errors
* snappy uses Uvi to encode/decode length prefix
* Add more comments
* Run fmt
Co-authored-by: Grant Wuerker <gwuerker@gmail.com>
Co-authored-by: Age Manning <Age@AgeManning.com>
Co-authored-by: Michael Sproul <micsproul@gmail.com>
Co-authored-by: Michael Sproul <michael@sigmaprime.io>
Co-authored-by: b-m-f <max@ehlers.berlin>
Co-authored-by: Paul Hauner <paul@paulhauner.com>
Co-authored-by: realbigsean <seananderson33@GMAIL.com>
2020-04-16 01:42:51 +00:00
|
|
|
|
|
|
|
let log = common::build_log(log_level, enable_logging);
|
|
|
|
let spec = E::default_spec();
|
|
|
|
|
2020-11-28 05:30:57 +00:00
|
|
|
let rt = Arc::new(Runtime::new().unwrap());
|
Add snappy compression support (#866)
* notes from call
* should_forward function
* should_forward_block cleanup
* some cleanup and notes
* complete draft
* corrections
* some cleanup
* ran cargo fmt
* Revert "ran cargo fmt"
This reverts commit 464a5c4b62401c68fff7d36d3cd8069560a57713.
* ran cargo fmt after updating
* proposer index mods
* fmt
* new way of reading state
* fmt
* fmt
* compiles again
* fmt
* Correct stream timeout delay queue handling
* Correct small typo
* Support BlocksByRange step parameter
* Add initial docs to syncing
* Initial start of naive-attestation-aggregation
* Remove ping protocol
* Initial renaming of network services
* Correct rebasing relative to latest master
* Adds HashMapDelay struct to utils
* Initial network restructure
* Network restructure. Adds new types for v0.2.0
* Removes build artefacts
* Shift validation to beacon chain
* Temporarily remove gossip validation
This is to be updated to match current optimisation efforts.
* Adds AggregateAndProof
* Begin rebuilding pubsub encoding/decoding
* Temp commit
* Shift gossipsup decoding in eth2_libp2p
* Shifts block encoding/decoding into RPC
* Progress on attestation service
* Initial work on removing libp2p lock
* Add LRU caches to store (rollup)
* Update attestation validation for DB changes (WIP)
* Initial version of should_forward_block
* Scaffold
* Progress on attestation validation
Also, consolidate prod+testing slot clocks so that they share much
of the same implementation and can both handle sub-slot time changes.
* Removes lock from libp2p service
* Completed network lock removal
* Finish(?) attestation processing
* Correct network termination future
* Add slot check to block check
* Correct fmt issues
* Remove Drop implementation for network service
* Address reviewers suggestions
* Modification of validator for subscriptions
* Add slot signing to validator client
* Further progress on validation subscription
* Register SSZ snappy protocol messages
* Add initial idea of snappy compressed Codec by replacing the UVI codec with the snap library
* Fix matching against protocol string
* Adds TODOs for implementation work
* Implements check against max packet size before attempting to decode snappy compressed packages
* Add ssz_snappy codec
* Adds necessary validator subscription functionality
* Progress on snappy codec
* Clean up validator <-> beacon node http types
* Add aggregator status to ValidatorDuty
* clear buffer after decoding
* Impl Clone for manual slot clock
* Fix minor errors
* Further progress validator client subscription
* Initial subscription and aggregation handling
* Progress to modifying val client for attestation aggregation
* First draft of validator client upgrade for aggregate attestations
* Trying something
* Length prefix compressed data
* Fix gossipsub tests
* Working snappy frames with compressed length prefix
* Removes lock on a network channel
* Partially implement beacon node subscription http api
* Uncompressed length prefix working
* Cleanup
* Remove Testing request and response
* Return codec from match statement; reduce code duplication
* Fix unsafe unwrap in Outbound decode
* Add length checks
* All encode/decode functions use snappy frame format
* Add a `full` BeaconBlock method
* Add executable to test rpc against other impls
* Remove unused code; minor fixes
* Add PH & MS slot clock changes
* Account for genesis time
* Use checked mul
* Account for genesis slot
* Change API
* Refactor "duration to..." functions
* Re-merge updated block processing to v0.2.0 (#962)
* Start updating types
* WIP
* Signature hacking
* Existing EF tests passing with fake_crypto
* Updates
* Delete outdated API spec
* The refactor continues
* It compiles
* WIP test fixes
* All release tests passing bar genesis state parsing
* Update and test YamlConfig
* Update to spec v0.10 compatible BLS
* Updates to BLS EF tests
* Add EF test for AggregateVerify
And delete unused hash2curve tests for uncompressed points
* Update EF tests to v0.10.1
* Use optional block root correctly in block proc
* Use genesis fork in deposit domain. All tests pass
* Cargo fmt
* Fast aggregate verify test
* Update REST API docs
* Cargo fmt
* Fix unused import
* Bump spec tags to v0.10.1
* Add `seconds_per_eth1_block` to chainspec
* Update to timestamp based eth1 voting scheme
* Return None from `get_votes_to_consider` if block cache is empty
* Handle overflows in `is_candidate_block`
* Revert to failing tests
* Fix eth1 data sets test
* Choose default vote according to spec
* Fix collect_valid_votes tests
* Fix `get_votes_to_consider` to choose all eligible blocks
* Uncomment winning_vote tests
* Add comments; remove unused code
* Reduce seconds_per_eth1_block for simulation
* Addressed review comments
* Add test for default vote case
* Fix logs
* Remove unused functions
* Meter default eth1 votes
* Fix comments
* Address review comments; remove unused dependency
* Add first attempt at attestation proc. re-write
* Add version 2 of attestation processing
* Minor fixes
* Add validator pubkey cache
* Make get_indexed_attestation take a committee
* Link signature processing into new attn verification
* First working version
* Ensure pubkey cache is updated
* Add more metrics, slight optimizations
* Clone committee cache during attestation processing
* Update shuffling cache during block processing
* Remove old commented-out code
* Fix shuffling cache insert bug
* Used indexed attestation in fork choice
* Restructure attn processing, add metrics
* Add more detailed metrics
* Tidy, fix failing tests
* Fix failing tests, tidy
* Disable/delete two outdated tests
* Add new Pubkeys struct to signature_sets
* Refactor with functional approach
* Update beacon chain
* Remove decompressed member from pubkey bytes
* Add hashmap for indices lookup
* Add state cache, remove store cache
* Only build the head committee cache
* Change `get_attesting_indices` to use Vec
* Fix failing test
* Tidy
* Add pubkey cache persistence file
* Add more comments
* Integrate persistence file into builder
* Add pubkey cache tests
* Add data_dir to beacon chain builder
* Remove Option in pubkey cache persistence file
* Ensure consistency between datadir/data_dir
* Fix failing network test
* Tidy
* Fix todos
* Improve tests
* Fix compile error
* Fix compile error from merge
* Split up block processing metrics
* Tidy
* Refactor get_pubkey_from_state
* Remove commented-out code
* Rename state_cache -> checkpoint_cache
* Rename Checkpoint -> Snapshot
* Tidy, add comments
* Tidy up find_head function
* Change some checkpoint -> snapshot
* Add tests
* Expose max_len
* Remove dead code
* Tidy
* Fix bug
* Add sync-speed metric
* Add first attempt at VerifiableBlock
* Start integrating into beacon chain
* Integrate VerifiableBlock
* Rename VerifableBlock -> PartialBlockVerification
* Add start of typed methods
* Add progress
* Add further progress
* Rename structs
* Add full block verification to block_processing.rs
* Further beacon chain integration
* Update checks for gossip
* Add todo
* Start adding segement verification
* Add passing chain segement test
* Initial integration with batch sync
* Minor changes
* Tidy, add more error checking
* Start adding chain_segment tests
* Finish invalid signature tests
* Include single and gossip verified blocks in tests
* Add gossip verification tests
* Start adding docs
* Finish adding comments to block_processing.rs
* Rename block_processing.rs -> block_verification
* Start removing old block processing code
* Fixes beacon_chain compilation
* Fix project-wide compile errors
* Remove old code
* Fix bug with beacon proposer index
* Fix shim for BlockProcessingError
* Only process one epoch at a time
* Fix loop in chain segment processing
* Add caching for state.eth1_data_votes
* Add BeaconChain::validator_pubkey
* Revert "Add caching for state.eth1_data_votes"
This reverts commit cd73dcd6434fb8d8e6bf30c5356355598ea7b78e.
* Add sync-speed metric (#898)
* Add PH & MS slot clock changes
* Account for genesis time
* Use checked mul
* Account for genesis slot
* Change API
* Allow for clock disparity
* Refactor "duration to..." functions
* Ensure errors are returned during batch processing
Co-authored-by: Michael Sproul <michael@sigmaprime.io>
Co-authored-by: Michael Sproul <micsproul@gmail.com>
Co-authored-by: pawan <pawandhananjay@gmail.com>
Co-authored-by: Paul Hauner <paul@paulhauner.com>
* Enr fork (#967)
* Start fixing enr-fork-id
* Fix time-until-next-fork logic
* Remove fork crate
* Update any references to beacon_chain.spec.milliseconds_per_slot to beacon_chain.slot_clock.slot_diration().as_nillis() in the attestation service. (#968)
* Process network attestations (#966)
* Start updating types
* WIP
* Signature hacking
* Existing EF tests passing with fake_crypto
* Updates
* Delete outdated API spec
* The refactor continues
* It compiles
* WIP test fixes
* All release tests passing bar genesis state parsing
* Update and test YamlConfig
* Update to spec v0.10 compatible BLS
* Updates to BLS EF tests
* Add EF test for AggregateVerify
And delete unused hash2curve tests for uncompressed points
* Update EF tests to v0.10.1
* Use optional block root correctly in block proc
* Use genesis fork in deposit domain. All tests pass
* Cargo fmt
* Fast aggregate verify test
* Update REST API docs
* Cargo fmt
* Fix unused import
* Bump spec tags to v0.10.1
* Add `seconds_per_eth1_block` to chainspec
* Update to timestamp based eth1 voting scheme
* Return None from `get_votes_to_consider` if block cache is empty
* Handle overflows in `is_candidate_block`
* Revert to failing tests
* Fix eth1 data sets test
* Choose default vote according to spec
* Fix collect_valid_votes tests
* Fix `get_votes_to_consider` to choose all eligible blocks
* Uncomment winning_vote tests
* Add comments; remove unused code
* Reduce seconds_per_eth1_block for simulation
* Addressed review comments
* Add test for default vote case
* Fix logs
* Remove unused functions
* Meter default eth1 votes
* Fix comments
* Address review comments; remove unused dependency
* Add first attempt at attestation proc. re-write
* Add version 2 of attestation processing
* Minor fixes
* Add validator pubkey cache
* Make get_indexed_attestation take a committee
* Link signature processing into new attn verification
* First working version
* Ensure pubkey cache is updated
* Add more metrics, slight optimizations
* Clone committee cache during attestation processing
* Update shuffling cache during block processing
* Remove old commented-out code
* Fix shuffling cache insert bug
* Used indexed attestation in fork choice
* Restructure attn processing, add metrics
* Add more detailed metrics
* Tidy, fix failing tests
* Fix failing tests, tidy
* Disable/delete two outdated tests
* Add new Pubkeys struct to signature_sets
* Refactor with functional approach
* Update beacon chain
* Remove decompressed member from pubkey bytes
* Add hashmap for indices lookup
* Add state cache, remove store cache
* Only build the head committee cache
* Change `get_attesting_indices` to use Vec
* Fix failing test
* Tidy
* Add pubkey cache persistence file
* Add more comments
* Integrate persistence file into builder
* Add pubkey cache tests
* Add data_dir to beacon chain builder
* Remove Option in pubkey cache persistence file
* Ensure consistency between datadir/data_dir
* Fix failing network test
* Tidy
* Fix todos
* Improve tests
* Fix compile error
* Fix compile error from merge
* Split up block processing metrics
* Tidy
* Refactor get_pubkey_from_state
* Remove commented-out code
* Rename state_cache -> checkpoint_cache
* Rename Checkpoint -> Snapshot
* Tidy, add comments
* Tidy up find_head function
* Change some checkpoint -> snapshot
* Add tests
* Expose max_len
* Remove dead code
* Tidy
* Fix bug
* Add sync-speed metric
* Add first attempt at VerifiableBlock
* Start integrating into beacon chain
* Integrate VerifiableBlock
* Rename VerifableBlock -> PartialBlockVerification
* Add start of typed methods
* Add progress
* Add further progress
* Rename structs
* Add full block verification to block_processing.rs
* Further beacon chain integration
* Update checks for gossip
* Add todo
* Start adding segement verification
* Add passing chain segement test
* Initial integration with batch sync
* Minor changes
* Tidy, add more error checking
* Start adding chain_segment tests
* Finish invalid signature tests
* Include single and gossip verified blocks in tests
* Add gossip verification tests
* Start adding docs
* Finish adding comments to block_processing.rs
* Rename block_processing.rs -> block_verification
* Start removing old block processing code
* Fixes beacon_chain compilation
* Fix project-wide compile errors
* Remove old code
* Fix bug with beacon proposer index
* Fix shim for BlockProcessingError
* Only process one epoch at a time
* Fix loop in chain segment processing
* Add caching for state.eth1_data_votes
* Add BeaconChain::validator_pubkey
* Revert "Add caching for state.eth1_data_votes"
This reverts commit cd73dcd6434fb8d8e6bf30c5356355598ea7b78e.
* Allow for clock disparity
* Ensure errors are returned during batch processing
* Add block gossip verification
* Connect attestation processing to beacon chain
* Optimistically subscribe to subnets on the same slot
Co-authored-by: Michael Sproul <michael@sigmaprime.io>
Co-authored-by: Michael Sproul <micsproul@gmail.com>
Co-authored-by: pawan <pawandhananjay@gmail.com>
Co-authored-by: Paul Hauner <paul@paulhauner.com>
* Update /validator/subscribe (#969)
* Add progress on duties refactor
* Add simple is_aggregator bool to val subscription
* Remove unused function
* Upgrade sim (#972)
* Add progress on duties refactor
* Add simple is_aggregator bool to val subscription
* Add the no-eth1-sim, refactor sim
* Sends discovery for persistent subnets (#973)
* main takes cmdline arguments
* Add test script
* Fix errors
* snappy uses Uvi to encode/decode length prefix
* Add more comments
* Run fmt
Co-authored-by: Grant Wuerker <gwuerker@gmail.com>
Co-authored-by: Age Manning <Age@AgeManning.com>
Co-authored-by: Michael Sproul <micsproul@gmail.com>
Co-authored-by: Michael Sproul <michael@sigmaprime.io>
Co-authored-by: b-m-f <max@ehlers.berlin>
Co-authored-by: Paul Hauner <paul@paulhauner.com>
Co-authored-by: realbigsean <seananderson33@GMAIL.com>
2020-04-16 01:42:51 +00:00
|
|
|
// get sender/receiver
|
2020-11-28 05:30:57 +00:00
|
|
|
rt.block_on(async {
|
|
|
|
let (mut sender, mut receiver) = common::build_node_pair(Arc::downgrade(&rt), &log).await;
|
|
|
|
|
|
|
|
// BlocksByRoot Request
|
|
|
|
let rpc_request = Request::BlocksByRoot(BlocksByRootRequest {
|
|
|
|
block_roots: VariableList::from(vec![
|
|
|
|
Hash256::from_low_u64_be(0),
|
|
|
|
Hash256::from_low_u64_be(0),
|
|
|
|
Hash256::from_low_u64_be(0),
|
2021-09-17 01:11:16 +00:00
|
|
|
Hash256::from_low_u64_be(0),
|
|
|
|
Hash256::from_low_u64_be(0),
|
|
|
|
Hash256::from_low_u64_be(0),
|
2020-11-28 05:30:57 +00:00
|
|
|
]),
|
|
|
|
});
|
|
|
|
|
|
|
|
// BlocksByRoot Response
|
2021-08-04 01:44:57 +00:00
|
|
|
let full_block = BeaconBlock::Base(BeaconBlockBase::<E>::full(&spec));
|
2021-07-09 06:15:32 +00:00
|
|
|
let signed_full_block = SignedBeaconBlock::from_block(full_block, Signature::empty());
|
2021-08-04 01:44:57 +00:00
|
|
|
let rpc_response_base = Response::BlocksByRoot(Some(Box::new(signed_full_block)));
|
|
|
|
|
|
|
|
let full_block = BeaconBlock::Altair(BeaconBlockAltair::<E>::full(&spec));
|
|
|
|
let signed_full_block = SignedBeaconBlock::from_block(full_block, Signature::empty());
|
|
|
|
let rpc_response_altair = Response::BlocksByRoot(Some(Box::new(signed_full_block)));
|
2020-11-28 05:30:57 +00:00
|
|
|
|
2021-12-02 02:00:39 +00:00
|
|
|
let full_block = merge_block_small(&common::fork_context());
|
2021-11-09 16:42:02 +00:00
|
|
|
let signed_full_block = SignedBeaconBlock::from_block(full_block, Signature::empty());
|
|
|
|
let rpc_response_merge_small = Response::BlocksByRoot(Some(Box::new(signed_full_block)));
|
|
|
|
|
2020-11-28 05:30:57 +00:00
|
|
|
// keep count of the number of messages received
|
|
|
|
let mut messages_received = 0;
|
|
|
|
// build the sender future
|
|
|
|
let sender_future = async {
|
|
|
|
loop {
|
|
|
|
match sender.next_event().await {
|
2021-07-13 00:48:33 +00:00
|
|
|
Libp2pEvent::Behaviour(BehaviourEvent::PeerConnectedOutgoing(peer_id)) => {
|
2020-11-28 05:30:57 +00:00
|
|
|
// Send a STATUS message
|
|
|
|
debug!(log, "Sending RPC");
|
2021-06-03 01:11:33 +00:00
|
|
|
sender.swarm.behaviour_mut().send_request(
|
2020-11-28 05:30:57 +00:00
|
|
|
peer_id,
|
2021-11-09 16:42:02 +00:00
|
|
|
RequestId::Sync(6),
|
2020-11-28 05:30:57 +00:00
|
|
|
rpc_request.clone(),
|
|
|
|
);
|
Add snappy compression support (#866)
* notes from call
* should_forward function
* should_forward_block cleanup
* some cleanup and notes
* complete draft
* corrections
* some cleanup
* ran cargo fmt
* Revert "ran cargo fmt"
This reverts commit 464a5c4b62401c68fff7d36d3cd8069560a57713.
* ran cargo fmt after updating
* proposer index mods
* fmt
* new way of reading state
* fmt
* fmt
* compiles again
* fmt
* Correct stream timeout delay queue handling
* Correct small typo
* Support BlocksByRange step parameter
* Add initial docs to syncing
* Initial start of naive-attestation-aggregation
* Remove ping protocol
* Initial renaming of network services
* Correct rebasing relative to latest master
* Adds HashMapDelay struct to utils
* Initial network restructure
* Network restructure. Adds new types for v0.2.0
* Removes build artefacts
* Shift validation to beacon chain
* Temporarily remove gossip validation
This is to be updated to match current optimisation efforts.
* Adds AggregateAndProof
* Begin rebuilding pubsub encoding/decoding
* Temp commit
* Shift gossipsup decoding in eth2_libp2p
* Shifts block encoding/decoding into RPC
* Progress on attestation service
* Initial work on removing libp2p lock
* Add LRU caches to store (rollup)
* Update attestation validation for DB changes (WIP)
* Initial version of should_forward_block
* Scaffold
* Progress on attestation validation
Also, consolidate prod+testing slot clocks so that they share much
of the same implementation and can both handle sub-slot time changes.
* Removes lock from libp2p service
* Completed network lock removal
* Finish(?) attestation processing
* Correct network termination future
* Add slot check to block check
* Correct fmt issues
* Remove Drop implementation for network service
* Address reviewers suggestions
* Modification of validator for subscriptions
* Add slot signing to validator client
* Further progress on validation subscription
* Register SSZ snappy protocol messages
* Add initial idea of snappy compressed Codec by replacing the UVI codec with the snap library
* Fix matching against protocol string
* Adds TODOs for implementation work
* Implements check against max packet size before attempting to decode snappy compressed packages
* Add ssz_snappy codec
* Adds necessary validator subscription functionality
* Progress on snappy codec
* Clean up validator <-> beacon node http types
* Add aggregator status to ValidatorDuty
* clear buffer after decoding
* Impl Clone for manual slot clock
* Fix minor errors
* Further progress validator client subscription
* Initial subscription and aggregation handling
* Progress to modifying val client for attestation aggregation
* First draft of validator client upgrade for aggregate attestations
* Trying something
* Length prefix compressed data
* Fix gossipsub tests
* Working snappy frames with compressed length prefix
* Removes lock on a network channel
* Partially implement beacon node subscription http api
* Uncompressed length prefix working
* Cleanup
* Remove Testing request and response
* Return codec from match statement; reduce code duplication
* Fix unsafe unwrap in Outbound decode
* Add length checks
* All encode/decode functions use snappy frame format
* Add a `full` BeaconBlock method
* Add executable to test rpc against other impls
* Remove unused code; minor fixes
* Add PH & MS slot clock changes
* Account for genesis time
* Use checked mul
* Account for genesis slot
* Change API
* Refactor "duration to..." functions
* Re-merge updated block processing to v0.2.0 (#962)
* Start updating types
* WIP
* Signature hacking
* Existing EF tests passing with fake_crypto
* Updates
* Delete outdated API spec
* The refactor continues
* It compiles
* WIP test fixes
* All release tests passing bar genesis state parsing
* Update and test YamlConfig
* Update to spec v0.10 compatible BLS
* Updates to BLS EF tests
* Add EF test for AggregateVerify
And delete unused hash2curve tests for uncompressed points
* Update EF tests to v0.10.1
* Use optional block root correctly in block proc
* Use genesis fork in deposit domain. All tests pass
* Cargo fmt
* Fast aggregate verify test
* Update REST API docs
* Cargo fmt
* Fix unused import
* Bump spec tags to v0.10.1
* Add `seconds_per_eth1_block` to chainspec
* Update to timestamp based eth1 voting scheme
* Return None from `get_votes_to_consider` if block cache is empty
* Handle overflows in `is_candidate_block`
* Revert to failing tests
* Fix eth1 data sets test
* Choose default vote according to spec
* Fix collect_valid_votes tests
* Fix `get_votes_to_consider` to choose all eligible blocks
* Uncomment winning_vote tests
* Add comments; remove unused code
* Reduce seconds_per_eth1_block for simulation
* Addressed review comments
* Add test for default vote case
* Fix logs
* Remove unused functions
* Meter default eth1 votes
* Fix comments
* Address review comments; remove unused dependency
* Add first attempt at attestation proc. re-write
* Add version 2 of attestation processing
* Minor fixes
* Add validator pubkey cache
* Make get_indexed_attestation take a committee
* Link signature processing into new attn verification
* First working version
* Ensure pubkey cache is updated
* Add more metrics, slight optimizations
* Clone committee cache during attestation processing
* Update shuffling cache during block processing
* Remove old commented-out code
* Fix shuffling cache insert bug
* Used indexed attestation in fork choice
* Restructure attn processing, add metrics
* Add more detailed metrics
* Tidy, fix failing tests
* Fix failing tests, tidy
* Disable/delete two outdated tests
* Add new Pubkeys struct to signature_sets
* Refactor with functional approach
* Update beacon chain
* Remove decompressed member from pubkey bytes
* Add hashmap for indices lookup
* Add state cache, remove store cache
* Only build the head committee cache
* Change `get_attesting_indices` to use Vec
* Fix failing test
* Tidy
* Add pubkey cache persistence file
* Add more comments
* Integrate persistence file into builder
* Add pubkey cache tests
* Add data_dir to beacon chain builder
* Remove Option in pubkey cache persistence file
* Ensure consistency between datadir/data_dir
* Fix failing network test
* Tidy
* Fix todos
* Improve tests
* Fix compile error
* Fix compile error from merge
* Split up block processing metrics
* Tidy
* Refactor get_pubkey_from_state
* Remove commented-out code
* Rename state_cache -> checkpoint_cache
* Rename Checkpoint -> Snapshot
* Tidy, add comments
* Tidy up find_head function
* Change some checkpoint -> snapshot
* Add tests
* Expose max_len
* Remove dead code
* Tidy
* Fix bug
* Add sync-speed metric
* Add first attempt at VerifiableBlock
* Start integrating into beacon chain
* Integrate VerifiableBlock
* Rename VerifableBlock -> PartialBlockVerification
* Add start of typed methods
* Add progress
* Add further progress
* Rename structs
* Add full block verification to block_processing.rs
* Further beacon chain integration
* Update checks for gossip
* Add todo
* Start adding segement verification
* Add passing chain segement test
* Initial integration with batch sync
* Minor changes
* Tidy, add more error checking
* Start adding chain_segment tests
* Finish invalid signature tests
* Include single and gossip verified blocks in tests
* Add gossip verification tests
* Start adding docs
* Finish adding comments to block_processing.rs
* Rename block_processing.rs -> block_verification
* Start removing old block processing code
* Fixes beacon_chain compilation
* Fix project-wide compile errors
* Remove old code
* Fix bug with beacon proposer index
* Fix shim for BlockProcessingError
* Only process one epoch at a time
* Fix loop in chain segment processing
* Add caching for state.eth1_data_votes
* Add BeaconChain::validator_pubkey
* Revert "Add caching for state.eth1_data_votes"
This reverts commit cd73dcd6434fb8d8e6bf30c5356355598ea7b78e.
* Add sync-speed metric (#898)
* Add PH & MS slot clock changes
* Account for genesis time
* Use checked mul
* Account for genesis slot
* Change API
* Allow for clock disparity
* Refactor "duration to..." functions
* Ensure errors are returned during batch processing
Co-authored-by: Michael Sproul <michael@sigmaprime.io>
Co-authored-by: Michael Sproul <micsproul@gmail.com>
Co-authored-by: pawan <pawandhananjay@gmail.com>
Co-authored-by: Paul Hauner <paul@paulhauner.com>
* Enr fork (#967)
* Start fixing enr-fork-id
* Fix time-until-next-fork logic
* Remove fork crate
* Update any references to beacon_chain.spec.milliseconds_per_slot to beacon_chain.slot_clock.slot_diration().as_nillis() in the attestation service. (#968)
* Process network attestations (#966)
* Start updating types
* WIP
* Signature hacking
* Existing EF tests passing with fake_crypto
* Updates
* Delete outdated API spec
* The refactor continues
* It compiles
* WIP test fixes
* All release tests passing bar genesis state parsing
* Update and test YamlConfig
* Update to spec v0.10 compatible BLS
* Updates to BLS EF tests
* Add EF test for AggregateVerify
And delete unused hash2curve tests for uncompressed points
* Update EF tests to v0.10.1
* Use optional block root correctly in block proc
* Use genesis fork in deposit domain. All tests pass
* Cargo fmt
* Fast aggregate verify test
* Update REST API docs
* Cargo fmt
* Fix unused import
* Bump spec tags to v0.10.1
* Add `seconds_per_eth1_block` to chainspec
* Update to timestamp based eth1 voting scheme
* Return None from `get_votes_to_consider` if block cache is empty
* Handle overflows in `is_candidate_block`
* Revert to failing tests
* Fix eth1 data sets test
* Choose default vote according to spec
* Fix collect_valid_votes tests
* Fix `get_votes_to_consider` to choose all eligible blocks
* Uncomment winning_vote tests
* Add comments; remove unused code
* Reduce seconds_per_eth1_block for simulation
* Addressed review comments
* Add test for default vote case
* Fix logs
* Remove unused functions
* Meter default eth1 votes
* Fix comments
* Address review comments; remove unused dependency
* Add first attempt at attestation proc. re-write
* Add version 2 of attestation processing
* Minor fixes
* Add validator pubkey cache
* Make get_indexed_attestation take a committee
* Link signature processing into new attn verification
* First working version
* Ensure pubkey cache is updated
* Add more metrics, slight optimizations
* Clone committee cache during attestation processing
* Update shuffling cache during block processing
* Remove old commented-out code
* Fix shuffling cache insert bug
* Used indexed attestation in fork choice
* Restructure attn processing, add metrics
* Add more detailed metrics
* Tidy, fix failing tests
* Fix failing tests, tidy
* Disable/delete two outdated tests
* Add new Pubkeys struct to signature_sets
* Refactor with functional approach
* Update beacon chain
* Remove decompressed member from pubkey bytes
* Add hashmap for indices lookup
* Add state cache, remove store cache
* Only build the head committee cache
* Change `get_attesting_indices` to use Vec
* Fix failing test
* Tidy
* Add pubkey cache persistence file
* Add more comments
* Integrate persistence file into builder
* Add pubkey cache tests
* Add data_dir to beacon chain builder
* Remove Option in pubkey cache persistence file
* Ensure consistency between datadir/data_dir
* Fix failing network test
* Tidy
* Fix todos
* Improve tests
* Fix compile error
* Fix compile error from merge
* Split up block processing metrics
* Tidy
* Refactor get_pubkey_from_state
* Remove commented-out code
* Rename state_cache -> checkpoint_cache
* Rename Checkpoint -> Snapshot
* Tidy, add comments
* Tidy up find_head function
* Change some checkpoint -> snapshot
* Add tests
* Expose max_len
* Remove dead code
* Tidy
* Fix bug
* Add sync-speed metric
* Add first attempt at VerifiableBlock
* Start integrating into beacon chain
* Integrate VerifiableBlock
* Rename VerifableBlock -> PartialBlockVerification
* Add start of typed methods
* Add progress
* Add further progress
* Rename structs
* Add full block verification to block_processing.rs
* Further beacon chain integration
* Update checks for gossip
* Add todo
* Start adding segement verification
* Add passing chain segement test
* Initial integration with batch sync
* Minor changes
* Tidy, add more error checking
* Start adding chain_segment tests
* Finish invalid signature tests
* Include single and gossip verified blocks in tests
* Add gossip verification tests
* Start adding docs
* Finish adding comments to block_processing.rs
* Rename block_processing.rs -> block_verification
* Start removing old block processing code
* Fixes beacon_chain compilation
* Fix project-wide compile errors
* Remove old code
* Fix bug with beacon proposer index
* Fix shim for BlockProcessingError
* Only process one epoch at a time
* Fix loop in chain segment processing
* Add caching for state.eth1_data_votes
* Add BeaconChain::validator_pubkey
* Revert "Add caching for state.eth1_data_votes"
This reverts commit cd73dcd6434fb8d8e6bf30c5356355598ea7b78e.
* Allow for clock disparity
* Ensure errors are returned during batch processing
* Add block gossip verification
* Connect attestation processing to beacon chain
* Optimistically subscribe to subnets on the same slot
Co-authored-by: Michael Sproul <michael@sigmaprime.io>
Co-authored-by: Michael Sproul <micsproul@gmail.com>
Co-authored-by: pawan <pawandhananjay@gmail.com>
Co-authored-by: Paul Hauner <paul@paulhauner.com>
* Update /validator/subscribe (#969)
* Add progress on duties refactor
* Add simple is_aggregator bool to val subscription
* Remove unused function
* Upgrade sim (#972)
* Add progress on duties refactor
* Add simple is_aggregator bool to val subscription
* Add the no-eth1-sim, refactor sim
* Sends discovery for persistent subnets (#973)
* main takes cmdline arguments
* Add test script
* Fix errors
* snappy uses Uvi to encode/decode length prefix
* Add more comments
* Run fmt
Co-authored-by: Grant Wuerker <gwuerker@gmail.com>
Co-authored-by: Age Manning <Age@AgeManning.com>
Co-authored-by: Michael Sproul <micsproul@gmail.com>
Co-authored-by: Michael Sproul <michael@sigmaprime.io>
Co-authored-by: b-m-f <max@ehlers.berlin>
Co-authored-by: Paul Hauner <paul@paulhauner.com>
Co-authored-by: realbigsean <seananderson33@GMAIL.com>
2020-04-16 01:42:51 +00:00
|
|
|
}
|
2020-11-28 05:30:57 +00:00
|
|
|
Libp2pEvent::Behaviour(BehaviourEvent::ResponseReceived {
|
|
|
|
peer_id: _,
|
2021-11-09 16:42:02 +00:00
|
|
|
id: RequestId::Sync(6),
|
2020-11-28 05:30:57 +00:00
|
|
|
response,
|
|
|
|
}) => match response {
|
|
|
|
Response::BlocksByRoot(Some(_)) => {
|
2021-11-09 16:42:02 +00:00
|
|
|
if messages_received < 2 {
|
2021-08-04 01:44:57 +00:00
|
|
|
assert_eq!(response, rpc_response_base.clone());
|
2021-11-09 16:42:02 +00:00
|
|
|
} else if messages_received < 4 {
|
2021-08-04 01:44:57 +00:00
|
|
|
assert_eq!(response, rpc_response_altair.clone());
|
2021-11-09 16:42:02 +00:00
|
|
|
} else {
|
|
|
|
assert_eq!(response, rpc_response_merge_small.clone());
|
2021-08-04 01:44:57 +00:00
|
|
|
}
|
2020-11-28 05:30:57 +00:00
|
|
|
messages_received += 1;
|
|
|
|
debug!(log, "Chunk received");
|
|
|
|
}
|
|
|
|
Response::BlocksByRoot(None) => {
|
|
|
|
// should be exactly messages_to_send
|
|
|
|
assert_eq!(messages_received, messages_to_send);
|
|
|
|
// end the test
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
_ => {} // Ignore other RPC messages
|
|
|
|
},
|
|
|
|
_ => {} // Ignore other behaviour events
|
|
|
|
}
|
2020-05-17 11:16:48 +00:00
|
|
|
}
|
2020-11-28 05:30:57 +00:00
|
|
|
};
|
|
|
|
|
|
|
|
// build the receiver future
|
|
|
|
let receiver_future = async {
|
|
|
|
loop {
|
|
|
|
match receiver.next_event().await {
|
|
|
|
Libp2pEvent::Behaviour(BehaviourEvent::RequestReceived {
|
|
|
|
peer_id,
|
|
|
|
id,
|
|
|
|
request,
|
|
|
|
}) => {
|
|
|
|
if request == rpc_request {
|
|
|
|
// send the response
|
|
|
|
debug!(log, "Receiver got request");
|
|
|
|
|
2021-08-04 01:44:57 +00:00
|
|
|
for i in 0..messages_to_send {
|
2021-11-09 16:42:02 +00:00
|
|
|
// Send equal base, altair and merge blocks
|
|
|
|
let rpc_response = if i < 2 {
|
2021-08-04 01:44:57 +00:00
|
|
|
rpc_response_base.clone()
|
2021-11-09 16:42:02 +00:00
|
|
|
} else if i < 4 {
|
2021-08-04 01:44:57 +00:00
|
|
|
rpc_response_altair.clone()
|
2021-11-09 16:42:02 +00:00
|
|
|
} else {
|
|
|
|
rpc_response_merge_small.clone()
|
2021-08-04 01:44:57 +00:00
|
|
|
};
|
2021-06-03 01:11:33 +00:00
|
|
|
receiver.swarm.behaviour_mut().send_successful_response(
|
2021-01-28 23:31:06 +00:00
|
|
|
peer_id,
|
2020-11-28 05:30:57 +00:00
|
|
|
id,
|
2021-08-04 01:44:57 +00:00
|
|
|
rpc_response,
|
2020-11-28 05:30:57 +00:00
|
|
|
);
|
|
|
|
debug!(log, "Sending message");
|
|
|
|
}
|
|
|
|
// send the stream termination
|
2021-06-03 01:11:33 +00:00
|
|
|
receiver.swarm.behaviour_mut().send_successful_response(
|
2020-11-28 05:30:57 +00:00
|
|
|
peer_id,
|
2020-06-05 03:07:59 +00:00
|
|
|
id,
|
2020-11-28 05:30:57 +00:00
|
|
|
Response::BlocksByRange(None),
|
2020-06-05 03:07:59 +00:00
|
|
|
);
|
2020-11-28 05:30:57 +00:00
|
|
|
debug!(log, "Send stream term");
|
Add snappy compression support (#866)
* notes from call
* should_forward function
* should_forward_block cleanup
* some cleanup and notes
* complete draft
* corrections
* some cleanup
* ran cargo fmt
* Revert "ran cargo fmt"
This reverts commit 464a5c4b62401c68fff7d36d3cd8069560a57713.
* ran cargo fmt after updating
* proposer index mods
* fmt
* new way of reading state
* fmt
* fmt
* compiles again
* fmt
* Correct stream timeout delay queue handling
* Correct small typo
* Support BlocksByRange step parameter
* Add initial docs to syncing
* Initial start of naive-attestation-aggregation
* Remove ping protocol
* Initial renaming of network services
* Correct rebasing relative to latest master
* Adds HashMapDelay struct to utils
* Initial network restructure
* Network restructure. Adds new types for v0.2.0
* Removes build artefacts
* Shift validation to beacon chain
* Temporarily remove gossip validation
This is to be updated to match current optimisation efforts.
* Adds AggregateAndProof
* Begin rebuilding pubsub encoding/decoding
* Temp commit
* Shift gossipsup decoding in eth2_libp2p
* Shifts block encoding/decoding into RPC
* Progress on attestation service
* Initial work on removing libp2p lock
* Add LRU caches to store (rollup)
* Update attestation validation for DB changes (WIP)
* Initial version of should_forward_block
* Scaffold
* Progress on attestation validation
Also, consolidate prod+testing slot clocks so that they share much
of the same implementation and can both handle sub-slot time changes.
* Removes lock from libp2p service
* Completed network lock removal
* Finish(?) attestation processing
* Correct network termination future
* Add slot check to block check
* Correct fmt issues
* Remove Drop implementation for network service
* Address reviewers suggestions
* Modification of validator for subscriptions
* Add slot signing to validator client
* Further progress on validation subscription
* Register SSZ snappy protocol messages
* Add initial idea of snappy compressed Codec by replacing the UVI codec with the snap library
* Fix matching against protocol string
* Adds TODOs for implementation work
* Implements check against max packet size before attempting to decode snappy compressed packages
* Add ssz_snappy codec
* Adds necessary validator subscription functionality
* Progress on snappy codec
* Clean up validator <-> beacon node http types
* Add aggregator status to ValidatorDuty
* clear buffer after decoding
* Impl Clone for manual slot clock
* Fix minor errors
* Further progress validator client subscription
* Initial subscription and aggregation handling
* Progress to modifying val client for attestation aggregation
* First draft of validator client upgrade for aggregate attestations
* Trying something
* Length prefix compressed data
* Fix gossipsub tests
* Working snappy frames with compressed length prefix
* Removes lock on a network channel
* Partially implement beacon node subscription http api
* Uncompressed length prefix working
* Cleanup
* Remove Testing request and response
* Return codec from match statement; reduce code duplication
* Fix unsafe unwrap in Outbound decode
* Add length checks
* All encode/decode functions use snappy frame format
* Add a `full` BeaconBlock method
* Add executable to test rpc against other impls
* Remove unused code; minor fixes
* Add PH & MS slot clock changes
* Account for genesis time
* Use checked mul
* Account for genesis slot
* Change API
* Refactor "duration to..." functions
* Re-merge updated block processing to v0.2.0 (#962)
* Start updating types
* WIP
* Signature hacking
* Existing EF tests passing with fake_crypto
* Updates
* Delete outdated API spec
* The refactor continues
* It compiles
* WIP test fixes
* All release tests passing bar genesis state parsing
* Update and test YamlConfig
* Update to spec v0.10 compatible BLS
* Updates to BLS EF tests
* Add EF test for AggregateVerify
And delete unused hash2curve tests for uncompressed points
* Update EF tests to v0.10.1
* Use optional block root correctly in block proc
* Use genesis fork in deposit domain. All tests pass
* Cargo fmt
* Fast aggregate verify test
* Update REST API docs
* Cargo fmt
* Fix unused import
* Bump spec tags to v0.10.1
* Add `seconds_per_eth1_block` to chainspec
* Update to timestamp based eth1 voting scheme
* Return None from `get_votes_to_consider` if block cache is empty
* Handle overflows in `is_candidate_block`
* Revert to failing tests
* Fix eth1 data sets test
* Choose default vote according to spec
* Fix collect_valid_votes tests
* Fix `get_votes_to_consider` to choose all eligible blocks
* Uncomment winning_vote tests
* Add comments; remove unused code
* Reduce seconds_per_eth1_block for simulation
* Addressed review comments
* Add test for default vote case
* Fix logs
* Remove unused functions
* Meter default eth1 votes
* Fix comments
* Address review comments; remove unused dependency
* Add first attempt at attestation proc. re-write
* Add version 2 of attestation processing
* Minor fixes
* Add validator pubkey cache
* Make get_indexed_attestation take a committee
* Link signature processing into new attn verification
* First working version
* Ensure pubkey cache is updated
* Add more metrics, slight optimizations
* Clone committee cache during attestation processing
* Update shuffling cache during block processing
* Remove old commented-out code
* Fix shuffling cache insert bug
* Used indexed attestation in fork choice
* Restructure attn processing, add metrics
* Add more detailed metrics
* Tidy, fix failing tests
* Fix failing tests, tidy
* Disable/delete two outdated tests
* Add new Pubkeys struct to signature_sets
* Refactor with functional approach
* Update beacon chain
* Remove decompressed member from pubkey bytes
* Add hashmap for indices lookup
* Add state cache, remove store cache
* Only build the head committee cache
* Change `get_attesting_indices` to use Vec
* Fix failing test
* Tidy
* Add pubkey cache persistence file
* Add more comments
* Integrate persistence file into builder
* Add pubkey cache tests
* Add data_dir to beacon chain builder
* Remove Option in pubkey cache persistence file
* Ensure consistency between datadir/data_dir
* Fix failing network test
* Tidy
* Fix todos
* Improve tests
* Fix compile error
* Fix compile error from merge
* Split up block processing metrics
* Tidy
* Refactor get_pubkey_from_state
* Remove commented-out code
* Rename state_cache -> checkpoint_cache
* Rename Checkpoint -> Snapshot
* Tidy, add comments
* Tidy up find_head function
* Change some checkpoint -> snapshot
* Add tests
* Expose max_len
* Remove dead code
* Tidy
* Fix bug
* Add sync-speed metric
* Add first attempt at VerifiableBlock
* Start integrating into beacon chain
* Integrate VerifiableBlock
* Rename VerifableBlock -> PartialBlockVerification
* Add start of typed methods
* Add progress
* Add further progress
* Rename structs
* Add full block verification to block_processing.rs
* Further beacon chain integration
* Update checks for gossip
* Add todo
* Start adding segement verification
* Add passing chain segement test
* Initial integration with batch sync
* Minor changes
* Tidy, add more error checking
* Start adding chain_segment tests
* Finish invalid signature tests
* Include single and gossip verified blocks in tests
* Add gossip verification tests
* Start adding docs
* Finish adding comments to block_processing.rs
* Rename block_processing.rs -> block_verification
* Start removing old block processing code
* Fixes beacon_chain compilation
* Fix project-wide compile errors
* Remove old code
* Fix bug with beacon proposer index
* Fix shim for BlockProcessingError
* Only process one epoch at a time
* Fix loop in chain segment processing
* Add caching for state.eth1_data_votes
* Add BeaconChain::validator_pubkey
* Revert "Add caching for state.eth1_data_votes"
This reverts commit cd73dcd6434fb8d8e6bf30c5356355598ea7b78e.
* Add sync-speed metric (#898)
* Add PH & MS slot clock changes
* Account for genesis time
* Use checked mul
* Account for genesis slot
* Change API
* Allow for clock disparity
* Refactor "duration to..." functions
* Ensure errors are returned during batch processing
Co-authored-by: Michael Sproul <michael@sigmaprime.io>
Co-authored-by: Michael Sproul <micsproul@gmail.com>
Co-authored-by: pawan <pawandhananjay@gmail.com>
Co-authored-by: Paul Hauner <paul@paulhauner.com>
* Enr fork (#967)
* Start fixing enr-fork-id
* Fix time-until-next-fork logic
* Remove fork crate
* Update any references to beacon_chain.spec.milliseconds_per_slot to beacon_chain.slot_clock.slot_diration().as_nillis() in the attestation service. (#968)
* Process network attestations (#966)
* Start updating types
* WIP
* Signature hacking
* Existing EF tests passing with fake_crypto
* Updates
* Delete outdated API spec
* The refactor continues
* It compiles
* WIP test fixes
* All release tests passing bar genesis state parsing
* Update and test YamlConfig
* Update to spec v0.10 compatible BLS
* Updates to BLS EF tests
* Add EF test for AggregateVerify
And delete unused hash2curve tests for uncompressed points
* Update EF tests to v0.10.1
* Use optional block root correctly in block proc
* Use genesis fork in deposit domain. All tests pass
* Cargo fmt
* Fast aggregate verify test
* Update REST API docs
* Cargo fmt
* Fix unused import
* Bump spec tags to v0.10.1
* Add `seconds_per_eth1_block` to chainspec
* Update to timestamp based eth1 voting scheme
* Return None from `get_votes_to_consider` if block cache is empty
* Handle overflows in `is_candidate_block`
* Revert to failing tests
* Fix eth1 data sets test
* Choose default vote according to spec
* Fix collect_valid_votes tests
* Fix `get_votes_to_consider` to choose all eligible blocks
* Uncomment winning_vote tests
* Add comments; remove unused code
* Reduce seconds_per_eth1_block for simulation
* Addressed review comments
* Add test for default vote case
* Fix logs
* Remove unused functions
* Meter default eth1 votes
* Fix comments
* Address review comments; remove unused dependency
* Add first attempt at attestation proc. re-write
* Add version 2 of attestation processing
* Minor fixes
* Add validator pubkey cache
* Make get_indexed_attestation take a committee
* Link signature processing into new attn verification
* First working version
* Ensure pubkey cache is updated
* Add more metrics, slight optimizations
* Clone committee cache during attestation processing
* Update shuffling cache during block processing
* Remove old commented-out code
* Fix shuffling cache insert bug
* Used indexed attestation in fork choice
* Restructure attn processing, add metrics
* Add more detailed metrics
* Tidy, fix failing tests
* Fix failing tests, tidy
* Disable/delete two outdated tests
* Add new Pubkeys struct to signature_sets
* Refactor with functional approach
* Update beacon chain
* Remove decompressed member from pubkey bytes
* Add hashmap for indices lookup
* Add state cache, remove store cache
* Only build the head committee cache
* Change `get_attesting_indices` to use Vec
* Fix failing test
* Tidy
* Add pubkey cache persistence file
* Add more comments
* Integrate persistence file into builder
* Add pubkey cache tests
* Add data_dir to beacon chain builder
* Remove Option in pubkey cache persistence file
* Ensure consistency between datadir/data_dir
* Fix failing network test
* Tidy
* Fix todos
* Improve tests
* Fix compile error
* Fix compile error from merge
* Split up block processing metrics
* Tidy
* Refactor get_pubkey_from_state
* Remove commented-out code
* Rename state_cache -> checkpoint_cache
* Rename Checkpoint -> Snapshot
* Tidy, add comments
* Tidy up find_head function
* Change some checkpoint -> snapshot
* Add tests
* Expose max_len
* Remove dead code
* Tidy
* Fix bug
* Add sync-speed metric
* Add first attempt at VerifiableBlock
* Start integrating into beacon chain
* Integrate VerifiableBlock
* Rename VerifableBlock -> PartialBlockVerification
* Add start of typed methods
* Add progress
* Add further progress
* Rename structs
* Add full block verification to block_processing.rs
* Further beacon chain integration
* Update checks for gossip
* Add todo
* Start adding segement verification
* Add passing chain segement test
* Initial integration with batch sync
* Minor changes
* Tidy, add more error checking
* Start adding chain_segment tests
* Finish invalid signature tests
* Include single and gossip verified blocks in tests
* Add gossip verification tests
* Start adding docs
* Finish adding comments to block_processing.rs
* Rename block_processing.rs -> block_verification
* Start removing old block processing code
* Fixes beacon_chain compilation
* Fix project-wide compile errors
* Remove old code
* Fix bug with beacon proposer index
* Fix shim for BlockProcessingError
* Only process one epoch at a time
* Fix loop in chain segment processing
* Add caching for state.eth1_data_votes
* Add BeaconChain::validator_pubkey
* Revert "Add caching for state.eth1_data_votes"
This reverts commit cd73dcd6434fb8d8e6bf30c5356355598ea7b78e.
* Allow for clock disparity
* Ensure errors are returned during batch processing
* Add block gossip verification
* Connect attestation processing to beacon chain
* Optimistically subscribe to subnets on the same slot
Co-authored-by: Michael Sproul <michael@sigmaprime.io>
Co-authored-by: Michael Sproul <micsproul@gmail.com>
Co-authored-by: pawan <pawandhananjay@gmail.com>
Co-authored-by: Paul Hauner <paul@paulhauner.com>
* Update /validator/subscribe (#969)
* Add progress on duties refactor
* Add simple is_aggregator bool to val subscription
* Remove unused function
* Upgrade sim (#972)
* Add progress on duties refactor
* Add simple is_aggregator bool to val subscription
* Add the no-eth1-sim, refactor sim
* Sends discovery for persistent subnets (#973)
* main takes cmdline arguments
* Add test script
* Fix errors
* snappy uses Uvi to encode/decode length prefix
* Add more comments
* Run fmt
Co-authored-by: Grant Wuerker <gwuerker@gmail.com>
Co-authored-by: Age Manning <Age@AgeManning.com>
Co-authored-by: Michael Sproul <micsproul@gmail.com>
Co-authored-by: Michael Sproul <michael@sigmaprime.io>
Co-authored-by: b-m-f <max@ehlers.berlin>
Co-authored-by: Paul Hauner <paul@paulhauner.com>
Co-authored-by: realbigsean <seananderson33@GMAIL.com>
2020-04-16 01:42:51 +00:00
|
|
|
}
|
|
|
|
}
|
2020-11-28 05:30:57 +00:00
|
|
|
_ => {} // Ignore other events
|
2020-05-17 11:16:48 +00:00
|
|
|
}
|
2020-11-28 05:30:57 +00:00
|
|
|
}
|
|
|
|
};
|
|
|
|
tokio::select! {
|
|
|
|
_ = sender_future => {}
|
|
|
|
_ = receiver_future => {}
|
|
|
|
_ = sleep(Duration::from_secs(30)) => {
|
2021-11-09 16:42:02 +00:00
|
|
|
panic!("Future timed out");
|
Add snappy compression support (#866)
* notes from call
* should_forward function
* should_forward_block cleanup
* some cleanup and notes
* complete draft
* corrections
* some cleanup
* ran cargo fmt
* Revert "ran cargo fmt"
This reverts commit 464a5c4b62401c68fff7d36d3cd8069560a57713.
* ran cargo fmt after updating
* proposer index mods
* fmt
* new way of reading state
* fmt
* fmt
* compiles again
* fmt
* Correct stream timeout delay queue handling
* Correct small typo
* Support BlocksByRange step parameter
* Add initial docs to syncing
* Initial start of naive-attestation-aggregation
* Remove ping protocol
* Initial renaming of network services
* Correct rebasing relative to latest master
* Adds HashMapDelay struct to utils
* Initial network restructure
* Network restructure. Adds new types for v0.2.0
* Removes build artefacts
* Shift validation to beacon chain
* Temporarily remove gossip validation
This is to be updated to match current optimisation efforts.
* Adds AggregateAndProof
* Begin rebuilding pubsub encoding/decoding
* Temp commit
* Shift gossipsup decoding in eth2_libp2p
* Shifts block encoding/decoding into RPC
* Progress on attestation service
* Initial work on removing libp2p lock
* Add LRU caches to store (rollup)
* Update attestation validation for DB changes (WIP)
* Initial version of should_forward_block
* Scaffold
* Progress on attestation validation
Also, consolidate prod+testing slot clocks so that they share much
of the same implementation and can both handle sub-slot time changes.
* Removes lock from libp2p service
* Completed network lock removal
* Finish(?) attestation processing
* Correct network termination future
* Add slot check to block check
* Correct fmt issues
* Remove Drop implementation for network service
* Address reviewers suggestions
* Modification of validator for subscriptions
* Add slot signing to validator client
* Further progress on validation subscription
* Register SSZ snappy protocol messages
* Add initial idea of snappy compressed Codec by replacing the UVI codec with the snap library
* Fix matching against protocol string
* Adds TODOs for implementation work
* Implements check against max packet size before attempting to decode snappy compressed packages
* Add ssz_snappy codec
* Adds necessary validator subscription functionality
* Progress on snappy codec
* Clean up validator <-> beacon node http types
* Add aggregator status to ValidatorDuty
* clear buffer after decoding
* Impl Clone for manual slot clock
* Fix minor errors
* Further progress validator client subscription
* Initial subscription and aggregation handling
* Progress to modifying val client for attestation aggregation
* First draft of validator client upgrade for aggregate attestations
* Trying something
* Length prefix compressed data
* Fix gossipsub tests
* Working snappy frames with compressed length prefix
* Removes lock on a network channel
* Partially implement beacon node subscription http api
* Uncompressed length prefix working
* Cleanup
* Remove Testing request and response
* Return codec from match statement; reduce code duplication
* Fix unsafe unwrap in Outbound decode
* Add length checks
* All encode/decode functions use snappy frame format
* Add a `full` BeaconBlock method
* Add executable to test rpc against other impls
* Remove unused code; minor fixes
* Add PH & MS slot clock changes
* Account for genesis time
* Use checked mul
* Account for genesis slot
* Change API
* Refactor "duration to..." functions
* Re-merge updated block processing to v0.2.0 (#962)
* Start updating types
* WIP
* Signature hacking
* Existing EF tests passing with fake_crypto
* Updates
* Delete outdated API spec
* The refactor continues
* It compiles
* WIP test fixes
* All release tests passing bar genesis state parsing
* Update and test YamlConfig
* Update to spec v0.10 compatible BLS
* Updates to BLS EF tests
* Add EF test for AggregateVerify
And delete unused hash2curve tests for uncompressed points
* Update EF tests to v0.10.1
* Use optional block root correctly in block proc
* Use genesis fork in deposit domain. All tests pass
* Cargo fmt
* Fast aggregate verify test
* Update REST API docs
* Cargo fmt
* Fix unused import
* Bump spec tags to v0.10.1
* Add `seconds_per_eth1_block` to chainspec
* Update to timestamp based eth1 voting scheme
* Return None from `get_votes_to_consider` if block cache is empty
* Handle overflows in `is_candidate_block`
* Revert to failing tests
* Fix eth1 data sets test
* Choose default vote according to spec
* Fix collect_valid_votes tests
* Fix `get_votes_to_consider` to choose all eligible blocks
* Uncomment winning_vote tests
* Add comments; remove unused code
* Reduce seconds_per_eth1_block for simulation
* Addressed review comments
* Add test for default vote case
* Fix logs
* Remove unused functions
* Meter default eth1 votes
* Fix comments
* Address review comments; remove unused dependency
* Add first attempt at attestation proc. re-write
* Add version 2 of attestation processing
* Minor fixes
* Add validator pubkey cache
* Make get_indexed_attestation take a committee
* Link signature processing into new attn verification
* First working version
* Ensure pubkey cache is updated
* Add more metrics, slight optimizations
* Clone committee cache during attestation processing
* Update shuffling cache during block processing
* Remove old commented-out code
* Fix shuffling cache insert bug
* Used indexed attestation in fork choice
* Restructure attn processing, add metrics
* Add more detailed metrics
* Tidy, fix failing tests
* Fix failing tests, tidy
* Disable/delete two outdated tests
* Add new Pubkeys struct to signature_sets
* Refactor with functional approach
* Update beacon chain
* Remove decompressed member from pubkey bytes
* Add hashmap for indices lookup
* Add state cache, remove store cache
* Only build the head committee cache
* Change `get_attesting_indices` to use Vec
* Fix failing test
* Tidy
* Add pubkey cache persistence file
* Add more comments
* Integrate persistence file into builder
* Add pubkey cache tests
* Add data_dir to beacon chain builder
* Remove Option in pubkey cache persistence file
* Ensure consistency between datadir/data_dir
* Fix failing network test
* Tidy
* Fix todos
* Improve tests
* Fix compile error
* Fix compile error from merge
* Split up block processing metrics
* Tidy
* Refactor get_pubkey_from_state
* Remove commented-out code
* Rename state_cache -> checkpoint_cache
* Rename Checkpoint -> Snapshot
* Tidy, add comments
* Tidy up find_head function
* Change some checkpoint -> snapshot
* Add tests
* Expose max_len
* Remove dead code
* Tidy
* Fix bug
* Add sync-speed metric
* Add first attempt at VerifiableBlock
* Start integrating into beacon chain
* Integrate VerifiableBlock
* Rename VerifableBlock -> PartialBlockVerification
* Add start of typed methods
* Add progress
* Add further progress
* Rename structs
* Add full block verification to block_processing.rs
* Further beacon chain integration
* Update checks for gossip
* Add todo
* Start adding segement verification
* Add passing chain segement test
* Initial integration with batch sync
* Minor changes
* Tidy, add more error checking
* Start adding chain_segment tests
* Finish invalid signature tests
* Include single and gossip verified blocks in tests
* Add gossip verification tests
* Start adding docs
* Finish adding comments to block_processing.rs
* Rename block_processing.rs -> block_verification
* Start removing old block processing code
* Fixes beacon_chain compilation
* Fix project-wide compile errors
* Remove old code
* Fix bug with beacon proposer index
* Fix shim for BlockProcessingError
* Only process one epoch at a time
* Fix loop in chain segment processing
* Add caching for state.eth1_data_votes
* Add BeaconChain::validator_pubkey
* Revert "Add caching for state.eth1_data_votes"
This reverts commit cd73dcd6434fb8d8e6bf30c5356355598ea7b78e.
* Add sync-speed metric (#898)
* Add PH & MS slot clock changes
* Account for genesis time
* Use checked mul
* Account for genesis slot
* Change API
* Allow for clock disparity
* Refactor "duration to..." functions
* Ensure errors are returned during batch processing
Co-authored-by: Michael Sproul <michael@sigmaprime.io>
Co-authored-by: Michael Sproul <micsproul@gmail.com>
Co-authored-by: pawan <pawandhananjay@gmail.com>
Co-authored-by: Paul Hauner <paul@paulhauner.com>
* Enr fork (#967)
* Start fixing enr-fork-id
* Fix time-until-next-fork logic
* Remove fork crate
* Update any references to beacon_chain.spec.milliseconds_per_slot to beacon_chain.slot_clock.slot_diration().as_nillis() in the attestation service. (#968)
* Process network attestations (#966)
* Start updating types
* WIP
* Signature hacking
* Existing EF tests passing with fake_crypto
* Updates
* Delete outdated API spec
* The refactor continues
* It compiles
* WIP test fixes
* All release tests passing bar genesis state parsing
* Update and test YamlConfig
* Update to spec v0.10 compatible BLS
* Updates to BLS EF tests
* Add EF test for AggregateVerify
And delete unused hash2curve tests for uncompressed points
* Update EF tests to v0.10.1
* Use optional block root correctly in block proc
* Use genesis fork in deposit domain. All tests pass
* Cargo fmt
* Fast aggregate verify test
* Update REST API docs
* Cargo fmt
* Fix unused import
* Bump spec tags to v0.10.1
* Add `seconds_per_eth1_block` to chainspec
* Update to timestamp based eth1 voting scheme
* Return None from `get_votes_to_consider` if block cache is empty
* Handle overflows in `is_candidate_block`
* Revert to failing tests
* Fix eth1 data sets test
* Choose default vote according to spec
* Fix collect_valid_votes tests
* Fix `get_votes_to_consider` to choose all eligible blocks
* Uncomment winning_vote tests
* Add comments; remove unused code
* Reduce seconds_per_eth1_block for simulation
* Addressed review comments
* Add test for default vote case
* Fix logs
* Remove unused functions
* Meter default eth1 votes
* Fix comments
* Address review comments; remove unused dependency
* Add first attempt at attestation proc. re-write
* Add version 2 of attestation processing
* Minor fixes
* Add validator pubkey cache
* Make get_indexed_attestation take a committee
* Link signature processing into new attn verification
* First working version
* Ensure pubkey cache is updated
* Add more metrics, slight optimizations
* Clone committee cache during attestation processing
* Update shuffling cache during block processing
* Remove old commented-out code
* Fix shuffling cache insert bug
* Used indexed attestation in fork choice
* Restructure attn processing, add metrics
* Add more detailed metrics
* Tidy, fix failing tests
* Fix failing tests, tidy
* Disable/delete two outdated tests
* Add new Pubkeys struct to signature_sets
* Refactor with functional approach
* Update beacon chain
* Remove decompressed member from pubkey bytes
* Add hashmap for indices lookup
* Add state cache, remove store cache
* Only build the head committee cache
* Change `get_attesting_indices` to use Vec
* Fix failing test
* Tidy
* Add pubkey cache persistence file
* Add more comments
* Integrate persistence file into builder
* Add pubkey cache tests
* Add data_dir to beacon chain builder
* Remove Option in pubkey cache persistence file
* Ensure consistency between datadir/data_dir
* Fix failing network test
* Tidy
* Fix todos
* Improve tests
* Fix compile error
* Fix compile error from merge
* Split up block processing metrics
* Tidy
* Refactor get_pubkey_from_state
* Remove commented-out code
* Rename state_cache -> checkpoint_cache
* Rename Checkpoint -> Snapshot
* Tidy, add comments
* Tidy up find_head function
* Change some checkpoint -> snapshot
* Add tests
* Expose max_len
* Remove dead code
* Tidy
* Fix bug
* Add sync-speed metric
* Add first attempt at VerifiableBlock
* Start integrating into beacon chain
* Integrate VerifiableBlock
* Rename VerifableBlock -> PartialBlockVerification
* Add start of typed methods
* Add progress
* Add further progress
* Rename structs
* Add full block verification to block_processing.rs
* Further beacon chain integration
* Update checks for gossip
* Add todo
* Start adding segement verification
* Add passing chain segement test
* Initial integration with batch sync
* Minor changes
* Tidy, add more error checking
* Start adding chain_segment tests
* Finish invalid signature tests
* Include single and gossip verified blocks in tests
* Add gossip verification tests
* Start adding docs
* Finish adding comments to block_processing.rs
* Rename block_processing.rs -> block_verification
* Start removing old block processing code
* Fixes beacon_chain compilation
* Fix project-wide compile errors
* Remove old code
* Fix bug with beacon proposer index
* Fix shim for BlockProcessingError
* Only process one epoch at a time
* Fix loop in chain segment processing
* Add caching for state.eth1_data_votes
* Add BeaconChain::validator_pubkey
* Revert "Add caching for state.eth1_data_votes"
This reverts commit cd73dcd6434fb8d8e6bf30c5356355598ea7b78e.
* Allow for clock disparity
* Ensure errors are returned during batch processing
* Add block gossip verification
* Connect attestation processing to beacon chain
* Optimistically subscribe to subnets on the same slot
Co-authored-by: Michael Sproul <michael@sigmaprime.io>
Co-authored-by: Michael Sproul <micsproul@gmail.com>
Co-authored-by: pawan <pawandhananjay@gmail.com>
Co-authored-by: Paul Hauner <paul@paulhauner.com>
* Update /validator/subscribe (#969)
* Add progress on duties refactor
* Add simple is_aggregator bool to val subscription
* Remove unused function
* Upgrade sim (#972)
* Add progress on duties refactor
* Add simple is_aggregator bool to val subscription
* Add the no-eth1-sim, refactor sim
* Sends discovery for persistent subnets (#973)
* main takes cmdline arguments
* Add test script
* Fix errors
* snappy uses Uvi to encode/decode length prefix
* Add more comments
* Run fmt
Co-authored-by: Grant Wuerker <gwuerker@gmail.com>
Co-authored-by: Age Manning <Age@AgeManning.com>
Co-authored-by: Michael Sproul <micsproul@gmail.com>
Co-authored-by: Michael Sproul <michael@sigmaprime.io>
Co-authored-by: b-m-f <max@ehlers.berlin>
Co-authored-by: Paul Hauner <paul@paulhauner.com>
Co-authored-by: realbigsean <seananderson33@GMAIL.com>
2020-04-16 01:42:51 +00:00
|
|
|
}
|
|
|
|
}
|
2020-11-28 05:30:57 +00:00
|
|
|
})
|
2019-11-27 01:47:46 +00:00
|
|
|
}
|
|
|
|
|
2020-05-18 11:41:01 +00:00
|
|
|
// Tests a streamed, chunked BlocksByRoot RPC Message terminates when all expected reponses have been received
|
2020-11-28 05:30:57 +00:00
|
|
|
#[test]
|
|
|
|
fn test_blocks_by_root_chunked_rpc_terminates_correctly() {
|
2020-05-18 11:41:01 +00:00
|
|
|
// set up the logging. The level and enabled logging or not
|
|
|
|
let log_level = Level::Debug;
|
|
|
|
let enable_logging = false;
|
|
|
|
|
|
|
|
let messages_to_send: u64 = 10;
|
|
|
|
let extra_messages_to_send: u64 = 10;
|
|
|
|
|
|
|
|
let log = common::build_log(log_level, enable_logging);
|
|
|
|
let spec = E::default_spec();
|
|
|
|
|
2020-11-28 05:30:57 +00:00
|
|
|
let rt = Arc::new(Runtime::new().unwrap());
|
2020-05-18 11:41:01 +00:00
|
|
|
// get sender/receiver
|
2020-11-28 05:30:57 +00:00
|
|
|
rt.block_on(async {
|
|
|
|
let (mut sender, mut receiver) = common::build_node_pair(Arc::downgrade(&rt), &log).await;
|
|
|
|
|
|
|
|
// BlocksByRoot Request
|
|
|
|
let rpc_request = Request::BlocksByRoot(BlocksByRootRequest {
|
|
|
|
block_roots: VariableList::from(vec![
|
|
|
|
Hash256::from_low_u64_be(0),
|
|
|
|
Hash256::from_low_u64_be(0),
|
|
|
|
Hash256::from_low_u64_be(0),
|
|
|
|
Hash256::from_low_u64_be(0),
|
|
|
|
Hash256::from_low_u64_be(0),
|
|
|
|
Hash256::from_low_u64_be(0),
|
|
|
|
Hash256::from_low_u64_be(0),
|
|
|
|
Hash256::from_low_u64_be(0),
|
|
|
|
Hash256::from_low_u64_be(0),
|
|
|
|
Hash256::from_low_u64_be(0),
|
|
|
|
]),
|
|
|
|
});
|
|
|
|
|
|
|
|
// BlocksByRoot Response
|
2021-08-04 01:44:57 +00:00
|
|
|
let full_block = BeaconBlock::Base(BeaconBlockBase::<E>::full(&spec));
|
2021-07-09 06:15:32 +00:00
|
|
|
let signed_full_block = SignedBeaconBlock::from_block(full_block, Signature::empty());
|
2020-11-28 05:30:57 +00:00
|
|
|
let rpc_response = Response::BlocksByRoot(Some(Box::new(signed_full_block)));
|
|
|
|
|
|
|
|
// keep count of the number of messages received
|
|
|
|
let mut messages_received = 0;
|
|
|
|
// build the sender future
|
|
|
|
let sender_future = async {
|
|
|
|
loop {
|
|
|
|
match sender.next_event().await {
|
2021-07-13 00:48:33 +00:00
|
|
|
Libp2pEvent::Behaviour(BehaviourEvent::PeerConnectedOutgoing(peer_id)) => {
|
2020-11-28 05:30:57 +00:00
|
|
|
// Send a STATUS message
|
|
|
|
debug!(log, "Sending RPC");
|
2021-06-03 01:11:33 +00:00
|
|
|
sender.swarm.behaviour_mut().send_request(
|
2020-11-28 05:30:57 +00:00
|
|
|
peer_id,
|
|
|
|
RequestId::Sync(10),
|
|
|
|
rpc_request.clone(),
|
|
|
|
);
|
|
|
|
}
|
|
|
|
Libp2pEvent::Behaviour(BehaviourEvent::ResponseReceived {
|
|
|
|
peer_id: _,
|
|
|
|
id: RequestId::Sync(10),
|
|
|
|
response,
|
|
|
|
}) => {
|
|
|
|
debug!(log, "Sender received a response");
|
|
|
|
match response {
|
|
|
|
Response::BlocksByRoot(Some(_)) => {
|
|
|
|
assert_eq!(response, rpc_response.clone());
|
|
|
|
messages_received += 1;
|
|
|
|
debug!(log, "Chunk received");
|
|
|
|
}
|
|
|
|
Response::BlocksByRoot(None) => {
|
|
|
|
// should be exactly messages_to_send
|
|
|
|
assert_eq!(messages_received, messages_to_send);
|
|
|
|
// end the test
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
_ => {} // Ignore other RPC messages
|
2020-05-18 11:41:01 +00:00
|
|
|
}
|
|
|
|
}
|
2020-11-28 05:30:57 +00:00
|
|
|
_ => {} // Ignore other behaviour events
|
2020-06-05 03:07:59 +00:00
|
|
|
}
|
2020-05-18 11:41:01 +00:00
|
|
|
}
|
2020-11-28 05:30:57 +00:00
|
|
|
};
|
|
|
|
|
|
|
|
// determine messages to send (PeerId, RequestId). If some, indicates we still need to send
|
|
|
|
// messages
|
|
|
|
let mut message_info = None;
|
|
|
|
// the number of messages we've sent
|
|
|
|
let mut messages_sent = 0;
|
|
|
|
let receiver_future = async {
|
|
|
|
loop {
|
|
|
|
// this future either drives the sending/receiving or times out allowing messages to be
|
|
|
|
// sent in the timeout
|
|
|
|
match futures::future::select(
|
|
|
|
Box::pin(receiver.next_event()),
|
2021-02-10 23:29:49 +00:00
|
|
|
Box::pin(tokio::time::sleep(Duration::from_secs(1))),
|
2020-11-28 05:30:57 +00:00
|
|
|
)
|
|
|
|
.await
|
|
|
|
{
|
|
|
|
futures::future::Either::Left((
|
|
|
|
Libp2pEvent::Behaviour(BehaviourEvent::RequestReceived {
|
|
|
|
peer_id,
|
|
|
|
id,
|
|
|
|
request,
|
|
|
|
}),
|
|
|
|
_,
|
|
|
|
)) => {
|
|
|
|
if request == rpc_request {
|
|
|
|
// send the response
|
|
|
|
warn!(log, "Receiver got request");
|
|
|
|
message_info = Some((peer_id, id));
|
|
|
|
}
|
2020-05-18 11:41:01 +00:00
|
|
|
}
|
2020-11-28 05:30:57 +00:00
|
|
|
futures::future::Either::Right((_, _)) => {} // The timeout hit, send messages if required
|
|
|
|
_ => continue,
|
2020-05-18 11:41:01 +00:00
|
|
|
}
|
|
|
|
|
2020-11-28 05:30:57 +00:00
|
|
|
// if we need to send messages send them here. This will happen after a delay
|
|
|
|
if message_info.is_some() {
|
|
|
|
messages_sent += 1;
|
|
|
|
let (peer_id, stream_id) = message_info.as_ref().unwrap();
|
2021-06-03 01:11:33 +00:00
|
|
|
receiver.swarm.behaviour_mut().send_successful_response(
|
2021-01-28 23:31:06 +00:00
|
|
|
*peer_id,
|
|
|
|
*stream_id,
|
2020-11-28 05:30:57 +00:00
|
|
|
rpc_response.clone(),
|
|
|
|
);
|
|
|
|
debug!(log, "Sending message {}", messages_sent);
|
|
|
|
if messages_sent == messages_to_send + extra_messages_to_send {
|
|
|
|
// stop sending messages
|
|
|
|
return;
|
|
|
|
}
|
2020-05-18 11:41:01 +00:00
|
|
|
}
|
|
|
|
}
|
2020-11-28 05:30:57 +00:00
|
|
|
};
|
2020-05-18 11:41:01 +00:00
|
|
|
|
2020-11-28 05:30:57 +00:00
|
|
|
tokio::select! {
|
|
|
|
_ = sender_future => {}
|
|
|
|
_ = receiver_future => {}
|
|
|
|
_ = sleep(Duration::from_secs(30)) => {
|
|
|
|
panic!("Future timed out");
|
|
|
|
}
|
2020-05-18 11:41:01 +00:00
|
|
|
}
|
2020-11-28 05:30:57 +00:00
|
|
|
})
|
2020-05-18 11:41:01 +00:00
|
|
|
}
|
|
|
|
|
2019-11-27 01:47:46 +00:00
|
|
|
// Tests a Goodbye RPC message
|
2020-11-28 05:30:57 +00:00
|
|
|
#[test]
|
2021-01-28 23:31:06 +00:00
|
|
|
#[allow(clippy::single_match)]
|
2020-11-28 05:30:57 +00:00
|
|
|
fn test_goodbye_rpc() {
|
2019-11-27 01:47:46 +00:00
|
|
|
// set up the logging. The level and enabled logging or not
|
|
|
|
let log_level = Level::Trace;
|
|
|
|
let enable_logging = false;
|
|
|
|
|
|
|
|
let log = common::build_log(log_level, enable_logging);
|
|
|
|
|
2020-11-28 05:30:57 +00:00
|
|
|
let rt = Arc::new(Runtime::new().unwrap());
|
2019-11-27 01:47:46 +00:00
|
|
|
// get sender/receiver
|
2020-11-28 05:30:57 +00:00
|
|
|
rt.block_on(async {
|
|
|
|
let (mut sender, mut receiver) = common::build_node_pair(Arc::downgrade(&rt), &log).await;
|
|
|
|
|
|
|
|
// build the sender future
|
|
|
|
let sender_future = async {
|
|
|
|
loop {
|
|
|
|
match sender.next_event().await {
|
2021-07-13 00:48:33 +00:00
|
|
|
Libp2pEvent::Behaviour(BehaviourEvent::PeerConnectedOutgoing(peer_id)) => {
|
2020-11-28 05:30:57 +00:00
|
|
|
// Send a goodbye and disconnect
|
|
|
|
debug!(log, "Sending RPC");
|
2021-06-03 01:11:33 +00:00
|
|
|
sender.swarm.behaviour_mut().goodbye_peer(
|
2020-12-08 03:55:50 +00:00
|
|
|
&peer_id,
|
|
|
|
GoodbyeReason::IrrelevantNetwork,
|
|
|
|
ReportSource::SyncService,
|
|
|
|
);
|
2020-11-28 05:30:57 +00:00
|
|
|
}
|
|
|
|
Libp2pEvent::Behaviour(BehaviourEvent::PeerDisconnected(_)) => {
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
_ => {} // Ignore other RPC messages
|
2019-11-27 01:47:46 +00:00
|
|
|
}
|
2020-05-17 11:16:48 +00:00
|
|
|
}
|
2020-11-28 05:30:57 +00:00
|
|
|
};
|
|
|
|
|
|
|
|
// build the receiver future
|
|
|
|
let receiver_future = async {
|
|
|
|
loop {
|
|
|
|
match receiver.next_event().await {
|
|
|
|
Libp2pEvent::Behaviour(BehaviourEvent::PeerDisconnected(_)) => {
|
|
|
|
// Should receive sent RPC request
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
_ => {} // Ignore other events
|
2020-05-17 11:16:48 +00:00
|
|
|
}
|
2019-11-27 01:47:46 +00:00
|
|
|
}
|
2020-11-28 05:30:57 +00:00
|
|
|
};
|
2019-11-27 01:47:46 +00:00
|
|
|
|
2020-11-28 05:30:57 +00:00
|
|
|
let total_future = futures::future::join(sender_future, receiver_future);
|
2020-07-07 00:13:16 +00:00
|
|
|
|
2020-11-28 05:30:57 +00:00
|
|
|
tokio::select! {
|
|
|
|
_ = total_future => {}
|
|
|
|
_ = sleep(Duration::from_secs(30)) => {
|
|
|
|
panic!("Future timed out");
|
|
|
|
}
|
2020-05-17 11:16:48 +00:00
|
|
|
}
|
2020-11-28 05:30:57 +00:00
|
|
|
})
|
2019-11-27 01:47:46 +00:00
|
|
|
}
|