cerc-io/lighthouse
11
0
Fork 0
Code Issues 1 Pull Requests Packages Projects Releases Wiki Activity

Initial work towards v0.2.0 (#924)

Browse Source 
* Remove ping protocol

* Initial renaming of network services

* Correct rebasing relative to latest master

* Start updating types

* Adds HashMapDelay struct to utils

* Initial network restructure

* Network restructure. Adds new types for v0.2.0

* Removes build artefacts

* Shift validation to beacon chain

* Temporarily remove gossip validation

This is to be updated to match current optimisation efforts.

* Adds AggregateAndProof

* Begin rebuilding pubsub encoding/decoding

* Signature hacking

* Shift gossipsup decoding into eth2_libp2p

* Existing EF tests passing with fake_crypto

* Shifts block encoding/decoding into RPC

* Delete outdated API spec

* All release tests passing bar genesis state parsing

* Update and test YamlConfig

* Update to spec v0.10 compatible BLS

* Updates to BLS EF tests

* Add EF test for AggregateVerify

And delete unused hash2curve tests for uncompressed points

* Update EF tests to v0.10.1

* Use optional block root correctly in block proc

* Use genesis fork in deposit domain. All tests pass

* Fast aggregate verify test

* Update REST API docs

* Fix unused import

* Bump spec tags to v0.10.1

* Add `seconds_per_eth1_block` to chainspec

* Update to timestamp based eth1 voting scheme

* Return None from `get_votes_to_consider` if block cache is empty

* Handle overflows in `is_candidate_block`

* Revert to failing tests

* Fix eth1 data sets test

* Choose default vote according to spec

* Fix collect_valid_votes tests

* Fix `get_votes_to_consider` to choose all eligible blocks

* Uncomment winning_vote tests

* Add comments; remove unused code

* Reduce seconds_per_eth1_block for simulation

* Addressed review comments

* Add test for default vote case

* Fix logs

* Remove unused functions

* Meter default eth1 votes

* Fix comments

* Progress on attestation service

* Address review comments; remove unused dependency

* Initial work on removing libp2p lock

* Add LRU caches to store (rollup)

* Update attestation validation for DB changes (WIP)

* Initial version of should_forward_block

* Scaffold

* Progress on attestation validation

Also, consolidate prod+testing slot clocks so that they share much
of the same implementation and can both handle sub-slot time changes.

* Removes lock from libp2p service

* Completed network lock removal

* Finish(?) attestation processing

* Correct network termination future

* Add slot check to block check

* Correct fmt issues

* Remove Drop implementation for network service

* Add first attempt at attestation proc. re-write

* Add version 2 of attestation processing

* Minor fixes

* Add validator pubkey cache

* Make get_indexed_attestation take a committee

* Link signature processing into new attn verification

* First working version

* Ensure pubkey cache is updated

* Add more metrics, slight optimizations

* Clone committee cache during attestation processing

* Update shuffling cache during block processing

* Remove old commented-out code

* Fix shuffling cache insert bug

* Used indexed attestation in fork choice

* Restructure attn processing, add metrics

* Add more detailed metrics

* Tidy, fix failing tests

* Fix failing tests, tidy

* Address reviewers suggestions

* Disable/delete two outdated tests

* Modification of validator for subscriptions

* Add slot signing to validator client

* Further progress on validation subscription

* Adds necessary validator subscription functionality

* Add new Pubkeys struct to signature_sets

* Refactor with functional approach

* Update beacon chain

* Clean up validator <-> beacon node http types

* Add aggregator status to ValidatorDuty

* Impl Clone for manual slot clock

* Fix minor errors

* Further progress validator client subscription

* Initial subscription and aggregation handling

* Remove decompressed member from pubkey bytes

* Progress to modifying val client for attestation aggregation

* First draft of validator client upgrade for aggregate attestations

* Add hashmap for indices lookup

* Add state cache, remove store cache

* Only build the head committee cache

* Removes lock on a network channel

* Partially implement beacon node subscription http api

* Correct compilation issues

* Change `get_attesting_indices` to use Vec

* Fix failing test

* Partial implementation of timer

* Adds timer, removes exit_future, http api to op pool

* Partial multiple aggregate attestation handling

* Permits bulk messages accross gossipsub network channel

* Correct compile issues

* Improve gosispsub messaging and correct rest api helpers

* Added global gossipsub subscriptions

* Update validator subscriptions data structs

* Tidy

* Re-structure validator subscriptions

* Initial handling of subscriptions

* Re-structure network service

* Add pubkey cache persistence file

* Add more comments

* Integrate persistence file into builder

* Add pubkey cache tests

* Add HashSetDelay and introduce into attestation service

* Handles validator subscriptions

* Add data_dir to beacon chain builder

* Remove Option in pubkey cache persistence file

* Ensure consistency between datadir/data_dir

* Fix failing network test

* Peer subnet discovery gets queued for future subscriptions

* Reorganise attestation service functions

* Initial wiring of attestation service

* First draft of attestation service timing logic

* Correct minor typos

* Tidy

* Fix todos

* Improve tests

* Add PeerInfo to connected peers mapping

* Fix compile error

* Fix compile error from merge

* Split up block processing metrics

* Tidy

* Refactor get_pubkey_from_state

* Remove commented-out code

* Rename state_cache -> checkpoint_cache

* Rename Checkpoint -> Snapshot

* Tidy, add comments

* Tidy up find_head function

* Change some checkpoint -> snapshot

* Add tests

* Expose max_len

* Remove dead code

* Tidy

* Fix bug

* Add sync-speed metric

* Add first attempt at VerifiableBlock

* Start integrating into beacon chain

* Integrate VerifiableBlock

* Rename VerifableBlock -> PartialBlockVerification

* Add start of typed methods

* Add progress

* Add further progress

* Rename structs

* Add full block verification to block_processing.rs

* Further beacon chain integration

* Update checks for gossip

* Add todo

* Start adding segement verification

* Add passing chain segement test

* Initial integration with batch sync

* Minor changes

* Tidy, add more error checking

* Start adding chain_segment tests

* Finish invalid signature tests

* Include single and gossip verified blocks in tests

* Add gossip verification tests

* Start adding docs

* Finish adding comments to block_processing.rs

* Rename block_processing.rs -> block_verification

* Start removing old block processing code

* Fixes beacon_chain compilation

* Fix project-wide compile errors

* Remove old code

* Correct code to pass all tests

* Fix bug with beacon proposer index

* Fix shim for BlockProcessingError

* Only process one epoch at a time

* Fix loop in chain segment processing

* Correct tests from master merge

* Add caching for state.eth1_data_votes

* Add BeaconChain::validator_pubkey

* Revert "Add caching for state.eth1_data_votes"

This reverts commit cd73dcd6434fb8d8e6bf30c5356355598ea7b78e.

Co-authored-by: Grant Wuerker <gwuerker@gmail.com>
Co-authored-by: Michael Sproul <michael@sigmaprime.io>
Co-authored-by: Michael Sproul <micsproul@gmail.com>
Co-authored-by: pawan <pawandhananjay@gmail.com>
Co-authored-by: Paul Hauner <paul@paulhauner.com>
This commit is contained in:
Age Manning
2020-03-17 17:24:44 +11:00
committed by GitHub
co-authored by Grant Wuerker Michael Sproul Michael Sproul pawan Paul Hauner
parent c198bddf9e
commit 95c8e476bc
161 changed files with 9771 additions and 5266 deletions
Download Patch File Download Diff File Expand all files Collapse all files
beacon_node/eth2-libp2p/tests/common/mod.rs
+17 -7
View File
@@ -5,6 +5,9 @@ use eth2_libp2p::NetworkConfig;
use eth2_libp2p::Service as LibP2PService;
use slog::{debug, error, o, Drain};
use std::time::Duration;
use types::MinimalEthSpec;
type E = MinimalEthSpec;
use tempdir::TempDir;
pub fn build_log(level: slog::Level, enabled: bool) -> slog::Logger {
@@ -43,7 +46,7 @@ pub fn build_libp2p_instance(
boot_nodes: Vec<Enr>,
secret_key: Option<String>,
log: slog::Logger,
) -> LibP2PService {
) -> LibP2PService<E> {
let config = build_config(port, boot_nodes, secret_key);
// launch libp2p service
LibP2PService::new(&config, log.clone())
@@ -52,15 +55,19 @@ pub fn build_libp2p_instance(
}
#[allow(dead_code)]
pub fn get_enr(node: &LibP2PService) -> Enr {
pub fn get_enr(node: &LibP2PService<E>) -> Enr {
node.swarm.discovery().local_enr().clone()
}
// Returns `n` libp2p peers in fully connected topology.
#[allow(dead_code)]
pub fn build_full_mesh(log: slog::Logger, n: usize, start_port: Option<u16>) -> Vec<LibP2PService> {
pub fn build_full_mesh(
log: slog::Logger,
n: usize,
start_port: Option<u16>,
) -> Vec<LibP2PService<E>> {
let base_port = start_port.unwrap_or(9000);
let mut nodes: Vec<LibP2PService> = (base_port..base_port + n as u16)
let mut nodes: Vec<LibP2PService<E>> = (base_port..base_port + n as u16)
.map(|p| build_libp2p_instance(p, vec![], None, log.clone()))
.collect();
let multiaddrs: Vec<Multiaddr> = nodes
@@ -84,7 +91,10 @@ pub fn build_full_mesh(log: slog::Logger, n: usize, start_port: Option<u16>) ->
// Constructs a pair of nodes with seperate loggers. The sender dials the receiver.
// This returns a (sender, receiver) pair.
#[allow(dead_code)]
pub fn build_node_pair(log: &slog::Logger, start_port: u16) -> (LibP2PService, LibP2PService) {
pub fn build_node_pair(
log: &slog::Logger,
start_port: u16,
) -> (LibP2PService<E>, LibP2PService<E>) {
let sender_log = log.new(o!("who" => "sender"));
let receiver_log = log.new(o!("who" => "receiver"));
@@ -101,9 +111,9 @@ pub fn build_node_pair(log: &slog::Logger, start_port: u16) -> (LibP2PService, L
// Returns `n` peers in a linear topology
#[allow(dead_code)]
pub fn build_linear(log: slog::Logger, n: usize, start_port: Option<u16>) -> Vec<LibP2PService> {
pub fn build_linear(log: slog::Logger, n: usize, start_port: Option<u16>) -> Vec<LibP2PService<E>> {
let base_port = start_port.unwrap_or(9000);
let mut nodes: Vec<LibP2PService> = (base_port..base_port + n as u16)
let mut nodes: Vec<LibP2PService<E>> = (base_port..base_port + n as u16)
.map(|p| build_libp2p_instance(p, vec![], None, log.clone()))
.collect();
let multiaddrs: Vec<Multiaddr> = nodes
beacon_node/eth2-libp2p/tests/gossipsub_tests.rs
+22 -9
View File
@@ -1,8 +1,12 @@
#![cfg(test)]
use crate::types::GossipEncoding;
use ::types::{BeaconBlock, EthSpec, MinimalEthSpec, Signature, SignedBeaconBlock};
use eth2_libp2p::*;
use futures::prelude::*;
use slog::{debug, Level};
type E = MinimalEthSpec;
mod common;
/* Gossipsub tests */
@@ -23,7 +27,14 @@ fn test_gossipsub_forward() {
let num_nodes = 20;
let mut nodes = common::build_linear(log.clone(), num_nodes, Some(19000));
let mut received_count = 0;
let pubsub_message = PubsubMessage::Block(vec![0; 4]);
let spec = E::default_spec();
let empty_block = BeaconBlock::empty(&spec);
let signed_block = SignedBeaconBlock {
message: empty_block,
signature: Signature::empty_signature(),
};
let data = PubsubData::BeaconBlock(Box::new(signed_block));
let pubsub_message = PubsubMessage::new(GossipEncoding::SSZ, data);
let publishing_topic: String = "/eth2/beacon_block/ssz".into();
let mut subscribed_count = 0;
tokio::run(futures::future::poll_fn(move || -> Result<_, ()> {
@@ -61,10 +72,7 @@ fn test_gossipsub_forward() {
subscribed_count += 1;
// Every node except the corner nodes are connected to 2 nodes.
if subscribed_count == (num_nodes * 2) - 2 {
node.swarm.publish(
&[Topic::new(topic.into_string())],
pubsub_message.clone(),
);
node.swarm.publish(vec![pubsub_message.clone()]);
}
}
}
@@ -90,7 +98,14 @@ fn test_gossipsub_full_mesh_publish() {
let num_nodes = 12;
let mut nodes = common::build_full_mesh(log, num_nodes, Some(11320));
let mut publishing_node = nodes.pop().unwrap();
let pubsub_message = PubsubMessage::Block(vec![0; 4]);
let spec = E::default_spec();
let empty_block = BeaconBlock::empty(&spec);
let signed_block = SignedBeaconBlock {
message: empty_block,
signature: Signature::empty_signature(),
};
let data = PubsubData::BeaconBlock(Box::new(signed_block));
let pubsub_message = PubsubMessage::new(GossipEncoding::SSZ, data);
let publishing_topic: String = "/eth2/beacon_block/ssz".into();
let mut subscribed_count = 0;
let mut received_count = 0;
@@ -123,9 +138,7 @@ fn test_gossipsub_full_mesh_publish() {
if topic == TopicHash::from_raw("/eth2/beacon_block/ssz") {
subscribed_count += 1;
if subscribed_count == num_nodes - 1 {
publishing_node
.swarm
.publish(&[Topic::new(topic.into_string())], pubsub_message.clone());
publishing_node.swarm.publish(vec![pubsub_message.clone()]);
}
}
}
beacon_node/eth2-libp2p/tests/noise.rs
+9 -2
View File
@@ -1,6 +1,7 @@
#![cfg(test)]
use crate::behaviour::{Behaviour, BehaviourEvent};
use crate::multiaddr::Protocol;
use ::types::MinimalEthSpec;
use eth2_libp2p::*;
use futures::prelude::*;
use libp2p::core::identity::Keypair;
@@ -16,10 +17,12 @@ use std::sync::Arc;
use std::time::Duration;
use tokio::prelude::*;
type TSpec = MinimalEthSpec;
mod common;
type Libp2pStream = Boxed<(PeerId, StreamMuxerBox), Error>;
type Libp2pBehaviour = Behaviour<Substream<StreamMuxerBox>>;
type Libp2pBehaviour = Behaviour<Substream<StreamMuxerBox>, TSpec>;
/// Build and return a eth2_libp2p Swarm with only secio support.
fn build_secio_swarm(
@@ -29,7 +32,11 @@ fn build_secio_swarm(
let local_keypair = Keypair::generate_secp256k1();
let local_peer_id = PeerId::from(local_keypair.public());
let network_globals = Arc::new(NetworkGlobals::new(local_peer_id.clone()));
let network_globals = Arc::new(NetworkGlobals::new(
local_peer_id.clone(),
config.libp2p_port,
config.discovery_port,
));
let mut swarm = {
// Set up the transport - tcp/ws with secio and mplex/yamux
beacon_node/eth2-libp2p/tests/rpc_tests.rs
+21 -131
View File
@@ -7,10 +7,14 @@ use std::sync::atomic::{AtomicBool, Ordering::Relaxed};
use std::sync::{Arc, Mutex};
use std::time::Duration;
use tokio::prelude::*;
use types::{Epoch, Hash256, Slot};
use types::{
BeaconBlock, Epoch, EthSpec, Hash256, MinimalEthSpec, Signature, SignedBeaconBlock, Slot,
};
mod common;
type E = MinimalEthSpec;
#[test]
// Tests the STATUS RPC message
fn test_status_rpc() {
@@ -73,7 +77,7 @@ fn test_status_rpc() {
warn!(sender_log, "Sender Completed");
return Ok(Async::Ready(true));
}
_ => panic!("Received invalid RPC message"),
e => panic!("Received invalid RPC message {}", e),
},
Async::Ready(Some(_)) => (),
Async::Ready(None) | Async::NotReady => return Ok(Async::NotReady),
@@ -98,7 +102,7 @@ fn test_status_rpc() {
RPCEvent::Response(id, RPCErrorResponse::Success(rpc_response.clone())),
);
}
_ => panic!("Received invalid RPC message"),
e => panic!("Received invalid RPC message {}", e),
},
Async::Ready(Some(_)) => (),
Async::Ready(None) | Async::NotReady => return Ok(Async::NotReady),
@@ -145,7 +149,13 @@ fn test_blocks_by_range_chunked_rpc() {
});
// BlocksByRange Response
let rpc_response = RPCResponse::BlocksByRange(vec![13, 13, 13]);
let spec = E::default_spec();
let empty_block = BeaconBlock::empty(&spec);
let empty_signed = SignedBeaconBlock {
message: empty_block,
signature: Signature::empty_signature(),
};
let rpc_response = RPCResponse::BlocksByRange(Box::new(empty_signed));
let sender_request = rpc_request.clone();
let sender_log = log.clone();
@@ -272,7 +282,13 @@ fn test_blocks_by_range_single_empty_rpc() {
});
// BlocksByRange Response
let rpc_response = RPCResponse::BlocksByRange(vec![]);
let spec = E::default_spec();
let empty_block = BeaconBlock::empty(&spec);
let empty_signed = SignedBeaconBlock {
message: empty_block,
signature: Signature::empty_signature(),
};
let rpc_response = RPCResponse::BlocksByRange(Box::new(empty_signed));
let sender_request = rpc_request.clone();
let sender_log = log.clone();
@@ -373,132 +389,6 @@ fn test_blocks_by_range_single_empty_rpc() {
assert!(test_result.load(Relaxed));
}
#[test]
// Tests a streamed, chunked BlocksByRoot RPC Message
fn test_blocks_by_root_chunked_rpc() {
// set up the logging. The level and enabled logging or not
let log_level = Level::Trace;
let enable_logging = false;
let messages_to_send = 3;
let log = common::build_log(log_level, enable_logging);
// get sender/receiver
let (mut sender, mut receiver) = common::build_node_pair(&log, 10515);
// BlocksByRoot Request
let rpc_request = RPCRequest::BlocksByRoot(BlocksByRootRequest {
block_roots: vec![Hash256::from_low_u64_be(0), Hash256::from_low_u64_be(0)],
});
// BlocksByRoot Response
let rpc_response = RPCResponse::BlocksByRoot(vec![13, 13, 13]);
let sender_request = rpc_request.clone();
let sender_log = log.clone();
let sender_response = rpc_response.clone();
// keep count of the number of messages received
let messages_received = Arc::new(Mutex::new(0));
// build the sender future
let sender_future = future::poll_fn(move || -> Poll<bool, ()> {
loop {
match sender.poll().unwrap() {
Async::Ready(Some(Libp2pEvent::PeerDialed(peer_id))) => {
// Send a BlocksByRoot request
warn!(sender_log, "Sender sending RPC request");
sender
.swarm
.send_rpc(peer_id, RPCEvent::Request(1, sender_request.clone()));
}
Async::Ready(Some(Libp2pEvent::RPC(_, event))) => match event {
// Should receive the RPC response
RPCEvent::Response(id, response) => {
warn!(sender_log, "Sender received a response");
assert_eq!(id, 1);
match response {
RPCErrorResponse::Success(res) => {
assert_eq!(res, sender_response.clone());
*messages_received.lock().unwrap() += 1;
warn!(sender_log, "Chunk received");
}
RPCErrorResponse::StreamTermination(
ResponseTermination::BlocksByRoot,
) => {
// should be exactly 10 messages before terminating
assert_eq!(*messages_received.lock().unwrap(), messages_to_send);
// end the test
return Ok(Async::Ready(true));
}
m => panic!("Invalid RPC received: {}", m),
}
}
m => panic!("Received invalid RPC message: {}", m),
},
Async::Ready(Some(_)) => {}
Async::Ready(None) | Async::NotReady => return Ok(Async::NotReady),
};
}
});
// build the receiver future
let receiver_future = future::poll_fn(move || -> Poll<bool, ()> {
loop {
match receiver.poll().unwrap() {
Async::Ready(Some(Libp2pEvent::RPC(peer_id, event))) => match event {
// Should receive the sent RPC request
RPCEvent::Request(id, request) => {
assert_eq!(id, 1);
assert_eq!(rpc_request.clone(), request);
// send the response
warn!(log, "Receiver got request");
for _ in 1..=messages_to_send {
receiver.swarm.send_rpc(
peer_id.clone(),
RPCEvent::Response(
id,
RPCErrorResponse::Success(rpc_response.clone()),
),
);
}
// send the stream termination
receiver.swarm.send_rpc(
peer_id,
RPCEvent::Response(
id,
RPCErrorResponse::StreamTermination(
ResponseTermination::BlocksByRoot,
),
),
);
}
_ => panic!("Received invalid RPC message"),
},
Async::Ready(Some(_)) => (),
Async::Ready(None) | Async::NotReady => return Ok(Async::NotReady),
}
}
});
// execute the futures and check the result
let test_result = Arc::new(AtomicBool::new(false));
let error_result = test_result.clone();
let thread_result = test_result.clone();
tokio::run(
sender_future
.select(receiver_future)
.timeout(Duration::from_millis(1000))
.map_err(move |_| error_result.store(false, Relaxed))
.map(move |result| {
thread_result.store(result.0, Relaxed);
}),
);
assert!(test_result.load(Relaxed));
}
#[test]
// Tests a Goodbye RPC message
fn test_goodbye_rpc() {