From 4cb2efb0420918324d91681c157440010b4d4f95 Mon Sep 17 00:00:00 2001 From: Chris Natoli Date: Tue, 18 Dec 2018 11:39:53 +1100 Subject: [PATCH 01/10] Start #93; Update documentation. --- docs/onboarding.md | 292 +++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 271 insertions(+), 21 deletions(-) diff --git a/docs/onboarding.md b/docs/onboarding.md index 61ed1e07d..90b38df97 100644 --- a/docs/onboarding.md +++ b/docs/onboarding.md @@ -1,36 +1,286 @@ -# Learn how to contribute to ETH 2.0! +# Contributing to Lighthouse -Lighthouse is an Ethereum 2.0 client built in Rust. +Lighthouse is an open-source Ethereum Serenity client built in +[Rust](https://www.rust-lang.org/). -If you are interested in contributing to the Ethereum ecosystem, and you want to learn Rust, Lighthouse is a great project to work on. +Lighthouse welcomes all contributions with open arms. If you are interested in +contributing to the Ethereum ecosystem, and you want to learn Rust, Lighthouse +is a great project to work on. -Initially this doc will contain reading material to help get you started in Rust and Ethereum. Eventually it will have guides specific to Lighthouse. +This documentation aims to provide a smooth on-boarding for all who wish to +help contribute to Lighthouse. Whether it is helping with the mountain of +documentation, writing extra tests or developing components, all help is +appreciated and your contributions will help not only the community but all +the contributors. -## Learn Rust +If you have any additional questions, please feel free to jump on the +[gitter](https://gitter.im/sigp/lighthouse) and have a chat with all of us. -* [The Rust Programming Language](https://doc.rust-lang.org/book/2018-edition/index.html) -* [Learning Rust With Entirely Too Many Linked Lists](http://cglab.ca/~abeinges/blah/too-many-lists/book/) -* [Rustlings](https://github.com/rustlings/rustlings) - Small exercises to get you used to reading and writing Rust code. -* [Rust Exercism](https://exercism.io/tracks/rust) - 88 exercises to learn Rust. +## Ideology -## Learn Ethereum +### Never Panic + +Lighthouse will be the gateway interacting with the Proof-of-Stake system +employed by Ethereum. This requires the validation and proposal of blocks +and extremely timely responses. As part of this, Lighthouse aims to ensure +the most uptime as possible, meaning minimising the amount of +exceptions and gracefully handling any issues. + +Rust's `panic` provides the ability to throw an exception and exit, this +will terminate the running processes. Thus, Lighthouse aims to use `panic` +as little as possible to minimise the possible termination cases. + +### Security First Mindset + +Lighthouse aims to provide a safe, secure Serenity client for the Ethereum +ecosystem. At each step of development, the aim is to have a security-first +mindset and always ensure you are following the safe, secure mindset. When +contributing to any part of the Lighthouse client, through any development, +always ensure you understand each aspect thoroughly and cover all potential +security considerations of your code. + +### Functions aren't completed until they are tested + +As part of the Security First mindset, we want to aim to cover as many distinct +cases. A function being developed is not considered "completed" until tests +exist for that function. The tests not only help show the correctness of the +function, but also provide a way for new developers to understand how the +function is to be called and how it works. + +## Understanding Serenity + +Ethereum's Serenity is based on a Proof-of-Stake based sharded beacon chain. + +(*If you don't know what that is, don't `panic`, that's what this documentation +is for!* :smile:) + +### Ethereum + +Ethereum is an open blockchain protocol, allowing for the building and use of +decentralized applications that run on blockchain technology. The blockchain can +be seen as a decentralized, distributed ledger of transactions. + +General Ethereum Introduction: -#### General Ethereum Resources * [What is Ethereum](http://ethdocs.org/en/latest/introduction/what-is-ethereum.html) * [Ethereum Introduction](https://github.com/ethereum/wiki/wiki/Ethereum-introduction) -#### Ethereum 2.0 -* [Ethereum 2.0 Spec - Casper and Sharding](https://github.com/ethereum/eth2.0-specs/blob/master/specs/beacon-chain.md) -#### Sharding +### Proof-of-Work and the current state of Ethereum. -* [How to Scale Ethereum: Sharding Explained](https://medium.com/prysmatic-labs/how-to-scale-ethereum-sharding-explained-ba2e283b7fce) +Currently, Ethereum is based on the Proof-of-Work model, a Sybil resilient +mechanism to allow nodes to propose blocks to the network. Although it provides +properties that allow the blockchain to operate in an open, public +(permissionless) network, it faces it's challenges and as a result impacts +the operation of the blockchain. -#### Casper +The main goals to advance Ethereum is to (1) increase the scalability and +overall transaction processing power of the Ethereum world computer and (2) +find a suitable replacement for Proof-of-Work that still provides the necessary +properties that we need. -* [Proof of Stake - Casper FFG](https://www.youtube.com/watch?v=uQ3IqLDf-oo) -* [Beacon Casper Chain](https://www.youtube.com/watch?v=GAywmwGToUI) +* [Proof-of-Work in Cryptocurrencies: an accessible introduction](https://blog.sigmaprime.io/what-is-proof-of-work.html) -### TODO -- add reading material as we discover. -- start developing guides specific to lighthouse. +### Serenity + +As part of the original Ethereum roadmap +[\[1\]](https://blog.ethereum.org/2015/03/03/ethereum-launch-process/) +[\[2\]](http://ethdocs.org/en/latest/introduction/the-homestead-release.html), +the Proof-of-Stake integration falls under **Release Step 4:*Serenity***. With +this, a number of changes are to be made to the current Ethereum protocol to +incorporate some of the new Proof-of-Stake mechanisms as well as improve on +some of the hindrances faced by the current Proof-of-Work chain. + +To now advance the current Ethereum, the decision is made to move to a sharded +Beacon chain structure where multiple shard-chains will be operating and +interacting with a central beacon chain. + +(Be mindful, the specifications change occasionally, so check these to keep up +to date) + +* Current Specifications: + * [Danny Ryan's "State of the Spec"](https://notes.ethereum.org/s/BJEZWNoyE) (A nice summary of the current specifications) + * [Ethereum Serenity - Phase 0: Beacon Chain Spec](https://github.com/ethereum/eth2.0-specs/blob/master/specs/core/0_beacon-chain.md) + * [Ethereum Serenity - Phase 1: Sharded Data Chains](https://github.com/ethereum/eth2.0-specs/blob/master/specs/core/1_shard-data-chains.md) + * [Beacon Chain - Vitalik Buterin and Justin Drake explain](https://www.youtube.com/watch?v=GAywmwGToUI) +* Understanding Sharding: + * [Prysmatic Labs: Sharding Explained](https://medium.com/prysmatic-labs/how-to-scale-ethereum-sharding-explained-ba2e283b7fce) +* Other relevant resources + * [Proof of Stake - Casper FFG](https://www.youtube.com/watch?v=uQ3IqLDf-oo) + +## Development Onboarding + +If you would like to contribute and develop Lighthouse, there are only a few +things to go through (and then you're on your way!). + +### Understanding Rust + +Rust is an extremely powerful, low-level programming language that provides +freedom and performance to create powerful projects. The [Rust +Book](https://doc.rust-lang.org/stable/book/) provides insight into the Rust +language and some of the coding style to follow (As well as acting as a great +introduction and tutorial for the language.) + +Rust has a steep learning curve, but there are many resources to help you! + +* [Rust Book](https://doc.rust-lang.org/stable/book/) +* [Rust by example](https://doc.rust-lang.org/stable/rust-by-example/) +* [Learning Rust With Entirely Too Many Linked Lists](http://cglab.ca/~abeinges/blah/too-many-lists/book/) +* [Rustlings](https://github.com/rustlings/rustlings) +* [Rust Exercism](https://exercism.io/tracks/rust) +* [Learn X in Y minutes - Rust](https://learnxinyminutes.com/docs/rust/) + + +#### Getting Started and installing Rust + +We recommend installing Rust using [**rustup**](https://rustup.rs/). Rustup +allows you to easily install versions of rust. + +**Linux/Unix/Mac:** + +``` +$ curl https://sh.rustup.rs -sSf | sh +``` + +**Windows (You need a bit more):** + +* Install the Visual Studio 2015 with C++ support +* Install Rustup using: https://static.rust-lang.org/rustup/dist/x86_64-pc-windows-msvc/rustup-init.exe +* You can then use the ``VS2015 x64 Native Tools Command Prompt`` and run: + +``` +rustup default stable-x86-64-pc-windows-msvc +``` + +#### Getting ready with Cargo + +[Cargo](https://doc.rust-lang.org/cargo/) is the package manager for Rust, and +allows to extend to a number of packages and external libraries. It's also extremely +handy for handling dependencies and helping to modularise your project better. + +*Note: If you've installed rust through rustup, you should have ``cargo`` +installed.* + + +#### Rust Terminology + +When developing rust, you'll come across some terminology that differs to +other programming languages you may have used. + +* **Trait**: A trait is a collection of methods defined for a type, they can be +implemented for any data type. +* **Struct**: A custom data type that lets us name and package together +multiple related values that make a meaninguful group. +* **Crate**: A crate is synonymous with a *library* or *package* in other +languages. They can produce an executable or library depending on the +project. +* **Module**: A collection of items: functions, structs, traits, and even other +modules. Modules allow you to hierarchically split code into logical units +and manage visibility. +* **Attribute**: Metadaata applied to some module, crate or item. +* **Macros**: Macros are powerful meta-programming statements that get expanded +into source code that gets compiled with the rest of the code (Unlike `C` +macros that are pre-processed, Rust macros form an Abstract Syntax Tree). + + +Other good appendix resources: + +* [Keywords](https://doc.rust-lang.org/book/appendix-01-keywords.html) +* [Operators/Symbols](https://doc.rust-lang.org/book/appendix-02-operators.html) +* [Traits](https://doc.rust-lang.org/book/appendix-03-derivable-traits.html) + + +### Understanding the Git Workflow + +Lighthouse utilises git as the primary open-source development tool. To help +with your contributions, it is great to understand the processes used to ensure +everything remains in sync and there's as little conflict as possible when +working on similar files. + +Lighthouse uses the **feature branch** workflow, where each issue, or each +feature, is developed on its own branch and then merged in via a pull-request. + +* [Feature Branch Tutorial](https://www.atlassian.com/git/tutorials/comparing-workflows/feature-branch-workflow) + +## Code Conventions/Styleguide and Ethos + +### Ethos + +**Pull Requests** + +Pull requests should be reviewed by **at least** one "*core developer*" +(someone with write-access to the repo). This should ensure bugs are caught and +the code is kept in a consistent state that follows all conventions and style. + +All discussion (whether in PRs or Issues or in the Gitter) should be respectful +and intellectual. Have fun, but always respect the limits of other people. + +**Testing** + +Generally, tests can be self-contained in the same file. Integration tests +should be added into the ``tests/`` directory in the crate's **root**. + +Large line-count tests should be in a separate file. + +### Rust StyleGuide + +Lighthouse adheres to Rust code conventions as outlined in the [**Rust +Styleguide**](https://github.com/rust-dev-tools/fmt-rfcs/blob/master/guide/guide.md). + +Ensure you use [Clippy](https://github.com/rust-lang/rust-clippy) to lint and +check your code. + +| Code Aspect | Guideline Format | +|:--------------------|:-------------------------------| +| Types | ``UpperCamelCase`` | +| Enums/Enum Variants | ``UpperCamelCase`` | +| Struct Fields | ``snake_case`` | +| Function / Method | ``snake_case`` | +| Macro Names | ``snake_case`` | +| Constants | ``SCREAMING_SNAKE_CASE`` | +| Forbidden name | Trialing Underscore: ``name_`` | + +Other general rust docs: + +* [Rust Other Style Advice](https://github.com/rust-dev-tools/fmt-rfcs/blob/master/guide/advice.md) +* [Cargo.toml Conventions](https://github.com/rust-dev-tools/fmt-rfcs/blob/master/guide/cargo.md) + +### TODOs + +All `TODO` statements should be accompanied by a GitHub issue. + +```rust +pub fn my_function(&mut self, _something &[u8]) -> Result { + // TODO: something_here + // https://github.com/sigp/lighthouse/issues/XX +} +``` + +### Comments + +**General Comments** + +* Prefer line (``//``) comments to block comments (``/* ... */``) +* Comments can appear on the line prior to the item or after a trailing space. +```rust +// Comment for this struct +struct Lighthouse {} + +fn make_blockchain() {} // A comment on the same line after a space +``` + +**Doc Comments** + +* The ``///`` is used to generate comments for Docs. +* The comments should come before attributes. + +```rust +/// Stores the core configuration for this Lighthouse instance. +/// This struct is general, other components may implement more +/// specialized config structs. +#[derive(Clone)] +pub struct LighthouseConfig { + pub data_dir: PathBuf, + pub p2p_listen_port: u16, +} +``` From d0422eed059661dcf0ac1960d0a2aebe3571fcba Mon Sep 17 00:00:00 2001 From: Paul Hauner Date: Fri, 28 Dec 2018 19:38:59 +1100 Subject: [PATCH 02/10] Add new ssz error type --- beacon_chain/utils/ssz/src/decode.rs | 1 + 1 file changed, 1 insertion(+) diff --git a/beacon_chain/utils/ssz/src/decode.rs b/beacon_chain/utils/ssz/src/decode.rs index 7f92c8dd7..0ab02b458 100644 --- a/beacon_chain/utils/ssz/src/decode.rs +++ b/beacon_chain/utils/ssz/src/decode.rs @@ -4,6 +4,7 @@ use super::LENGTH_BYTES; pub enum DecodeError { TooShort, TooLong, + Invalid, } pub trait Decodable: Sized { From d9c472ec37639a350f576ba919d5e60598b5082e Mon Sep 17 00:00:00 2001 From: Paul Hauner Date: Fri, 28 Dec 2018 19:39:25 +1100 Subject: [PATCH 03/10] Run rustfmt on ssz --- beacon_chain/utils/ssz/src/decode.rs | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/beacon_chain/utils/ssz/src/decode.rs b/beacon_chain/utils/ssz/src/decode.rs index 0ab02b458..426baeace 100644 --- a/beacon_chain/utils/ssz/src/decode.rs +++ b/beacon_chain/utils/ssz/src/decode.rs @@ -149,7 +149,8 @@ mod tests { 0, 0, 0, 16, 0, 0, 0, 10, 0, 0, 0, 10, 0, 0, 0, 10, 0, 0, 0, 10, ], 0, - ).unwrap(); + ) + .unwrap(); assert_eq!(decoded.0, v); assert_eq!(decoded.1, 20); @@ -161,7 +162,8 @@ mod tests { 10, 0, 0, 0, 0, 0, 0, 0, 10, ], 0, - ).unwrap(); + ) + .unwrap(); assert_eq!(decoded.0, v); assert_eq!(decoded.1, 36); @@ -173,7 +175,8 @@ mod tests { 0, 0, 15, 0, 0, 0, 0, 0, 0, 0, 15, 0, 0, 0, 0, 0, 0, 0, 15, ], 10, - ).unwrap(); + ) + .unwrap(); assert_eq!(decoded.0, v); assert_eq!(decoded.1, 46); From e6977b5f4b837205049cf64e113854e1e4e5554c Mon Sep 17 00:00:00 2001 From: Paul Hauner Date: Fri, 28 Dec 2018 19:39:44 +1100 Subject: [PATCH 04/10] Impl ssz Enc/Decode on Address. Decode on Vec --- beacon_chain/utils/ssz/src/impl_decode.rs | 12 +++++++++++- beacon_chain/utils/ssz/src/impl_encode.rs | 17 ++++++++++++++++- 2 files changed, 27 insertions(+), 2 deletions(-) diff --git a/beacon_chain/utils/ssz/src/impl_decode.rs b/beacon_chain/utils/ssz/src/impl_decode.rs index b926c23ef..134e438e1 100644 --- a/beacon_chain/utils/ssz/src/impl_decode.rs +++ b/beacon_chain/utils/ssz/src/impl_decode.rs @@ -1,5 +1,5 @@ use super::decode::decode_ssz_list; -use super::ethereum_types::H256; +use super::ethereum_types::{Address, H256}; use super::{Decodable, DecodeError}; macro_rules! impl_decodable_for_uint { @@ -49,6 +49,16 @@ impl Decodable for H256 { } } +impl Decodable for Address { + fn ssz_decode(bytes: &[u8], index: usize) -> Result<(Self, usize), DecodeError> { + if bytes.len() < 20 || bytes.len() - 20 < index { + Err(DecodeError::TooShort) + } else { + Ok((Address::from(&bytes[index..(index + 20)]), index + 20)) + } + } +} + impl Decodable for Vec where T: Decodable, diff --git a/beacon_chain/utils/ssz/src/impl_encode.rs b/beacon_chain/utils/ssz/src/impl_encode.rs index f316a21ea..63de18058 100644 --- a/beacon_chain/utils/ssz/src/impl_encode.rs +++ b/beacon_chain/utils/ssz/src/impl_encode.rs @@ -1,7 +1,7 @@ extern crate bytes; use self::bytes::{BufMut, BytesMut}; -use super::ethereum_types::H256; +use super::ethereum_types::{Address, H256}; use super::{Encodable, SszStream}; /* @@ -52,6 +52,21 @@ impl Encodable for H256 { } } +impl Encodable for Address { + fn ssz_append(&self, s: &mut SszStream) { + s.append_encoded_raw(&self.to_vec()); + } +} + +impl Encodable for Vec +where + T: Encodable, +{ + fn ssz_append(&self, s: &mut SszStream) { + s.append_vec(&self); + } +} + #[cfg(test)] mod tests { use super::*; From e4e729d309f2552164ed388c122ff5f2e16e23c2 Mon Sep 17 00:00:00 2001 From: Paul Hauner Date: Fri, 28 Dec 2018 19:45:59 +1100 Subject: [PATCH 05/10] Add newtype for bls::PublicKey --- beacon_chain/utils/bls/src/lib.rs | 6 ++- beacon_chain/utils/bls/src/public_key.rs | 52 ++++++++++++++++++++++++ beacon_chain/utils/bls/src/signature.rs | 7 ++-- 3 files changed, 60 insertions(+), 5 deletions(-) create mode 100644 beacon_chain/utils/bls/src/public_key.rs diff --git a/beacon_chain/utils/bls/src/lib.rs b/beacon_chain/utils/bls/src/lib.rs index d3a01fb10..2b5d56a7c 100644 --- a/beacon_chain/utils/bls/src/lib.rs +++ b/beacon_chain/utils/bls/src/lib.rs @@ -3,20 +3,22 @@ extern crate hashing; extern crate ssz; mod aggregate_signature; +mod public_key; mod signature; pub use crate::aggregate_signature::AggregateSignature; pub use crate::signature::Signature; +pub use crate::public_key::PublicKey; pub use self::bls_aggregates::AggregatePublicKey; pub use self::bls_aggregates::Keypair; -pub use self::bls_aggregates::PublicKey; pub use self::bls_aggregates::SecretKey; pub const BLS_AGG_SIG_BYTE_SIZE: usize = 97; use hashing::canonical_hash; use std::default::Default; +use ssz::ssz_encode; fn extend_if_needed(hash: &mut Vec) { // NOTE: bls_aggregates crate demands 48 bytes, this may be removed as we get closer to production @@ -26,7 +28,7 @@ fn extend_if_needed(hash: &mut Vec) { /// For some signature and public key, ensure that the signature message was the public key and it /// was signed by the secret key that corresponds to that public key. pub fn verify_proof_of_possession(sig: &Signature, pubkey: &PublicKey) -> bool { - let mut hash = canonical_hash(&pubkey.as_bytes()); + let mut hash = canonical_hash(&ssz_encode(pubkey)); extend_if_needed(&mut hash); sig.verify_hashed(&hash, &pubkey) } diff --git a/beacon_chain/utils/bls/src/public_key.rs b/beacon_chain/utils/bls/src/public_key.rs new file mode 100644 index 000000000..b5d85eff7 --- /dev/null +++ b/beacon_chain/utils/bls/src/public_key.rs @@ -0,0 +1,52 @@ +use super::SecretKey; +use bls_aggregates::PublicKey as RawPublicKey; +use ssz::{decode_ssz_list, Decodable, DecodeError, Encodable, SszStream}; + +/// A single BLS signature. +/// +/// This struct is a wrapper upon a base type and provides helper functions (e.g., SSZ +/// serialization). +#[derive(Debug, PartialEq, Clone)] +pub struct PublicKey(RawPublicKey); + +impl PublicKey { + pub fn from_secret_key(secret_key: &SecretKey) -> Self { + PublicKey(RawPublicKey::from_secret_key(secret_key)) + } + + /// Returns the underlying signature. + pub fn as_raw(&self) -> &RawPublicKey { + &self.0 + } +} + +impl Encodable for PublicKey { + fn ssz_append(&self, s: &mut SszStream) { + s.append_vec(&self.0.as_bytes()); + } +} + +impl Decodable for PublicKey { + fn ssz_decode(bytes: &[u8], i: usize) -> Result<(Self, usize), DecodeError> { + let (sig_bytes, i) = decode_ssz_list(bytes, i)?; + let raw_sig = RawPublicKey::from_bytes(&sig_bytes).map_err(|_| DecodeError::TooShort)?; + Ok((PublicKey(raw_sig), i)) + } +} + +#[cfg(test)] +mod tests { + use super::super::ssz::ssz_encode; + use super::*; + + #[test] + pub fn test_ssz_round_trip() { + let sk = SecretKey::random(); + let original = PublicKey::from_secret_key(&sk); + + let bytes = ssz_encode(&original); + let (decoded, _) = PublicKey::ssz_decode(&bytes, 0).unwrap(); + + assert_eq!(original, decoded); + } +} diff --git a/beacon_chain/utils/bls/src/signature.rs b/beacon_chain/utils/bls/src/signature.rs index ebdb5b817..8854d9019 100644 --- a/beacon_chain/utils/bls/src/signature.rs +++ b/beacon_chain/utils/bls/src/signature.rs @@ -1,5 +1,6 @@ use super::ssz::{decode_ssz_list, Decodable, DecodeError, Encodable, SszStream}; -use bls_aggregates::{PublicKey, SecretKey, Signature as RawSignature}; +use super::PublicKey; +use bls_aggregates::{SecretKey, Signature as RawSignature}; /// A single BLS signature. /// @@ -22,12 +23,12 @@ impl Signature { /// Verify the Signature against a PublicKey. pub fn verify(&self, msg: &[u8], pk: &PublicKey) -> bool { - self.0.verify(msg, pk) + self.0.verify(msg, pk.as_raw()) } /// Verify the Signature against a PublicKey, where the message has already been hashed. pub fn verify_hashed(&self, msg_hash: &[u8], pk: &PublicKey) -> bool { - self.0.verify_hashed(msg_hash, pk) + self.0.verify_hashed(msg_hash, pk.as_raw()) } /// Returns the underlying signature. From 39f217105357ceb8d4e2733e5b071e1a23bbe389 Mon Sep 17 00:00:00 2001 From: Paul Hauner Date: Fri, 28 Dec 2018 19:46:33 +1100 Subject: [PATCH 06/10] Implement ssz Enc/Decode for BeaconState + more --- beacon_chain/types/src/beacon_state.rs | 114 ++++++++++++++ .../src/candidate_pow_receipt_root_record.rs | 53 +++++++ beacon_chain/types/src/crosslink_record.rs | 53 +++++++ beacon_chain/types/src/deposit_input.rs | 7 +- beacon_chain/types/src/fork_data.rs | 58 +++++++ .../types/src/pending_attestation_record.rs | 61 ++++++++ beacon_chain/types/src/shard_committee.rs | 47 ++++-- .../types/src/shard_reassignment_record.rs | 58 +++++++ beacon_chain/types/src/test_utils/address.rs | 11 ++ beacon_chain/types/src/test_utils/mod.rs | 7 + beacon_chain/types/src/validator_record.rs | 148 ++++++++++++++---- .../types/src/validator_registration.rs | 16 +- 12 files changed, 574 insertions(+), 59 deletions(-) create mode 100644 beacon_chain/types/src/test_utils/address.rs diff --git a/beacon_chain/types/src/beacon_state.rs b/beacon_chain/types/src/beacon_state.rs index a3e83a523..7c61d1d20 100644 --- a/beacon_chain/types/src/beacon_state.rs +++ b/beacon_chain/types/src/beacon_state.rs @@ -6,6 +6,9 @@ use super::shard_committee::ShardCommittee; use super::shard_reassignment_record::ShardReassignmentRecord; use super::validator_record::ValidatorRecord; use super::Hash256; +use crate::test_utils::TestRandom; +use rand::RngCore; +use ssz::{Decodable, DecodeError, Encodable, SszStream}; #[derive(Debug, PartialEq, Clone)] pub struct BeaconState { @@ -51,3 +54,114 @@ impl BeaconState { Hash256::zero() } } + +impl Encodable for BeaconState { + fn ssz_append(&self, s: &mut SszStream) { + s.append(&self.slot); + s.append(&self.genesis_time); + s.append(&self.fork_data); + s.append(&self.validator_registry); + s.append(&self.validator_registry_latest_change_slot); + s.append(&self.validator_registry_exit_count); + s.append(&self.validator_registry_delta_chain_tip); + s.append(&self.randao_mix); + s.append(&self.next_seed); + s.append(&self.shard_committees_at_slots); + s.append(&self.persistent_committees); + s.append(&self.persistent_committee_reassignments); + s.append(&self.previous_justified_slot); + s.append(&self.justified_slot); + s.append(&self.justification_bitfield); + s.append(&self.finalized_slot); + s.append(&self.latest_crosslinks); + s.append(&self.latest_block_roots); + s.append(&self.latest_penalized_exit_balances); + s.append(&self.latest_attestations); + s.append(&self.processed_pow_receipt_root); + s.append(&self.candidate_pow_receipt_roots); + } +} + +impl Decodable for BeaconState { + fn ssz_decode(bytes: &[u8], i: usize) -> Result<(Self, usize), DecodeError> { + let (slot, i) = <_>::ssz_decode(bytes, i)?; + let (genesis_time, i) = <_>::ssz_decode(bytes, i)?; + let (fork_data, i) = <_>::ssz_decode(bytes, i)?; + let (validator_registry, i) = <_>::ssz_decode(bytes, i)?; + let (validator_registry_latest_change_slot, i) = <_>::ssz_decode(bytes, i)?; + let (validator_registry_exit_count, i) = <_>::ssz_decode(bytes, i)?; + let (validator_registry_delta_chain_tip, i) = <_>::ssz_decode(bytes, i)?; + let (randao_mix, i) = <_>::ssz_decode(bytes, i)?; + let (next_seed, i) = <_>::ssz_decode(bytes, i)?; + let (shard_committees_at_slots, i) = <_>::ssz_decode(bytes, i)?; + let (persistent_committees, i) = <_>::ssz_decode(bytes, i)?; + let (persistent_committee_reassignments, i) = <_>::ssz_decode(bytes, i)?; + let (previous_justified_slot, i) = <_>::ssz_decode(bytes, i)?; + let (justified_slot, i) = <_>::ssz_decode(bytes, i)?; + let (justification_bitfield, i) = <_>::ssz_decode(bytes, i)?; + let (finalized_slot, i) = <_>::ssz_decode(bytes, i)?; + let (latest_crosslinks, i) = <_>::ssz_decode(bytes, i)?; + let (latest_block_roots, i) = <_>::ssz_decode(bytes, i)?; + let (latest_penalized_exit_balances, i) = <_>::ssz_decode(bytes, i)?; + let (latest_attestations, i) = <_>::ssz_decode(bytes, i)?; + let (processed_pow_receipt_root, i) = <_>::ssz_decode(bytes, i)?; + let (candidate_pow_receipt_roots, i) = <_>::ssz_decode(bytes, i)?; + + Ok(( + Self { + slot, + genesis_time, + fork_data, + validator_registry, + validator_registry_latest_change_slot, + validator_registry_exit_count, + validator_registry_delta_chain_tip, + randao_mix, + next_seed, + shard_committees_at_slots, + persistent_committees, + persistent_committee_reassignments, + previous_justified_slot, + justified_slot, + justification_bitfield, + finalized_slot, + latest_crosslinks, + latest_block_roots, + latest_penalized_exit_balances, + latest_attestations, + processed_pow_receipt_root, + candidate_pow_receipt_roots, + }, + i, + )) + } +} + +impl TestRandom for BeaconState { + fn random_for_test(rng: &mut T) -> Self { + Self { + slot: <_>::random_for_test(rng), + genesis_time: <_>::random_for_test(rng), + fork_data: <_>::random_for_test(rng), + validator_registry: <_>::random_for_test(rng), + validator_registry_latest_change_slot: <_>::random_for_test(rng), + validator_registry_exit_count: <_>::random_for_test(rng), + validator_registry_delta_chain_tip: <_>::random_for_test(rng), + randao_mix: <_>::random_for_test(rng), + next_seed: <_>::random_for_test(rng), + shard_committees_at_slots: <_>::random_for_test(rng), + persistent_committees: <_>::random_for_test(rng), + persistent_committee_reassignments: <_>::random_for_test(rng), + previous_justified_slot: <_>::random_for_test(rng), + justified_slot: <_>::random_for_test(rng), + justification_bitfield: <_>::random_for_test(rng), + finalized_slot: <_>::random_for_test(rng), + latest_crosslinks: <_>::random_for_test(rng), + latest_block_roots: <_>::random_for_test(rng), + latest_penalized_exit_balances: <_>::random_for_test(rng), + latest_attestations: <_>::random_for_test(rng), + processed_pow_receipt_root: <_>::random_for_test(rng), + candidate_pow_receipt_roots: <_>::random_for_test(rng), + } + } +} diff --git a/beacon_chain/types/src/candidate_pow_receipt_root_record.rs b/beacon_chain/types/src/candidate_pow_receipt_root_record.rs index a06e5e794..663c3e1db 100644 --- a/beacon_chain/types/src/candidate_pow_receipt_root_record.rs +++ b/beacon_chain/types/src/candidate_pow_receipt_root_record.rs @@ -1,7 +1,60 @@ +use super::ssz::{Decodable, DecodeError, Encodable, SszStream}; use super::Hash256; +use crate::test_utils::TestRandom; +use rand::RngCore; #[derive(Debug, PartialEq, Clone)] pub struct CandidatePoWReceiptRootRecord { pub candidate_pow_receipt_root: Hash256, pub votes: u64, } + +impl Encodable for CandidatePoWReceiptRootRecord { + fn ssz_append(&self, s: &mut SszStream) { + s.append(&self.candidate_pow_receipt_root); + s.append(&self.votes); + } +} + +impl Decodable for CandidatePoWReceiptRootRecord { + fn ssz_decode(bytes: &[u8], i: usize) -> Result<(Self, usize), DecodeError> { + let (candidate_pow_receipt_root, i) = <_>::ssz_decode(bytes, i)?; + let (votes, i) = <_>::ssz_decode(bytes, i)?; + + Ok(( + Self { + candidate_pow_receipt_root, + votes, + }, + i, + )) + } +} + +impl TestRandom for CandidatePoWReceiptRootRecord { + fn random_for_test(rng: &mut T) -> Self { + Self { + candidate_pow_receipt_root: <_>::random_for_test(rng), + votes: <_>::random_for_test(rng), + } + } +} + +#[cfg(test)] +mod tests { + use super::super::ssz::ssz_encode; + use super::*; + use crate::test_utils::TestRandom; + use rand::{prng::XorShiftRng, SeedableRng}; + + #[test] + pub fn test_ssz_round_trip() { + let mut rng = XorShiftRng::from_seed([42; 16]); + let original = CandidatePoWReceiptRootRecord::random_for_test(&mut rng); + + let bytes = ssz_encode(&original); + let (decoded, _) = <_>::ssz_decode(&bytes, 0).unwrap(); + + assert_eq!(original, decoded); + } +} diff --git a/beacon_chain/types/src/crosslink_record.rs b/beacon_chain/types/src/crosslink_record.rs index 6a70daade..9e525664a 100644 --- a/beacon_chain/types/src/crosslink_record.rs +++ b/beacon_chain/types/src/crosslink_record.rs @@ -1,4 +1,7 @@ +use super::ssz::{Decodable, DecodeError, Encodable, SszStream}; use super::Hash256; +use crate::test_utils::TestRandom; +use rand::RngCore; #[derive(Clone, Debug, PartialEq)] pub struct CrosslinkRecord { @@ -15,3 +18,53 @@ impl CrosslinkRecord { } } } + +impl Encodable for CrosslinkRecord { + fn ssz_append(&self, s: &mut SszStream) { + s.append(&self.slot); + s.append(&self.shard_block_root); + } +} + +impl Decodable for CrosslinkRecord { + fn ssz_decode(bytes: &[u8], i: usize) -> Result<(Self, usize), DecodeError> { + let (slot, i) = <_>::ssz_decode(bytes, i)?; + let (shard_block_root, i) = <_>::ssz_decode(bytes, i)?; + + Ok(( + Self { + slot, + shard_block_root, + }, + i, + )) + } +} + +impl TestRandom for CrosslinkRecord { + fn random_for_test(rng: &mut T) -> Self { + Self { + slot: <_>::random_for_test(rng), + shard_block_root: <_>::random_for_test(rng), + } + } +} + +#[cfg(test)] +mod tests { + use super::super::ssz::ssz_encode; + use super::*; + use crate::test_utils::TestRandom; + use rand::{prng::XorShiftRng, SeedableRng}; + + #[test] + pub fn test_ssz_round_trip() { + let mut rng = XorShiftRng::from_seed([42; 16]); + let original = CrosslinkRecord::random_for_test(&mut rng); + + let bytes = ssz_encode(&original); + let (decoded, _) = <_>::ssz_decode(&bytes, 0).unwrap(); + + assert_eq!(original, decoded); + } +} diff --git a/beacon_chain/types/src/deposit_input.rs b/beacon_chain/types/src/deposit_input.rs index 41b579cda..3a9a08412 100644 --- a/beacon_chain/types/src/deposit_input.rs +++ b/beacon_chain/types/src/deposit_input.rs @@ -1,4 +1,4 @@ -use super::ssz::{decode_ssz_list, Decodable, DecodeError, Encodable, SszStream}; +use super::ssz::{Decodable, DecodeError, Encodable, SszStream}; use super::Hash256; use crate::test_utils::TestRandom; use bls::{PublicKey, Signature}; @@ -14,7 +14,7 @@ pub struct DepositInput { impl Encodable for DepositInput { fn ssz_append(&self, s: &mut SszStream) { - s.append_vec(&self.pubkey.as_bytes()); + s.append(&self.pubkey); s.append(&self.withdrawal_credentials); s.append(&self.randao_commitment); s.append(&self.proof_of_possession); @@ -23,8 +23,7 @@ impl Encodable for DepositInput { impl Decodable for DepositInput { fn ssz_decode(bytes: &[u8], i: usize) -> Result<(Self, usize), DecodeError> { - let (pubkey_bytes, i) = decode_ssz_list(bytes, i)?; - let pubkey = PublicKey::from_bytes(&pubkey_bytes).map_err(|_| DecodeError::TooShort)?; + let (pubkey, i) = <_>::ssz_decode(bytes, i)?; let (withdrawal_credentials, i) = <_>::ssz_decode(bytes, i)?; let (randao_commitment, i) = <_>::ssz_decode(bytes, i)?; let (proof_of_possession, i) = <_>::ssz_decode(bytes, i)?; diff --git a/beacon_chain/types/src/fork_data.rs b/beacon_chain/types/src/fork_data.rs index 4c3541c3f..e779c797e 100644 --- a/beacon_chain/types/src/fork_data.rs +++ b/beacon_chain/types/src/fork_data.rs @@ -1,6 +1,64 @@ +use super::ssz::{Decodable, DecodeError, Encodable, SszStream}; +use crate::test_utils::TestRandom; +use rand::RngCore; + #[derive(Debug, Clone, PartialEq)] pub struct ForkData { pub pre_fork_version: u64, pub post_fork_version: u64, pub fork_slot: u64, } + +impl Encodable for ForkData { + fn ssz_append(&self, s: &mut SszStream) { + s.append(&self.pre_fork_version); + s.append(&self.post_fork_version); + s.append(&self.fork_slot); + } +} + +impl Decodable for ForkData { + fn ssz_decode(bytes: &[u8], i: usize) -> Result<(Self, usize), DecodeError> { + let (pre_fork_version, i) = <_>::ssz_decode(bytes, i)?; + let (post_fork_version, i) = <_>::ssz_decode(bytes, i)?; + let (fork_slot, i) = <_>::ssz_decode(bytes, i)?; + + Ok(( + Self { + pre_fork_version, + post_fork_version, + fork_slot, + }, + i, + )) + } +} + +impl TestRandom for ForkData { + fn random_for_test(rng: &mut T) -> Self { + Self { + pre_fork_version: <_>::random_for_test(rng), + post_fork_version: <_>::random_for_test(rng), + fork_slot: <_>::random_for_test(rng), + } + } +} + +#[cfg(test)] +mod tests { + use super::super::ssz::ssz_encode; + use super::*; + use crate::test_utils::TestRandom; + use rand::{prng::XorShiftRng, SeedableRng}; + + #[test] + pub fn test_ssz_round_trip() { + let mut rng = XorShiftRng::from_seed([42; 16]); + let original = ForkData::random_for_test(&mut rng); + + let bytes = ssz_encode(&original); + let (decoded, _) = <_>::ssz_decode(&bytes, 0).unwrap(); + + assert_eq!(original, decoded); + } +} diff --git a/beacon_chain/types/src/pending_attestation_record.rs b/beacon_chain/types/src/pending_attestation_record.rs index 16f77c90c..aa289e904 100644 --- a/beacon_chain/types/src/pending_attestation_record.rs +++ b/beacon_chain/types/src/pending_attestation_record.rs @@ -1,4 +1,7 @@ +use super::ssz::{Decodable, DecodeError, Encodable, SszStream}; use super::{AttestationData, Bitfield}; +use crate::test_utils::TestRandom; +use rand::RngCore; #[derive(Debug, Clone, PartialEq)] pub struct PendingAttestationRecord { @@ -7,3 +10,61 @@ pub struct PendingAttestationRecord { pub custody_bitfield: Bitfield, pub slot_included: u64, } + +impl Encodable for PendingAttestationRecord { + fn ssz_append(&self, s: &mut SszStream) { + s.append(&self.data); + s.append(&self.participation_bitfield); + s.append(&self.custody_bitfield); + s.append(&self.slot_included); + } +} + +impl Decodable for PendingAttestationRecord { + fn ssz_decode(bytes: &[u8], i: usize) -> Result<(Self, usize), DecodeError> { + let (data, i) = <_>::ssz_decode(bytes, i)?; + let (participation_bitfield, i) = <_>::ssz_decode(bytes, i)?; + let (custody_bitfield, i) = <_>::ssz_decode(bytes, i)?; + let (slot_included, i) = <_>::ssz_decode(bytes, i)?; + + Ok(( + Self { + data, + participation_bitfield, + custody_bitfield, + slot_included, + }, + i, + )) + } +} + +impl TestRandom for PendingAttestationRecord { + fn random_for_test(rng: &mut T) -> Self { + Self { + data: <_>::random_for_test(rng), + participation_bitfield: <_>::random_for_test(rng), + custody_bitfield: <_>::random_for_test(rng), + slot_included: <_>::random_for_test(rng), + } + } +} + +#[cfg(test)] +mod tests { + use super::super::ssz::ssz_encode; + use super::*; + use crate::test_utils::TestRandom; + use rand::{prng::XorShiftRng, SeedableRng}; + + #[test] + pub fn test_ssz_round_trip() { + let mut rng = XorShiftRng::from_seed([42; 16]); + let original = PendingAttestationRecord::random_for_test(&mut rng); + + let bytes = ssz_encode(&original); + let (decoded, _) = <_>::ssz_decode(&bytes, 0).unwrap(); + + assert_eq!(original, decoded); + } +} diff --git a/beacon_chain/types/src/shard_committee.rs b/beacon_chain/types/src/shard_committee.rs index 75208670f..073dd0176 100644 --- a/beacon_chain/types/src/shard_committee.rs +++ b/beacon_chain/types/src/shard_committee.rs @@ -1,28 +1,53 @@ +use super::ssz::{Decodable, DecodeError, Encodable, SszStream}; +use crate::test_utils::TestRandom; +use rand::RngCore; + #[derive(Clone, Debug, PartialEq)] pub struct ShardCommittee { - pub shard: u16, + pub shard: u64, pub committee: Vec, } -impl ShardCommittee { - /// Returns a new instance where the `shard_id` is zero and the - /// committee is an empty vector. - pub fn zero() -> Self { +impl Encodable for ShardCommittee { + fn ssz_append(&self, s: &mut SszStream) { + s.append(&self.shard); + s.append(&self.committee); + } +} + +impl Decodable for ShardCommittee { + fn ssz_decode(bytes: &[u8], i: usize) -> Result<(Self, usize), DecodeError> { + let (shard, i) = <_>::ssz_decode(bytes, i)?; + let (committee, i) = <_>::ssz_decode(bytes, i)?; + + Ok((Self { shard, committee }, i)) + } +} + +impl TestRandom for ShardCommittee { + fn random_for_test(rng: &mut T) -> Self { Self { - shard: 0, - committee: vec![], + shard: <_>::random_for_test(rng), + committee: <_>::random_for_test(rng), } } } #[cfg(test)] mod tests { + use super::super::ssz::ssz_encode; use super::*; + use crate::test_utils::TestRandom; + use rand::{prng::XorShiftRng, SeedableRng}; #[test] - fn test_shard_and_committee_zero() { - let s = ShardCommittee::zero(); - assert_eq!(s.shard, 0); - assert_eq!(s.committee.len(), 0); + pub fn test_ssz_round_trip() { + let mut rng = XorShiftRng::from_seed([42; 16]); + let original = ShardCommittee::random_for_test(&mut rng); + + let bytes = ssz_encode(&original); + let (decoded, _) = <_>::ssz_decode(&bytes, 0).unwrap(); + + assert_eq!(original, decoded); } } diff --git a/beacon_chain/types/src/shard_reassignment_record.rs b/beacon_chain/types/src/shard_reassignment_record.rs index c9a17741c..e990afeac 100644 --- a/beacon_chain/types/src/shard_reassignment_record.rs +++ b/beacon_chain/types/src/shard_reassignment_record.rs @@ -1,6 +1,64 @@ +use super::ssz::{Decodable, DecodeError, Encodable, SszStream}; +use crate::test_utils::TestRandom; +use rand::RngCore; + #[derive(Debug, PartialEq, Clone)] pub struct ShardReassignmentRecord { pub validator_index: u64, pub shard: u64, pub slot: u64, } + +impl Encodable for ShardReassignmentRecord { + fn ssz_append(&self, s: &mut SszStream) { + s.append(&self.validator_index); + s.append(&self.shard); + s.append(&self.slot); + } +} + +impl Decodable for ShardReassignmentRecord { + fn ssz_decode(bytes: &[u8], i: usize) -> Result<(Self, usize), DecodeError> { + let (validator_index, i) = <_>::ssz_decode(bytes, i)?; + let (shard, i) = <_>::ssz_decode(bytes, i)?; + let (slot, i) = <_>::ssz_decode(bytes, i)?; + + Ok(( + Self { + validator_index, + shard, + slot, + }, + i, + )) + } +} + +impl TestRandom for ShardReassignmentRecord { + fn random_for_test(rng: &mut T) -> Self { + Self { + validator_index: <_>::random_for_test(rng), + shard: <_>::random_for_test(rng), + slot: <_>::random_for_test(rng), + } + } +} + +#[cfg(test)] +mod tests { + use super::super::ssz::ssz_encode; + use super::*; + use crate::test_utils::TestRandom; + use rand::{prng::XorShiftRng, SeedableRng}; + + #[test] + pub fn test_ssz_round_trip() { + let mut rng = XorShiftRng::from_seed([42; 16]); + let original = ShardReassignmentRecord::random_for_test(&mut rng); + + let bytes = ssz_encode(&original); + let (decoded, _) = <_>::ssz_decode(&bytes, 0).unwrap(); + + assert_eq!(original, decoded); + } +} diff --git a/beacon_chain/types/src/test_utils/address.rs b/beacon_chain/types/src/test_utils/address.rs new file mode 100644 index 000000000..2d60b72da --- /dev/null +++ b/beacon_chain/types/src/test_utils/address.rs @@ -0,0 +1,11 @@ +use super::TestRandom; +use crate::Address; +use rand::RngCore; + +impl TestRandom for Address { + fn random_for_test(rng: &mut T) -> Self { + let mut key_bytes = vec![0; 20]; + rng.fill_bytes(&mut key_bytes); + Address::from(&key_bytes[..]) + } +} diff --git a/beacon_chain/types/src/test_utils/mod.rs b/beacon_chain/types/src/test_utils/mod.rs index 131ff9aac..f3e382e6e 100644 --- a/beacon_chain/types/src/test_utils/mod.rs +++ b/beacon_chain/types/src/test_utils/mod.rs @@ -2,6 +2,7 @@ use rand::RngCore; pub use rand::{prng::XorShiftRng, SeedableRng}; +pub mod address; pub mod aggregate_signature; pub mod bitfield; pub mod hash256; @@ -27,6 +28,12 @@ impl TestRandom for u32 { } } +impl TestRandom for usize { + fn random_for_test(rng: &mut T) -> Self { + rng.next_u32() as usize + } +} + impl TestRandom for Vec where U: TestRandom { diff --git a/beacon_chain/types/src/validator_record.rs b/beacon_chain/types/src/validator_record.rs index 00c4637bf..7da858676 100644 --- a/beacon_chain/types/src/validator_record.rs +++ b/beacon_chain/types/src/validator_record.rs @@ -1,5 +1,8 @@ -use super::bls::{Keypair, PublicKey}; +use super::bls::PublicKey; use super::{Address, Hash256}; +use crate::test_utils::TestRandom; +use rand::RngCore; +use ssz::{Decodable, DecodeError, Encodable, SszStream}; use std::convert; #[derive(Debug, PartialEq, Clone)] @@ -38,44 +41,131 @@ pub struct ValidatorRecord { pub exit_slot: u64, } -impl ValidatorRecord { - /// Generates a new instance where the keypair is generated using - /// `rand::thread_rng` entropy and all other fields are set to zero. - /// - /// Returns the new instance and new keypair. - pub fn zero_with_thread_rand_keypair() -> (Self, Keypair) { - let keypair = Keypair::random(); - let s = Self { - pubkey: keypair.pk.clone(), - withdrawal_shard: 0, - withdrawal_address: Address::zero(), - randao_commitment: Hash256::zero(), - randao_last_change: 0, - balance: 0, - status: From::from(0), - exit_slot: 0, +impl Encodable for ValidatorStatus { + fn ssz_append(&self, s: &mut SszStream) { + let byte: u8 = match self { + ValidatorStatus::PendingActivation => 0, + ValidatorStatus::Active => 1, + ValidatorStatus::PendingExit => 2, + ValidatorStatus::PendingWithdraw => 3, + ValidatorStatus::Withdrawn => 5, + ValidatorStatus::Penalized => 127, }; - (s, keypair) + s.append(&byte); } +} - pub fn status_is(&self, status: ValidatorStatus) -> bool { - self.status == status +impl Decodable for ValidatorStatus { + fn ssz_decode(bytes: &[u8], i: usize) -> Result<(Self, usize), DecodeError> { + let (byte, i) = u8::ssz_decode(bytes, i)?; + let status = match byte { + 0 => ValidatorStatus::PendingActivation, + 1 => ValidatorStatus::Active, + 2 => ValidatorStatus::PendingExit, + 3 => ValidatorStatus::PendingWithdraw, + 5 => ValidatorStatus::Withdrawn, + 127 => ValidatorStatus::Penalized, + _ => return Err(DecodeError::Invalid), + }; + Ok((status, i)) + } +} + +impl TestRandom for ValidatorStatus { + fn random_for_test(rng: &mut T) -> Self { + let options = vec![ + ValidatorStatus::PendingActivation, + ValidatorStatus::Active, + ValidatorStatus::PendingExit, + ValidatorStatus::PendingWithdraw, + ValidatorStatus::Withdrawn, + ValidatorStatus::Penalized, + ]; + options[(rng.next_u32() as usize) % options.len()].clone() + } +} + +impl Encodable for ValidatorRecord { + fn ssz_append(&self, s: &mut SszStream) { + s.append(&self.pubkey); + s.append(&self.withdrawal_shard); + s.append(&self.withdrawal_address); + s.append(&self.randao_commitment); + s.append(&self.randao_last_change); + s.append(&self.balance); + s.append(&self.status); + s.append(&self.exit_slot); + } +} + +impl Decodable for ValidatorRecord { + fn ssz_decode(bytes: &[u8], i: usize) -> Result<(Self, usize), DecodeError> { + let (pubkey, i) = <_>::ssz_decode(bytes, i)?; + let (withdrawal_shard, i) = <_>::ssz_decode(bytes, i)?; + let (withdrawal_address, i) = <_>::ssz_decode(bytes, i)?; + let (randao_commitment, i) = <_>::ssz_decode(bytes, i)?; + let (randao_last_change, i) = <_>::ssz_decode(bytes, i)?; + let (balance, i) = <_>::ssz_decode(bytes, i)?; + let (status, i) = <_>::ssz_decode(bytes, i)?; + let (exit_slot, i) = <_>::ssz_decode(bytes, i)?; + + Ok(( + Self { + pubkey, + withdrawal_shard, + withdrawal_address, + randao_commitment, + randao_last_change, + balance, + status, + exit_slot, + }, + i, + )) + } +} + +impl TestRandom for ValidatorRecord { + fn random_for_test(rng: &mut T) -> Self { + Self { + pubkey: <_>::random_for_test(rng), + withdrawal_shard: <_>::random_for_test(rng), + withdrawal_address: <_>::random_for_test(rng), + randao_commitment: <_>::random_for_test(rng), + randao_last_change: <_>::random_for_test(rng), + balance: <_>::random_for_test(rng), + status: <_>::random_for_test(rng), + exit_slot: <_>::random_for_test(rng), + } } } #[cfg(test)] mod tests { + use super::super::ssz::ssz_encode; use super::*; + use crate::test_utils::TestRandom; + use rand::{prng::XorShiftRng, SeedableRng}; #[test] - fn test_validator_record_zero_rand_keypair() { - let (v, _kp) = ValidatorRecord::zero_with_thread_rand_keypair(); - assert_eq!(v.withdrawal_shard, 0); - assert!(v.withdrawal_address.is_zero()); - assert!(v.randao_commitment.is_zero()); - assert_eq!(v.randao_last_change, 0); - assert_eq!(v.balance, 0); - assert_eq!(v.status, From::from(0)); - assert_eq!(v.exit_slot, 0); + pub fn test_ssz_round_trip() { + let mut rng = XorShiftRng::from_seed([42; 16]); + let original = ValidatorRecord::random_for_test(&mut rng); + + let bytes = ssz_encode(&original); + let (decoded, _) = <_>::ssz_decode(&bytes, 0).unwrap(); + + assert_eq!(original, decoded); + } + + #[test] + pub fn test_validator_status_ssz_round_trip() { + let mut rng = XorShiftRng::from_seed([42; 16]); + let original = ValidatorStatus::random_for_test(&mut rng); + + let bytes = ssz_encode(&original); + let (decoded, _) = <_>::ssz_decode(&bytes, 0).unwrap(); + + assert_eq!(original, decoded); } } diff --git a/beacon_chain/types/src/validator_registration.rs b/beacon_chain/types/src/validator_registration.rs index 11f6d11c5..964b80b8a 100644 --- a/beacon_chain/types/src/validator_registration.rs +++ b/beacon_chain/types/src/validator_registration.rs @@ -1,5 +1,5 @@ use super::{Address, Hash256}; -use bls::{create_proof_of_possession, Keypair, PublicKey, Signature}; +use bls::{PublicKey, Signature}; /// The information gathered from the PoW chain validator registration function. #[derive(Debug, Clone, PartialEq)] @@ -10,17 +10,3 @@ pub struct ValidatorRegistration { pub randao_commitment: Hash256, pub proof_of_possession: Signature, } - -impl ValidatorRegistration { - pub fn random() -> Self { - let keypair = Keypair::random(); - - Self { - pubkey: keypair.pk.clone(), - withdrawal_shard: 0, - withdrawal_address: Address::random(), - randao_commitment: Hash256::random(), - proof_of_possession: create_proof_of_possession(&keypair), - } - } -} From 4757b35ed2826a01dd69a691768d6de2dd3e6c46 Mon Sep 17 00:00:00 2001 From: Paul Hauner Date: Sat, 29 Dec 2018 14:31:16 +1100 Subject: [PATCH 07/10] Wrap BLS keypair, pubkey and privkey in newtypes --- .../attestation_validation/src/signature.rs | 8 ++- beacon_chain/utils/bls/src/keypair.rs | 16 +++++ beacon_chain/utils/bls/src/lib.rs | 10 ++-- beacon_chain/utils/bls/src/public_key.rs | 2 +- beacon_chain/utils/bls/src/secret_key.rs | 59 +++++++++++++++++++ beacon_chain/utils/bls/src/signature.rs | 8 +-- beacon_chain/validator_change/Cargo.toml | 1 + beacon_chain/validator_change/src/lib.rs | 5 +- lighthouse/db/src/stores/validator_store.rs | 50 +++++++++------- 9 files changed, 125 insertions(+), 34 deletions(-) create mode 100644 beacon_chain/utils/bls/src/keypair.rs create mode 100644 beacon_chain/utils/bls/src/secret_key.rs diff --git a/beacon_chain/attestation_validation/src/signature.rs b/beacon_chain/attestation_validation/src/signature.rs index 050f9cdf9..7d08be337 100644 --- a/beacon_chain/attestation_validation/src/signature.rs +++ b/beacon_chain/attestation_validation/src/signature.rs @@ -27,7 +27,7 @@ where .get_public_key_by_index(validator)? .ok_or(Error::NoPublicKeyForValidator)?; // Aggregate the public key. - agg_pub_key.add(&pub_key); + agg_pub_key.add(&pub_key.as_raw()); } } @@ -129,7 +129,8 @@ mod tests { &agg_sig, &attestation_indices, &store, - ).unwrap(); + ) + .unwrap(); assert_eq!(outcome, Outcome::Valid); /* @@ -143,7 +144,8 @@ mod tests { &agg_sig, &attestation_indices, &store, - ).unwrap(); + ) + .unwrap(); assert_eq!(outcome, Outcome::Invalid(Invalid::SignatureInvalid)); } } diff --git a/beacon_chain/utils/bls/src/keypair.rs b/beacon_chain/utils/bls/src/keypair.rs new file mode 100644 index 000000000..51a61f8f7 --- /dev/null +++ b/beacon_chain/utils/bls/src/keypair.rs @@ -0,0 +1,16 @@ +use super::{PublicKey, SecretKey}; + +#[derive(Debug, Clone, PartialEq)] +pub struct Keypair { + pub sk: SecretKey, + pub pk: PublicKey, +} + +impl Keypair { + /// Instantiate a Keypair using SecretKey::random(). + pub fn random() -> Self { + let sk = SecretKey::random(); + let pk = PublicKey::from_secret_key(&sk); + Keypair { sk, pk } + } +} diff --git a/beacon_chain/utils/bls/src/lib.rs b/beacon_chain/utils/bls/src/lib.rs index 2b5d56a7c..9d27d870b 100644 --- a/beacon_chain/utils/bls/src/lib.rs +++ b/beacon_chain/utils/bls/src/lib.rs @@ -3,16 +3,18 @@ extern crate hashing; extern crate ssz; mod aggregate_signature; +mod keypair; mod public_key; +mod secret_key; mod signature; pub use crate::aggregate_signature::AggregateSignature; -pub use crate::signature::Signature; +pub use crate::keypair::Keypair; pub use crate::public_key::PublicKey; +pub use crate::secret_key::SecretKey; +pub use crate::signature::Signature; pub use self::bls_aggregates::AggregatePublicKey; -pub use self::bls_aggregates::Keypair; -pub use self::bls_aggregates::SecretKey; pub const BLS_AGG_SIG_BYTE_SIZE: usize = 97; @@ -34,7 +36,7 @@ pub fn verify_proof_of_possession(sig: &Signature, pubkey: &PublicKey) -> bool { } pub fn create_proof_of_possession(keypair: &Keypair) -> Signature { - let mut hash = canonical_hash(&keypair.pk.as_bytes()); + let mut hash = canonical_hash(&ssz_encode(&keypair.pk)); extend_if_needed(&mut hash); Signature::new_hashed(&hash, &keypair.sk) } diff --git a/beacon_chain/utils/bls/src/public_key.rs b/beacon_chain/utils/bls/src/public_key.rs index b5d85eff7..49dbc9e4b 100644 --- a/beacon_chain/utils/bls/src/public_key.rs +++ b/beacon_chain/utils/bls/src/public_key.rs @@ -11,7 +11,7 @@ pub struct PublicKey(RawPublicKey); impl PublicKey { pub fn from_secret_key(secret_key: &SecretKey) -> Self { - PublicKey(RawPublicKey::from_secret_key(secret_key)) + PublicKey(RawPublicKey::from_secret_key(secret_key.as_raw())) } /// Returns the underlying signature. diff --git a/beacon_chain/utils/bls/src/secret_key.rs b/beacon_chain/utils/bls/src/secret_key.rs new file mode 100644 index 000000000..7c4383695 --- /dev/null +++ b/beacon_chain/utils/bls/src/secret_key.rs @@ -0,0 +1,59 @@ +use bls_aggregates::{DecodeError as BlsDecodeError, SecretKey as RawSecretKey}; +use ssz::{decode_ssz_list, Decodable, DecodeError, Encodable, SszStream}; + +/// A single BLS signature. +/// +/// This struct is a wrapper upon a base type and provides helper functions (e.g., SSZ +/// serialization). +#[derive(Debug, PartialEq, Clone)] +pub struct SecretKey(RawSecretKey); + +impl SecretKey { + pub fn random() -> Self { + SecretKey(RawSecretKey::random()) + } + + /// Instantiate a SecretKey from existing bytes. + /// + /// Note: this is _not_ SSZ decoding. + pub fn from_bytes(bytes: &[u8]) -> Result { + Ok(SecretKey(RawSecretKey::from_bytes(bytes)?)) + } + + /// Returns the underlying secret key. + pub fn as_raw(&self) -> &RawSecretKey { + &self.0 + } +} + +impl Encodable for SecretKey { + fn ssz_append(&self, s: &mut SszStream) { + s.append_vec(&self.0.as_bytes()); + } +} + +impl Decodable for SecretKey { + fn ssz_decode(bytes: &[u8], i: usize) -> Result<(Self, usize), DecodeError> { + let (sig_bytes, i) = decode_ssz_list(bytes, i)?; + let raw_sig = RawSecretKey::from_bytes(&sig_bytes).map_err(|_| DecodeError::TooShort)?; + Ok((SecretKey(raw_sig), i)) + } +} + +#[cfg(test)] +mod tests { + use super::super::ssz::ssz_encode; + use super::*; + + #[test] + pub fn test_ssz_round_trip() { + let original = + SecretKey::from_bytes("jzjxxgjajfjrmgodszzsgqccmhnyvetcuxobhtynojtpdtbj".as_bytes()) + .unwrap(); + + let bytes = ssz_encode(&original); + let (decoded, _) = SecretKey::ssz_decode(&bytes, 0).unwrap(); + + assert_eq!(original, decoded); + } +} diff --git a/beacon_chain/utils/bls/src/signature.rs b/beacon_chain/utils/bls/src/signature.rs index 8854d9019..242908e21 100644 --- a/beacon_chain/utils/bls/src/signature.rs +++ b/beacon_chain/utils/bls/src/signature.rs @@ -1,6 +1,6 @@ use super::ssz::{decode_ssz_list, Decodable, DecodeError, Encodable, SszStream}; -use super::PublicKey; -use bls_aggregates::{SecretKey, Signature as RawSignature}; +use super::{PublicKey, SecretKey}; +use bls_aggregates::Signature as RawSignature; /// A single BLS signature. /// @@ -12,13 +12,13 @@ pub struct Signature(RawSignature); impl Signature { /// Instantiate a new Signature from a message and a SecretKey. pub fn new(msg: &[u8], sk: &SecretKey) -> Self { - Signature(RawSignature::new(msg, sk)) + Signature(RawSignature::new(msg, sk.as_raw())) } /// Instantiate a new Signature from a message and a SecretKey, where the message has already /// been hashed. pub fn new_hashed(msg_hashed: &[u8], sk: &SecretKey) -> Self { - Signature(RawSignature::new_hashed(msg_hashed, sk)) + Signature(RawSignature::new_hashed(msg_hashed, sk.as_raw())) } /// Verify the Signature against a PublicKey. diff --git a/beacon_chain/validator_change/Cargo.toml b/beacon_chain/validator_change/Cargo.toml index 4574a115f..a1c499340 100644 --- a/beacon_chain/validator_change/Cargo.toml +++ b/beacon_chain/validator_change/Cargo.toml @@ -7,4 +7,5 @@ edition = "2018" [dependencies] bytes = "0.4.10" hashing = { path = "../utils/hashing" } +ssz = { path = "../utils/ssz" } types = { path = "../types" } diff --git a/beacon_chain/validator_change/src/lib.rs b/beacon_chain/validator_change/src/lib.rs index 99687e30a..7c13b168a 100644 --- a/beacon_chain/validator_change/src/lib.rs +++ b/beacon_chain/validator_change/src/lib.rs @@ -4,6 +4,7 @@ extern crate types; use bytes::{BufMut, BytesMut}; use hashing::canonical_hash; +use ssz::ssz_encode; use std::cmp::max; use types::{Hash256, ValidatorRecord, ValidatorStatus}; @@ -70,7 +71,7 @@ pub fn update_validator_set( */ if new_total_changed <= max_allowable_change { v.status = ValidatorStatus::Active; - hasher.extend(i, &v.pubkey.as_bytes(), VALIDATOR_FLAG_ENTRY); + hasher.extend(i, &ssz_encode(&v.pubkey), VALIDATOR_FLAG_ENTRY); total_changed = new_total_changed; } else { // Entering the validator would exceed the balance delta. @@ -91,7 +92,7 @@ pub fn update_validator_set( if new_total_changed <= max_allowable_change { v.status = ValidatorStatus::PendingWithdraw; v.exit_slot = present_slot; - hasher.extend(i, &v.pubkey.as_bytes(), VALIDATOR_FLAG_EXIT); + hasher.extend(i, &ssz_encode(&v.pubkey), VALIDATOR_FLAG_EXIT); total_changed = new_total_changed; } else { // Exiting the validator would exceed the balance delta. diff --git a/lighthouse/db/src/stores/validator_store.rs b/lighthouse/db/src/stores/validator_store.rs index 518d92608..500bb50af 100644 --- a/lighthouse/db/src/stores/validator_store.rs +++ b/lighthouse/db/src/stores/validator_store.rs @@ -4,6 +4,7 @@ use self::bytes::{BufMut, BytesMut}; use super::bls::PublicKey; use super::VALIDATOR_DB_COLUMN as DB_COLUMN; use super::{ClientDB, DBError}; +use ssz::{ssz_encode, Decodable}; use std::sync::Arc; #[derive(Debug, PartialEq)] @@ -54,7 +55,7 @@ impl ValidatorStore { public_key: &PublicKey, ) -> Result<(), ValidatorStoreError> { let key = self.get_db_key_for_index(&KeyPrefixes::PublicKey, index); - let val = public_key.as_bytes(); + let val = ssz_encode(public_key); self.db .put(DB_COLUMN, &key[..], &val[..]) .map_err(ValidatorStoreError::from) @@ -68,8 +69,8 @@ impl ValidatorStore { let val = self.db.get(DB_COLUMN, &key[..])?; match val { None => Ok(None), - Some(val) => match PublicKey::from_bytes(&val) { - Ok(key) => Ok(Some(key)), + Some(val) => match PublicKey::ssz_decode(&val, 0) { + Ok((key, _)) => Ok(Some(key)), Err(_) => Err(ValidatorStoreError::DecodeError), }, } @@ -87,7 +88,10 @@ mod tests { let db = Arc::new(MemoryDB::open()); let store = ValidatorStore::new(db.clone()); - assert_eq!(store.prefix_bytes(&KeyPrefixes::PublicKey), b"pubkey".to_vec()); + assert_eq!( + store.prefix_bytes(&KeyPrefixes::PublicKey), + b"pubkey".to_vec() + ); } #[test] @@ -98,7 +102,10 @@ mod tests { let mut buf = BytesMut::with_capacity(6 + 8); buf.put(b"pubkey".to_vec()); buf.put_u64_be(42); - assert_eq!(store.get_db_key_for_index(&KeyPrefixes::PublicKey, 42), buf.take().to_vec()) + assert_eq!( + store.get_db_key_for_index(&KeyPrefixes::PublicKey, 42), + buf.take().to_vec() + ) } #[test] @@ -110,12 +117,15 @@ mod tests { let public_key = Keypair::random().pk; store.put_public_key_by_index(index, &public_key).unwrap(); - let public_key_at_index = db.get( - DB_COLUMN, - &store.get_db_key_for_index(&KeyPrefixes::PublicKey, index)[..] - ).unwrap().unwrap(); + let public_key_at_index = db + .get( + DB_COLUMN, + &store.get_db_key_for_index(&KeyPrefixes::PublicKey, index)[..], + ) + .unwrap() + .unwrap(); - assert_eq!(public_key_at_index, public_key.as_bytes()); + assert_eq!(public_key_at_index, ssz_encode(&public_key)); } #[test] @@ -129,8 +139,9 @@ mod tests { db.put( DB_COLUMN, &store.get_db_key_for_index(&KeyPrefixes::PublicKey, index)[..], - &public_key.as_bytes()[..] - ).unwrap(); + &ssz_encode(&public_key)[..], + ) + .unwrap(); let public_key_at_index = store.get_public_key_by_index(index).unwrap().unwrap(); assert_eq!(public_key_at_index, public_key); @@ -146,8 +157,9 @@ mod tests { db.put( DB_COLUMN, &store.get_db_key_for_index(&KeyPrefixes::PublicKey, 3)[..], - &public_key.as_bytes()[..] - ).unwrap(); + &ssz_encode(&public_key)[..], + ) + .unwrap(); let public_key_at_index = store.get_public_key_by_index(4).unwrap(); assert_eq!(public_key_at_index, None); @@ -195,11 +207,9 @@ mod tests { /* * Check that an index that wasn't stored returns None. */ - assert!( - store - .get_public_key_by_index(keys.len() + 1) - .unwrap() - .is_none() - ); + assert!(store + .get_public_key_by_index(keys.len() + 1) + .unwrap() + .is_none()); } } From 9efc7a0def7db48bf7030d944356a9655fb2badb Mon Sep 17 00:00:00 2001 From: Paul Hauner Date: Sat, 29 Dec 2018 14:33:48 +1100 Subject: [PATCH 08/10] Remove unused types::common mod --- beacon_chain/types/src/common/maps.rs | 7 ------- beacon_chain/types/src/common/mod.rs | 9 --------- 2 files changed, 16 deletions(-) delete mode 100644 beacon_chain/types/src/common/maps.rs delete mode 100644 beacon_chain/types/src/common/mod.rs diff --git a/beacon_chain/types/src/common/maps.rs b/beacon_chain/types/src/common/maps.rs deleted file mode 100644 index 6b7c5da19..000000000 --- a/beacon_chain/types/src/common/maps.rs +++ /dev/null @@ -1,7 +0,0 @@ -use std::collections::HashMap; - -/// Maps a (slot, shard_id) to attestation_indices. -pub type AttesterMap = HashMap<(u64, u16), Vec>; - -/// Maps a slot to a block proposer. -pub type ProposerMap = HashMap; diff --git a/beacon_chain/types/src/common/mod.rs b/beacon_chain/types/src/common/mod.rs deleted file mode 100644 index 76d294b23..000000000 --- a/beacon_chain/types/src/common/mod.rs +++ /dev/null @@ -1,9 +0,0 @@ -mod delegation; -mod shuffling; - -pub mod maps; -pub mod attestation_parent_hashes; - -use super::utils; -use super::utils::types::Hash256; -pub use self::shuffling::shuffle; From b365bb87739f4427a26652f88403e401f2ec8dee Mon Sep 17 00:00:00 2001 From: Paul Hauner Date: Sat, 29 Dec 2018 14:34:22 +1100 Subject: [PATCH 09/10] Re-add ValidatorRecord::status_is() fn It was accidentally removed. --- beacon_chain/types/src/validator_record.rs | 6 ++++++ beacon_chain/validator_induction/src/inductor.rs | 15 +++++++++------ beacon_chain/validator_shuffling/src/shuffle.rs | 2 +- 3 files changed, 16 insertions(+), 7 deletions(-) diff --git a/beacon_chain/types/src/validator_record.rs b/beacon_chain/types/src/validator_record.rs index 7da858676..45b6c301a 100644 --- a/beacon_chain/types/src/validator_record.rs +++ b/beacon_chain/types/src/validator_record.rs @@ -41,6 +41,12 @@ pub struct ValidatorRecord { pub exit_slot: u64, } +impl ValidatorRecord { + pub fn status_is(&self, status: ValidatorStatus) -> bool { + self.status == status + } +} + impl Encodable for ValidatorStatus { fn ssz_append(&self, s: &mut SszStream) { let byte: u8 = match self { diff --git a/beacon_chain/validator_induction/src/inductor.rs b/beacon_chain/validator_induction/src/inductor.rs index 79d8fc256..3d3b15ce3 100644 --- a/beacon_chain/validator_induction/src/inductor.rs +++ b/beacon_chain/validator_induction/src/inductor.rs @@ -110,8 +110,8 @@ impl ValidatorInductor { mod tests { use super::*; - use bls::{create_proof_of_possession, Keypair, Signature}; - use hashing::canonical_hash; + use bls::{create_proof_of_possession, Keypair}; + use types::test_utils::{SeedableRng, TestRandom, XorShiftRng}; use types::{Address, Hash256}; fn registration_equals_record(reg: &ValidatorRegistration, rec: &ValidatorRecord) -> bool { @@ -167,9 +167,10 @@ mod tests { #[test] fn test_validator_inductor_valid_all_active_validators() { + let mut rng = XorShiftRng::from_seed([42; 16]); let mut validators = vec![]; for _ in 0..5 { - let (mut v, _) = ValidatorRecord::zero_with_thread_rand_keypair(); + let mut v = ValidatorRecord::random_for_test(&mut rng); v.status = ValidatorStatus::Active; validators.push(v); } @@ -187,12 +188,13 @@ mod tests { #[test] fn test_validator_inductor_valid_all_second_validator_withdrawn() { + let mut rng = XorShiftRng::from_seed([42; 16]); let mut validators = vec![]; - let (mut v, _) = ValidatorRecord::zero_with_thread_rand_keypair(); + let mut v = ValidatorRecord::random_for_test(&mut rng); v.status = ValidatorStatus::Active; validators.push(v); for _ in 0..4 { - let (mut v, _) = ValidatorRecord::zero_with_thread_rand_keypair(); + let mut v = ValidatorRecord::random_for_test(&mut rng); v.status = ValidatorStatus::Withdrawn; validators.push(v); } @@ -210,9 +212,10 @@ mod tests { #[test] fn test_validator_inductor_valid_all_withdrawn_validators() { + let mut rng = XorShiftRng::from_seed([42; 16]); let mut validators = vec![]; for _ in 0..5 { - let (mut v, _) = ValidatorRecord::zero_with_thread_rand_keypair(); + let mut v = ValidatorRecord::random_for_test(&mut rng); v.status = ValidatorStatus::Withdrawn; validators.push(v); } diff --git a/beacon_chain/validator_shuffling/src/shuffle.rs b/beacon_chain/validator_shuffling/src/shuffle.rs index ee7389949..5b31f8e03 100644 --- a/beacon_chain/validator_shuffling/src/shuffle.rs +++ b/beacon_chain/validator_shuffling/src/shuffle.rs @@ -95,7 +95,7 @@ fn generate_cycle( .honey_badger_split(committees_per_slot) .enumerate() .map(|(j, shard_indices)| ShardCommittee { - shard: ((shard_start + j) % shard_count) as u16, + shard: ((shard_start + j) % shard_count) as u64, committee: shard_indices.to_vec(), }) .collect() From 8308ea4932d4ce4d320d3c91a08aad1cb31d931b Mon Sep 17 00:00:00 2001 From: Paul Hauner Date: Sat, 29 Dec 2018 14:35:03 +1100 Subject: [PATCH 10/10] Fix old instances of shards being u16 --- beacon_chain/chain/src/maps.rs | 2 +- beacon_chain/types/src/lib.rs | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/beacon_chain/chain/src/maps.rs b/beacon_chain/chain/src/maps.rs index cecd87c8d..4c737cb44 100644 --- a/beacon_chain/chain/src/maps.rs +++ b/beacon_chain/chain/src/maps.rs @@ -45,7 +45,7 @@ mod tests { use super::*; fn sac_generator( - shard_count: u16, + shard_count: u64, slot_count: usize, sac_per_slot: usize, committee_size: usize, diff --git a/beacon_chain/types/src/lib.rs b/beacon_chain/types/src/lib.rs index a4f52701a..08b871c1c 100644 --- a/beacon_chain/types/src/lib.rs +++ b/beacon_chain/types/src/lib.rs @@ -65,7 +65,7 @@ pub type Bitfield = boolean_bitfield::BooleanBitfield; pub type BitfieldError = boolean_bitfield::Error; /// Maps a (slot, shard_id) to attestation_indices. -pub type AttesterMap = HashMap<(u64, u16), Vec>; +pub type AttesterMap = HashMap<(u64, u64), Vec>; /// Maps a slot to a block proposer. pub type ProposerMap = HashMap;