Commit Graph

30 Commits

Author SHA1 Message Date
Kirk Baird
611a0c7d19
Arbitrary trait for eth2/types (#1040)
* Add the arbitrary type to eth2/types and their deps

Signed-off-by: Kirk Baird <baird.k@outlook.com>

* Wrap arbitrary in a feature flag

Signed-off-by: Kirk Baird <baird.k@outlook.com>

* Fix feature for types

Signed-off-by: Kirk Baird <baird.k@outlook.com>

* Fix comment

Signed-off-by: Kirk Baird <baird.k@outlook.com>

* Patch versioning

Signed-off-by: Kirk Baird <baird.k@outlook.com>

* Allow expanded crate reference for arbitrary 0.4.3

Signed-off-by: Kirk Baird <baird.k@outlook.com>

* Add arbitrary to remaining types

Signed-off-by: Kirk Baird <baird.k@outlook.com>

* use cmp::min

Signed-off-by: Kirk Baird <baird.k@outlook.com>

* Derive Arbitrary trait for ValidatorStatus, TotalBalances and InclusionInfo

* Add CI check for state processing arbitrary faetures

Signed-off-by: Kirk Baird <baird.k@outlook.com>

* Fix indentation

Signed-off-by: Kirk Baird <baird.k@outlook.com>

Co-authored-by: Mehdi Zerouali <mehdi@sigmaprime.io>
2020-05-06 09:12:28 +10:00
Kirk Baird
c102d9d1f9
Update SecretKey to 32 bytes (#1072)
* Update SecretKey to 32 bytes

Signed-off-by: Kirk Baird <baird.k@outlook.com>

* Stop test from padding to 48 bytes

Signed-off-by: Kirk Baird <baird.k@outlook.com>

* Fix keypair file secret key length

Signed-off-by: Kirk Baird <baird.k@outlook.com>
2020-04-28 18:27:33 +10:00
Kirk Baird
1abb54dabd
Milagro BLS update (#985)
* Start updating types

* WIP

* Signature hacking

* Existing EF tests passing with fake_crypto

* Updates

* Delete outdated API spec

* The refactor continues

* It compiles

* WIP test fixes

* All release tests passing bar genesis state parsing

* Update and test YamlConfig

* Update to spec v0.10 compatible BLS

* Updates to BLS EF tests

* Add EF test for AggregateVerify

And delete unused hash2curve tests for uncompressed points

* Update EF tests to v0.10.1

* Use optional block root correctly in block proc

* Use genesis fork in deposit domain. All tests pass

* Cargo fmt

* Fast aggregate verify test

* Update REST API docs

* Cargo fmt

* Fix unused import

* Bump spec tags to v0.10.1

* Add `seconds_per_eth1_block` to chainspec

* Update to timestamp based eth1 voting scheme

* Return None from `get_votes_to_consider` if block cache is empty

* Handle overflows in `is_candidate_block`

* Revert to failing tests

* Fix eth1 data sets test

* Choose default vote according to spec

* Fix collect_valid_votes tests

* Fix `get_votes_to_consider` to choose all eligible blocks

* Uncomment winning_vote tests

* Add comments; remove unused code

* Reduce seconds_per_eth1_block for simulation

* Addressed review comments

* Add test for default vote case

* Fix logs

* Remove unused functions

* Meter default eth1 votes

* Fix comments

* Address review comments; remove unused dependency

* Add first attempt at attestation proc. re-write

* Add version 2 of attestation processing

* Minor fixes

* Add validator pubkey cache

* Make get_indexed_attestation take a committee

* Link signature processing into new attn verification

* First working version

* Ensure pubkey cache is updated

* Add more metrics, slight optimizations

* Clone committee cache during attestation processing

* Update shuffling cache during block processing

* Remove old commented-out code

* Fix shuffling cache insert bug

* Used indexed attestation in fork choice

* Restructure attn processing, add metrics

* Add more detailed metrics

* Tidy, fix failing tests

* Fix failing tests, tidy

* Disable/delete two outdated tests

* Add new Pubkeys struct to signature_sets

* Refactor with functional approach

* Update beacon chain

* Remove decompressed member from pubkey bytes

* Add hashmap for indices lookup

* Change `get_attesting_indices` to use Vec

* Fix failing test

* Tidy

* Add pubkey cache persistence file

* Add more comments

* Integrate persistence file into builder

* Add pubkey cache tests

* Add data_dir to beacon chain builder

* Remove Option in pubkey cache persistence file

* Ensure consistency between datadir/data_dir

* Fix failing network test

* Tidy

* Fix todos

* Improve tests

* Split up block processing metrics

* Tidy

* Refactor get_pubkey_from_state

* Remove commented-out code

* Add BeaconChain::validator_pubkey

* Update milagro_bls

Signed-off-by: Kirk Baird <baird.k@outlook.com>

* Cargo fmt

Signed-off-by: Kirk Baird <baird.k@outlook.com>

* Use Option::filter

* Remove Box

* Comment out tests that fail due to hard-coded

* Fix fake crypto

Signed-off-by: Kirk Baird <baird.k@outlook.com>

* Fix Cow::Borrowed

Signed-off-by: Kirk Baird <baird.k@outlook.com>

* Cargo fmt

Signed-off-by: Kirk Baird <baird.k@outlook.com>

Co-authored-by: Michael Sproul <michael@sigmaprime.io>
Co-authored-by: Michael Sproul <micsproul@gmail.com>
Co-authored-by: pawan <pawandhananjay@gmail.com>
Co-authored-by: Paul Hauner <paul@paulhauner.com>
2020-04-28 13:15:46 +10:00
Age Manning
95c8e476bc
Initial work towards v0.2.0 (#924)
* Remove ping protocol

* Initial renaming of network services

* Correct rebasing relative to latest master

* Start updating types

* Adds HashMapDelay struct to utils

* Initial network restructure

* Network restructure. Adds new types for v0.2.0

* Removes build artefacts

* Shift validation to beacon chain

* Temporarily remove gossip validation

This is to be updated to match current optimisation efforts.

* Adds AggregateAndProof

* Begin rebuilding pubsub encoding/decoding

* Signature hacking

* Shift gossipsup decoding into eth2_libp2p

* Existing EF tests passing with fake_crypto

* Shifts block encoding/decoding into RPC

* Delete outdated API spec

* All release tests passing bar genesis state parsing

* Update and test YamlConfig

* Update to spec v0.10 compatible BLS

* Updates to BLS EF tests

* Add EF test for AggregateVerify

And delete unused hash2curve tests for uncompressed points

* Update EF tests to v0.10.1

* Use optional block root correctly in block proc

* Use genesis fork in deposit domain. All tests pass

* Fast aggregate verify test

* Update REST API docs

* Fix unused import

* Bump spec tags to v0.10.1

* Add `seconds_per_eth1_block` to chainspec

* Update to timestamp based eth1 voting scheme

* Return None from `get_votes_to_consider` if block cache is empty

* Handle overflows in `is_candidate_block`

* Revert to failing tests

* Fix eth1 data sets test

* Choose default vote according to spec

* Fix collect_valid_votes tests

* Fix `get_votes_to_consider` to choose all eligible blocks

* Uncomment winning_vote tests

* Add comments; remove unused code

* Reduce seconds_per_eth1_block for simulation

* Addressed review comments

* Add test for default vote case

* Fix logs

* Remove unused functions

* Meter default eth1 votes

* Fix comments

* Progress on attestation service

* Address review comments; remove unused dependency

* Initial work on removing libp2p lock

* Add LRU caches to store (rollup)

* Update attestation validation for DB changes (WIP)

* Initial version of should_forward_block

* Scaffold

* Progress on attestation validation

Also, consolidate prod+testing slot clocks so that they share much
of the same implementation and can both handle sub-slot time changes.

* Removes lock from libp2p service

* Completed network lock removal

* Finish(?) attestation processing

* Correct network termination future

* Add slot check to block check

* Correct fmt issues

* Remove Drop implementation for network service

* Add first attempt at attestation proc. re-write

* Add version 2 of attestation processing

* Minor fixes

* Add validator pubkey cache

* Make get_indexed_attestation take a committee

* Link signature processing into new attn verification

* First working version

* Ensure pubkey cache is updated

* Add more metrics, slight optimizations

* Clone committee cache during attestation processing

* Update shuffling cache during block processing

* Remove old commented-out code

* Fix shuffling cache insert bug

* Used indexed attestation in fork choice

* Restructure attn processing, add metrics

* Add more detailed metrics

* Tidy, fix failing tests

* Fix failing tests, tidy

* Address reviewers suggestions

* Disable/delete two outdated tests

* Modification of validator for subscriptions

* Add slot signing to validator client

* Further progress on validation subscription

* Adds necessary validator subscription functionality

* Add new Pubkeys struct to signature_sets

* Refactor with functional approach

* Update beacon chain

* Clean up validator <-> beacon node http types

* Add aggregator status to ValidatorDuty

* Impl Clone for manual slot clock

* Fix minor errors

* Further progress validator client subscription

* Initial subscription and aggregation handling

* Remove decompressed member from pubkey bytes

* Progress to modifying val client for attestation aggregation

* First draft of validator client upgrade for aggregate attestations

* Add hashmap for indices lookup

* Add state cache, remove store cache

* Only build the head committee cache

* Removes lock on a network channel

* Partially implement beacon node subscription http api

* Correct compilation issues

* Change `get_attesting_indices` to use Vec

* Fix failing test

* Partial implementation of timer

* Adds timer, removes exit_future, http api to op pool

* Partial multiple aggregate attestation handling

* Permits bulk messages accross gossipsub network channel

* Correct compile issues

* Improve gosispsub messaging and correct rest api helpers

* Added global gossipsub subscriptions

* Update validator subscriptions data structs

* Tidy

* Re-structure validator subscriptions

* Initial handling of subscriptions

* Re-structure network service

* Add pubkey cache persistence file

* Add more comments

* Integrate persistence file into builder

* Add pubkey cache tests

* Add HashSetDelay and introduce into attestation service

* Handles validator subscriptions

* Add data_dir to beacon chain builder

* Remove Option in pubkey cache persistence file

* Ensure consistency between datadir/data_dir

* Fix failing network test

* Peer subnet discovery gets queued for future subscriptions

* Reorganise attestation service functions

* Initial wiring of attestation service

* First draft of attestation service timing logic

* Correct minor typos

* Tidy

* Fix todos

* Improve tests

* Add PeerInfo to connected peers mapping

* Fix compile error

* Fix compile error from merge

* Split up block processing metrics

* Tidy

* Refactor get_pubkey_from_state

* Remove commented-out code

* Rename state_cache -> checkpoint_cache

* Rename Checkpoint -> Snapshot

* Tidy, add comments

* Tidy up find_head function

* Change some checkpoint -> snapshot

* Add tests

* Expose max_len

* Remove dead code

* Tidy

* Fix bug

* Add sync-speed metric

* Add first attempt at VerifiableBlock

* Start integrating into beacon chain

* Integrate VerifiableBlock

* Rename VerifableBlock -> PartialBlockVerification

* Add start of typed methods

* Add progress

* Add further progress

* Rename structs

* Add full block verification to block_processing.rs

* Further beacon chain integration

* Update checks for gossip

* Add todo

* Start adding segement verification

* Add passing chain segement test

* Initial integration with batch sync

* Minor changes

* Tidy, add more error checking

* Start adding chain_segment tests

* Finish invalid signature tests

* Include single and gossip verified blocks in tests

* Add gossip verification tests

* Start adding docs

* Finish adding comments to block_processing.rs

* Rename block_processing.rs -> block_verification

* Start removing old block processing code

* Fixes beacon_chain compilation

* Fix project-wide compile errors

* Remove old code

* Correct code to pass all tests

* Fix bug with beacon proposer index

* Fix shim for BlockProcessingError

* Only process one epoch at a time

* Fix loop in chain segment processing

* Correct tests from master merge

* Add caching for state.eth1_data_votes

* Add BeaconChain::validator_pubkey

* Revert "Add caching for state.eth1_data_votes"

This reverts commit cd73dcd6434fb8d8e6bf30c5356355598ea7b78e.

Co-authored-by: Grant Wuerker <gwuerker@gmail.com>
Co-authored-by: Michael Sproul <michael@sigmaprime.io>
Co-authored-by: Michael Sproul <micsproul@gmail.com>
Co-authored-by: pawan <pawandhananjay@gmail.com>
Co-authored-by: Paul Hauner <paul@paulhauner.com>
2020-03-17 17:24:44 +11:00
Michael Sproul
371e5adcf8
Update to Spec v0.10 (#817)
* Start updating types

* WIP

* Signature hacking

* Existing EF tests passing with fake_crypto

* Updates

* Delete outdated API spec

* The refactor continues

* It compiles

* WIP test fixes

* All release tests passing bar genesis state parsing

* Update and test YamlConfig

* Update to spec v0.10 compatible BLS

* Updates to BLS EF tests

* Add EF test for AggregateVerify

And delete unused hash2curve tests for uncompressed points

* Update EF tests to v0.10.1

* Use optional block root correctly in block proc

* Use genesis fork in deposit domain. All tests pass

* Cargo fmt

* Fast aggregate verify test

* Update REST API docs

* Cargo fmt

* Fix unused import

* Bump spec tags to v0.10.1

* Add `seconds_per_eth1_block` to chainspec

* Update to timestamp based eth1 voting scheme

* Return None from `get_votes_to_consider` if block cache is empty

* Handle overflows in `is_candidate_block`

* Revert to failing tests

* Fix eth1 data sets test

* Choose default vote according to spec

* Fix collect_valid_votes tests

* Fix `get_votes_to_consider` to choose all eligible blocks

* Uncomment winning_vote tests

* Add comments; remove unused code

* Reduce seconds_per_eth1_block for simulation

* Addressed review comments

* Add test for default vote case

* Fix logs

* Remove unused functions

* Meter default eth1 votes

* Fix comments

* Address review comments; remove unused dependency

* Disable/delete two outdated tests

* Bump eth1 default vote warn to error

* Delete outdated eth1 test

Co-authored-by: Pawan Dhananjay <pawandhananjay@gmail.com>
2020-02-11 10:19:36 +11:00
Kirk Baird
cd401147ea Bump bls version (#768)
Signed-off-by: Kirk Baird <baird.k@outlook.com>
2020-01-28 15:19:45 +11:00
Age Manning
5c97ed3562
Updates external dependencies (#577)
* Updates external dependencies

* Correct fmt formatting
2019-10-30 12:22:18 +11:00
Michael Sproul
96a327fe50
Update Milagro BLS to v0.11.0 (#554) 2019-10-02 12:04:17 +10:00
Michael Sproul
24ebad9168
Merge remote-tracking branch 'origin/master' into v0.8.3 2019-09-05 12:27:58 +10:00
Michael Sproul
81cafdc804
Shuffling and sanity tests 2019-08-29 17:41:20 +10:00
Paul Hauner
bcffe42712
Bulk signature verification (#507)
* Add basic block processing benches

* Start reviving state processing benches

* Fix old block builders

* Add optimization for faster pubkey add

* Tidy benches, add another

* Add extra block processing bench

* Start working on faster BLS scheme

* Add partially complete sig verify optimization

* Add .gitignore to state processing

* Add progress on faster signature verification

* Fix SignatureSet for fake_crypto

* Tidy attester slashings sig set

* Tidy bulk signature verifier

* Refactor signature sets to be cleaner

* Start threading SignatureStrategy through code

* Add (empty) test dir

* Move BenchingBlockBuilder

* Add initial block signature verification tests

* Add tests for bulk signature verification

* Start threading SignatureStrategy in block proc.

* Refactor per_block_processing errors

* Use sig set tuples instead of lists of two

* Remove dead code

* Thread VerifySignatures through per_block_processing

* Add bulk signature verification

* Introduce parallel bulk signature verification

* Expand state processing benches

* Fix additional compile errors

* Fix issue where par iter chunks is 0

* Update milagro_bls dep

* Remove debugs, code fragment in beacon chain

* Tidy, add comments to block sig verifier

* Fix various PR comments

* Add block_root option to per_block_processing

* Fix comment in block signature verifier

* Fix comments from PR review

* Remove old comment

* Fix comment
2019-08-29 11:34:25 +10:00
Paul Hauner
8cfa36fedd
Publish ssz_types (and deps) to crates.io (#468)
* Rename `hashing` crate to `eth2_hashing`

* Add license, desc to eth2_hashing Cargo.toml

* Remove merkle root from eth2 hashing

* Remove old benches folder (zombied from old branch)

* Add docs to eth2_hashing

* Prepare tree_hash for publishing on crates.io

* Update deps to use crates.io instead of paths

* Update all crates to pull ssz from crates.io

* Remove cached_tree_hash, add patches to manifest

* Fix compile error in benches

* Remove unused code

* Fix fake_crypto compile error
2019-08-08 11:39:47 +10:00
Michael Sproul
a236003a7b Update to frozen spec ❄️ (v0.8.1) (#444)
* types: first updates for v0.8

* state_processing: epoch processing v0.8.0

* state_processing: block processing v0.8.0

* tree_hash_derive: support generics in SignedRoot

* types v0.8: update to use ssz_types

* state_processing v0.8: use ssz_types

* ssz_types: add bitwise methods and from_elem

* types: fix v0.8 FIXMEs

* ssz_types: add bitfield shift_up

* ssz_types: iterators and DerefMut for VariableList

* types,state_processing: use VariableList

* ssz_types: fix BitVector Decode impl

Fixed a typo in the implementation of ssz::Decode for BitVector, which caused it
to be considered variable length!

* types: fix test modules for v0.8 update

* types: remove slow type-level arithmetic

* state_processing: fix tests for v0.8

* op_pool: update for v0.8

* ssz_types: Bitfield difference length-independent

Allow computing the difference of two bitfields of different lengths.

* Implement compact committee support

* epoch_processing: committee & active index roots

* state_processing: genesis state builder v0.8

* state_processing: implement v0.8.1

* Further improve tree_hash

* Strip examples, tests from cached_tree_hash

* Update TreeHash, un-impl CachedTreeHash

* Update bitfield TreeHash, un-impl CachedTreeHash

* Update FixedLenVec TreeHash, unimpl CachedTreeHash

* Update update tree_hash_derive for new TreeHash

* Fix TreeHash, un-impl CachedTreeHash for ssz_types

* Remove fixed_len_vec, ssz benches

SSZ benches relied upon fixed_len_vec -- it is easier to just delete
them and rebuild them later (when necessary)

* Remove boolean_bitfield crate

* Fix fake_crypto BLS compile errors

* Update ef_tests for new v.8 type params

* Update ef_tests submodule to v0.8.1 tag

* Make fixes to support parsing ssz ef_tests

* `compact_committee...` to `compact_committees...`

* Derive more traits for `CompactCommittee`

* Flip bitfield byte-endianness

* Fix tree_hash for bitfields

* Modify CLI output for ef_tests

* Bump ssz crate version

* Update ssz_types doc comment

* Del cached tree hash tests from ssz_static tests

* Tidy SSZ dependencies

* Rename ssz_types crate to eth2_ssz_types

* validator_client: update for v0.8

* ssz_types: update union/difference for bit order swap

* beacon_node: update for v0.8, EthSpec

* types: disable cached tree hash, update min spec

* state_processing: fix slot bug in committee update

* tests: temporarily disable fork choice harness test

See #447

* committee cache: prevent out-of-bounds access

In the case where we tried to access the committee of a shard that didn't have a committee in the
current epoch, we were accessing elements beyond the end of the shuffling vector and panicking! This
commit adds a check to make the failure safe and explicit.

* fix bug in get_indexed_attestation and simplify

There was a bug in our implementation of get_indexed_attestation whereby
incorrect "committee indices" were used to index into the custody bitfield. The
bug was only observable in the case where some bits of the custody bitfield were
set to 1. The implementation has been simplified to remove the bug, and a test
added.

* state_proc: workaround for compact committees bug

https://github.com/ethereum/eth2.0-specs/issues/1315

* v0.8: updates to make the EF tests pass

* Remove redundant max operation checks.
* Always supply both messages when checking attestation signatures -- allowing
  verification of an attestation with no signatures.
* Swap the order of the fork and domain constant in `get_domain`, to match
  the spec.

* rustfmt

* ef_tests: add new epoch processing tests

* Integrate v0.8 into master (compiles)

* Remove unused crates, fix clippy lints

* Replace v0.6.3 tags w/ v0.8.1

* Remove old comment

* Ensure lmd ghost tests only run in release

* Update readme
2019-07-30 12:44:51 +10:00
Paul Hauner
027f0a539d
Prepare ssz for publishing on crates.io 2019-07-03 16:06:20 +10:00
Paul Hauner
10a7936c37
Update BLS library 2019-06-03 13:26:22 +10:00
Paul Hauner
f65e981f6f
Begin implementing cached hashing in types 2019-04-26 15:24:18 +10:00
Michael Sproul
71a0fed8eb
testing: add a fake_crypto feature 2019-04-17 12:00:22 +10:00
Paul Hauner
b8c4c3308a
Update types to new tree_hash crate 2019-04-16 14:14:38 +10:00
Paul Hauner
236b97476a
Marge fixes to test_harness, add serdehex crate 2019-03-15 13:31:30 +11:00
Paul Hauner
65e3b388a0
Update signature-scheme to v0.6.1 2019-03-14 18:17:32 +11:00
Kirk Baird
c92f867cd8
Upgrade to signature scheme 0.6.0 2019-03-12 16:01:09 +11:00
Paul Hauner
3b6431b4b4
Impl serde ser and deser for bls keypairs 2019-03-08 13:15:41 +11:00
Kirk Baird
9c4a1f1d1f
Update to signature-scheme 0.5.2 2019-02-18 10:50:40 +11:00
Kirk Baird
977f3edfb6
Add domain to all signature funcitons, modify validate_proof_of_possession() 2019-02-15 13:58:14 +11:00
Paul Hauner
35c914baa6
Revert "Merge pull request #200 from sigp/new-structure"
This reverts commit d7a3545be1, reversing
changes made to 1da06c156c.
2019-02-14 12:09:18 +11:00
Paul Hauner
1d5ff4359a
Strip out old code
All of these files have been moved to either:

- https://github.com/sigp/lighthouse-beacon
- https://github.com/sigp/lighthouse-validator
- https://github.com/sigp/lighthouse-common

For rationale, see: https://github.com/sigp/lighthouse/issues/197
2019-02-13 14:15:53 +11:00
Paul Hauner
0120082f6b
Merge branch 'master' into add-chain-benchmarks 2019-02-05 05:49:45 +11:00
Grant Wuerker
0cbc2936ea
added version to signature-schemes dependency
using tag instead of branch

changed signature-schemes tag to 0.3.0
2019-01-27 21:12:00 -06:00
Paul Hauner
90ae2298ab
Impl serde::Serialize for all types 2019-01-26 08:25:31 +11:00
Paul Hauner
e16f9e0aec
Rename beacon_chain/ -> eth2/ 2019-01-22 16:16:02 +11:00