Commit Graph

13 Commits

Author SHA1 Message Date
Kirk Baird
611a0c7d19
Arbitrary trait for eth2/types (#1040)
* Add the arbitrary type to eth2/types and their deps

Signed-off-by: Kirk Baird <baird.k@outlook.com>

* Wrap arbitrary in a feature flag

Signed-off-by: Kirk Baird <baird.k@outlook.com>

* Fix feature for types

Signed-off-by: Kirk Baird <baird.k@outlook.com>

* Fix comment

Signed-off-by: Kirk Baird <baird.k@outlook.com>

* Patch versioning

Signed-off-by: Kirk Baird <baird.k@outlook.com>

* Allow expanded crate reference for arbitrary 0.4.3

Signed-off-by: Kirk Baird <baird.k@outlook.com>

* Add arbitrary to remaining types

Signed-off-by: Kirk Baird <baird.k@outlook.com>

* use cmp::min

Signed-off-by: Kirk Baird <baird.k@outlook.com>

* Derive Arbitrary trait for ValidatorStatus, TotalBalances and InclusionInfo

* Add CI check for state processing arbitrary faetures

Signed-off-by: Kirk Baird <baird.k@outlook.com>

* Fix indentation

Signed-off-by: Kirk Baird <baird.k@outlook.com>

Co-authored-by: Mehdi Zerouali <mehdi@sigmaprime.io>
2020-05-06 09:12:28 +10:00
Paul Hauner
54782d896c
Use global pubkey cache for block processing (#849)
* Start updating types

* WIP

* Signature hacking

* Existing EF tests passing with fake_crypto

* Updates

* Delete outdated API spec

* The refactor continues

* It compiles

* WIP test fixes

* All release tests passing bar genesis state parsing

* Update and test YamlConfig

* Update to spec v0.10 compatible BLS

* Updates to BLS EF tests

* Add EF test for AggregateVerify

And delete unused hash2curve tests for uncompressed points

* Update EF tests to v0.10.1

* Use optional block root correctly in block proc

* Use genesis fork in deposit domain. All tests pass

* Cargo fmt

* Fast aggregate verify test

* Update REST API docs

* Cargo fmt

* Fix unused import

* Bump spec tags to v0.10.1

* Add `seconds_per_eth1_block` to chainspec

* Update to timestamp based eth1 voting scheme

* Return None from `get_votes_to_consider` if block cache is empty

* Handle overflows in `is_candidate_block`

* Revert to failing tests

* Fix eth1 data sets test

* Choose default vote according to spec

* Fix collect_valid_votes tests

* Fix `get_votes_to_consider` to choose all eligible blocks

* Uncomment winning_vote tests

* Add comments; remove unused code

* Reduce seconds_per_eth1_block for simulation

* Addressed review comments

* Add test for default vote case

* Fix logs

* Remove unused functions

* Meter default eth1 votes

* Fix comments

* Address review comments; remove unused dependency

* Add first attempt at attestation proc. re-write

* Add version 2 of attestation processing

* Minor fixes

* Add validator pubkey cache

* Make get_indexed_attestation take a committee

* Link signature processing into new attn verification

* First working version

* Ensure pubkey cache is updated

* Add more metrics, slight optimizations

* Clone committee cache during attestation processing

* Update shuffling cache during block processing

* Remove old commented-out code

* Fix shuffling cache insert bug

* Used indexed attestation in fork choice

* Restructure attn processing, add metrics

* Add more detailed metrics

* Tidy, fix failing tests

* Fix failing tests, tidy

* Disable/delete two outdated tests

* Add new Pubkeys struct to signature_sets

* Refactor with functional approach

* Update beacon chain

* Remove decompressed member from pubkey bytes

* Add hashmap for indices lookup

* Change `get_attesting_indices` to use Vec

* Fix failing test

* Tidy

* Add pubkey cache persistence file

* Add more comments

* Integrate persistence file into builder

* Add pubkey cache tests

* Add data_dir to beacon chain builder

* Remove Option in pubkey cache persistence file

* Ensure consistency between datadir/data_dir

* Fix failing network test

* Tidy

* Fix todos

* Improve tests

* Split up block processing metrics

* Tidy

* Refactor get_pubkey_from_state

* Remove commented-out code

* Add BeaconChain::validator_pubkey

* Use Option::filter

* Remove Box

* Comment out tests that fail due to hard-coded

Co-authored-by: Michael Sproul <michael@sigmaprime.io>
Co-authored-by: Michael Sproul <micsproul@gmail.com>
Co-authored-by: pawan <pawandhananjay@gmail.com>
2020-04-06 14:13:19 +10:00
Paul Hauner
7f6ae4c2f5
Refactor tree hashing (#861)
* Pre-allocated tree hash caches

* Add SmallVec to tree hash cache

* Avoid allocation for validator.pubkey

* Avoid iterator which seems to be doing heap alloc

* Add more smallvecs

* MOAR SMALLVEC

* Move non-test code to Hash256 tree hash

* Fix byte ordering error

* Add incomplete but working merkle stream impl

* Fix zero hash error

* Add zero hash fn

* Add MerkleStream comments

* Add smallvec, tidy

* Integrate into tree hash derive

* Update ssz_types tree hash

* Don't heap alloc for mix in length

* Add byte-level streaming to MerkleStream

* Avoid recursion in write method

* Update BLS to MerkleStream

* Fix some not-compiling tests

* Remove debug profiling

* Remove code duplication

* Move beacon state tree hash to new hasher

* Fix failing tests

* Update comments

* Add some fast-paths to tree_hash::merkle_root

* Remove unncessary test

* Rename MerkleStream -> MerkleHasher

* Rename new_with_leaf_count -> with_leaves

* Tidy

* Remove NonZeroUsize

* Remove todo

* Update smallvec
2020-03-05 08:07:27 +11:00
Paul Hauner
370c658c7c
Optimization: reduce BLS decompression (#766)
* Add RwLock-style caching for BLS pubkeys

* Tidy docker ignore

* Remove RwLocks

* Merge in master
2020-01-10 15:32:10 +11:00
Paul Hauner
251aea645c
Make API friendly to block explorers (#702)
* Add validator index to duties response

* Add `get_state` method to beacon chain

* Improve /beacon/validators endpoint

* Add validators/all and validators/active endpoints

* Start refactor of HTTP docs

* Document /beacon/heads endpoint

* Remove some unused API endpoints

* Improve API docs

* Add methods to get all validator duties

* Improve docs

* Remove dead links

* Make tables left-justified

* Add /consensus/vote_count endpoint

* Add /consensus/individual_votes endpoint

* Update formatting

* Tidy

* Add committees endpoint

* Strictly require 0x prefix for serde in BLS

* Update docs to have 0x prefix

* Fix failing tests

* Add unfinished code

* Improve testing, fix bugs

* Tidy, ensure all beacon endpoints smoke tested

* Fix pubkey cache error

* Address comments with docs
2019-12-19 11:45:28 +11:00
Paul Hauner
75efed305c
Faster BeaconState enc/dec (#671)
* Add state enc/dec benches

* Add example for flamegraph

* Use `PublicKeyBytes` for `Validator`

* Ripple PublicKeyBytes change through codebase

* Add benches, optimizations to store BeaconState

* Store BeaconState in StorageContainer too

* Optimize StorageContainer with std::mem magic

* Fix rest_api tests
2019-12-06 16:44:03 +11:00
Michael Sproul
c1a2238f1a
Implement tree hash caching (#584)
* Implement basic tree hash caching

* Use spaces to indent top-level Cargo.toml

* Optimize BLS tree hash by hashing bytes directly

* Implement tree hash caching for validator registry

* Persist BeaconState tree hash cache to disk

* Address Paul's review comments
2019-11-05 15:46:52 +11:00
Paul Hauner
6a870d468c
Add ssz_fixed_len method to ssz::Encode 2019-09-08 12:23:37 -04:00
Paul Hauner
8cfa36fedd
Publish ssz_types (and deps) to crates.io (#468)
* Rename `hashing` crate to `eth2_hashing`

* Add license, desc to eth2_hashing Cargo.toml

* Remove merkle root from eth2 hashing

* Remove old benches folder (zombied from old branch)

* Add docs to eth2_hashing

* Prepare tree_hash for publishing on crates.io

* Update deps to use crates.io instead of paths

* Update all crates to pull ssz from crates.io

* Remove cached_tree_hash, add patches to manifest

* Fix compile error in benches

* Remove unused code

* Fix fake_crypto compile error
2019-08-08 11:39:47 +10:00
blacktemplar
01054ecf2f Use SignatureBytes and PublicKeyBytes for deposits (#472)
* Replace deposit signatures with SignatureBytes, a struct which lazyly parsers signatures only on demand.

* check byte length when parsing SignatureBytes

* add comment to struct

* distinguish BadSignature and BadSignatureBytes in verify_deposit_signature

* add test for valid signature

* Implements TryInto<Signature> for &SignatureBytes and From<Signature> for &SignatureBytes

* add and use PublicKeyBytes + fix formatting

* fix compiler warning + docs for macro generated structs

* adds tests to ensure correct byte lengths

* small style improvement as suggested by michaelsproul
2019-08-06 13:49:11 +10:00
Michael Sproul
a236003a7b Update to frozen spec ❄️ (v0.8.1) (#444)
* types: first updates for v0.8

* state_processing: epoch processing v0.8.0

* state_processing: block processing v0.8.0

* tree_hash_derive: support generics in SignedRoot

* types v0.8: update to use ssz_types

* state_processing v0.8: use ssz_types

* ssz_types: add bitwise methods and from_elem

* types: fix v0.8 FIXMEs

* ssz_types: add bitfield shift_up

* ssz_types: iterators and DerefMut for VariableList

* types,state_processing: use VariableList

* ssz_types: fix BitVector Decode impl

Fixed a typo in the implementation of ssz::Decode for BitVector, which caused it
to be considered variable length!

* types: fix test modules for v0.8 update

* types: remove slow type-level arithmetic

* state_processing: fix tests for v0.8

* op_pool: update for v0.8

* ssz_types: Bitfield difference length-independent

Allow computing the difference of two bitfields of different lengths.

* Implement compact committee support

* epoch_processing: committee & active index roots

* state_processing: genesis state builder v0.8

* state_processing: implement v0.8.1

* Further improve tree_hash

* Strip examples, tests from cached_tree_hash

* Update TreeHash, un-impl CachedTreeHash

* Update bitfield TreeHash, un-impl CachedTreeHash

* Update FixedLenVec TreeHash, unimpl CachedTreeHash

* Update update tree_hash_derive for new TreeHash

* Fix TreeHash, un-impl CachedTreeHash for ssz_types

* Remove fixed_len_vec, ssz benches

SSZ benches relied upon fixed_len_vec -- it is easier to just delete
them and rebuild them later (when necessary)

* Remove boolean_bitfield crate

* Fix fake_crypto BLS compile errors

* Update ef_tests for new v.8 type params

* Update ef_tests submodule to v0.8.1 tag

* Make fixes to support parsing ssz ef_tests

* `compact_committee...` to `compact_committees...`

* Derive more traits for `CompactCommittee`

* Flip bitfield byte-endianness

* Fix tree_hash for bitfields

* Modify CLI output for ef_tests

* Bump ssz crate version

* Update ssz_types doc comment

* Del cached tree hash tests from ssz_static tests

* Tidy SSZ dependencies

* Rename ssz_types crate to eth2_ssz_types

* validator_client: update for v0.8

* ssz_types: update union/difference for bit order swap

* beacon_node: update for v0.8, EthSpec

* types: disable cached tree hash, update min spec

* state_processing: fix slot bug in committee update

* tests: temporarily disable fork choice harness test

See #447

* committee cache: prevent out-of-bounds access

In the case where we tried to access the committee of a shard that didn't have a committee in the
current epoch, we were accessing elements beyond the end of the shuffling vector and panicking! This
commit adds a check to make the failure safe and explicit.

* fix bug in get_indexed_attestation and simplify

There was a bug in our implementation of get_indexed_attestation whereby
incorrect "committee indices" were used to index into the custody bitfield. The
bug was only observable in the case where some bits of the custody bitfield were
set to 1. The implementation has been simplified to remove the bug, and a test
added.

* state_proc: workaround for compact committees bug

https://github.com/ethereum/eth2.0-specs/issues/1315

* v0.8: updates to make the EF tests pass

* Remove redundant max operation checks.
* Always supply both messages when checking attestation signatures -- allowing
  verification of an attestation with no signatures.
* Swap the order of the fork and domain constant in `get_domain`, to match
  the spec.

* rustfmt

* ef_tests: add new epoch processing tests

* Integrate v0.8 into master (compiles)

* Remove unused crates, fix clippy lints

* Replace v0.6.3 tags w/ v0.8.1

* Remove old comment

* Ensure lmd ghost tests only run in release

* Update readme
2019-07-30 12:44:51 +10:00
Paul Hauner
a0148b5aae
Rename SSZ traits (Encodable -> Encode) 2019-05-13 15:12:19 +10:00
Paul Hauner
5550d14d62
Upgrade BLS to new SSZ 2019-05-06 15:03:20 +10:00