Commit Graph

39 Commits

Author SHA1 Message Date
Paul Hauner
7f6ae4c2f5
Refactor tree hashing (#861)
* Pre-allocated tree hash caches

* Add SmallVec to tree hash cache

* Avoid allocation for validator.pubkey

* Avoid iterator which seems to be doing heap alloc

* Add more smallvecs

* MOAR SMALLVEC

* Move non-test code to Hash256 tree hash

* Fix byte ordering error

* Add incomplete but working merkle stream impl

* Fix zero hash error

* Add zero hash fn

* Add MerkleStream comments

* Add smallvec, tidy

* Integrate into tree hash derive

* Update ssz_types tree hash

* Don't heap alloc for mix in length

* Add byte-level streaming to MerkleStream

* Avoid recursion in write method

* Update BLS to MerkleStream

* Fix some not-compiling tests

* Remove debug profiling

* Remove code duplication

* Move beacon state tree hash to new hasher

* Fix failing tests

* Update comments

* Add some fast-paths to tree_hash::merkle_root

* Remove unncessary test

* Rename MerkleStream -> MerkleHasher

* Rename new_with_leaf_count -> with_leaves

* Tidy

* Remove NonZeroUsize

* Remove todo

* Update smallvec
2020-03-05 08:07:27 +11:00
Paul Hauner
c3182e3c1c
Tree hash cache arena (#836)
* Start adding interop genesis state to lcli

* Use more efficient method to generate genesis state

* Remove duplicate int_to_bytes32

* Add lcli command to change state genesis time

* Add option to allow VC to start with unsynced BN

* Set VC to do parallel key loading

* Don't default to dummy eth1 backend

* Add endpoint to dump operation pool

* Add metrics for op pool

* Remove state clone for slot notifier

* Add mem size approximation for tree hash cache

* Avoid cloning tree hash when getting head

* Avoid cloning tree hash when getting head

* Add working arena-based cached tree hash

* Add another benchmark

* Add pre-allocation for caches

* Make cache nullable

* Fix bugs in cache tree hash

* Add validator tree hash optimization

* Optimize hash_concat

* Make hash32_concat return fixed-len array

* Fix failing API tests

* Add new beacon state cache struct

* Add validator-specific cache

* Separate list and values arenas

* Add parallel validator registry hashing

* Remove MultiTreeHashCache

* Remove cached tree hash macro

* Fix failing tree hash test

* Address Michael's comments

* Add CachedTreeHash impl for ef tests

* Fix messy merge conflict

* Rename cache struct, add comments

* Rename cache struct, add comments

* Remove unnecessary mutability

* Wrap iter in result

* Tidy cached tree hash

* Address Michael comments

* Address more comments

* Use ring::Context
2020-02-07 12:42:49 +11:00
Paul Hauner
f267bf2afe
Optimizations, disable val client sync check & additional lcli tools (#834)
* Start adding interop genesis state to lcli

* Use more efficient method to generate genesis state

* Remove duplicate int_to_bytes32

* Add lcli command to change state genesis time

* Add option to allow VC to start with unsynced BN

* Set VC to do parallel key loading

* Don't default to dummy eth1 backend

* Add endpoint to dump operation pool

* Add metrics for op pool

* Remove state clone for slot notifier

* Add mem size approximation for tree hash cache

* Avoid cloning tree hash when getting head

* Fix failing API tests

* Address Michael's comments

* Add HashMap::from_par_iter
2020-02-04 12:43:04 +11:00
pscott
7396cd2cab Fix clippy warnings (#813)
* Clippy account manager

* Clippy account_manager

* Clippy beacon_node/beacon_chain

* Clippy beacon_node/client

* Clippy beacon_node/eth1

* Clippy beacon_node/eth2-libp2p

* Clippy beacon_node/genesis

* Clippy beacon_node/network

* Clippy beacon_node/rest_api

* Clippy beacon_node/src

* Clippy beacon_node/store

* Clippy eth2/lmd_ghost

* Clippy eth2/operation_pool

* Clippy eth2/state_processing

* Clippy eth2/types

* Clippy eth2/utils/bls

* Clippy eth2/utils/cahced_tree_hash

* Clippy eth2/utils/deposit_contract

* Clippy eth2/utils/eth2_interop_keypairs

* Clippy eth2/utils/eth2_testnet_config

* Clippy eth2/utils/lighthouse_metrics

* Clippy eth2/utils/ssz

* Clippy eth2/utils/ssz_types

* Clippy eth2/utils/tree_hash_derive

* Clippy lcli

* Clippy tests/beacon_chain_sim

* Clippy validator_client

* Cargo fmt
2020-01-21 18:38:56 +11:00
pscott
8e1e6838d2 Add individual processing for ef tests (#745)
* Add Individual signature verification for ef tests

* Add clone state to avoid errors in ef tests

* Add state verification for VerifyIndividual in ef tests

* Add Clone derive for errors

* Update comments
2020-01-08 10:24:44 +11:00
Michael Sproul
c1a2238f1a
Implement tree hash caching (#584)
* Implement basic tree hash caching

* Use spaces to indent top-level Cargo.toml

* Optimize BLS tree hash by hashing bytes directly

* Implement tree hash caching for validator registry

* Persist BeaconState tree hash cache to disk

* Address Paul's review comments
2019-11-05 15:46:52 +11:00
Paul Hauner
8cfa36fedd
Publish ssz_types (and deps) to crates.io (#468)
* Rename `hashing` crate to `eth2_hashing`

* Add license, desc to eth2_hashing Cargo.toml

* Remove merkle root from eth2 hashing

* Remove old benches folder (zombied from old branch)

* Add docs to eth2_hashing

* Prepare tree_hash for publishing on crates.io

* Update deps to use crates.io instead of paths

* Update all crates to pull ssz from crates.io

* Remove cached_tree_hash, add patches to manifest

* Fix compile error in benches

* Remove unused code

* Fix fake_crypto compile error
2019-08-08 11:39:47 +10:00
Pawan Dhananjay
88e89f9ab2 Update ethereum types (#489)
* Update ethereum-types to version 0.6

* Fix tests

* Run rustfmt
2019-08-06 14:41:42 +10:00
Paul Hauner
81a089aa8b
Remove old benches (#465)
* Remove cached_tree_hash benches

* Remove state_processing benches
2019-07-30 17:19:03 +10:00
Michael Sproul
a236003a7b Update to frozen spec ❄️ (v0.8.1) (#444)
* types: first updates for v0.8

* state_processing: epoch processing v0.8.0

* state_processing: block processing v0.8.0

* tree_hash_derive: support generics in SignedRoot

* types v0.8: update to use ssz_types

* state_processing v0.8: use ssz_types

* ssz_types: add bitwise methods and from_elem

* types: fix v0.8 FIXMEs

* ssz_types: add bitfield shift_up

* ssz_types: iterators and DerefMut for VariableList

* types,state_processing: use VariableList

* ssz_types: fix BitVector Decode impl

Fixed a typo in the implementation of ssz::Decode for BitVector, which caused it
to be considered variable length!

* types: fix test modules for v0.8 update

* types: remove slow type-level arithmetic

* state_processing: fix tests for v0.8

* op_pool: update for v0.8

* ssz_types: Bitfield difference length-independent

Allow computing the difference of two bitfields of different lengths.

* Implement compact committee support

* epoch_processing: committee & active index roots

* state_processing: genesis state builder v0.8

* state_processing: implement v0.8.1

* Further improve tree_hash

* Strip examples, tests from cached_tree_hash

* Update TreeHash, un-impl CachedTreeHash

* Update bitfield TreeHash, un-impl CachedTreeHash

* Update FixedLenVec TreeHash, unimpl CachedTreeHash

* Update update tree_hash_derive for new TreeHash

* Fix TreeHash, un-impl CachedTreeHash for ssz_types

* Remove fixed_len_vec, ssz benches

SSZ benches relied upon fixed_len_vec -- it is easier to just delete
them and rebuild them later (when necessary)

* Remove boolean_bitfield crate

* Fix fake_crypto BLS compile errors

* Update ef_tests for new v.8 type params

* Update ef_tests submodule to v0.8.1 tag

* Make fixes to support parsing ssz ef_tests

* `compact_committee...` to `compact_committees...`

* Derive more traits for `CompactCommittee`

* Flip bitfield byte-endianness

* Fix tree_hash for bitfields

* Modify CLI output for ef_tests

* Bump ssz crate version

* Update ssz_types doc comment

* Del cached tree hash tests from ssz_static tests

* Tidy SSZ dependencies

* Rename ssz_types crate to eth2_ssz_types

* validator_client: update for v0.8

* ssz_types: update union/difference for bit order swap

* beacon_node: update for v0.8, EthSpec

* types: disable cached tree hash, update min spec

* state_processing: fix slot bug in committee update

* tests: temporarily disable fork choice harness test

See #447

* committee cache: prevent out-of-bounds access

In the case where we tried to access the committee of a shard that didn't have a committee in the
current epoch, we were accessing elements beyond the end of the shuffling vector and panicking! This
commit adds a check to make the failure safe and explicit.

* fix bug in get_indexed_attestation and simplify

There was a bug in our implementation of get_indexed_attestation whereby
incorrect "committee indices" were used to index into the custody bitfield. The
bug was only observable in the case where some bits of the custody bitfield were
set to 1. The implementation has been simplified to remove the bug, and a test
added.

* state_proc: workaround for compact committees bug

https://github.com/ethereum/eth2.0-specs/issues/1315

* v0.8: updates to make the EF tests pass

* Remove redundant max operation checks.
* Always supply both messages when checking attestation signatures -- allowing
  verification of an attestation with no signatures.
* Swap the order of the fork and domain constant in `get_domain`, to match
  the spec.

* rustfmt

* ef_tests: add new epoch processing tests

* Integrate v0.8 into master (compiles)

* Remove unused crates, fix clippy lints

* Replace v0.6.3 tags w/ v0.8.1

* Remove old comment

* Ensure lmd ghost tests only run in release

* Update readme
2019-07-30 12:44:51 +10:00
John Adler
fec7168512
Fix lots of typos. 2019-07-26 15:26:06 -04:00
Michael Sproul
3a0f00215d
spec: ensure types up to date with v0.6.1 2019-05-15 11:48:09 +10:00
Luke Anderson
93a4bef41c
Very small wording/typo fixes for docs. 2019-05-10 13:49:01 +10:00
Paul Hauner
3be80ba9a9
Marginally improve cached_tree_hash docs 2019-05-10 13:29:43 +10:00
Paul Hauner
84d72cfed6
Tidy and add docs for cached tree hash 2019-04-29 17:46:01 +10:00
Paul Hauner
f622aa0b65
Add doc comments to vec impl 2019-04-29 15:45:02 +10:00
Paul Hauner
f20314bd87
Fix clippy lints, impl treehash for slices 2019-04-29 15:32:41 +10:00
Paul Hauner
240d1e197a
Ignore clippy range_plus_one lints 2019-04-29 14:24:57 +10:00
Paul Hauner
a90bbbfd82
Address various clippy lints, improve comments 2019-04-29 14:04:52 +10:00
Paul Hauner
fbf8fad4f1
Add counter-resets to reset_modifications 2019-04-29 10:57:19 +10:00
Paul Hauner
0599d3f1f8
Remove redundant type param fom CachedTreeHash 2019-04-29 07:48:48 +10:00
Paul Hauner
68b36787e2
Move leaf node creation into separate fn 2019-04-28 20:30:48 +10:00
Paul Hauner
58308e3dc5
Modify example execution counts 2019-04-28 19:10:59 +10:00
Paul Hauner
a4559e798d
Add benches, examples for cached hashing.
Ignore flamegraph files
2019-04-28 11:38:32 +10:00
Paul Hauner
c58723350c
Improve cached hashing performance 2019-04-28 11:33:29 +10:00
Paul Hauner
89d64b007f
Run cargofmt --all 2019-04-27 19:04:26 +10:00
Paul Hauner
6c9be1a73c
Add tree hash cache as field to BeaconState. 2019-04-27 19:02:52 +10:00
Paul Hauner
b70ebd09ea
Increase detail of cached hashing testing 2019-04-27 16:33:31 +10:00
Paul Hauner
80fa5d08c5
Fix bug with cached tree hash, passes tests 2019-04-27 16:24:29 +10:00
Paul Hauner
b86e118062
Add specific failing cache test 2019-04-26 18:19:13 +10:00
Paul Hauner
a425beb42a
Fix chunk index bug with structs 2019-04-26 17:39:38 +10:00
Paul Hauner
d3309b9f7e
Fix bug with bitfield tree hash caching 2019-04-26 17:27:39 +10:00
Paul Hauner
dddcc91ef3
Fix bug with shrinking list. 2019-04-26 16:56:01 +10:00
Paul Hauner
f65e981f6f
Begin implementing cached hashing in types 2019-04-26 15:24:18 +10:00
Paul Hauner
ecff8f0007
Update CachedTreeHash trait to return schema 2019-04-26 12:27:04 +10:00
Paul Hauner
8976e652d2
Introduce BTreeSchema 2019-04-26 11:55:26 +10:00
Paul Hauner
794b48078c
Remove num_items from BTreeOverlay 2019-04-26 11:39:33 +10:00
Paul Hauner
15f81c0907
Implement CachedTreeHash for TreeHashVector 2019-04-26 11:15:17 +10:00
Paul Hauner
b213a5ade4
Create cached_tree_hash crate. 2019-04-26 09:55:03 +10:00