Commit Graph

42 Commits

Author SHA1 Message Date
Paul Hauner
c3182e3c1c
Tree hash cache arena (#836)
* Start adding interop genesis state to lcli

* Use more efficient method to generate genesis state

* Remove duplicate int_to_bytes32

* Add lcli command to change state genesis time

* Add option to allow VC to start with unsynced BN

* Set VC to do parallel key loading

* Don't default to dummy eth1 backend

* Add endpoint to dump operation pool

* Add metrics for op pool

* Remove state clone for slot notifier

* Add mem size approximation for tree hash cache

* Avoid cloning tree hash when getting head

* Avoid cloning tree hash when getting head

* Add working arena-based cached tree hash

* Add another benchmark

* Add pre-allocation for caches

* Make cache nullable

* Fix bugs in cache tree hash

* Add validator tree hash optimization

* Optimize hash_concat

* Make hash32_concat return fixed-len array

* Fix failing API tests

* Add new beacon state cache struct

* Add validator-specific cache

* Separate list and values arenas

* Add parallel validator registry hashing

* Remove MultiTreeHashCache

* Remove cached tree hash macro

* Fix failing tree hash test

* Address Michael's comments

* Add CachedTreeHash impl for ef tests

* Fix messy merge conflict

* Rename cache struct, add comments

* Rename cache struct, add comments

* Remove unnecessary mutability

* Wrap iter in result

* Tidy cached tree hash

* Address Michael comments

* Address more comments

* Use ring::Context
2020-02-07 12:42:49 +11:00
Michael Sproul
c1a2238f1a
Implement tree hash caching (#584)
* Implement basic tree hash caching

* Use spaces to indent top-level Cargo.toml

* Optimize BLS tree hash by hashing bytes directly

* Implement tree hash caching for validator registry

* Persist BeaconState tree hash cache to disk

* Address Paul's review comments
2019-11-05 15:46:52 +11:00
Michael Sproul
d511c939eb
SSZ generic tests for big uints 2019-09-04 14:32:04 +10:00
Paul Hauner
8cfa36fedd
Publish ssz_types (and deps) to crates.io (#468)
* Rename `hashing` crate to `eth2_hashing`

* Add license, desc to eth2_hashing Cargo.toml

* Remove merkle root from eth2 hashing

* Remove old benches folder (zombied from old branch)

* Add docs to eth2_hashing

* Prepare tree_hash for publishing on crates.io

* Update deps to use crates.io instead of paths

* Update all crates to pull ssz from crates.io

* Remove cached_tree_hash, add patches to manifest

* Fix compile error in benches

* Remove unused code

* Fix fake_crypto compile error
2019-08-08 11:39:47 +10:00
Michael Sproul
a236003a7b Update to frozen spec ❄️ (v0.8.1) (#444)
* types: first updates for v0.8

* state_processing: epoch processing v0.8.0

* state_processing: block processing v0.8.0

* tree_hash_derive: support generics in SignedRoot

* types v0.8: update to use ssz_types

* state_processing v0.8: use ssz_types

* ssz_types: add bitwise methods and from_elem

* types: fix v0.8 FIXMEs

* ssz_types: add bitfield shift_up

* ssz_types: iterators and DerefMut for VariableList

* types,state_processing: use VariableList

* ssz_types: fix BitVector Decode impl

Fixed a typo in the implementation of ssz::Decode for BitVector, which caused it
to be considered variable length!

* types: fix test modules for v0.8 update

* types: remove slow type-level arithmetic

* state_processing: fix tests for v0.8

* op_pool: update for v0.8

* ssz_types: Bitfield difference length-independent

Allow computing the difference of two bitfields of different lengths.

* Implement compact committee support

* epoch_processing: committee & active index roots

* state_processing: genesis state builder v0.8

* state_processing: implement v0.8.1

* Further improve tree_hash

* Strip examples, tests from cached_tree_hash

* Update TreeHash, un-impl CachedTreeHash

* Update bitfield TreeHash, un-impl CachedTreeHash

* Update FixedLenVec TreeHash, unimpl CachedTreeHash

* Update update tree_hash_derive for new TreeHash

* Fix TreeHash, un-impl CachedTreeHash for ssz_types

* Remove fixed_len_vec, ssz benches

SSZ benches relied upon fixed_len_vec -- it is easier to just delete
them and rebuild them later (when necessary)

* Remove boolean_bitfield crate

* Fix fake_crypto BLS compile errors

* Update ef_tests for new v.8 type params

* Update ef_tests submodule to v0.8.1 tag

* Make fixes to support parsing ssz ef_tests

* `compact_committee...` to `compact_committees...`

* Derive more traits for `CompactCommittee`

* Flip bitfield byte-endianness

* Fix tree_hash for bitfields

* Modify CLI output for ef_tests

* Bump ssz crate version

* Update ssz_types doc comment

* Del cached tree hash tests from ssz_static tests

* Tidy SSZ dependencies

* Rename ssz_types crate to eth2_ssz_types

* validator_client: update for v0.8

* ssz_types: update union/difference for bit order swap

* beacon_node: update for v0.8, EthSpec

* types: disable cached tree hash, update min spec

* state_processing: fix slot bug in committee update

* tests: temporarily disable fork choice harness test

See #447

* committee cache: prevent out-of-bounds access

In the case where we tried to access the committee of a shard that didn't have a committee in the
current epoch, we were accessing elements beyond the end of the shuffling vector and panicking! This
commit adds a check to make the failure safe and explicit.

* fix bug in get_indexed_attestation and simplify

There was a bug in our implementation of get_indexed_attestation whereby
incorrect "committee indices" were used to index into the custody bitfield. The
bug was only observable in the case where some bits of the custody bitfield were
set to 1. The implementation has been simplified to remove the bug, and a test
added.

* state_proc: workaround for compact committees bug

https://github.com/ethereum/eth2.0-specs/issues/1315

* v0.8: updates to make the EF tests pass

* Remove redundant max operation checks.
* Always supply both messages when checking attestation signatures -- allowing
  verification of an attestation with no signatures.
* Swap the order of the fork and domain constant in `get_domain`, to match
  the spec.

* rustfmt

* ef_tests: add new epoch processing tests

* Integrate v0.8 into master (compiles)

* Remove unused crates, fix clippy lints

* Replace v0.6.3 tags w/ v0.8.1

* Remove old comment

* Ensure lmd ghost tests only run in release

* Update readme
2019-07-30 12:44:51 +10:00
John Adler
fec7168512
Fix lots of typos. 2019-07-26 15:26:06 -04:00
Paul Hauner
88c6d15c32
Padding efficent merkle root algo (#436)
* Add initial work on padding efficent merkle roots

* Improve merklize_padded

* Improve tree_hash crate -- fix bugs, docs

* Update codebase for tree_hash API change

* Remove dbg statements, fix import error

* Fix clippy lints, doc error

* Tidy tree hash comments

* Increase tree_hash max tree height

* Fix PR review comments

* Fix typos

* Fix cache access off-by-one in tree hash

* Set max tree depth to 48 (from 64)
2019-07-16 14:40:56 +10:00
Michael Sproul
3a0f00215d
spec: ensure types up to date with v0.6.1 2019-05-15 11:48:09 +10:00
Paul Hauner
f20314bd87
Fix clippy lints, impl treehash for slices 2019-04-29 15:32:41 +10:00
Paul Hauner
a90bbbfd82
Address various clippy lints, improve comments 2019-04-29 14:04:52 +10:00
Paul Hauner
f65e981f6f
Begin implementing cached hashing in types 2019-04-26 15:24:18 +10:00
Paul Hauner
15f81c0907
Implement CachedTreeHash for TreeHashVector 2019-04-26 11:15:17 +10:00
Paul Hauner
f1d8224d89
Further tidy tree_hash crate. 2019-04-26 10:37:50 +10:00
Paul Hauner
b213a5ade4
Create cached_tree_hash crate. 2019-04-26 09:55:03 +10:00
Paul Hauner
827e1c62d9
Add extra tests, all passing 2019-04-25 12:11:31 +10:00
Paul Hauner
cab5e59a6f
Rename CachedTreeHash trait, add readme 2019-04-24 18:23:58 +10:00
Paul Hauner
7563755b15
Fix mix-in-length issue 2019-04-24 18:14:00 +10:00
Paul Hauner
e12fa58e6e
Fix failing test, add hacky fix 2019-04-24 14:56:39 +10:00
Paul Hauner
e19abee7f9
Fix bug with num_nodes/num_chunks 2019-04-24 10:17:05 +10:00
Paul Hauner
a84a063c25
Update depth variable 2019-04-24 09:29:32 +10:00
Paul Hauner
ec43a4085c
Introduce failing test 2019-04-22 21:31:39 +10:00
Paul Hauner
2f69185ccb
Fix tree hash bug 2019-04-22 16:58:40 +10:00
Paul Hauner
7c64a5a21b
Add tests, fix bugs 2019-04-22 16:09:29 +10:00
Paul Hauner
2c12aabf04
Implement further cache tests and bug fixes 2019-04-22 09:20:13 +10:00
Paul Hauner
4aeadfa60f
Remove "old" item requirement from treehash 2019-04-21 12:12:47 +10:00
Paul Hauner
10a5d2657c
Encode bitfield as list not vector 2019-04-17 11:57:57 +10:00
Paul Hauner
10eeced227
Remove SSZ dep from tree_hash 2019-04-17 11:18:00 +10:00
Paul Hauner
af39f096e7
Add vector type to tree hashing 2019-04-17 10:58:03 +10:00
Paul Hauner
b8c4c3308a
Update types to new tree_hash crate 2019-04-16 14:14:38 +10:00
Paul Hauner
3eaa06d758
Remove tree hashing from ssz crate 2019-04-16 12:29:39 +10:00
Paul Hauner
024b9e315a
Add signed_root to tree_hash crate 2019-04-16 11:14:28 +10:00
Paul Hauner
8a1bde3e2f
Update naming for tree_hash fns/structs/traits 2019-04-16 10:48:33 +10:00
Paul Hauner
d311b48a9f
Unify tree hash methods 2019-04-16 09:34:23 +10:00
Paul Hauner
93f3fc858d
Add uncached tree hashing 2019-04-16 09:14:33 +10:00
Paul Hauner
2be05a466f
Add tree_hash_derive crate 2019-04-15 15:45:05 +10:00
Paul Hauner
354f823c16
Tidy tree hash cache, add new trait 2019-04-15 15:13:02 +10:00
Paul Hauner
8e5b79452a
Further tidy cached tree hash 2019-04-15 12:01:12 +10:00
Paul Hauner
c18cdf2abf
Remove num_bytes method 2019-04-15 11:55:56 +10:00
Paul Hauner
cb9b59b93d
Remove panics from packed_encoding 2019-04-15 11:49:50 +10:00
Paul Hauner
e6c33e1b60
Remove child_nodes method from CachedTreeHash 2019-04-15 11:44:44 +10:00
Paul Hauner
c87a0fc588
Tidy CachedTreeHash trait 2019-04-15 11:37:29 +10:00
Paul Hauner
0b5c10212d
Move tree_hash from ssz into own crate 2019-04-15 11:14:30 +10:00