Commit Graph

3192 Commits

Author SHA1 Message Date
Michael Sproul
869b0621d6
Update benchmarks, add to CI (#988) 2020-04-06 20:16:08 +10:00
divma
4cba745df6
make sync sim end faster by checking sync every epoch (#986) 2020-04-06 17:04:06 +10:00
Akihito Nakano
c188227cc2
Fix Eth1 data underflow (#977)
* Fix Eth1 data underflow #540

* Refactor: smart transformation from Option to Result

* Add tests for BeaconState::get_outstanding_deposit_len()
2020-04-06 15:36:34 +10:00
Paul Hauner
54782d896c
Use global pubkey cache for block processing (#849)
* Start updating types

* WIP

* Signature hacking

* Existing EF tests passing with fake_crypto

* Updates

* Delete outdated API spec

* The refactor continues

* It compiles

* WIP test fixes

* All release tests passing bar genesis state parsing

* Update and test YamlConfig

* Update to spec v0.10 compatible BLS

* Updates to BLS EF tests

* Add EF test for AggregateVerify

And delete unused hash2curve tests for uncompressed points

* Update EF tests to v0.10.1

* Use optional block root correctly in block proc

* Use genesis fork in deposit domain. All tests pass

* Cargo fmt

* Fast aggregate verify test

* Update REST API docs

* Cargo fmt

* Fix unused import

* Bump spec tags to v0.10.1

* Add `seconds_per_eth1_block` to chainspec

* Update to timestamp based eth1 voting scheme

* Return None from `get_votes_to_consider` if block cache is empty

* Handle overflows in `is_candidate_block`

* Revert to failing tests

* Fix eth1 data sets test

* Choose default vote according to spec

* Fix collect_valid_votes tests

* Fix `get_votes_to_consider` to choose all eligible blocks

* Uncomment winning_vote tests

* Add comments; remove unused code

* Reduce seconds_per_eth1_block for simulation

* Addressed review comments

* Add test for default vote case

* Fix logs

* Remove unused functions

* Meter default eth1 votes

* Fix comments

* Address review comments; remove unused dependency

* Add first attempt at attestation proc. re-write

* Add version 2 of attestation processing

* Minor fixes

* Add validator pubkey cache

* Make get_indexed_attestation take a committee

* Link signature processing into new attn verification

* First working version

* Ensure pubkey cache is updated

* Add more metrics, slight optimizations

* Clone committee cache during attestation processing

* Update shuffling cache during block processing

* Remove old commented-out code

* Fix shuffling cache insert bug

* Used indexed attestation in fork choice

* Restructure attn processing, add metrics

* Add more detailed metrics

* Tidy, fix failing tests

* Fix failing tests, tidy

* Disable/delete two outdated tests

* Add new Pubkeys struct to signature_sets

* Refactor with functional approach

* Update beacon chain

* Remove decompressed member from pubkey bytes

* Add hashmap for indices lookup

* Change `get_attesting_indices` to use Vec

* Fix failing test

* Tidy

* Add pubkey cache persistence file

* Add more comments

* Integrate persistence file into builder

* Add pubkey cache tests

* Add data_dir to beacon chain builder

* Remove Option in pubkey cache persistence file

* Ensure consistency between datadir/data_dir

* Fix failing network test

* Tidy

* Fix todos

* Improve tests

* Split up block processing metrics

* Tidy

* Refactor get_pubkey_from_state

* Remove commented-out code

* Add BeaconChain::validator_pubkey

* Use Option::filter

* Remove Box

* Comment out tests that fail due to hard-coded

Co-authored-by: Michael Sproul <michael@sigmaprime.io>
Co-authored-by: Michael Sproul <micsproul@gmail.com>
Co-authored-by: pawan <pawandhananjay@gmail.com>
2020-04-06 14:13:19 +10:00
Paul Hauner
6f3503b0e9
Add FAQ section to the book (#982)
* Add FAQ section

* Tidy
2020-04-06 10:54:10 +10:00
Paul Hauner
2fb6b7c793
Add no-copy block processing cache (#863)
* Add state cache, remove store cache

* Only build the head committee cache

* Fix compile error

* Fix compile error from merge

* Rename state_cache -> checkpoint_cache

* Rename Checkpoint -> Snapshot

* Tidy, add comments

* Tidy up find_head function

* Change some checkpoint -> snapshot

* Add tests

* Expose max_len

* Remove dead code

* Tidy

* Fix bug
2020-04-06 10:53:33 +10:00
Akihito Nakano
93bcee147d
Tidy Eth2Config generation at runtime (#912)
* Move the codes that loads Eth2Config from config to environment

* Move the codes that setups Eth2Config for testnet

* Move the codes that creates a new ChainSpec

* Remove unused `mut`

* Reduce local variable number

* Remove unused outputs of config::get_configs()

* Change the method name from plural to singular

* DRY the const `ETH2_CONFIG_FILENAME`

* Add comments

* Remove unnecessary blank line

* cargo fmt

* Add tests for EnvironmentBuilder::setup_eth2_config()

* Remove the comment that have been fixed

* Reduce local variable

* Remove redundant local variable

* Remove prysm-specific codes

Now the spec is in the eth2-testnets repo
2020-04-02 18:47:00 +11:00
Michael Sproul
26bdc2927b
Update to spec v0.11 (#959)
* Update process_final_updates() hysteresis computation

* Update core to v0.11.1

* Bump tags to v0.11.1

* Update docs and deposit contract

* Add compute_fork_digest

* Address review comments

Co-authored-by: Herman Alonso Junge <alonso.junge@gmail.com>
2020-04-01 22:03:03 +11:00
Paul Hauner
e04fc8ddb4
Don't override the fork in lcli eth1-genesis (#948)
* Don't override the fork in lcli

* Add spec flag to lcli
2020-04-01 17:42:24 +11:00
Paul Hauner
88d37e96fa
Fix bug when resuming from DB prior to genesis (#946)
* Use default client genesis if no chain in store

* Always use resume if there is a beacon chain
2020-04-01 17:41:59 +11:00
Paul Hauner
0759806c89
Enable eth1 flag when eth1-endpoint is suppilied (#941) 2020-04-01 17:41:19 +11:00
Paul Hauner
5b984ad394
Add lcli tool for checking deposit data (#940)
* Add check-deposit-data tool

* Update help text

* Update function name
2020-04-01 17:40:32 +11:00
Paul Hauner
11a238900a
Allow slot clock to handle clock disparity (#929)
* Add PH & MS slot clock changes

* Account for genesis time

* Use checked mul

* Account for genesis slot

* Change API

* Refactor "duration to..." functions
2020-04-01 17:40:04 +11:00
Paul Hauner
5c397c49d8
Fix error in /consensus/individual_votes (#859)
* Add patch and commented-out tests

* Delete commented-out tests
2020-04-01 17:39:28 +11:00
Paul Hauner
5a6e90428b
Ensure attestations are created with empty signature (#960)
* Ensure attestations are created with empty sig

* Update docs
2020-03-30 09:29:29 +11:00
Paul Hauner
784997b09b
Ensure correct path for genesis state file (#951) 2020-03-25 12:31:11 +11:00
Sacha Saint-Leger
f6a6de2c5d
Become a Validator guides: update (#928)
* Edit become-a-validator-docker.md

* Update Become a Validator guides

* Update Become a Validator guides

* Update Become a Validator guides

* fix inconsistency
2020-03-23 19:21:53 +11:00
Age Manning
f8bc045a01
Fix race condition in the syncing sim (#944)
* Fix race condition in the syncing sim

* Update another strategy
2020-03-23 18:28:13 +11:00
divma
47aef629d1
move the parent lookup process to a dedicated thread (#906)
* Upgrade the parent lookup logic

* Apply reviewer suggestions

* move the parent lookup process to a dedicated thread

* move the logic of parent lookup and range syncing to a block processor

* review suggestions

* more review suggestions

* Add small logging changes

* Process parent lookups in reverse

Co-authored-by: Age Manning <Age@AgeManning.com>
2020-03-23 12:07:41 +11:00
/raw PONG _GHMoaCXLT
a5fbaef469
Github: Add version to issue template (#937) 2020-03-20 12:18:31 +11:00
Akihito Nakano
8c716b2e92
Fix incomplete build in case of the machine is offline (#935) 2020-03-19 11:22:15 +11:00
Herman Junge
70e39cc6a1
Update reward calculations to v0.11.0 - Handle u64 overflow (#920) (#921)
* Reference https://github.com/ethereum/eth2.0-specs/pull/1635

* Suffix '_ebi' on each effected variable
2020-03-19 11:02:42 +11:00
Age Manning
f160f7a21b
Book corrections to allow http api access for docker use (#911)
* Book corrections to allow http api access for docker use

* Limit beacon node API to localhost

* Add localhost to comment
2020-03-18 16:42:49 +11:00
Paul Hauner
3c8fe00510
Add sync-speed metric (#898) 2020-03-18 09:25:29 +11:00
FawadHa1der
c198bddf9e
fix typo in httpnetwork doc (#904) 2020-03-17 11:07:59 +11:00
Age Manning
756b110854
Fix master failing test (#915)
* Use a valid secret key in ssz test

* Correct test_random secret key tests
2020-03-17 11:07:52 +11:00
Age Manning
b1d23ec294
Adds valid block response to http book example (#910) 2020-03-16 11:38:55 +11:00
Age Manning
5d8a085880
Upgrade the parent lookup logic (#895)
* Upgrade the parent lookup logic
2020-03-13 15:51:03 +11:00
Michael Sproul
6b2e9ff246
Less noisy logs for unaligned finalized blocks (#901) 2020-03-12 12:11:46 +11:00
Paul Hauner
8c5bcfe53a
Optimise beacon chain persistence (#851)
* Unfinished progress

* Update more persistence code

* Start fixing tests

* Combine persist head and fork choice

* Persist head on reorg

* Gracefully handle op pool and eth1 cache missing

* Fix test failure

* Address Michael's comments
2020-03-06 16:09:41 +11:00
Paul Hauner
a87e8c55fc
Add a GitHub action to publish The Book (#877) 2020-03-06 10:54:07 +11:00
Paul Hauner
6656cb00e4
Optimize attestation processing (#841)
* Start updating types

* WIP

* Signature hacking

* Existing EF tests passing with fake_crypto

* Updates

* Delete outdated API spec

* The refactor continues

* It compiles

* WIP test fixes

* All release tests passing bar genesis state parsing

* Update and test YamlConfig

* Update to spec v0.10 compatible BLS

* Updates to BLS EF tests

* Add EF test for AggregateVerify

And delete unused hash2curve tests for uncompressed points

* Update EF tests to v0.10.1

* Use optional block root correctly in block proc

* Use genesis fork in deposit domain. All tests pass

* Cargo fmt

* Fast aggregate verify test

* Update REST API docs

* Cargo fmt

* Fix unused import

* Bump spec tags to v0.10.1

* Add `seconds_per_eth1_block` to chainspec

* Update to timestamp based eth1 voting scheme

* Return None from `get_votes_to_consider` if block cache is empty

* Handle overflows in `is_candidate_block`

* Revert to failing tests

* Fix eth1 data sets test

* Choose default vote according to spec

* Fix collect_valid_votes tests

* Fix `get_votes_to_consider` to choose all eligible blocks

* Uncomment winning_vote tests

* Add comments; remove unused code

* Reduce seconds_per_eth1_block for simulation

* Addressed review comments

* Add test for default vote case

* Fix logs

* Remove unused functions

* Meter default eth1 votes

* Fix comments

* Address review comments; remove unused dependency

* Add first attempt at attestation proc. re-write

* Add version 2 of attestation processing

* Minor fixes

* Add validator pubkey cache

* Make get_indexed_attestation take a committee

* Link signature processing into new attn verification

* First working version

* Ensure pubkey cache is updated

* Add more metrics, slight optimizations

* Clone committee cache during attestation processing

* Update shuffling cache during block processing

* Remove old commented-out code

* Fix shuffling cache insert bug

* Used indexed attestation in fork choice

* Restructure attn processing, add metrics

* Add more detailed metrics

* Tidy, fix failing tests

* Fix failing tests, tidy

* Disable/delete two outdated tests

* Tidy

* Add pubkey cache persistence file

* Add more comments

* Integrate persistence file into builder

* Add pubkey cache tests

* Add data_dir to beacon chain builder

* Remove Option in pubkey cache persistence file

* Ensure consistency between datadir/data_dir

* Fix failing network test

* Tidy

* Fix todos

* Add attestation processing tests

* Add another test

* Only run attestation tests in release

* Make attestation tests MainnetEthSpec

* Address Michael's comments

* Remove redundant check

* Fix warning

* Fix failing test

Co-authored-by: Michael Sproul <micsproul@gmail.com>
Co-authored-by: Pawan Dhananjay <pawandhananjay@gmail.com>
2020-03-05 17:19:35 +11:00
Paul Hauner
c141f1cc03
Use --locked, upgrade Dockerfile Rust version (#894)
* Upgrade docker Rust version

* Use --locked for cargo install
2020-03-05 12:50:24 +11:00
Paul Hauner
08ca9504aa
Fix compile issue (#893) 2020-03-05 10:35:39 +11:00
Paul Hauner
64ec3bd671
Testnet5 docs (#892)
* Merge in ethdever book changes

* Remove ethdenver docs
2020-03-05 10:12:34 +11:00
Paul Hauner
7f6ae4c2f5
Refactor tree hashing (#861)
* Pre-allocated tree hash caches

* Add SmallVec to tree hash cache

* Avoid allocation for validator.pubkey

* Avoid iterator which seems to be doing heap alloc

* Add more smallvecs

* MOAR SMALLVEC

* Move non-test code to Hash256 tree hash

* Fix byte ordering error

* Add incomplete but working merkle stream impl

* Fix zero hash error

* Add zero hash fn

* Add MerkleStream comments

* Add smallvec, tidy

* Integrate into tree hash derive

* Update ssz_types tree hash

* Don't heap alloc for mix in length

* Add byte-level streaming to MerkleStream

* Avoid recursion in write method

* Update BLS to MerkleStream

* Fix some not-compiling tests

* Remove debug profiling

* Remove code duplication

* Move beacon state tree hash to new hasher

* Fix failing tests

* Update comments

* Add some fast-paths to tree_hash::merkle_root

* Remove unncessary test

* Rename MerkleStream -> MerkleHasher

* Rename new_with_leaf_count -> with_leaves

* Tidy

* Remove NonZeroUsize

* Remove todo

* Update smallvec
2020-03-05 08:07:27 +11:00
Paul Hauner
12999fb06c
Faster attestation production (#838)
* Start adding interop genesis state to lcli

* Use more efficient method to generate genesis state

* Remove duplicate int_to_bytes32

* Add lcli command to change state genesis time

* Add option to allow VC to start with unsynced BN

* Set VC to do parallel key loading

* Don't default to dummy eth1 backend

* Add endpoint to dump operation pool

* Add metrics for op pool

* Remove state clone for slot notifier

* Add mem size approximation for tree hash cache

* Avoid cloning tree hash when getting head

* Avoid cloning tree hash when getting head

* Add working arena-based cached tree hash

* Add another benchmark

* Add pre-allocation for caches

* Make cache nullable

* Fix bugs in cache tree hash

* Add validator tree hash optimization

* Optimize hash_concat

* Make hash32_concat return fixed-len array

* Fix failing API tests

* Add new beacon state cache struct

* Add validator-specific cache

* Separate list and values arenas

* Add parallel validator registry hashing

* Remove MultiTreeHashCache

* Remove cached tree hash macro

* Fix failing tree hash test

* Address Michael's comments

* Add CachedTreeHash impl for ef tests

* Fix messy merge conflict

* Optimize attestation production

* Add first basic optimizations

* Fix SlotOutOfBounds error

* Resolved missed merge conflicts

* Fix another missed merge conflict

* Fix more merge conflict issues

* Add `StateSkipConfig`

* Fix test compile errors

* Add failing test

* Fix bug, make tests pass

* Add comment

* Delete unused function

* Replace deleted comment
2020-03-04 17:10:22 +11:00
Michael Sproul
1f16d8fe4d
Add methods to delete blocks and states from disk (#843)
Closes #833
2020-03-04 16:48:35 +11:00
Akihito Nakano
12d9b42188
Fix incorrect error message in ClientBuilder (#888)
* Fix incorrect error message in ClientBuilder

* Run cargo fmt --all
2020-03-04 16:37:13 +11:00
Paul Hauner
ca0314ee55
Add "new-testnet" command to lcli (#853)
* Add new command to lcli

* Add lcli to dockerfile

* Add min validator count param

* Fix bug in arg parsing

* Fix 0x address prefix issue

* Add effective balance increment

* Add ejection balance

* Fix PR comments
2020-03-04 14:28:02 +11:00
Paul Hauner
871163aecc
Add optimized SSZ decoding for fixed-len items (#865)
* Add custom SSZ decode for Validator

* Move efficient decode into macro

* Don't allocate SSZ offset to heap

* Use smallvec in SszDecoder

* Fix test compile error
2020-03-04 11:45:01 +11:00
Paul Hauner
58fb144276
Remove testnet dir from repo (#857)
* Pull testnet dir with build script

* Add comment

* Don't hardcode the genesis fork

* Re-enable hard-coded test
2020-03-04 10:02:44 +11:00
realbigsean
6368be148d
Add checks for eth1 chain connection before inserting to op pool (#868) 2020-03-02 14:04:49 +11:00
Paul Hauner
fbb630793e
Attempt to remove a tree hash from block replaying (#862)
* Attempt to remove a tree hash from block replaying

* Add missed thing
2020-03-02 13:40:58 +11:00
Adam Szkoda
f6f924a7a6
Correct typo (#872) 2020-03-02 13:40:19 +11:00
Pawan Dhananjay
e85e337540
Change port in test to some fairly random number (#874) 2020-03-02 13:36:34 +11:00
Pawan Dhananjay
4d60694443
Add support for noise protocol (#873)
* Add noise support with fallback to secio

* Add config parameter for noise support

* Add secio/noise compatibility test

* Cleanup

* Remove config parameter for noise support

* Modify test to work between a secio swarm and a noise libp2p service

* Minor fixes
2020-03-02 13:35:20 +11:00
Paul Hauner
0c96c515a0
Update book deposit contract address (#858) 2020-02-25 10:15:11 +11:00
Paul Hauner
1cf8769c07
Ensure deposits are generated with correct fork (#855)
* Ensure genesis fork is applied to spec

* Ensure account man uses testnet spec
2020-02-25 09:01:32 +11:00
Paul Hauner
123c63119d
Remove temporary heap allocations during shuffling (#867)
* Remove temp allocs in compute shuffled index

* Update shuffle list
2020-02-25 09:00:09 +11:00