Commit Graph

1043 Commits

Author SHA1 Message Date
Paul Hauner
2fb6b7c793
Add no-copy block processing cache (#863)
* Add state cache, remove store cache

* Only build the head committee cache

* Fix compile error

* Fix compile error from merge

* Rename state_cache -> checkpoint_cache

* Rename Checkpoint -> Snapshot

* Tidy, add comments

* Tidy up find_head function

* Change some checkpoint -> snapshot

* Add tests

* Expose max_len

* Remove dead code

* Tidy

* Fix bug
2020-04-06 10:53:33 +10:00
Akihito Nakano
93bcee147d
Tidy Eth2Config generation at runtime (#912)
* Move the codes that loads Eth2Config from config to environment

* Move the codes that setups Eth2Config for testnet

* Move the codes that creates a new ChainSpec

* Remove unused `mut`

* Reduce local variable number

* Remove unused outputs of config::get_configs()

* Change the method name from plural to singular

* DRY the const `ETH2_CONFIG_FILENAME`

* Add comments

* Remove unnecessary blank line

* cargo fmt

* Add tests for EnvironmentBuilder::setup_eth2_config()

* Remove the comment that have been fixed

* Reduce local variable

* Remove redundant local variable

* Remove prysm-specific codes

Now the spec is in the eth2-testnets repo
2020-04-02 18:47:00 +11:00
Michael Sproul
26bdc2927b
Update to spec v0.11 (#959)
* Update process_final_updates() hysteresis computation

* Update core to v0.11.1

* Bump tags to v0.11.1

* Update docs and deposit contract

* Add compute_fork_digest

* Address review comments

Co-authored-by: Herman Alonso Junge <alonso.junge@gmail.com>
2020-04-01 22:03:03 +11:00
Paul Hauner
88d37e96fa
Fix bug when resuming from DB prior to genesis (#946)
* Use default client genesis if no chain in store

* Always use resume if there is a beacon chain
2020-04-01 17:41:59 +11:00
Paul Hauner
0759806c89
Enable eth1 flag when eth1-endpoint is suppilied (#941) 2020-04-01 17:41:19 +11:00
Paul Hauner
5c397c49d8
Fix error in /consensus/individual_votes (#859)
* Add patch and commented-out tests

* Delete commented-out tests
2020-04-01 17:39:28 +11:00
Paul Hauner
5a6e90428b
Ensure attestations are created with empty signature (#960)
* Ensure attestations are created with empty sig

* Update docs
2020-03-30 09:29:29 +11:00
divma
47aef629d1
move the parent lookup process to a dedicated thread (#906)
* Upgrade the parent lookup logic

* Apply reviewer suggestions

* move the parent lookup process to a dedicated thread

* move the logic of parent lookup and range syncing to a block processor

* review suggestions

* more review suggestions

* Add small logging changes

* Process parent lookups in reverse

Co-authored-by: Age Manning <Age@AgeManning.com>
2020-03-23 12:07:41 +11:00
Paul Hauner
3c8fe00510
Add sync-speed metric (#898) 2020-03-18 09:25:29 +11:00
Age Manning
5d8a085880
Upgrade the parent lookup logic (#895)
* Upgrade the parent lookup logic
2020-03-13 15:51:03 +11:00
Michael Sproul
6b2e9ff246
Less noisy logs for unaligned finalized blocks (#901) 2020-03-12 12:11:46 +11:00
Paul Hauner
8c5bcfe53a
Optimise beacon chain persistence (#851)
* Unfinished progress

* Update more persistence code

* Start fixing tests

* Combine persist head and fork choice

* Persist head on reorg

* Gracefully handle op pool and eth1 cache missing

* Fix test failure

* Address Michael's comments
2020-03-06 16:09:41 +11:00
Paul Hauner
6656cb00e4
Optimize attestation processing (#841)
* Start updating types

* WIP

* Signature hacking

* Existing EF tests passing with fake_crypto

* Updates

* Delete outdated API spec

* The refactor continues

* It compiles

* WIP test fixes

* All release tests passing bar genesis state parsing

* Update and test YamlConfig

* Update to spec v0.10 compatible BLS

* Updates to BLS EF tests

* Add EF test for AggregateVerify

And delete unused hash2curve tests for uncompressed points

* Update EF tests to v0.10.1

* Use optional block root correctly in block proc

* Use genesis fork in deposit domain. All tests pass

* Cargo fmt

* Fast aggregate verify test

* Update REST API docs

* Cargo fmt

* Fix unused import

* Bump spec tags to v0.10.1

* Add `seconds_per_eth1_block` to chainspec

* Update to timestamp based eth1 voting scheme

* Return None from `get_votes_to_consider` if block cache is empty

* Handle overflows in `is_candidate_block`

* Revert to failing tests

* Fix eth1 data sets test

* Choose default vote according to spec

* Fix collect_valid_votes tests

* Fix `get_votes_to_consider` to choose all eligible blocks

* Uncomment winning_vote tests

* Add comments; remove unused code

* Reduce seconds_per_eth1_block for simulation

* Addressed review comments

* Add test for default vote case

* Fix logs

* Remove unused functions

* Meter default eth1 votes

* Fix comments

* Address review comments; remove unused dependency

* Add first attempt at attestation proc. re-write

* Add version 2 of attestation processing

* Minor fixes

* Add validator pubkey cache

* Make get_indexed_attestation take a committee

* Link signature processing into new attn verification

* First working version

* Ensure pubkey cache is updated

* Add more metrics, slight optimizations

* Clone committee cache during attestation processing

* Update shuffling cache during block processing

* Remove old commented-out code

* Fix shuffling cache insert bug

* Used indexed attestation in fork choice

* Restructure attn processing, add metrics

* Add more detailed metrics

* Tidy, fix failing tests

* Fix failing tests, tidy

* Disable/delete two outdated tests

* Tidy

* Add pubkey cache persistence file

* Add more comments

* Integrate persistence file into builder

* Add pubkey cache tests

* Add data_dir to beacon chain builder

* Remove Option in pubkey cache persistence file

* Ensure consistency between datadir/data_dir

* Fix failing network test

* Tidy

* Fix todos

* Add attestation processing tests

* Add another test

* Only run attestation tests in release

* Make attestation tests MainnetEthSpec

* Address Michael's comments

* Remove redundant check

* Fix warning

* Fix failing test

Co-authored-by: Michael Sproul <micsproul@gmail.com>
Co-authored-by: Pawan Dhananjay <pawandhananjay@gmail.com>
2020-03-05 17:19:35 +11:00
Paul Hauner
08ca9504aa
Fix compile issue (#893) 2020-03-05 10:35:39 +11:00
Paul Hauner
7f6ae4c2f5
Refactor tree hashing (#861)
* Pre-allocated tree hash caches

* Add SmallVec to tree hash cache

* Avoid allocation for validator.pubkey

* Avoid iterator which seems to be doing heap alloc

* Add more smallvecs

* MOAR SMALLVEC

* Move non-test code to Hash256 tree hash

* Fix byte ordering error

* Add incomplete but working merkle stream impl

* Fix zero hash error

* Add zero hash fn

* Add MerkleStream comments

* Add smallvec, tidy

* Integrate into tree hash derive

* Update ssz_types tree hash

* Don't heap alloc for mix in length

* Add byte-level streaming to MerkleStream

* Avoid recursion in write method

* Update BLS to MerkleStream

* Fix some not-compiling tests

* Remove debug profiling

* Remove code duplication

* Move beacon state tree hash to new hasher

* Fix failing tests

* Update comments

* Add some fast-paths to tree_hash::merkle_root

* Remove unncessary test

* Rename MerkleStream -> MerkleHasher

* Rename new_with_leaf_count -> with_leaves

* Tidy

* Remove NonZeroUsize

* Remove todo

* Update smallvec
2020-03-05 08:07:27 +11:00
Paul Hauner
12999fb06c
Faster attestation production (#838)
* Start adding interop genesis state to lcli

* Use more efficient method to generate genesis state

* Remove duplicate int_to_bytes32

* Add lcli command to change state genesis time

* Add option to allow VC to start with unsynced BN

* Set VC to do parallel key loading

* Don't default to dummy eth1 backend

* Add endpoint to dump operation pool

* Add metrics for op pool

* Remove state clone for slot notifier

* Add mem size approximation for tree hash cache

* Avoid cloning tree hash when getting head

* Avoid cloning tree hash when getting head

* Add working arena-based cached tree hash

* Add another benchmark

* Add pre-allocation for caches

* Make cache nullable

* Fix bugs in cache tree hash

* Add validator tree hash optimization

* Optimize hash_concat

* Make hash32_concat return fixed-len array

* Fix failing API tests

* Add new beacon state cache struct

* Add validator-specific cache

* Separate list and values arenas

* Add parallel validator registry hashing

* Remove MultiTreeHashCache

* Remove cached tree hash macro

* Fix failing tree hash test

* Address Michael's comments

* Add CachedTreeHash impl for ef tests

* Fix messy merge conflict

* Optimize attestation production

* Add first basic optimizations

* Fix SlotOutOfBounds error

* Resolved missed merge conflicts

* Fix another missed merge conflict

* Fix more merge conflict issues

* Add `StateSkipConfig`

* Fix test compile errors

* Add failing test

* Fix bug, make tests pass

* Add comment

* Delete unused function

* Replace deleted comment
2020-03-04 17:10:22 +11:00
Michael Sproul
1f16d8fe4d
Add methods to delete blocks and states from disk (#843)
Closes #833
2020-03-04 16:48:35 +11:00
Akihito Nakano
12d9b42188
Fix incorrect error message in ClientBuilder (#888)
* Fix incorrect error message in ClientBuilder

* Run cargo fmt --all
2020-03-04 16:37:13 +11:00
Paul Hauner
58fb144276
Remove testnet dir from repo (#857)
* Pull testnet dir with build script

* Add comment

* Don't hardcode the genesis fork

* Re-enable hard-coded test
2020-03-04 10:02:44 +11:00
realbigsean
6368be148d
Add checks for eth1 chain connection before inserting to op pool (#868) 2020-03-02 14:04:49 +11:00
Paul Hauner
fbb630793e
Attempt to remove a tree hash from block replaying (#862)
* Attempt to remove a tree hash from block replaying

* Add missed thing
2020-03-02 13:40:58 +11:00
Pawan Dhananjay
e85e337540
Change port in test to some fairly random number (#874) 2020-03-02 13:36:34 +11:00
Pawan Dhananjay
4d60694443
Add support for noise protocol (#873)
* Add noise support with fallback to secio

* Add config parameter for noise support

* Add secio/noise compatibility test

* Cleanup

* Remove config parameter for noise support

* Modify test to work between a secio swarm and a noise libp2p service

* Minor fixes
2020-03-02 13:35:20 +11:00
Age Manning
2a9c718a20
Remove network lock (#840)
* Initial work on removing libp2p lock

* Removes lock from libp2p service

* Completed network lock removal

* Correct network termination future

* Correct fmt issues

* Remove Drop implementation for network service

* Address reviewers suggestions

* Fix dht persistence test (#844)

* Fix persistence test

* Block until dht is persisted

* Fix libp2p test

* Correct test ordering check

* Remove expensive tests from debug testing

Co-authored-by: Pawan Dhananjay <pawandhananjay@gmail.com>
2020-02-19 22:12:25 +11:00
Age Manning
26be30bc4c
Handle simultaneous RPC negotiation errors (#846) 2020-02-19 15:39:20 +11:00
Pawan Dhananjay
74c34d1602
Add attester/proposer slashing endpoints (#856)
* Remove deprecated api_spec.yaml

* add prototype for proposer slashing

* remove clippy warnings

* Add proposer_slashing API

* Prototype for attester slashing API call

* Fix logic error in operation pool

* Finish test for attester_slashing api call

* Clean proposer_slashing test

* Cargo fmt

* Remove useless to_string after format! macro

* Cargo fmt

* Update book with new api calls

* Re-enable proposer slashing verification

* Update book with appropriate test example

* Fix proposer_slashing test

* Update comments and tests for clearer code

* Remove extraneous comments

* Fix test

* Minor fix

* Address reviewer comments

Co-authored-by: pscott <30843220+pscott@users.noreply.github.com>
2020-02-14 22:35:18 +11:00
Michael Sproul
371e5adcf8
Update to Spec v0.10 (#817)
* Start updating types

* WIP

* Signature hacking

* Existing EF tests passing with fake_crypto

* Updates

* Delete outdated API spec

* The refactor continues

* It compiles

* WIP test fixes

* All release tests passing bar genesis state parsing

* Update and test YamlConfig

* Update to spec v0.10 compatible BLS

* Updates to BLS EF tests

* Add EF test for AggregateVerify

And delete unused hash2curve tests for uncompressed points

* Update EF tests to v0.10.1

* Use optional block root correctly in block proc

* Use genesis fork in deposit domain. All tests pass

* Cargo fmt

* Fast aggregate verify test

* Update REST API docs

* Cargo fmt

* Fix unused import

* Bump spec tags to v0.10.1

* Add `seconds_per_eth1_block` to chainspec

* Update to timestamp based eth1 voting scheme

* Return None from `get_votes_to_consider` if block cache is empty

* Handle overflows in `is_candidate_block`

* Revert to failing tests

* Fix eth1 data sets test

* Choose default vote according to spec

* Fix collect_valid_votes tests

* Fix `get_votes_to_consider` to choose all eligible blocks

* Uncomment winning_vote tests

* Add comments; remove unused code

* Reduce seconds_per_eth1_block for simulation

* Addressed review comments

* Add test for default vote case

* Fix logs

* Remove unused functions

* Meter default eth1 votes

* Fix comments

* Address review comments; remove unused dependency

* Disable/delete two outdated tests

* Bump eth1 default vote warn to error

* Delete outdated eth1 test

Co-authored-by: Pawan Dhananjay <pawandhananjay@gmail.com>
2020-02-11 10:19:36 +11:00
Michael Sproul
e0b9fa599f
Add LRU cache to database (#837)
* Add LRU caches to store

* Improvements to LRU caches

* Take state by value in `Store::put_state`

* Store blocks by value, configurable cache sizes

* Use a StateBatch to efficiently store skip states

* Fix store tests

* Add CloneConfig test, remove unused metrics

* Use Mutexes instead of RwLocks for LRU caches
2020-02-10 11:30:21 +11:00
Paul Hauner
f267bf2afe
Optimizations, disable val client sync check & additional lcli tools (#834)
* Start adding interop genesis state to lcli

* Use more efficient method to generate genesis state

* Remove duplicate int_to_bytes32

* Add lcli command to change state genesis time

* Add option to allow VC to start with unsynced BN

* Set VC to do parallel key loading

* Don't default to dummy eth1 backend

* Add endpoint to dump operation pool

* Add metrics for op pool

* Remove state clone for slot notifier

* Add mem size approximation for tree hash cache

* Avoid cloning tree hash when getting head

* Fix failing API tests

* Address Michael's comments

* Add HashMap::from_par_iter
2020-02-04 12:43:04 +11:00
Age Manning
eef56e77ef
Patch discv5 - session establishment issues (#828) 2020-01-31 18:57:20 +11:00
Age Manning
8923f84af9
Prevent duplicate gossipsub loops/message amplification (#832)
* Increase gossipsub duplicate cache

* Hashes of messages are now stored in LRUCache
2020-01-31 18:56:49 +11:00
Paul Hauner
b771bbb60c
Add proto_array fork choice (#804)
* Start implementing proto_array

* Add progress

* Add unfinished progress

* Add further progress

* Add progress

* Add tree filtering

* Add half-finished modifications

* Add refactored version

* Tidy, add incomplete LmdGhost impl

* Move impls in LmdGhost trait def

* Remove old reduced_tree fork choice

* Combine two functions in to `compute_deltas`

* Start testing

* Add more compute_deltas tests

* Add fork choice testing

* Add more fork choice testing

* Add more fork choice tests

* Add more testing to proto-array

* Remove old tests

* Modify tests

* Add more tests

* Add more testing

* Add comments and fixes

* Re-organise crate

* Tidy, finish pruning tests

* Add ssz encoding, other pub fns

* Rename lmd_ghost > proto_array_fork_choice

* Integrate proto_array into lighthouse

* Add first pass at fixing filter

* Clean out old comments

* Add more comments

* Attempt to fix prune error

* Adjust TODO

* Fix test compile errors

* Add extra justification change check

* Update cargo.lock

* Fix fork choice test compile errors

* Most remove ffg_update_required

* Fix bug with epoch of attestation votes

* Start adding new test format

* Make fork choice tests declarative

* Create test def concept

* Move test defs into crate

* Add binary, re-org crate

* Shuffle files

* Start adding ffg tests

* Add more fork choice tests

* Add fork choice JSON dumping

* Add more detail to best node error

* Ensure fin+just checkpoints from from same block

* Rename JustificationManager

* Move checkpoint manager into own file

* Tidy

* Add targetted logging for sneaky sync bug

* Fix justified balances bug

* Add cache metrics

* Add metrics for log levels

* Fix bug in checkpoint manager

* Fix compile error in fork choice tests

* Ignore duplicate blocks in fork choice

* Add block to fock choice before db

* Rename on_new_block fn

* Fix spec inconsistency in `CheckpointManager`

* Remove BlockRootTree

* Remove old reduced_tree code fragment

* Add API endpoint for fork choice

* Add more ffg tests

* Remove block_root_tree reminents

* Ensure effective balances are used

* Remove old debugging code, fix API fault

* Add check to ensure parent block is in fork choice

* Update readme dates

* Fix readme

* Tidy checkpoint manager

* Remove fork choice yaml files from repo

* Remove fork choice yaml from repo

* General tidy

* Address majority of Michael's comments

* Tidy bin/lib business

* Remove dangling file

* Undo changes for rpc/handler from master

* Revert "Undo changes for rpc/handler from master"

This reverts commit 876edff0e4a501aafbb47113454852826dcc24e8.

Co-authored-by: Age Manning <Age@AgeManning.com>
2020-01-29 15:05:00 +11:00
pscott
d98c00389a Fix typo in test name; fix clippy warning (#826) 2020-01-23 19:28:23 +11:00
Age Manning
81b028b805
Advanced error handling for syncing (#819)
* Initial block processing thread design

* Correct compilation issues

* Increase logging and request from all given peers

* Patch peer request bug

* Adds fork choice to block processing

* Adds logging for bug isolation

* Patch syncing for chains with skip-slots

* Bump block processing error logs

* Improve logging for attestation processing

* Randomize peer selection during sync

* Resuming chains restarts from local finalized slot

* Downgrades Arc batches to Rc batches

* Add clippy fixes

* Add advanced error handling for invalid/malicious batches

* Downgrade Rc<Batch> to Option<Batch> to pass processed batches to chains

* Squash edge case rpc and syncing bugs

* Process empty batches which could end chains

* Removes last_processed_id concept to account for ending skip-slot batches

* Add logging for chain purges

* Adds retries to re-request batch logging

* Remove bug finding log

* Add reviewers suggestions

* Revert to master modifications

* Line wrapping

* Revert to master
2020-01-23 19:25:13 +11:00
Pawan Dhananjay
23a35c3767 Persist/load DHT on shutdown/startup (#659)
* Store dht enrs on shutdown

* Load enrs on startup and add tests

* Remove enr_entries from behavior

* Move all dht persisting logic to `NetworkService`

* Move `PersistedDht` from eth2-libp2p to network crate

* Add test to confirm dht persistence

* Add logging

* Remove extra call to beacon_chain persist

* Expose only mutable `add_enr` method from behaviour

* Fix tests

* Fix merge errors
2020-01-23 18:16:11 +11:00
Age Manning
8c96739cab
Correct discovery address CLI functionality (#818)
* Improve handling of discovery IP address CLI config

* Remove excess debug logging

* Add reviewers suggestions
2020-01-23 17:31:08 +11:00
Age Manning
fdb6e28f94
Super/Silky smooth syncs (#816)
* Initial block processing thread design

* Correct compilation issues

* Increase logging and request from all given peers

* Patch peer request bug

* Adds fork choice to block processing

* Adds logging for bug isolation

* Patch syncing for chains with skip-slots

* Bump block processing error logs

* Improve logging for attestation processing

* Randomize peer selection during sync

* Resuming chains restarts from local finalized slot

* Downgrades Arc batches to Rc batches

* Add clippy fixes

* Downgrade Rc<Batch> to Option<Batch> to pass processed batches to chains

* Add reviewers suggestions
2020-01-23 17:30:49 +11:00
pscott
7396cd2cab Fix clippy warnings (#813)
* Clippy account manager

* Clippy account_manager

* Clippy beacon_node/beacon_chain

* Clippy beacon_node/client

* Clippy beacon_node/eth1

* Clippy beacon_node/eth2-libp2p

* Clippy beacon_node/genesis

* Clippy beacon_node/network

* Clippy beacon_node/rest_api

* Clippy beacon_node/src

* Clippy beacon_node/store

* Clippy eth2/lmd_ghost

* Clippy eth2/operation_pool

* Clippy eth2/state_processing

* Clippy eth2/types

* Clippy eth2/utils/bls

* Clippy eth2/utils/cahced_tree_hash

* Clippy eth2/utils/deposit_contract

* Clippy eth2/utils/eth2_interop_keypairs

* Clippy eth2/utils/eth2_testnet_config

* Clippy eth2/utils/lighthouse_metrics

* Clippy eth2/utils/ssz

* Clippy eth2/utils/ssz_types

* Clippy eth2/utils/tree_hash_derive

* Clippy lcli

* Clippy tests/beacon_chain_sim

* Clippy validator_client

* Cargo fmt
2020-01-21 18:38:56 +11:00
pscott
1abb964652 Update op_pool to use proper rewards (#707)
* Update op_pool to use proper rewards

* Fix missing use import for tests

* Address Michael's comments

* Revert to private ValidatorStatuses

* Rename variable for clearer code

* Fix update_cover function

* Remove expect

* Add WIP test for rewards

* Use aggregation_bits instead of earliest_attestation_validators

* Use earliest attestation in test and correct typo

* Fix op_pool test thanks to @michaelsproul 's help

* Change test name
2020-01-20 10:33:28 +11:00
Pawan Dhananjay
661ef65de8 Persist eth1 cache (#760)
* Add intermediate structures for bytes conversion

* Expose byte conversion methods from `Eth1Service`

* Add eth1 ssz containers

* Fix type errors

* Load eth1 cache on restart

* Fix compile errors

* Update Cargo.lock

* Add comments and minor formatting

* Add test for eth1 cache persistence

* Restrict Deposit and Block cache field visibility

* Add checks

* Fix `SszDepositCache` check

* Implement Encode/Decode directly on `BlockCache`
2020-01-20 07:52:58 +11:00
Age Manning
13e74e5352
Correct stream timeout delay queue handling (#803)
* Correct stream timeout delay queue handling

* Correct small typo
2020-01-15 15:50:01 +11:00
Age Manning
c184a98170
Sync fixes (#801)
* Randomize peer selection for batch errors

* Downgrade attestation logging

* Handle range sync errors

* Update lock file

* Downgrade logs

* Decrease batch size for better thread handling

* Optimise peer selection in range sync
2020-01-15 14:48:09 +11:00
Paul Hauner
271fdd4fb9
Bumb version tags (#792) 2020-01-10 16:01:37 +11:00
Paul Hauner
55a6dc9ae3
Notifier errors (#793)
* Handle notifier errors

* Log error for notifier errors
2020-01-10 16:01:22 +11:00
Michael Sproul
5a8f2dd961
Increase default slots per restore point to 2048 (#790)
This should reduce disk usage by 32x while keeping historical state queries to
less than 10s. If historical states are required quickly, the minimum SPRP of 32
can be set on the CLI.
2020-01-10 14:42:49 +11:00
Age Manning
b3712d8e9b
Discv5 upgrade (#788)
* Update discv5 to delay queues for timeouts

* Further discv5 bug fixes
2020-01-10 14:35:07 +11:00
Paul Hauner
698af6d7ec
Update docs for testnet3 (#773)
* Start docker docs

* Add progress

* Update docker.md

* Add note about geth syncing

* Fix env to use geth

* Update docs

* Remove sigp goerli node

* Change text about eth1 syncing

* Address comments from @pscott

* Add links to beaconcha.in

* Add instructions for starting an eth1 node

* Fix typo

* Re-organise Syncing log
2020-01-10 13:56:48 +11:00
Age Manning
1aaeb6b505
Upgrades trace logging for Sync RPC method (#784) 2020-01-10 02:01:03 +11:00
Michael Sproul
95fc840e2c
Fix off-by-one error in get_latest_restore_point (#787)
* Fix off-by-one error in get_latest_restore_point

* Tighten SPRP checks for succinct hot DB change
2020-01-09 21:05:56 +11:00
Michael Sproul
d9e9c17d3b
Avoid building caches during block replay (#783)
Also, make the ExitCache safe.
2020-01-09 11:43:11 +11:00