Commit Graph

3575 Commits

Author SHA1 Message Date
Age Manning
040628bf3e
Correct issues with the syncing sim (#1008) 2020-04-15 18:46:27 +10:00
Age Manning
d3dfd72f4d
Minor useability updates (#1005)
* Adds lcli to the makefile

* Update docs

* Set genesis fork version to default
2020-04-15 17:46:03 +10:00
Age Manning
be7aaa3dbe
Adds a workaround for peer manager integration (#1003) 2020-04-14 20:32:03 +10:00
Age Manning
e5874f4565
Global Sync access (#994)
* Connect sync logic to network globals

* Add further sync info to sync status

* Build new syncing HTTP API methods

* Fix bug in updating sync state

* Highest slot is current slot

* Update book for syncing API
2020-04-14 18:17:35 +10:00
Age Manning
db7847c34a
Update CLI defaults and book (#999) 2020-04-14 15:29:29 +10:00
Age Manning
bf361e5ca3
Fix issues with testnet dir and update docs (#992)
* Second update for this PR
2020-04-14 12:23:47 +10:00
ethDreamer
065ea15c9f
Added purge subcommand to purge beacon chain db (#971) 2020-04-14 10:45:02 +10:00
divma
fa9daa488d
add handling of failed batches that imported blocks (#996) 2020-04-13 23:23:44 +10:00
Age Manning
7bf1ea2356
Fix issues with testnet dir and update docs (#992) 2020-04-13 23:20:45 +10:00
Age Manning
19b8c5a9e0
Small bug fixes from initial sim tests (#993)
* Debug logging and fixes

* Minor fixes

* Remove debugging statements
2020-04-09 14:28:37 +10:00
Age Manning
1779aa6a8a
Merge latest master in v0.2.0 2020-04-08 16:46:37 +10:00
Age Manning
b23f19272d
v0.11.1 Network update (#989)
* Minor log bumps

* Initial building of extended RPC methods

* Wire in extended RPC methods

* Merge initial peer management template

* Add a PeerDB and give the peer manager some basic functions

* Initial connection of peer manager

* Add peer manager to lighthouse

* Connect peer manager with new RPC methods

* Correct tests and metadata RPC

Co-authored-by: Diva <divma@protonmail.com>
2020-04-08 01:08:05 +10:00
Michael Sproul
869b0621d6
Update benchmarks, add to CI (#988) 2020-04-06 20:16:08 +10:00
divma
4cba745df6
make sync sim end faster by checking sync every epoch (#986) 2020-04-06 17:04:06 +10:00
Akihito Nakano
c188227cc2
Fix Eth1 data underflow (#977)
* Fix Eth1 data underflow #540

* Refactor: smart transformation from Option to Result

* Add tests for BeaconState::get_outstanding_deposit_len()
2020-04-06 15:36:34 +10:00
Paul Hauner
54782d896c
Use global pubkey cache for block processing (#849)
* Start updating types

* WIP

* Signature hacking

* Existing EF tests passing with fake_crypto

* Updates

* Delete outdated API spec

* The refactor continues

* It compiles

* WIP test fixes

* All release tests passing bar genesis state parsing

* Update and test YamlConfig

* Update to spec v0.10 compatible BLS

* Updates to BLS EF tests

* Add EF test for AggregateVerify

And delete unused hash2curve tests for uncompressed points

* Update EF tests to v0.10.1

* Use optional block root correctly in block proc

* Use genesis fork in deposit domain. All tests pass

* Cargo fmt

* Fast aggregate verify test

* Update REST API docs

* Cargo fmt

* Fix unused import

* Bump spec tags to v0.10.1

* Add `seconds_per_eth1_block` to chainspec

* Update to timestamp based eth1 voting scheme

* Return None from `get_votes_to_consider` if block cache is empty

* Handle overflows in `is_candidate_block`

* Revert to failing tests

* Fix eth1 data sets test

* Choose default vote according to spec

* Fix collect_valid_votes tests

* Fix `get_votes_to_consider` to choose all eligible blocks

* Uncomment winning_vote tests

* Add comments; remove unused code

* Reduce seconds_per_eth1_block for simulation

* Addressed review comments

* Add test for default vote case

* Fix logs

* Remove unused functions

* Meter default eth1 votes

* Fix comments

* Address review comments; remove unused dependency

* Add first attempt at attestation proc. re-write

* Add version 2 of attestation processing

* Minor fixes

* Add validator pubkey cache

* Make get_indexed_attestation take a committee

* Link signature processing into new attn verification

* First working version

* Ensure pubkey cache is updated

* Add more metrics, slight optimizations

* Clone committee cache during attestation processing

* Update shuffling cache during block processing

* Remove old commented-out code

* Fix shuffling cache insert bug

* Used indexed attestation in fork choice

* Restructure attn processing, add metrics

* Add more detailed metrics

* Tidy, fix failing tests

* Fix failing tests, tidy

* Disable/delete two outdated tests

* Add new Pubkeys struct to signature_sets

* Refactor with functional approach

* Update beacon chain

* Remove decompressed member from pubkey bytes

* Add hashmap for indices lookup

* Change `get_attesting_indices` to use Vec

* Fix failing test

* Tidy

* Add pubkey cache persistence file

* Add more comments

* Integrate persistence file into builder

* Add pubkey cache tests

* Add data_dir to beacon chain builder

* Remove Option in pubkey cache persistence file

* Ensure consistency between datadir/data_dir

* Fix failing network test

* Tidy

* Fix todos

* Improve tests

* Split up block processing metrics

* Tidy

* Refactor get_pubkey_from_state

* Remove commented-out code

* Add BeaconChain::validator_pubkey

* Use Option::filter

* Remove Box

* Comment out tests that fail due to hard-coded

Co-authored-by: Michael Sproul <michael@sigmaprime.io>
Co-authored-by: Michael Sproul <micsproul@gmail.com>
Co-authored-by: pawan <pawandhananjay@gmail.com>
2020-04-06 14:13:19 +10:00
Paul Hauner
6f3503b0e9
Add FAQ section to the book (#982)
* Add FAQ section

* Tidy
2020-04-06 10:54:10 +10:00
Paul Hauner
2fb6b7c793
Add no-copy block processing cache (#863)
* Add state cache, remove store cache

* Only build the head committee cache

* Fix compile error

* Fix compile error from merge

* Rename state_cache -> checkpoint_cache

* Rename Checkpoint -> Snapshot

* Tidy, add comments

* Tidy up find_head function

* Change some checkpoint -> snapshot

* Add tests

* Expose max_len

* Remove dead code

* Tidy

* Fix bug
2020-04-06 10:53:33 +10:00
Pawan Dhananjay
d7e2938296
Add snappy encoding to gossipsub messages (#984)
* Add snappy encode/decode to gossip messages

* Fix gossipsub tests
2020-04-05 18:29:14 +10:00
realbigsean
cc63c2b769
order of operations fix in the duration_to_next_slot calculation (#981)
* order of operations fix in the duration_to_next_slot calculation

* use slot_clock.duration_to_slot() to simplify the duration_to_subscribe calculation

* fix error message
2020-04-04 12:09:50 +11:00
Akihito Nakano
93bcee147d
Tidy Eth2Config generation at runtime (#912)
* Move the codes that loads Eth2Config from config to environment

* Move the codes that setups Eth2Config for testnet

* Move the codes that creates a new ChainSpec

* Remove unused `mut`

* Reduce local variable number

* Remove unused outputs of config::get_configs()

* Change the method name from plural to singular

* DRY the const `ETH2_CONFIG_FILENAME`

* Add comments

* Remove unnecessary blank line

* cargo fmt

* Add tests for EnvironmentBuilder::setup_eth2_config()

* Remove the comment that have been fixed

* Reduce local variable

* Remove redundant local variable

* Remove prysm-specific codes

Now the spec is in the eth2-testnets repo
2020-04-02 18:47:00 +11:00
Michael Sproul
26bdc2927b
Update to spec v0.11 (#959)
* Update process_final_updates() hysteresis computation

* Update core to v0.11.1

* Bump tags to v0.11.1

* Update docs and deposit contract

* Add compute_fork_digest

* Address review comments

Co-authored-by: Herman Alonso Junge <alonso.junge@gmail.com>
2020-04-01 22:03:03 +11:00
Age Manning
88cecd6fb8
V0.11.0 network update (#976)
* Adjust RPC methods to match v0.11.1

* Adds fork handling for gossipsub topics

* Update gossipsub topics to v0.11.0
2020-04-01 17:54:49 +11:00
Age Manning
5eb4c7d682
Updates all discovery searches to predicate searches (#975)
* Update global gossip topic handling

* Adds ENR predicate searches to lighthouse

* Correct log
2020-04-01 17:54:49 +11:00
Age Manning
d766b7fa48
Sends discovery for persistent subnets (#973) 2020-04-01 17:54:49 +11:00
Paul Hauner
7305e9e5d9
Upgrade sim (#972)
* Add progress on duties refactor

* Add simple is_aggregator bool to val subscription

* Add the no-eth1-sim, refactor sim
2020-04-01 17:54:49 +11:00
Paul Hauner
aa6f838c3c
Update /validator/subscribe (#969)
* Add progress on duties refactor

* Add simple is_aggregator bool to val subscription

* Remove unused function
2020-04-01 17:54:49 +11:00
Age Manning
cf2cb26caa
Process network attestations (#966)
* Start updating types

* WIP

* Signature hacking

* Existing EF tests passing with fake_crypto

* Updates

* Delete outdated API spec

* The refactor continues

* It compiles

* WIP test fixes

* All release tests passing bar genesis state parsing

* Update and test YamlConfig

* Update to spec v0.10 compatible BLS

* Updates to BLS EF tests

* Add EF test for AggregateVerify

And delete unused hash2curve tests for uncompressed points

* Update EF tests to v0.10.1

* Use optional block root correctly in block proc

* Use genesis fork in deposit domain. All tests pass

* Cargo fmt

* Fast aggregate verify test

* Update REST API docs

* Cargo fmt

* Fix unused import

* Bump spec tags to v0.10.1

* Add `seconds_per_eth1_block` to chainspec

* Update to timestamp based eth1 voting scheme

* Return None from `get_votes_to_consider` if block cache is empty

* Handle overflows in `is_candidate_block`

* Revert to failing tests

* Fix eth1 data sets test

* Choose default vote according to spec

* Fix collect_valid_votes tests

* Fix `get_votes_to_consider` to choose all eligible blocks

* Uncomment winning_vote tests

* Add comments; remove unused code

* Reduce seconds_per_eth1_block for simulation

* Addressed review comments

* Add test for default vote case

* Fix logs

* Remove unused functions

* Meter default eth1 votes

* Fix comments

* Address review comments; remove unused dependency

* Add first attempt at attestation proc. re-write

* Add version 2 of attestation processing

* Minor fixes

* Add validator pubkey cache

* Make get_indexed_attestation take a committee

* Link signature processing into new attn verification

* First working version

* Ensure pubkey cache is updated

* Add more metrics, slight optimizations

* Clone committee cache during attestation processing

* Update shuffling cache during block processing

* Remove old commented-out code

* Fix shuffling cache insert bug

* Used indexed attestation in fork choice

* Restructure attn processing, add metrics

* Add more detailed metrics

* Tidy, fix failing tests

* Fix failing tests, tidy

* Disable/delete two outdated tests

* Add new Pubkeys struct to signature_sets

* Refactor with functional approach

* Update beacon chain

* Remove decompressed member from pubkey bytes

* Add hashmap for indices lookup

* Add state cache, remove store cache

* Only build the head committee cache

* Change `get_attesting_indices` to use Vec

* Fix failing test

* Tidy

* Add pubkey cache persistence file

* Add more comments

* Integrate persistence file into builder

* Add pubkey cache tests

* Add data_dir to beacon chain builder

* Remove Option in pubkey cache persistence file

* Ensure consistency between datadir/data_dir

* Fix failing network test

* Tidy

* Fix todos

* Improve tests

* Fix compile error

* Fix compile error from merge

* Split up block processing metrics

* Tidy

* Refactor get_pubkey_from_state

* Remove commented-out code

* Rename state_cache -> checkpoint_cache

* Rename Checkpoint -> Snapshot

* Tidy, add comments

* Tidy up find_head function

* Change some checkpoint -> snapshot

* Add tests

* Expose max_len

* Remove dead code

* Tidy

* Fix bug

* Add sync-speed metric

* Add first attempt at VerifiableBlock

* Start integrating into beacon chain

* Integrate VerifiableBlock

* Rename VerifableBlock -> PartialBlockVerification

* Add start of typed methods

* Add progress

* Add further progress

* Rename structs

* Add full block verification to block_processing.rs

* Further beacon chain integration

* Update checks for gossip

* Add todo

* Start adding segement verification

* Add passing chain segement test

* Initial integration with batch sync

* Minor changes

* Tidy, add more error checking

* Start adding chain_segment tests

* Finish invalid signature tests

* Include single and gossip verified blocks in tests

* Add gossip verification tests

* Start adding docs

* Finish adding comments to block_processing.rs

* Rename block_processing.rs -> block_verification

* Start removing old block processing code

* Fixes beacon_chain compilation

* Fix project-wide compile errors

* Remove old code

* Fix bug with beacon proposer index

* Fix shim for BlockProcessingError

* Only process one epoch at a time

* Fix loop in chain segment processing

* Add caching for state.eth1_data_votes

* Add BeaconChain::validator_pubkey

* Revert "Add caching for state.eth1_data_votes"

This reverts commit cd73dcd6434fb8d8e6bf30c5356355598ea7b78e.

* Allow for clock disparity

* Ensure errors are returned during batch processing

* Add block gossip verification

* Connect attestation processing to beacon chain

* Optimistically subscribe to subnets on the same slot

Co-authored-by: Michael Sproul <michael@sigmaprime.io>
Co-authored-by: Michael Sproul <micsproul@gmail.com>
Co-authored-by: pawan <pawandhananjay@gmail.com>
Co-authored-by: Paul Hauner <paul@paulhauner.com>
2020-04-01 17:54:48 +11:00
realbigsean
0d45250f80
Update any references to beacon_chain.spec.milliseconds_per_slot to beacon_chain.slot_clock.slot_diration().as_nillis() in the attestation service. (#968) 2020-04-01 17:54:48 +11:00
Paul Hauner
f26bafe436
Enr fork (#967)
* Start fixing enr-fork-id

* Fix time-until-next-fork logic

* Remove fork crate
2020-04-01 17:54:48 +11:00
Age Manning
bb065e3d00
Re-merge updated block processing to v0.2.0 (#962)
* Start updating types

* WIP

* Signature hacking

* Existing EF tests passing with fake_crypto

* Updates

* Delete outdated API spec

* The refactor continues

* It compiles

* WIP test fixes

* All release tests passing bar genesis state parsing

* Update and test YamlConfig

* Update to spec v0.10 compatible BLS

* Updates to BLS EF tests

* Add EF test for AggregateVerify

And delete unused hash2curve tests for uncompressed points

* Update EF tests to v0.10.1

* Use optional block root correctly in block proc

* Use genesis fork in deposit domain. All tests pass

* Cargo fmt

* Fast aggregate verify test

* Update REST API docs

* Cargo fmt

* Fix unused import

* Bump spec tags to v0.10.1

* Add `seconds_per_eth1_block` to chainspec

* Update to timestamp based eth1 voting scheme

* Return None from `get_votes_to_consider` if block cache is empty

* Handle overflows in `is_candidate_block`

* Revert to failing tests

* Fix eth1 data sets test

* Choose default vote according to spec

* Fix collect_valid_votes tests

* Fix `get_votes_to_consider` to choose all eligible blocks

* Uncomment winning_vote tests

* Add comments; remove unused code

* Reduce seconds_per_eth1_block for simulation

* Addressed review comments

* Add test for default vote case

* Fix logs

* Remove unused functions

* Meter default eth1 votes

* Fix comments

* Address review comments; remove unused dependency

* Add first attempt at attestation proc. re-write

* Add version 2 of attestation processing

* Minor fixes

* Add validator pubkey cache

* Make get_indexed_attestation take a committee

* Link signature processing into new attn verification

* First working version

* Ensure pubkey cache is updated

* Add more metrics, slight optimizations

* Clone committee cache during attestation processing

* Update shuffling cache during block processing

* Remove old commented-out code

* Fix shuffling cache insert bug

* Used indexed attestation in fork choice

* Restructure attn processing, add metrics

* Add more detailed metrics

* Tidy, fix failing tests

* Fix failing tests, tidy

* Disable/delete two outdated tests

* Add new Pubkeys struct to signature_sets

* Refactor with functional approach

* Update beacon chain

* Remove decompressed member from pubkey bytes

* Add hashmap for indices lookup

* Add state cache, remove store cache

* Only build the head committee cache

* Change `get_attesting_indices` to use Vec

* Fix failing test

* Tidy

* Add pubkey cache persistence file

* Add more comments

* Integrate persistence file into builder

* Add pubkey cache tests

* Add data_dir to beacon chain builder

* Remove Option in pubkey cache persistence file

* Ensure consistency between datadir/data_dir

* Fix failing network test

* Tidy

* Fix todos

* Improve tests

* Fix compile error

* Fix compile error from merge

* Split up block processing metrics

* Tidy

* Refactor get_pubkey_from_state

* Remove commented-out code

* Rename state_cache -> checkpoint_cache

* Rename Checkpoint -> Snapshot

* Tidy, add comments

* Tidy up find_head function

* Change some checkpoint -> snapshot

* Add tests

* Expose max_len

* Remove dead code

* Tidy

* Fix bug

* Add sync-speed metric

* Add first attempt at VerifiableBlock

* Start integrating into beacon chain

* Integrate VerifiableBlock

* Rename VerifableBlock -> PartialBlockVerification

* Add start of typed methods

* Add progress

* Add further progress

* Rename structs

* Add full block verification to block_processing.rs

* Further beacon chain integration

* Update checks for gossip

* Add todo

* Start adding segement verification

* Add passing chain segement test

* Initial integration with batch sync

* Minor changes

* Tidy, add more error checking

* Start adding chain_segment tests

* Finish invalid signature tests

* Include single and gossip verified blocks in tests

* Add gossip verification tests

* Start adding docs

* Finish adding comments to block_processing.rs

* Rename block_processing.rs -> block_verification

* Start removing old block processing code

* Fixes beacon_chain compilation

* Fix project-wide compile errors

* Remove old code

* Fix bug with beacon proposer index

* Fix shim for BlockProcessingError

* Only process one epoch at a time

* Fix loop in chain segment processing

* Add caching for state.eth1_data_votes

* Add BeaconChain::validator_pubkey

* Revert "Add caching for state.eth1_data_votes"

This reverts commit cd73dcd6434fb8d8e6bf30c5356355598ea7b78e.

* Add sync-speed metric (#898)

* Add PH & MS slot clock changes

* Account for genesis time

* Use checked mul

* Account for genesis slot

* Change API

* Allow for clock disparity

* Refactor "duration to..." functions

* Ensure errors are returned during batch processing

Co-authored-by: Michael Sproul <michael@sigmaprime.io>
Co-authored-by: Michael Sproul <micsproul@gmail.com>
Co-authored-by: pawan <pawandhananjay@gmail.com>
Co-authored-by: Paul Hauner <paul@paulhauner.com>
2020-04-01 17:54:48 +11:00
Paul Hauner
3606f0447d
Merge latest from branch clock-disparity
* Account for genesis time

* Use checked mul

* Account for genesis slot

* Change API

* Refactor "duration to..." functions
2020-04-01 17:53:55 +11:00
Paul Hauner
e04fc8ddb4
Don't override the fork in lcli eth1-genesis (#948)
* Don't override the fork in lcli

* Add spec flag to lcli
2020-04-01 17:42:24 +11:00
Paul Hauner
88d37e96fa
Fix bug when resuming from DB prior to genesis (#946)
* Use default client genesis if no chain in store

* Always use resume if there is a beacon chain
2020-04-01 17:41:59 +11:00
Paul Hauner
0759806c89
Enable eth1 flag when eth1-endpoint is suppilied (#941) 2020-04-01 17:41:19 +11:00
Paul Hauner
5b984ad394
Add lcli tool for checking deposit data (#940)
* Add check-deposit-data tool

* Update help text

* Update function name
2020-04-01 17:40:32 +11:00
Paul Hauner
11a238900a
Allow slot clock to handle clock disparity (#929)
* Add PH & MS slot clock changes

* Account for genesis time

* Use checked mul

* Account for genesis slot

* Change API

* Refactor "duration to..." functions
2020-04-01 17:40:04 +11:00
Paul Hauner
5c397c49d8
Fix error in /consensus/individual_votes (#859)
* Add patch and commented-out tests

* Delete commented-out tests
2020-04-01 17:39:28 +11:00
Paul Hauner
5a6e90428b
Ensure attestations are created with empty signature (#960)
* Ensure attestations are created with empty sig

* Update docs
2020-03-30 09:29:29 +11:00
Age Manning
6ca4f4709b
Connects the attestation service to network components (#961)
* Sends attestations to the attestation service for processing

* Adds 'attnets' field to local ENR

* Adds ENR bitfield modification logic

* Link attestation service to discovery

- Updates discv5
- Links discover events to discovery
- Support for ENRBitfield

* Adds discovery config params, correct warnings

* Rust fmt fixes

* Correct tests
2020-03-25 22:18:06 +11:00
Paul Hauner
fbcf0f8e2e
Update VC and BN APIs for naive aggregation (#950)
* Refactor `Attestation` production

* Add constant

* Start refactor for aggregation

* Return early when no attesting validators

* Refactor into individual functions

* Tidy, add comments

* Add first draft of NaiveAggregationPool

* Further progress on naive aggregation pool

* Fix compile errors in VC

* Change locking logic for naive pool

* Introduce AttesationType

* Add pruning, comments

* Add MAX_ATTESTATIONS_PER_SLOT restriction

* Add pruning based on slot

* Update BN for new aggregation fns

* Fix test compile errors

* Fix failing rest_api test

* Move SignedAggregateAndProof into own file

* Update docs, fix warning

* Tidy some formatting in validator API

* Remove T::default_spec from signing

* Fix failing rest test

* Tidy

* Add test, fix bug

* Improve naive pool tests

* Add max attestations test

* Revert changes to the op_pool

* Refactor timer
2020-03-25 21:14:05 +11:00
Paul Hauner
784997b09b
Ensure correct path for genesis state file (#951) 2020-03-25 12:31:11 +11:00
Age Manning
58111cddb2
Adds ENR "eth2" field and Fork logic to networking (#953)
* Merge #913

* Correct release tests

* Completed release test corrections

* Initial work on upgrading discovery

* Updates discovery to latest version

* Update ENR initialisation logic

* Remove debug statements

* Shifts timing units to slots

* Initial work

* Add initial fork versioning and EnrForkId

* Correct linking for EnrForkId

* Adds eth2 field to local ENR

* Initial work to eth2 field integration

* Integrate eth2 field into discovery

* temp commit

* Add a timer to adjust fork versions during a hard fork for the ENR
2020-03-24 21:45:53 +11:00
Age Manning
af1c5c326c
Handle timing underflows in attestation service (#952) 2020-03-24 21:05:22 +11:00
Age Manning
2b6da4b8de
Initial fork versioning (#934)
* Merge #913

* Correct release tests

* Completed release test corrections

* Initial work on upgrading discovery

* Updates discovery to latest version

* Update ENR initialisation logic

* Remove debug statements

* Shifts timing units to slots

* Initial work

* Add initial fork versioning and EnrForkId

* Correct linking for EnrForkId
2020-03-24 19:10:28 +11:00
Sacha Saint-Leger
f6a6de2c5d
Become a Validator guides: update (#928)
* Edit become-a-validator-docker.md

* Update Become a Validator guides

* Update Become a Validator guides

* Update Become a Validator guides

* fix inconsistency
2020-03-23 19:21:53 +11:00
Age Manning
f8bc045a01
Fix race condition in the syncing sim (#944)
* Fix race condition in the syncing sim

* Update another strategy
2020-03-23 18:28:13 +11:00
divma
47aef629d1
move the parent lookup process to a dedicated thread (#906)
* Upgrade the parent lookup logic

* Apply reviewer suggestions

* move the parent lookup process to a dedicated thread

* move the logic of parent lookup and range syncing to a block processor

* review suggestions

* more review suggestions

* Add small logging changes

* Process parent lookups in reverse

Co-authored-by: Age Manning <Age@AgeManning.com>
2020-03-23 12:07:41 +11:00
/raw PONG _GHMoaCXLT
a5fbaef469
Github: Add version to issue template (#937) 2020-03-20 12:18:31 +11:00
Age Manning
24a384d0d6
Attestation service slots (#931)
* Merge #913

* Correct release tests

* Completed release test corrections

* Initial work on upgrading discovery

* Updates discovery to latest version

* Update ENR initialisation logic

* Remove debug statements

* Shifts timing units to slots
2020-03-19 15:11:54 +11:00