Commit Graph

1007 Commits

Author SHA1 Message Date
Age Manning
fdb6e28f94
Super/Silky smooth syncs (#816)
* Initial block processing thread design

* Correct compilation issues

* Increase logging and request from all given peers

* Patch peer request bug

* Adds fork choice to block processing

* Adds logging for bug isolation

* Patch syncing for chains with skip-slots

* Bump block processing error logs

* Improve logging for attestation processing

* Randomize peer selection during sync

* Resuming chains restarts from local finalized slot

* Downgrades Arc batches to Rc batches

* Add clippy fixes

* Downgrade Rc<Batch> to Option<Batch> to pass processed batches to chains

* Add reviewers suggestions
2020-01-23 17:30:49 +11:00
pscott
7396cd2cab Fix clippy warnings (#813)
* Clippy account manager

* Clippy account_manager

* Clippy beacon_node/beacon_chain

* Clippy beacon_node/client

* Clippy beacon_node/eth1

* Clippy beacon_node/eth2-libp2p

* Clippy beacon_node/genesis

* Clippy beacon_node/network

* Clippy beacon_node/rest_api

* Clippy beacon_node/src

* Clippy beacon_node/store

* Clippy eth2/lmd_ghost

* Clippy eth2/operation_pool

* Clippy eth2/state_processing

* Clippy eth2/types

* Clippy eth2/utils/bls

* Clippy eth2/utils/cahced_tree_hash

* Clippy eth2/utils/deposit_contract

* Clippy eth2/utils/eth2_interop_keypairs

* Clippy eth2/utils/eth2_testnet_config

* Clippy eth2/utils/lighthouse_metrics

* Clippy eth2/utils/ssz

* Clippy eth2/utils/ssz_types

* Clippy eth2/utils/tree_hash_derive

* Clippy lcli

* Clippy tests/beacon_chain_sim

* Clippy validator_client

* Cargo fmt
2020-01-21 18:38:56 +11:00
pscott
1abb964652 Update op_pool to use proper rewards (#707)
* Update op_pool to use proper rewards

* Fix missing use import for tests

* Address Michael's comments

* Revert to private ValidatorStatuses

* Rename variable for clearer code

* Fix update_cover function

* Remove expect

* Add WIP test for rewards

* Use aggregation_bits instead of earliest_attestation_validators

* Use earliest attestation in test and correct typo

* Fix op_pool test thanks to @michaelsproul 's help

* Change test name
2020-01-20 10:33:28 +11:00
Pawan Dhananjay
661ef65de8 Persist eth1 cache (#760)
* Add intermediate structures for bytes conversion

* Expose byte conversion methods from `Eth1Service`

* Add eth1 ssz containers

* Fix type errors

* Load eth1 cache on restart

* Fix compile errors

* Update Cargo.lock

* Add comments and minor formatting

* Add test for eth1 cache persistence

* Restrict Deposit and Block cache field visibility

* Add checks

* Fix `SszDepositCache` check

* Implement Encode/Decode directly on `BlockCache`
2020-01-20 07:52:58 +11:00
Age Manning
13e74e5352
Correct stream timeout delay queue handling (#803)
* Correct stream timeout delay queue handling

* Correct small typo
2020-01-15 15:50:01 +11:00
Age Manning
c184a98170
Sync fixes (#801)
* Randomize peer selection for batch errors

* Downgrade attestation logging

* Handle range sync errors

* Update lock file

* Downgrade logs

* Decrease batch size for better thread handling

* Optimise peer selection in range sync
2020-01-15 14:48:09 +11:00
Paul Hauner
271fdd4fb9
Bumb version tags (#792) 2020-01-10 16:01:37 +11:00
Paul Hauner
55a6dc9ae3
Notifier errors (#793)
* Handle notifier errors

* Log error for notifier errors
2020-01-10 16:01:22 +11:00
Michael Sproul
5a8f2dd961
Increase default slots per restore point to 2048 (#790)
This should reduce disk usage by 32x while keeping historical state queries to
less than 10s. If historical states are required quickly, the minimum SPRP of 32
can be set on the CLI.
2020-01-10 14:42:49 +11:00
Age Manning
b3712d8e9b
Discv5 upgrade (#788)
* Update discv5 to delay queues for timeouts

* Further discv5 bug fixes
2020-01-10 14:35:07 +11:00
Paul Hauner
698af6d7ec
Update docs for testnet3 (#773)
* Start docker docs

* Add progress

* Update docker.md

* Add note about geth syncing

* Fix env to use geth

* Update docs

* Remove sigp goerli node

* Change text about eth1 syncing

* Address comments from @pscott

* Add links to beaconcha.in

* Add instructions for starting an eth1 node

* Fix typo

* Re-organise Syncing log
2020-01-10 13:56:48 +11:00
Age Manning
1aaeb6b505
Upgrades trace logging for Sync RPC method (#784) 2020-01-10 02:01:03 +11:00
Michael Sproul
95fc840e2c
Fix off-by-one error in get_latest_restore_point (#787)
* Fix off-by-one error in get_latest_restore_point

* Tighten SPRP checks for succinct hot DB change
2020-01-09 21:05:56 +11:00
Michael Sproul
d9e9c17d3b
Avoid building caches during block replay (#783)
Also, make the ExitCache safe.
2020-01-09 11:43:11 +11:00
Paul Hauner
da95a73605
Fix typo in eth1 service (#782) 2020-01-09 10:53:25 +11:00
Age Manning
30f51df4cf
Improved RPC timeout handling (#770)
* Handle slow app level responses for RPC

* Correct errors

* Terminates stream on error
2020-01-08 14:18:06 +11:00
Michael Sproul
f36a5a15d6
Store states efficiently in the hot database (#746)
* Sparse hot DB and block root tree

* Fix store_tests

* Ensure loads of hot states on boundaries are fast

* Milder error for unaligned finalized blocks
2020-01-08 13:58:01 +11:00
Paul Hauner
26dde26c48
Use Cow for checkpoint cache (#775) 2020-01-08 11:09:27 +11:00
pscott
95cc5dd22f Rename ssz_encode_state.rs to ssz_encode_state_container.rs (#774) 2020-01-07 10:33:58 +11:00
Paul Hauner
f04c55075e
Add timeouts to canonical head rwlock (#759)
* Add TimeoutRwLock to BeaconChain

* Update network crate

* Update rest api

* Fix beacon chain tests

* Fix rest api tests

* Set test back to !debug_assertions
2020-01-06 17:30:37 +11:00
Paul Hauner
b0c8b2b700
Change genesis fork for testnets (#750)
* Change genesis fork for testnets

* Fix genesis fork in lcli
2020-01-06 17:30:08 +11:00
Age Manning
55680ab1d3
Update libp2p and updates gossipsub (#764)
* Update gossipsub, adds anonymous source

* Remove temp protobuf fix
2020-01-03 17:56:03 +11:00
Pawan Dhananjay
7320f8497f Remove the logic allowing lighthouse to update it's own ENR (#682)
* Set random port when zero-port option is set

* Remove logic allowing lighthouse to update its own ENR

* Discovery address is set to localhost by default

* Return error if discovery-addr isn't explicit
2020-01-03 15:37:05 +11:00
Paul Hauner
647034b637
Optimization: avoid recomputing known state roots (#762)
* Start adding optimization

* Add temp fix for protobuf issue

* Fix compile errors

* Fix protobuf import
2020-01-03 15:09:00 +11:00
pscott
39df89521f Fix --datadir cli option (#753)
* Fix datadir bug; fix clippy warnings; update error message

* Cargo fmt

* Update error message
2020-01-03 11:00:13 +11:00
Age Manning
01ac60d638
Documents and simplifies syncing code (#738)
* Add duplication prevention to gossipsub

* Clean up topic logs

* Add content addressed messages for gossip

* Support BlocksByRange step parameter

* Add initial docs to syncing

* Adds documentation and simplify code in sync

* Remove unnecessary comment

* Fix fmt issue

* Add batch log to sync
2019-12-23 21:23:14 +11:00
Age Manning
41d6d5fafd
Adds support for the step parameter in BlocksByRange request (#737)
* Add duplication prevention to gossipsub

* Clean up topic logs

* Add content addressed messages for gossip

* Support BlocksByRange step parameter
2019-12-20 17:12:16 +11:00
Age Manning
45271abc16
Filter gossipsub message duplication (#736)
* Add duplication prevention to gossipsub

* Clean up topic logs

* Add content addressed messages for gossip
2019-12-20 16:26:30 +11:00
Pawan Dhananjay
74b327b50d Make eth1 caching work with fast synced node (#709)
* Add functions to get deposit_count and deposit_root from deposit cache

* Fetch deposit root and deposit count from cache

* Fix bugs

* Add test

* Compare deposit_count between the caching and http eth1 blocks

* Revert "Compare deposit_count between the caching and http eth1 blocks"

This reverts commit e3d0325ae6d1040976d37eddf35fcc7ff2333cb8.

* Fetch deposit cache using binary search instead of linear search

* BlockCache waits till DepositCache is in sync

* Truncate required_blocks in block_cache upto latest_processed_block in deposit cache

* Clean up

* Handled getting deposit count before deploying deposit contract

* More cleanup

* Remove calls to http get deposit/count

* Fix block cache tests

* Minor changes

* Fix bootnode ports

* Address some of Paul's comments

* Optimize `get_deposit_root` by caching `DepositDataTree`

* Fix comments and minor changes

* Change eth1 default config parameters

* Use `Vec` instead of `HashMap` to store `deposit_roots`

* Minor renaming
2019-12-20 09:37:39 +11:00
Paul Hauner
251aea645c
Make API friendly to block explorers (#702)
* Add validator index to duties response

* Add `get_state` method to beacon chain

* Improve /beacon/validators endpoint

* Add validators/all and validators/active endpoints

* Start refactor of HTTP docs

* Document /beacon/heads endpoint

* Remove some unused API endpoints

* Improve API docs

* Add methods to get all validator duties

* Improve docs

* Remove dead links

* Make tables left-justified

* Add /consensus/vote_count endpoint

* Add /consensus/individual_votes endpoint

* Update formatting

* Tidy

* Add committees endpoint

* Strictly require 0x prefix for serde in BLS

* Update docs to have 0x prefix

* Fix failing tests

* Add unfinished code

* Improve testing, fix bugs

* Tidy, ensure all beacon endpoints smoke tested

* Fix pubkey cache error

* Address comments with docs
2019-12-19 11:45:28 +11:00
pscott
e9149f4580 Enable bulk signature verification (#739)
* Change verification strategy

* Update EF tests
2019-12-18 12:22:50 +11:00
pscott
e7de1b3339 Delete outdated deposits handling in operation pool (#719) 2019-12-17 10:37:12 +11:00
Paul Hauner
34f003adb8
Add more eth1 metrics (#728)
* Add metrics for junk eth1 votes

* Add eth1 cache metrics
2019-12-17 10:20:27 +11:00
pscott
61be1491a1 Add support for gzip (#641)
* add support for gzip

* Fix clippy warnings

* Fix additional clippy warnings and optimized get_deposits function

* Fix get_deposits function call

* Add simulator to CI

* Install ganache before running sim
2019-12-17 09:04:50 +11:00
Michael Sproul
5e7803f00b Clean up database metrics, add freezer DB size (#715)
* Clean up database metrics, add freezer DB size

* Address review comments
2019-12-13 13:30:58 +11:00
Pawan Dhananjay
b9d00ee8b8 Reduced num_nodes in full_mesh gossipsub test (#718) 2019-12-12 22:38:38 +11:00
pscott
ed9643b846 Add maximum slot limit when processing blocks (#668)
* Add maximum slot limit when processing blocks

* Fix clippy warning

* Update comments

* Change MAXIMUM_BLOCK_SLOT_NUMBER to 2^32

* Update graffiti versioning
2019-12-12 17:05:07 +11:00
Michael Sproul
4eba26572b
Store states in the DB before their blocks (#712)
Partial fix for #692
2019-12-12 12:48:18 +11:00
Paul Hauner
8a62f3f456
Hopefully fix NaN issue in sync speed (#701) 2019-12-11 11:02:54 +11:00
Paul Hauner
10a134792b
Testnet2 (#685)
* Apply clippy lints to beacon node

* Remove unnecessary logging and correct formatting

* Initial bones of load-balanced range-sync

* Port bump meshsup tests

* Further structure and network handling logic added

* Basic structure, ignoring error handling

* Correct max peers delay bug

* Clean up and re-write message processor and sync manager

* Restructure directory, correct type issues

* Fix compiler issues

* Completed first testing of new sync

* Correct merge issues

* Clean up warnings

* Push attestation processed log down to dbg

* Add state enc/dec benches

* Correct math error, downgraded logs

* Add example for flamegraph

* Use `PublicKeyBytes` for `Validator`

* Ripple PublicKeyBytes change through codebase

* Add RPC error handling and improved syncing code

* Add benches, optimizations to store BeaconState

* Store BeaconState in StorageContainer too

* Optimize StorageContainer with std::mem magic

* Add libp2p stream error handling and dropping of invalid peers

* Lower logs

* Update lcli to parse spec at boot, remove pycli

* Fix issues when starting with mainnet spec

* Set default spec to mainnet

* Fix lcli --spec param

* Add discovery tweak

* Ensure ETH1_FOLLOW_DISTANCE is in YamlConfig

* Set testnet ETH1_FOLLOW_DISTANCE to 16

* Fix rest_api tests

* Set testnet min validator count

* Update with new testnet dir

* Remove some dbg, println

* Add timeout when notifier waits for libp2p lock

* Add validator count CLI flag to lcli contract deploy

* Extend genesis delay time

* Correct libp2p service locking

* Update testnet dir

* Add basic block/state caching on beacon chain

* Add decimals display to notifier sync speed

* Try merge in change to reduce fork choice calls

* Remove fork choice from process block

* Minor log fix

* Check successes > 0

* Adds checkpoint cache

* Stop storing the tree hash cache in the db

* Handles peer disconnects for sync

* Fix failing beacon chain tests

* Change eth2_testnet_config tests to Mainnet

* Add logs downgrade discovery log

* Remove dedunant beacon state write

* Fix re-org warnings

* Use caching get methods in fork choice

* Fix mistake in prev commit

* Use caching state getting in state_by_slot

* Add state.cacheless_clone

* Less fork choice (#679)

* Try merge in change to reduce fork choice calls

* Remove fork choice from process block

* Minor log fix

* Check successes > 0

* Fix failing beacon chain tests

* Fix re-org warnings

* Fix mistake in prev commit

* Attempt to improve attestation processing times

* Introduce HeadInfo struct

* Used cache tree hash for block processing

* Use cached tree hash for block production too

* Range sync refactor

- Introduces `ChainCollection`
- Correct Disconnect node handling
- Removes duplicate code

* Add more logging for DB

* Various bug fixes

* Remove unnecessary logs

* Maintain syncing state in the transition from finalied to head

* Improved disconnect handling

* Add `Speedo` struct

* Fix bugs in speedo

* Fix bug in speedo

* Fix rounding bug in speedo

* Move code around, reduce speedo observation count

* Adds forwards block interator

* Fix inf NaN

* Add first draft of validator onboarding

* Update docs

* Add documentation link to main README

* Continue docs development

* Update book readme

* Update docs

* Allow vc to run without testnet subcommand

* Small change to onboarding docs

* Tidy CLI help messages

* Update docs

* Add check to val client see if beacon node is synced

* Attempt to fix NaN bug

* Fix compile bug

* Add notifier service to validator client

* Re-order onboarding steps

* Update deposit contract address

* Update testnet dir

* Add note about public eth1 node

* Fix installation link

* Set default eth1 endpoint to sigp

* Fix broken test

* Try fix eth1 cache locking

* Be more specific about eth1 endpoint

* Increase gas limit for deposit

* Fix default deposit amount

* Fix re-org log
2019-12-09 23:14:13 +11:00
Paul Hauner
3c6c06a505
Validator on-boarding docs (#656)
* Add first draft of validator onboarding

* Update docs

* Add documentation link to main README

* Continue docs development

* Update book readme

* Update docs

* Allow vc to run without testnet subcommand

* Small change to onboarding docs

* Tidy CLI help messages

* Update docs

* Add check to val client see if beacon node is synced

* Add notifier service to validator client

* Re-order onboarding steps

* Update deposit contract address

* Update testnet dir

* Add note about public eth1 node

* Set default eth1 endpoint to sigp

* Fix broken test

* Try fix eth1 cache locking

* Be more specific about eth1 endpoint

* Increase gas limit for deposit

* Fix default deposit amount
2019-12-09 22:42:36 +11:00
Paul Hauner
fc75dca9a3
Add test for when skip slots kill all validators (#657) 2019-12-09 21:17:57 +11:00
Age Manning
5853326342
Sync Re-Write (#663)
* Apply clippy lints to beacon node

* Remove unnecessary logging and correct formatting

* Initial bones of load-balanced range-sync

* Port bump meshsup tests

* Further structure and network handling logic added

* Basic structure, ignoring error handling

* Correct max peers delay bug

* Clean up and re-write message processor and sync manager

* Restructure directory, correct type issues

* Fix compiler issues

* Completed first testing of new sync

* Correct merge issues

* Clean up warnings

* Push attestation processed log down to dbg

* Correct math error, downgraded logs

* Add RPC error handling and improved syncing code

* Add libp2p stream error handling and dropping of invalid peers

* Lower logs

* Add discovery tweak

* Correct libp2p service locking

* Handles peer disconnects for sync

* Add logs downgrade discovery log

* Less fork choice (#679)

* Try merge in change to reduce fork choice calls

* Remove fork choice from process block

* Minor log fix

* Check successes > 0

* Fix failing beacon chain tests

* Fix re-org warnings

* Fix mistake in prev commit

* Range sync refactor

- Introduces `ChainCollection`
- Correct Disconnect node handling
- Removes duplicate code

* Various bug fixes

* Remove unnecessary logs

* Maintain syncing state in the transition from finalied to head

* Improved disconnect handling

* Adds forwards block interator

* Notifies lighthouse on stream timeouts

* Apply new gossipsub updates
2019-12-09 18:50:21 +11:00
Paul Hauner
988059bc9c
Fix issues when starting with mainnet spec (#674)
* Update lcli to parse spec at boot, remove pycli

* Fix issues when starting with mainnet spec

* Set default spec to mainnet

* Ensure ETH1_FOLLOW_DISTANCE is in YamlConfig

* Set testnet ETH1_FOLLOW_DISTANCE to 16

* Set testnet min validator count

* Add validator count CLI flag to lcli contract deploy

* Extend genesis delay time
2019-12-09 17:23:56 +11:00
Paul Hauner
bfbb556f02
Improve sync speed indicator (#683)
* Add `Speedo` struct

* Fix bugs in speedo

* Fix bug in speedo

* Fix rounding bug in speedo

* Move code around, reduce speedo observation count

* Fix inf NaN

* Attempt to fix NaN bug

* Fix compile bug
2019-12-09 17:23:43 +11:00
Paul Hauner
2bfc512fb6
Add block/state caching on beacon chain (#677)
* Add basic block/state caching on beacon chain

* Adds checkpoint cache

* Stop storing the tree hash cache in the db

* Remove dedunant beacon state write

* Use caching get methods in fork choice

* Use caching state getting in state_by_slot

* Add state.cacheless_clone

* Attempt to improve attestation processing times

* Introduce HeadInfo struct

* Used cache tree hash for block processing

* Use cached tree hash for block production too
2019-12-09 14:20:25 +11:00
Paul Hauner
36624b1d14 Add timeout when notifier waits for libp2p lock (#676) 2019-12-07 10:11:31 +11:00
Paul Hauner
0164742610 Add decimals display to notifier sync speed (#678) 2019-12-07 10:09:41 +11:00
Paul Hauner
d4b28d48f8
Remove some dbg, println (#675) 2019-12-07 07:29:20 +11:00
Michael Sproul
bd1b61a5b1 Forwards block root iterators (#672)
* Implement forwards block root iterators

* Clean up errors and docs
2019-12-06 18:52:11 +11:00