Commit Graph

3236 Commits

Author SHA1 Message Date
Paul Hauner
4d0ef41a2c
Fix testnet docs (#747)
* Fix testnet docs

* Minor fixes

* More fixes

* Describe http flag

* Tidy

* Revert change

* Address suggestion from Pawan
2020-01-06 14:31:43 +11:00
pscott
5427664cf4 Add log-format option to CLI (#744)
* Add log-format CLI option

* Cargo fmt

* Add log format logic for file logging. Add doc

* Review comment

* Fix compilation errors

* Remove Mutex from logger
2020-01-06 12:26:30 +11:00
Age Manning
55680ab1d3
Update libp2p and updates gossipsub (#764)
* Update gossipsub, adds anonymous source

* Remove temp protobuf fix
2020-01-03 17:56:03 +11:00
Pawan Dhananjay
7320f8497f Remove the logic allowing lighthouse to update it's own ENR (#682)
* Set random port when zero-port option is set

* Remove logic allowing lighthouse to update its own ENR

* Discovery address is set to localhost by default

* Return error if discovery-addr isn't explicit
2020-01-03 15:37:05 +11:00
Paul Hauner
647034b637
Optimization: avoid recomputing known state roots (#762)
* Start adding optimization

* Add temp fix for protobuf issue

* Fix compile errors

* Fix protobuf import
2020-01-03 15:09:00 +11:00
pscott
39df89521f Fix --datadir cli option (#753)
* Fix datadir bug; fix clippy warnings; update error message

* Cargo fmt

* Update error message
2020-01-03 11:00:13 +11:00
pscott
50ea669e1e Update CONTRIBUTING.md (#751)
* Replace gitter with Discord; fix typos; Specify the need to sign commits

* Add link to contributing document in lighthouse book
2020-01-03 10:45:53 +11:00
Age Manning
01ac60d638
Documents and simplifies syncing code (#738)
* Add duplication prevention to gossipsub

* Clean up topic logs

* Add content addressed messages for gossip

* Support BlocksByRange step parameter

* Add initial docs to syncing

* Adds documentation and simplify code in sync

* Remove unnecessary comment

* Fix fmt issue

* Add batch log to sync
2019-12-23 21:23:14 +11:00
mkinney
3acb3cc640 got Docker image down to 138mb from 1.8gb (#749) 2019-12-20 18:30:38 +11:00
Age Manning
41d6d5fafd
Adds support for the step parameter in BlocksByRange request (#737)
* Add duplication prevention to gossipsub

* Clean up topic logs

* Add content addressed messages for gossip

* Support BlocksByRange step parameter
2019-12-20 17:12:16 +11:00
Age Manning
45271abc16
Filter gossipsub message duplication (#736)
* Add duplication prevention to gossipsub

* Clean up topic logs

* Add content addressed messages for gossip
2019-12-20 16:26:30 +11:00
Pawan Dhananjay
74b327b50d Make eth1 caching work with fast synced node (#709)
* Add functions to get deposit_count and deposit_root from deposit cache

* Fetch deposit root and deposit count from cache

* Fix bugs

* Add test

* Compare deposit_count between the caching and http eth1 blocks

* Revert "Compare deposit_count between the caching and http eth1 blocks"

This reverts commit e3d0325ae6d1040976d37eddf35fcc7ff2333cb8.

* Fetch deposit cache using binary search instead of linear search

* BlockCache waits till DepositCache is in sync

* Truncate required_blocks in block_cache upto latest_processed_block in deposit cache

* Clean up

* Handled getting deposit count before deploying deposit contract

* More cleanup

* Remove calls to http get deposit/count

* Fix block cache tests

* Minor changes

* Fix bootnode ports

* Address some of Paul's comments

* Optimize `get_deposit_root` by caching `DepositDataTree`

* Fix comments and minor changes

* Change eth1 default config parameters

* Use `Vec` instead of `HashMap` to store `deposit_roots`

* Minor renaming
2019-12-20 09:37:39 +11:00
Paul Hauner
251aea645c
Make API friendly to block explorers (#702)
* Add validator index to duties response

* Add `get_state` method to beacon chain

* Improve /beacon/validators endpoint

* Add validators/all and validators/active endpoints

* Start refactor of HTTP docs

* Document /beacon/heads endpoint

* Remove some unused API endpoints

* Improve API docs

* Add methods to get all validator duties

* Improve docs

* Remove dead links

* Make tables left-justified

* Add /consensus/vote_count endpoint

* Add /consensus/individual_votes endpoint

* Update formatting

* Tidy

* Add committees endpoint

* Strictly require 0x prefix for serde in BLS

* Update docs to have 0x prefix

* Fix failing tests

* Add unfinished code

* Improve testing, fix bugs

* Tidy, ensure all beacon endpoints smoke tested

* Fix pubkey cache error

* Address comments with docs
2019-12-19 11:45:28 +11:00
Pawan Dhananjay
d756bc9ecd Fix bootnode ports (#743) 2019-12-18 22:40:19 +11:00
pscott
e9149f4580 Enable bulk signature verification (#739)
* Change verification strategy

* Update EF tests
2019-12-18 12:22:50 +11:00
Paul Hauner
03ed66e933 Add banner for testnet being down (#741)
LGTM!
2019-12-18 10:44:54 +11:00
Paul Hauner
49eef178cf
Clean cargo in dockerfile (#735) 2019-12-18 10:36:42 +11:00
pscott
e7de1b3339 Delete outdated deposits handling in operation pool (#719) 2019-12-17 10:37:12 +11:00
Paul Hauner
34f003adb8
Add more eth1 metrics (#728)
* Add metrics for junk eth1 votes

* Add eth1 cache metrics
2019-12-17 10:20:27 +11:00
pscott
61be1491a1 Add support for gzip (#641)
* add support for gzip

* Fix clippy warnings

* Fix additional clippy warnings and optimized get_deposits function

* Fix get_deposits function call

* Add simulator to CI

* Install ganache before running sim
2019-12-17 09:04:50 +11:00
Paul Hauner
5e4cf2f3da
Add simulator to CI (#723)
* Add simulator to CI

* Install ganache before running sim
2019-12-17 08:48:20 +11:00
pscott
0694d1d0ec Create backup instead of overwriting log files (#686)
* Create backup instead of overwriting log files

* Clean variable names

* Return string instead of expect on SystemTimeError

* Cargo fmt after merge
2019-12-15 05:59:20 +11:00
pscott
b1d4284524 Add logging level when using logfile (#721) 2019-12-13 16:11:43 +11:00
Michael Sproul
5e7803f00b Clean up database metrics, add freezer DB size (#715)
* Clean up database metrics, add freezer DB size

* Address review comments
2019-12-13 13:30:58 +11:00
Pawan Dhananjay
b9d00ee8b8 Reduced num_nodes in full_mesh gossipsub test (#718) 2019-12-12 22:38:38 +11:00
Jaden McConkey
ba1cbbba55 specify exact version for rust in Dockerfile (#703)
signed commit
2019-12-12 17:11:49 +11:00
pscott
7f484db524 Move max_seed_lookahead to used constants (#705) 2019-12-12 17:11:14 +11:00
pscott
ed9643b846 Add maximum slot limit when processing blocks (#668)
* Add maximum slot limit when processing blocks

* Fix clippy warning

* Update comments

* Change MAXIMUM_BLOCK_SLOT_NUMBER to 2^32

* Update graffiti versioning
2019-12-12 17:05:07 +11:00
Michael Sproul
4eba26572b
Store states in the DB before their blocks (#712)
Partial fix for #692
2019-12-12 12:48:18 +11:00
Paul Hauner
8a62f3f456
Hopefully fix NaN issue in sync speed (#701) 2019-12-11 11:02:54 +11:00
pscott
8e7b737912 Add note about rustup update (#696) 2019-12-11 09:31:25 +11:00
ethers
182b66f396 est_time for syncing is lower than distance (#697)
If I understood correctly, this should help avoid misunderstanding of how long a sync might take.

(Another option may be a code change that log for INFO Syncing will display est_time first, before the distance.)
2019-12-11 09:31:05 +11:00
pscott
e4c56e6c5c Fix two minor typos (#688) 2019-12-10 12:40:10 +11:00
Paul Hauner
10a134792b
Testnet2 (#685)
* Apply clippy lints to beacon node

* Remove unnecessary logging and correct formatting

* Initial bones of load-balanced range-sync

* Port bump meshsup tests

* Further structure and network handling logic added

* Basic structure, ignoring error handling

* Correct max peers delay bug

* Clean up and re-write message processor and sync manager

* Restructure directory, correct type issues

* Fix compiler issues

* Completed first testing of new sync

* Correct merge issues

* Clean up warnings

* Push attestation processed log down to dbg

* Add state enc/dec benches

* Correct math error, downgraded logs

* Add example for flamegraph

* Use `PublicKeyBytes` for `Validator`

* Ripple PublicKeyBytes change through codebase

* Add RPC error handling and improved syncing code

* Add benches, optimizations to store BeaconState

* Store BeaconState in StorageContainer too

* Optimize StorageContainer with std::mem magic

* Add libp2p stream error handling and dropping of invalid peers

* Lower logs

* Update lcli to parse spec at boot, remove pycli

* Fix issues when starting with mainnet spec

* Set default spec to mainnet

* Fix lcli --spec param

* Add discovery tweak

* Ensure ETH1_FOLLOW_DISTANCE is in YamlConfig

* Set testnet ETH1_FOLLOW_DISTANCE to 16

* Fix rest_api tests

* Set testnet min validator count

* Update with new testnet dir

* Remove some dbg, println

* Add timeout when notifier waits for libp2p lock

* Add validator count CLI flag to lcli contract deploy

* Extend genesis delay time

* Correct libp2p service locking

* Update testnet dir

* Add basic block/state caching on beacon chain

* Add decimals display to notifier sync speed

* Try merge in change to reduce fork choice calls

* Remove fork choice from process block

* Minor log fix

* Check successes > 0

* Adds checkpoint cache

* Stop storing the tree hash cache in the db

* Handles peer disconnects for sync

* Fix failing beacon chain tests

* Change eth2_testnet_config tests to Mainnet

* Add logs downgrade discovery log

* Remove dedunant beacon state write

* Fix re-org warnings

* Use caching get methods in fork choice

* Fix mistake in prev commit

* Use caching state getting in state_by_slot

* Add state.cacheless_clone

* Less fork choice (#679)

* Try merge in change to reduce fork choice calls

* Remove fork choice from process block

* Minor log fix

* Check successes > 0

* Fix failing beacon chain tests

* Fix re-org warnings

* Fix mistake in prev commit

* Attempt to improve attestation processing times

* Introduce HeadInfo struct

* Used cache tree hash for block processing

* Use cached tree hash for block production too

* Range sync refactor

- Introduces `ChainCollection`
- Correct Disconnect node handling
- Removes duplicate code

* Add more logging for DB

* Various bug fixes

* Remove unnecessary logs

* Maintain syncing state in the transition from finalied to head

* Improved disconnect handling

* Add `Speedo` struct

* Fix bugs in speedo

* Fix bug in speedo

* Fix rounding bug in speedo

* Move code around, reduce speedo observation count

* Adds forwards block interator

* Fix inf NaN

* Add first draft of validator onboarding

* Update docs

* Add documentation link to main README

* Continue docs development

* Update book readme

* Update docs

* Allow vc to run without testnet subcommand

* Small change to onboarding docs

* Tidy CLI help messages

* Update docs

* Add check to val client see if beacon node is synced

* Attempt to fix NaN bug

* Fix compile bug

* Add notifier service to validator client

* Re-order onboarding steps

* Update deposit contract address

* Update testnet dir

* Add note about public eth1 node

* Fix installation link

* Set default eth1 endpoint to sigp

* Fix broken test

* Try fix eth1 cache locking

* Be more specific about eth1 endpoint

* Increase gas limit for deposit

* Fix default deposit amount

* Fix re-org log
2019-12-09 23:14:13 +11:00
Paul Hauner
3c6c06a505
Validator on-boarding docs (#656)
* Add first draft of validator onboarding

* Update docs

* Add documentation link to main README

* Continue docs development

* Update book readme

* Update docs

* Allow vc to run without testnet subcommand

* Small change to onboarding docs

* Tidy CLI help messages

* Update docs

* Add check to val client see if beacon node is synced

* Add notifier service to validator client

* Re-order onboarding steps

* Update deposit contract address

* Update testnet dir

* Add note about public eth1 node

* Set default eth1 endpoint to sigp

* Fix broken test

* Try fix eth1 cache locking

* Be more specific about eth1 endpoint

* Increase gas limit for deposit

* Fix default deposit amount
2019-12-09 22:42:36 +11:00
Paul Hauner
f1edca30ff Add Cargo.lock (#684) 2019-12-09 22:39:44 +11:00
Paul Hauner
fc75dca9a3
Add test for when skip slots kill all validators (#657) 2019-12-09 21:17:57 +11:00
Age Manning
5853326342
Sync Re-Write (#663)
* Apply clippy lints to beacon node

* Remove unnecessary logging and correct formatting

* Initial bones of load-balanced range-sync

* Port bump meshsup tests

* Further structure and network handling logic added

* Basic structure, ignoring error handling

* Correct max peers delay bug

* Clean up and re-write message processor and sync manager

* Restructure directory, correct type issues

* Fix compiler issues

* Completed first testing of new sync

* Correct merge issues

* Clean up warnings

* Push attestation processed log down to dbg

* Correct math error, downgraded logs

* Add RPC error handling and improved syncing code

* Add libp2p stream error handling and dropping of invalid peers

* Lower logs

* Add discovery tweak

* Correct libp2p service locking

* Handles peer disconnects for sync

* Add logs downgrade discovery log

* Less fork choice (#679)

* Try merge in change to reduce fork choice calls

* Remove fork choice from process block

* Minor log fix

* Check successes > 0

* Fix failing beacon chain tests

* Fix re-org warnings

* Fix mistake in prev commit

* Range sync refactor

- Introduces `ChainCollection`
- Correct Disconnect node handling
- Removes duplicate code

* Various bug fixes

* Remove unnecessary logs

* Maintain syncing state in the transition from finalied to head

* Improved disconnect handling

* Adds forwards block interator

* Notifies lighthouse on stream timeouts

* Apply new gossipsub updates
2019-12-09 18:50:21 +11:00
Paul Hauner
988059bc9c
Fix issues when starting with mainnet spec (#674)
* Update lcli to parse spec at boot, remove pycli

* Fix issues when starting with mainnet spec

* Set default spec to mainnet

* Ensure ETH1_FOLLOW_DISTANCE is in YamlConfig

* Set testnet ETH1_FOLLOW_DISTANCE to 16

* Set testnet min validator count

* Add validator count CLI flag to lcli contract deploy

* Extend genesis delay time
2019-12-09 17:23:56 +11:00
Paul Hauner
bfbb556f02
Improve sync speed indicator (#683)
* Add `Speedo` struct

* Fix bugs in speedo

* Fix bug in speedo

* Fix rounding bug in speedo

* Move code around, reduce speedo observation count

* Fix inf NaN

* Attempt to fix NaN bug

* Fix compile bug
2019-12-09 17:23:43 +11:00
Paul Hauner
2bfc512fb6
Add block/state caching on beacon chain (#677)
* Add basic block/state caching on beacon chain

* Adds checkpoint cache

* Stop storing the tree hash cache in the db

* Remove dedunant beacon state write

* Use caching get methods in fork choice

* Use caching state getting in state_by_slot

* Add state.cacheless_clone

* Attempt to improve attestation processing times

* Introduce HeadInfo struct

* Used cache tree hash for block processing

* Use cached tree hash for block production too
2019-12-09 14:20:25 +11:00
Paul Hauner
36624b1d14 Add timeout when notifier waits for libp2p lock (#676) 2019-12-07 10:11:31 +11:00
Paul Hauner
0164742610 Add decimals display to notifier sync speed (#678) 2019-12-07 10:09:41 +11:00
Paul Hauner
d4b28d48f8
Remove some dbg, println (#675) 2019-12-07 07:29:20 +11:00
Michael Sproul
bd1b61a5b1 Forwards block root iterators (#672)
* Implement forwards block root iterators

* Clean up errors and docs
2019-12-06 18:52:11 +11:00
Paul Hauner
779873680b
Improve beacon node logging (#660)
* Squish prev commits into a single atop master

* Enable http and ws during testing

* Checkout clobbered files from master

* Further un-clobber rest_api changes

* Tidy formatting

* Do rust-fmt's job for it :|
2019-12-06 18:44:38 +11:00
Paul Hauner
75efed305c
Faster BeaconState enc/dec (#671)
* Add state enc/dec benches

* Add example for flamegraph

* Use `PublicKeyBytes` for `Validator`

* Ripple PublicKeyBytes change through codebase

* Add benches, optimizations to store BeaconState

* Store BeaconState in StorageContainer too

* Optimize StorageContainer with std::mem magic

* Fix rest_api tests
2019-12-06 16:44:03 +11:00
Michael Sproul
d0319320ce Improve freezer DB efficiency with periodic restore points (#649)
* Draft of checkpoint freezer DB

* Fix bugs

* Adjust root iterators for checkpoint database

* Fix freezer state lookups with no slot hint

* Fix split comment

* Use "restore point" to refer to frozen states

* Resolve some FIXMEs

* Configurable slots per restore point

* Document new freezer DB functions

* Fix up StoreConfig

* Fix new test for merge

* Document SPRP default CLI flag, clarify tests
2019-12-06 14:29:06 +11:00
Age Manning
5a765396b7 Implements a timeout for peer banning (#665)
* Adds peer ban timeout of 30 seconds

* Remove delay queue from discovery
2019-12-06 14:13:43 +11:00
Michael Sproul
5e8f958977
Allow multiple proposals per validator per epoch (#662)
Closes #658
2019-12-05 10:56:37 +11:00