* Adds counter of received chunks to an OutboundSubstream. Ends the stream when the counter reaches the desired amount of chunks that where specified in a Request.
* Keeps track of remaining chunks for a stream, rather than expected ones and calculating the remainder on each received chunk
* WIP test, waiting for stable-futures to land in master
* Improve calculation for remaining chunks in response handler. Improve initial calculation for expected chunks in outbount substream
* Remove rebase artifact
* Fix compiler errors after rebasing on master
* Clone request to allow two accesses to it that move it when determining the amount of expected responses
* Correctly terminate the stream when all chunks have been received
* WIP: test that stream is terminated correctly
* Terminate stream with a termination response. Handle further received chunks in OutboundStream::Closing branch to return errors
* Remove request clone
* Report stream timeouts when closing
* Update rpc test
* Fix BlocksByRoot RPC test to request as many chunks as responses are expected
* Adds test for correctly termined BlocksByRoot rpc stream when all chunks have been received
Co-authored-by: Age Manning <Age@AgeManning.com>
* Implement slashing protection
Roll-up of #588 with some conflicts resolved
* WIP improvements
* Require slot uniqueness for blocks (rather than epochs)
* Native DB support for Slot and Epoch
* Simplify surrounding/surrounded-by queries
* Implement unified slashing protection database
A single SQL database saves on open file descriptors.
* Make slashing protection concurrency safe.
Revive tests, add parallel tests.
* Some simplifications
* Auto-registration, test clean-ups
* More tests, clean-ups, hardening
* Fix comments in BLS
* Optimise bulk validator registration
* Delete outdated tests
* Use bundled SQLite in slashing protection
* Auto-register validators in simulation
* Use real signing_root in slashing protection
* Update book for --auto-register
* Refine log messages and help flags
* Correct typo in Cargo.toml authors
* Fix merge conflicts
* Safer error handling in sqlite slot/epoch
* Address review comments
* Add attestation test mutating block root
Co-authored-by: pscott <scottpiriou@gmail.com>
* Cleanup `ExitCache`
Minor suggested cleanups after familiarising myself with the `ExitCache`.
* Remove "validators exiting/exited at a given epoch" comment in favour of the notion of exit epoch (less wishy-washy).
* Remove "or zero if not known" comment. The number of validators with that exit epoch is known, even in the case where it's zero.
* Rename `epoch` to `exit_epoch` for consistency and clarity.
* Rename `exits_per_epoch` to `exit_epoch_counts` for precision and clarity.
* Remove seemingly unnecessary complexity with `force_build`.
* Consider renaming `ExitCache` to `ExitEpochCache` for clarity.
* Update exit_cache.rs
* Port eth1 lib to use stable futures
* Port eth1_test_rig to stable futures
* Port eth1 tests to stable futures
* Port genesis service to stable futures
* Port genesis tests to stable futures
* Port beacon_chain to stable futures
* Port lcli to stable futures
* Fix eth1_test_rig (#1014)
* Fix lcli
* Port timer to stable futures
* Fix timer
* Port websocket_server to stable futures
* Port notifier to stable futures
* Add TODOS
* Update hashmap hashset to stable futures
* Adds panic test to hashset delay
* Port remote_beacon_node to stable futures
* Fix lcli merge conflicts
* Non rpc stuff compiles
* protocol.rs compiles
* Port websockets, timer and notifier to stable futures (#1035)
* Fix lcli
* Port timer to stable futures
* Fix timer
* Port websocket_server to stable futures
* Port notifier to stable futures
* Add TODOS
* Port remote_beacon_node to stable futures
* Partial eth2-libp2p stable future upgrade
* Finished first round of fighting RPC types
* Further progress towards porting eth2-libp2p adds caching to discovery
* Update behaviour
* RPC handler to stable futures
* Update RPC to master libp2p
* Network service additions
* Fix the fallback transport construction (#1102)
* Correct warning
* Remove hashmap delay
* Compiling version of eth2-libp2p
* Update all crates versions
* Fix conversion function and add tests (#1113)
* Port validator_client to stable futures (#1114)
* Add PH & MS slot clock changes
* Account for genesis time
* Add progress on duties refactor
* Add simple is_aggregator bool to val subscription
* Start work on attestation_verification.rs
* Add progress on ObservedAttestations
* Progress with ObservedAttestations
* Fix tests
* Add observed attestations to the beacon chain
* Add attestation observation to processing code
* Add progress on attestation verification
* Add first draft of ObservedAttesters
* Add more tests
* Add observed attesters to beacon chain
* Add observers to attestation processing
* Add more attestation verification
* Create ObservedAggregators map
* Remove commented-out code
* Add observed aggregators into chain
* Add progress
* Finish adding features to attestation verification
* Ensure beacon chain compiles
* Link attn verification into chain
* Integrate new attn verification in chain
* Remove old attestation processing code
* Start trying to fix beacon_chain tests
* Split adding into pools into two functions
* Add aggregation to harness
* Get test harness working again
* Adjust the number of aggregators for test harness
* Fix edge-case in harness
* Integrate new attn processing in network
* Fix compile bug in validator_client
* Update validator API endpoints
* Fix aggreagation in test harness
* Fix enum thing
* Fix attestation observation bug:
* Patch failing API tests
* Start adding comments to attestation verification
* Remove unused attestation field
* Unify "is block known" logic
* Update comments
* Supress fork choice errors for network processing
* Add todos
* Tidy
* Add gossip attn tests
* Disallow test harness to produce old attns
* Comment out in-progress tests
* Partially address pruning tests
* Fix failing store test
* Add aggregate tests
* Add comments about which spec conditions we check
* Dont re-aggregate
* Split apart test harness attn production
* Fix compile error in network
* Make progress on commented-out test
* Fix skipping attestation test
* Add fork choice verification tests
* Tidy attn tests, remove dead code
* Remove some accidentally added code
* Fix clippy lint
* Rename test file
* Add block tests, add cheap block proposer check
* Rename block testing file
* Add observed_block_producers
* Tidy
* Switch around block signature verification
* Finish block testing
* Remove gossip from signature tests
* First pass of self review
* Fix deviation in spec
* Update test spec tags
* Start moving over to hashset
* Finish moving observed attesters to hashmap
* Move aggregation pool over to hashmap
* Make fc attn borrow again
* Fix rest_api compile error
* Fix missing comments
* Fix monster test
* Uncomment increasing slots test
* Address remaining comments
* Remove unsafe, use cfg test
* Remove cfg test flag
* Fix dodgy comment
* Revert "Update hashmap hashset to stable futures"
This reverts commit d432378a3cc5cd67fc29c0b15b96b886c1323554.
* Revert "Adds panic test to hashset delay"
This reverts commit 281502396fc5b90d9c421a309c2c056982c9525b.
* Ported attestation_service
* Ported duties_service
* Ported fork_service
* More ports
* Port block_service
* Minor fixes
* VC compiles
* Update TODOS
* Borrow self where possible
* Ignore aggregates that are already known.
* Unify aggregator modulo logic
* Fix typo in logs
* Refactor validator subscription logic
* Avoid reproducing selection proof
* Skip HTTP call if no subscriptions
* Rename DutyAndState -> DutyAndProof
* Tidy logs
* Print root as dbg
* Fix compile errors in tests
* Fix compile error in test
* Re-Fix attestation and duties service
* Minor fixes
Co-authored-by: Paul Hauner <paul@paulhauner.com>
* Network crate update to stable futures
* Port account_manager to stable futures (#1121)
* Port account_manager to stable futures
* Run async fns in tokio environment
* Port rest_api crate to stable futures (#1118)
* Port rest_api lib to stable futures
* Reduce tokio features
* Update notifier to stable futures
* Builder update
* Further updates
* Convert self referential async functions
* stable futures fixes (#1124)
* Fix eth1 update functions
* Fix genesis and client
* Fix beacon node lib
* Return appropriate runtimes from environment
* Fix test rig
* Refactor eth1 service update
* Upgrade simulator to stable futures
* Lighthouse compiles on stable futures
* Remove println debugging statement
* Update libp2p service, start rpc test upgrade
* Update network crate for new libp2p
* Update tokio::codec to futures_codec (#1128)
* Further work towards RPC corrections
* Correct http timeout and network service select
* Use tokio runtime for libp2p
* Revert "Update tokio::codec to futures_codec (#1128)"
This reverts commit e57aea924acf5cbabdcea18895ac07e38a425ed7.
* Upgrade RPC libp2p tests
* Upgrade secio fallback test
* Upgrade gossipsub examples
* Clean up RPC protocol
* Test fixes (#1133)
* Correct websocket timeout and run on os thread
* Fix network test
* Clean up PR
* Correct tokio tcp move attestation service tests
* Upgrade attestation service tests
* Correct network test
* Correct genesis test
* Test corrections
* Log info when block is received
* Modify logs and update attester service events
* Stable futures: fixes to vc, eth1 and account manager (#1142)
* Add local testnet scripts
* Remove whiteblock script
* Rename local testnet script
* Move spawns onto handle
* Fix VC panic
* Initial fix to block production issue
* Tidy block producer fix
* Tidy further
* Add local testnet clean script
* Run cargo fmt
* Tidy duties service
* Tidy fork service
* Tidy ForkService
* Tidy AttestationService
* Tidy notifier
* Ensure await is not suppressed in eth1
* Ensure await is not suppressed in account_manager
* Use .ok() instead of .unwrap_or(())
* RPC decoding test for proto
* Update discv5 and eth2-libp2p deps
* Fix lcli double runtime issue (#1144)
* Handle stream termination and dialing peer errors
* Correct peer_info variant types
* Remove unnecessary warnings
* Handle subnet unsubscription removal and improve logigng
* Add logs around ping
* Upgrade discv5 and improve logging
* Handle peer connection status for multiple connections
* Improve network service logging
* Improve logging around peer manager
* Upgrade swarm poll centralise peer management
* Identify clients on error
* Fix `remove_peer` in sync (#1150)
* remove_peer removes from all chains
* Remove logs
* Fix early return from loop
* Improved logging, fix panic
* Partially correct tests
* Stable futures: Vc sync (#1149)
* Improve syncing heuristic
* Add comments
* Use safer method for tolerance
* Fix tests
* Stable futures: Fix VC bug, update agg pool, add more metrics (#1151)
* Expose epoch processing summary
* Expose participation metrics to prometheus
* Switch to f64
* Reduce precision
* Change precision
* Expose observed attesters metrics
* Add metrics for agg/unagg attn counts
* Add metrics for gossip rx
* Add metrics for gossip tx
* Adds ignored attns to prom
* Add attestation timing
* Add timer for aggregation pool sig agg
* Add write lock timer for agg pool
* Add more metrics to agg pool
* Change map lock code
* Add extra metric to agg pool
* Change lock handling in agg pool
* Change .write() to .read()
* Add another agg pool timer
* Fix for is_aggregator
* Fix pruning bug
Co-authored-by: pawan <pawandhananjay@gmail.com>
Co-authored-by: Paul Hauner <paul@paulhauner.com>
* Improve error handling in block iteration
* Introduce atomic DB operations
* Fix race condition
An invariant was violated: For every block hash in head_tracker, that
block is accessible from the store.
* Add test to understand flow of key storage
* First commit
* Committing to save trait stuff
* Working naive design
* Add keystore struct
* Move keystore files into their own module
* Add serde (de)serialize_with magic
* Add keystore test
* Fix tests
* Add comments and minor fixes
* Pass optional params to `to_keystore` function
* Add `path` field to keystore
* Add function to read Keystore from file
* Add test vectors and fix Version serialization
* Checksum params is empty object
* Add public key to Keystore
* Add function for saving keystore into file
* Deleted account_manager main.rs
* Move keystore module to validator_client
* Add save_keystore method to validator_directory
* Add load_keystore function. Minor refactorings
* Fixed dependencies
* Address some review comments
* Add Password newtype; derive Zeroize
* Fix test
* Move keystore into own crate
* Remove padding
* Add error enum, zeroize more things
* Fix comment
* Add keystore builder
* Remove keystore stuff from val client
* Add more tests, comments
* Add more comments, test vectors
* Progress on improving JSON validation
* More JSON verification
* Start moving JSON into own mod
* Remove old code
* Add more tests, reader/writers
* Tidy
* Move keystore into own file
* Move more logic into keystore file
* Tidy
* Tidy
* Allow for odd-character hex
* Add more json missing field checks
* Use scrypt by default
* Tidy, address comments
* Test path and uuid in vectors
* Fix comment
* Add checks for kdf params
* Enforce empty kdf message
* Expose json_keystore mod
* Split out encrypt/decrypt
* Replace some password usage with slice
* Expose PlainText struct
* First commits on path derivation
* Progress with implementation
* More progress
* Passing intermediate test vectors
* Tidy, add comments
* Add DerivedKey structs
* Move key derivation into own crate
* Add zeroize structs
* Return error for empty seed
* Add tests
* Tidy
* Expose consts, remove Password
* Expose SALT_SIZE
* Move dbg assert statement
* Ensure bigints are zeroized
* Try fix Cargo.toml
Co-authored-by: pawan <pawandhananjay@gmail.com>
* Add test to understand flow of key storage
* First commit
* Committing to save trait stuff
* Working naive design
* Add keystore struct
* Move keystore files into their own module
* Add serde (de)serialize_with magic
* Add keystore test
* Fix tests
* Add comments and minor fixes
* Pass optional params to `to_keystore` function
* Add `path` field to keystore
* Add function to read Keystore from file
* Add test vectors and fix Version serialization
* Checksum params is empty object
* Add public key to Keystore
* Add function for saving keystore into file
* Deleted account_manager main.rs
* Move keystore module to validator_client
* Add save_keystore method to validator_directory
* Add load_keystore function. Minor refactorings
* Fixed dependencies
* Address some review comments
* Add Password newtype; derive Zeroize
* Fix test
* Move keystore into own crate
* Remove padding
* Add error enum, zeroize more things
* Fix comment
* Add keystore builder
* Remove keystore stuff from val client
* Add more tests, comments
* Add more comments, test vectors
* Progress on improving JSON validation
* More JSON verification
* Start moving JSON into own mod
* Remove old code
* Add more tests, reader/writers
* Tidy
* Move keystore into own file
* Move more logic into keystore file
* Tidy
* Tidy
* Allow for odd-character hex
* Add more json missing field checks
* Use scrypt by default
* Tidy, address comments
* Test path and uuid in vectors
* Fix comment
* Add checks for kdf params
* Enforce empty kdf message
* Expose json_keystore mod
* Split out encrypt/decrypt
* Replace some password usage with slice
* Expose PlainText struct
* Expose consts, remove Password
* Expose SALT_SIZE
* Move dbg assert statement
* Fix dodgy json test
* Protect against n == 1
* Return error if n is not power of 2
* Add dklen checks
* Add note about panics
Co-authored-by: pawan <pawandhananjay@gmail.com>
* Fix parallelism bug in exit processing
Also:
* Remove parallelism for all other operations except deposit merkle proofs
* Improve exit tests
* Fix broken attestation test
Closes#1090
* Allow for generating block/pre/post states from some unit tests (#1123)
* Add post-state checks, comments
* Add state_transition_vectors crate
* Integrate new testing crate with CI
* Add readme
* Add additional valid tests
* Remove ExitTests (they were moved to new crate)
* Small test fixes
* Delete incorrect saturating_sub in slash_validator
And clean-up the balance increase/decrease functions to look more like the spec.
Co-authored-by: Paul Hauner <paul@paulhauner.com>
* Add PH & MS slot clock changes
* Account for genesis time
* Add progress on duties refactor
* Add simple is_aggregator bool to val subscription
* Start work on attestation_verification.rs
* Add progress on ObservedAttestations
* Progress with ObservedAttestations
* Fix tests
* Add observed attestations to the beacon chain
* Add attestation observation to processing code
* Add progress on attestation verification
* Add first draft of ObservedAttesters
* Add more tests
* Add observed attesters to beacon chain
* Add observers to attestation processing
* Add more attestation verification
* Create ObservedAggregators map
* Remove commented-out code
* Add observed aggregators into chain
* Add progress
* Finish adding features to attestation verification
* Ensure beacon chain compiles
* Link attn verification into chain
* Integrate new attn verification in chain
* Remove old attestation processing code
* Start trying to fix beacon_chain tests
* Split adding into pools into two functions
* Add aggregation to harness
* Get test harness working again
* Adjust the number of aggregators for test harness
* Fix edge-case in harness
* Integrate new attn processing in network
* Fix compile bug in validator_client
* Update validator API endpoints
* Fix aggreagation in test harness
* Fix enum thing
* Fix attestation observation bug:
* Patch failing API tests
* Start adding comments to attestation verification
* Remove unused attestation field
* Unify "is block known" logic
* Update comments
* Supress fork choice errors for network processing
* Add todos
* Tidy
* Add gossip attn tests
* Disallow test harness to produce old attns
* Comment out in-progress tests
* Partially address pruning tests
* Fix failing store test
* Add aggregate tests
* Add comments about which spec conditions we check
* Dont re-aggregate
* Split apart test harness attn production
* Fix compile error in network
* Make progress on commented-out test
* Fix skipping attestation test
* Add fork choice verification tests
* Tidy attn tests, remove dead code
* Remove some accidentally added code
* Fix clippy lint
* Rename test file
* Add block tests, add cheap block proposer check
* Rename block testing file
* Add observed_block_producers
* Tidy
* Switch around block signature verification
* Finish block testing
* Remove gossip from signature tests
* First pass of self review
* Fix deviation in spec
* Update test spec tags
* Start moving over to hashset
* Finish moving observed attesters to hashmap
* Move aggregation pool over to hashmap
* Make fc attn borrow again
* Fix rest_api compile error
* Fix missing comments
* Fix monster test
* Uncomment increasing slots test
* Address remaining comments
* Remove unsafe, use cfg test
* Remove cfg test flag
* Fix dodgy comment
* Ignore aggregates that are already known.
* Unify aggregator modulo logic
* Fix typo in logs
* Refactor validator subscription logic
* Avoid reproducing selection proof
* Skip HTTP call if no subscriptions
* Rename DutyAndState -> DutyAndProof
* Tidy logs
* Print root as dbg
* Fix compile errors in tests
* Fix compile error in test
* Add cli flag and parse dns address
* Fail if enr udp port isn't set
* Improve docs and address parsing
* address review comments
* Remove debug statements
* Add requires condition for enr-address
* Return address in error
* track whether we have aggregate validator subscriptions to exact subnets, so we know whether or not to drop incoming attestations
* fix is aggregator check
* fix CI
Co-authored-by: Age Manning <Age@AgeManning.com>
