realbigsean
334aa2eabd
Single lookup improvements ( #5488 )
...
* Fix unexpected `UnrequestedBlobId` and `ExtraBlocksReturned` errors due to race conditions.
* Continue chain segment processing and skip any blocks that are already known, rather than returning an error.
* more de-dup checking
* ensure we don't reset `requested_ids` during rpc download
* better fix
* Merge branch 'unstable' of https://github.com/sigp/lighthouse into more-dup-lookup-fixes
* remove chain hash check
* Merge branch 'fix-block-lookup-race' of https://github.com/jimmygchen/lighthouse into sean-test-lookups
* remove block check
* add back tests
* Log and CI fixes
* undue extra check
* Merge branch 'sean-test-lookups' of https://github.com/realbigsean/lighthouse into sean-test-lookups
* log improvements
* Improve logging
2024-03-27 10:01:27 +00:00
Eitan Seri-Levi
e4d4e439cb
Add Capella & Deneb light client support ( #4946 )
...
* rebase and add comment
* conditional test
* test
* optimistic chould be working now
* finality should be working now
* try again
* try again
* clippy fix
* add lc bootstrap beacon api
* add lc optimistic/finality update to events
* fmt
* That error isn't occuring on my computer but I think this should fix it
* Merge branch 'unstable' into light_client_beacon_api_1
# Conflicts:
# beacon_node/beacon_chain/src/events.rs
# beacon_node/http_api/src/lib.rs
# beacon_node/http_api/src/test_utils.rs
# beacon_node/http_api/tests/main.rs
# beacon_node/lighthouse_network/src/rpc/codec/ssz_snappy.rs
# beacon_node/lighthouse_network/src/rpc/methods.rs
# beacon_node/lighthouse_network/src/service/api_types.rs
# beacon_node/network/src/beacon_processor/worker/rpc_methods.rs
# beacon_node/tests/test.rs
# common/eth2/src/types.rs
# lighthouse/src/main.rs
* Add missing test file
* Update light client types to comply with Altair light client spec.
* Fix test compilation
* Merge branch 'unstable' into light_client_beacon_api_1
* Support deserializing light client structures for the Bellatrix fork
* Move `get_light_client_bootstrap` logic to `BeaconChain`. `LightClientBootstrap` API to return `ForkVersionedResponse`.
* Misc fixes.
- log cleanup
- move http_api config mutation to `config::get_config` for consistency
- fix light client API responses
* Add light client bootstrap API test and fix existing ones.
* Merge branch 'unstable' into light_client_beacon_api_1
* Fix test for `light-client-server` http api config.
* Appease clippy
* Add Altair light client SSZ tests
* Merge branch 'unstable' of https://github.com/sigp/lighthouse into light_client_beacon_api_1
* updates to light client header
* light client header from signed beacon block
* using options
* implement helper functions
* placeholder conversion from vec hash256 to exec branch
* add deneb
* using fixed vector
* remove unwraps
* by epoch
* compute merkle proof
* merkle proof
* update comments
* resolve merge conflicts
* linting
* Merge branch 'unstable' into light-client-ssz-tests
# Conflicts:
# beacon_node/beacon_chain/src/beacon_chain.rs
# consensus/types/src/light_client_bootstrap.rs
# consensus/types/src/light_client_header.rs
* superstruct attempt
* superstruct changes
* lint
* altair
* update
* update
* changes to light_client_optimistic_ and finality
* merge unstable
* refactor
* resolved merge conflicts
* Merge branch 'unstable' of https://github.com/sigp/lighthouse into capella_deneb_light_client_types
* block_to_light_client_header fork aware
* fmt
* comment fix
* comment fix
* include merge fork, update deserialize_by_fork, refactor
* fmt
* pass by ref to prevent clone
* rename merkle proof fn
* add FIXME
* LightClientHeader TestRandom
* fix comments
* fork version deserialize
* merge unstable
* move fn arguments, fork name calc
* use task executor
* remove unneeded fns
* remove dead code
* add manual ssz decoding/encoding and add ssz_tests_by_fork macro
* merge deneb types with tests
* merge ssz tests, revert code deletion, cleanup
* move chainspec
* update ssz tests
* fmt
* light client ssz tests
* change to superstruct
* changes from feedback
* linting
* Merge branch 'unstable' of https://github.com/sigp/lighthouse into capella_deneb_light_client_types
* test fix
* cleanup
* Remove unused `derive`.
* Merge branch 'unstable' of https://github.com/sigp/lighthouse into capella_deneb_light_client_types
* beta compiler fix
* merge
2024-03-25 11:01:56 +00:00
dknopik
0a6e4a11d7
Verify whether validators really are unknown during sync committee duty API request ( #5174 )
...
* Verify whether validators really are unknown during sync committee duty API request
* Merge branch 'unstable' into fix-4717
* Merge branch 'unstable' into fix-4717
* Merge branch 'unstable' of https://github.com/sigp/lighthouse into fix-4717
2024-03-23 00:01:11 +00:00
Lion - dapplion
306d3eb0eb
Delete unused incomplete_processing_components ( #5418 )
...
* Delete unused incomplete_processing_components
* lint
2024-03-23 00:00:35 +00:00
zhiqiangxu
5121d655f7
chore: reduce scope of commitment ( #5426 )
...
* reduce scope of commitment
* avoid clone for last reference
* Merge branch 'unstable' of https://github.com/sigp/lighthouse into mod_merge_single_blob
2024-03-22 18:16:52 +00:00
zhiqiangxu
5bfe6a8ae3
chore: remove stale comment ( #5440 )
...
* rm stale comment
* Merge branch 'unstable' of https://github.com/sigp/lighthouse into rm_irelevant_comment
2024-03-22 18:16:49 +00:00
realbigsean
21cdc64bfe
Improve parent lookup logging ( #5451 )
...
* upgrade parent lookup result processing logs to debug, use display instead of debug for BlockError in case a blob parent unknown error is hit, add block root to BlockIsAlreadyKnown
* fix compile
* fix compile
* fix compile
2024-03-22 18:16:13 +00:00
Michael Sproul
65a6118c53
Fix gossip verification of duplicate attester slashings ( #5385 )
...
* Fix gossip verification of duplicate attester slashings
2024-03-20 20:47:38 +00:00
Eitan Seri-Levi
01ec42e75a
Fix Rust beta compiler errors 1.78.0-beta.1 ( #5439 )
...
* remove redundant imports
* fix test
* contains key
* fmt
* Merge branch 'unstable' into fix-beta-compiler
2024-03-20 05:17:02 +00:00
Michael Sproul
f93844e63b
Optimise concurrent block production ( #5368 )
...
* Optimise concurrent block production
2024-03-08 05:15:28 +00:00
João Oliveira
65c4ff0775
remove exit-future ( #5183 )
...
* remove exit-future usage,
as it is non maintained, and replace with async-channel which is already in the repo.
* Merge branch 'unstable' of https://github.com/sigp/lighthouse into remove-exit-future
* Merge branch 'unstable' of https://github.com/sigp/lighthouse into remove-exit-future
2024-02-27 22:12:44 +00:00
Pawan Dhananjay
3ab9d3a84e
Add a cli option for the snapshot cache size ( #5270 )
...
* Add a cli option for snapshot cache size
* Remove junk
* Make snapshot_cache module public
* lint
* Update docs
2024-02-26 05:19:39 +00:00
Michael Sproul
c9702cb0a1
Download checkpoint blobs during checkpoint sync ( #5252 )
...
* MVP implementation (untested)
* update store checkpoint sync test
* update cli help
* Merge pull request #5253 from realbigsean/checkpoint-blobs-sean
Checkpoint blobs sean
* Warn only if blobs are missing from server
* Merge remote-tracking branch 'origin/unstable' into checkpoint-blobs
* Verify checkpoint blobs
* Move blob verification earlier
2024-02-19 02:22:23 +00:00
realbigsean
f21472991d
check the da cache and the attester cache in responding to RPC requests ( #5138 )
...
* check the da cache and the attester cache in responding to RPC requests
* use the processing cache instead
* update comment
* add da cache metrics
* rename early attester cache method
* Merge branch 'unstable' of https://github.com/sigp/lighthouse into check-da-cache-in-rpc-response
* make rustup update run on the runners
* Revert "make rustup update run on the runners"
This reverts commit d097e9bfa84a13b1d7813c03df38e7756fb0bfc5.
2024-02-19 02:22:15 +00:00
ethDreamer
a264afd19f
Verify Versioned Hashes During Optimistic Sync ( #4832 )
...
* Convert NewPayloadRequest to use Reference
* Refactor for Clarity
* Verify Versioned Hashes
* Added Tests for Version Hash Verification
* Added Moar Tests
* Fix Problems Caused By Merge
* Update to use Alloy Instead of Reth Crates (#14 )
* Update beacon_node/execution_layer/src/engine_api/new_payload_request.rs
Co-authored-by: realbigsean <seananderson33@GMAIL.com>
* Faster Versioned Hash Extraction
* Update to rust 1.75 & Pin alloy-consensus
2024-02-18 12:40:45 +00:00
realbigsean
4172d9f75c
Update to consensus spec v1.4.0-beta.6 ( #5094 )
...
* get latest ef tests passing
* fix tests
* Fix invalid payload recovery tests
* Merge branch 'unstable' into update-to-spec-v1.4.0-beta.6
* Revert "fix tests"
This reverts commit 0c875b02e032ebd9bebd822183a1c3d10d333a4c.
* Fix fork choice def. tests
* Update beacon_node/beacon_chain/tests/payload_invalidation.rs
2024-02-08 18:08:21 +00:00
Jimmy Chen
39e9f7dc6b
Fix Rust beta compiler errors (1.77) ( #5180 )
...
* Lint fixes
* More fixes for beta compiler.
* Format fixes
* Move `#[allow(dead_code)]` to field level.
* Remove old comment.
* Update beacon_node/execution_layer/src/test_utils/mod.rs
Co-authored-by: João Oliveira <hello@jxs.pt>
* remove duplicate line
2024-02-05 17:54:11 +00:00
Michael Sproul
0b6416c444
Re-disable block verification tests in debug ( #5155 )
...
* Re-disable block verification tests in debug
2024-02-01 08:35:10 +00:00
Age Manning
7582da7855
Test backfill ( #5109 )
...
* Test backfill
* Revert cargo.toml
* Update beacon_node/beacon_chain/src/builder.rs
Co-authored-by: João Oliveira <hello@jxs.pt>
* Remove redundant code
2024-01-31 18:11:45 +00:00
Lion - dapplion
b035638f9b
Compute recent lightclient updates ( #4969 )
...
* Compute recent lightclient updates
* Review PR
* Merge remote-tracking branch 'upstream/unstable' into lc-prod-recent-updates
* Review PR
* consistent naming
* add metrics
* revert dropping reprocessing queue
* Update light client optimistic update re-processing logic. (#7 )
* Add light client server simulator tests. Co-authored by @dapplion.
* Merge branch 'unstable' into fork/dapplion/lc-prod-recent-updates
* Fix lint
* Enable light client server in simulator test.
* Fix test for light client optimistic updates and finality updates.
2024-01-31 05:25:51 +00:00
Michael Sproul
6f3af67362
Fix off-by-one in backfill sig verification ( #5120 )
...
* Fix off-by-one in backfill sig verification
* Add self-referential PR link
2024-01-30 00:33:01 +00:00
realbigsean
a4fcf60bcc
Increase attestation cache sizes ( #5135 )
...
* increase observed attesters and aggregates cache sizes
* fix comment
2024-01-30 00:32:57 +00:00
Sergey Kisel
64efdaf39a
#5102 Fix load_state_for_block_production metric mapping ( #5103 )
...
* #5102 Fix load_state_for_block_production metric mapping
2024-01-30 00:32:52 +00:00
realbigsean
1cebf41452
Backfill blob storage fix ( #5119 )
...
* store blobs in the correct db in backfill
* add database migration
* add migration file
* remove log info suggesting deneb isn't schedule
* add batching in blob migration
2024-01-24 09:35:02 +11:00
Michael Sproul
a403138ed0
Reduce size of futures in HTTP API to prevent stack overflows ( #5104 )
...
* Box::pin a few big futures
* Arc the blocks early in publication
* Fix more tests
2024-01-23 15:32:07 +11:00
Lion - dapplion
585124fb2f
Hold HeadTracker lock until persisting to disk ( #5084 )
...
* Fix head tracker drop order on un-ordered shutdown
* lint
---------
Co-authored-by: Michael Sproul <michael@sigmaprime.io>
2024-01-22 15:14:11 +11:00
Mac L
a0b407c15d
Add Deneb readiness logging ( #5074 )
2024-01-18 15:21:38 -05:00
Paul Hauner
be79f74c6d
Fix IncorrectAttestationSource error in attn. simulator ( #5048 )
...
* Don't produce attestations when syncing
* Handle `IncorrectAttestationSource`
2024-01-10 10:47:00 +11:00
Michael Sproul
7e948eec9d
Fix block v3 reward encodings ( #5049 )
...
* Fix block v3 reward encodings
* Use crates.io version
2024-01-10 10:44:07 +11:00
realbigsean
b47e3f252e
Runtime rpc request sizes ( #4841 )
...
* add runtime variable list type
* add configs to ChainSpec
* git rid of max request blocks type
* fix tests and lints
* remove todos
* git rid of old const usage
* fix decode impl
* add new config to `Config` api struct
* add docs fix compilt
* move methods for per-fork-spec to chainspec
* get values off chain spec
* fix compile
* remove min by root size
* add tests for runtime var list
---------
Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>
2024-01-09 10:23:47 +11:00
realbigsean
f70c32ec70
create unified slashing cache ( #5033 )
...
* create unified slashing cache
* add observed slashable file
* fix broadcast validation tests
* revert block seen cache changes
* clean up slashable cache test
* check header signatures for RPC blobs
* don't throw error on RPC signature invalie
2024-01-08 10:30:57 -05:00
Eitan Seri-Levi
9c1505d082
Block v3 builder boost factor ( #5035 )
...
* builder boost factor
* default boost factor
* revert
* deprecate always_prefer_builder_payload, builder-profit-threshold, ignore_builder_override_suggestion_threshold and builder_comparison_factor flags
* revert
* set deprecated flags to no op, revert should_override_builder
* fix test, calc boosted relay value correctly, dont calculate if none
* Add deprecation warnings and restore CLI docs
2024-01-08 11:10:32 +11:00
Michael Sproul
af11e78ae1
Clean up blockv3 metadata and client ( #5015 )
...
* Improve block production v3 client
* Delete wayward line
* Overhaul JSON endpoint as well
* Rename timeout param
* Update tests
* I broke everything
* Ah this is an insane fix
* Remove unnecessary optionals
* Doc fix
2023-12-22 09:39:17 -05:00
realbigsean
c55608be10
suppress error on duplicate blobs ( #4995 )
2023-12-18 12:15:12 -05:00
Jimmy Chen
b0c374c1ca
Update dependencies to get rid of the yanked deps ( #4994 )
...
* Initial attempt to upgrade hashbrown to latest to get rid of the crate warnings.
* Replace `.expect()` usage with use of `const`.
* Update ahash 0.7 as well
* Remove unsafe code.
* Update `lru` to 0.12 and fix release test errors.
* Set non-blocking socket
* Bump testcontainers to 0.15.
* Fix lint
---------
Co-authored-by: Michael Sproul <michael@sigmaprime.io>
2023-12-15 18:31:59 +11:00
Michael Sproul
f1113540d8
Fix off-by-one bug in missed block detection ( #5012 )
...
* Regression test
* Fix the bug
2023-12-15 14:23:54 +11:00
Pawan Dhananjay
ae4a296089
Convert a FullPayload to a BlindedPayload correctly ( #5005 )
2023-12-14 08:44:14 -05:00
Joel Rousseau
189430a45c
Add attestation simulator ( #4880 )
...
* basic scaffold
* remove unnecessary ?
* check if committee cache is init
* typed ValidatorMonitor with ethspecs + store attestations within
* nits
* process unaggregated attestation
* typo
* extract in func
* add tests
* better naming
* better naming 2
* less verbose
* use same naming as validator monitor
* use attestation_simulator
* add metrics
* remove cache
* refacto flag_indices process
* add lag
* remove copying state
* clean and lint
* extract metrics
* nits
* compare prom metrics in tests
* implement lag
* nits
* nits
* add attestation simulator service
* fmt
* return beacon_chain as arc
* nit: debug
* sed s/unaggregated/unagg.//
* fmt
* fmt
* nit: remove unused comments
* increase max unaggregated attestation hashmap to 64
* nit: sed s/clone/copied//
* improve perf: remove unecessary hashmap copy
* fix flag indices comp
* start service in client builder
* remove //
* cargo fmt
* lint
* cloned keys
* fmt
* use Slot value instead of pointer
* Update beacon_node/beacon_chain/src/attestation_simulator.rs
Co-authored-by: Paul Hauner <paul@paulhauner.com>
---------
Co-authored-by: Paul Hauner <paul@paulhauner.com>
2023-12-14 11:44:56 +11:00
Pawan Dhananjay
a3a370302a
Use the block header to compute the canonical_root ( #5003 )
2023-12-14 09:24:36 +11:00
Gua00va
a3fb27c99b
add forK_choice_read_lock as parameter ( #4978 )
2023-12-13 16:06:00 +11:00
ethDreamer
78ffa378b4
Batch Verify RPC Blobs ( #4934 )
2023-12-08 16:48:03 -05:00
realbigsean
46184e5ce4
Remove delayed lookups ( #4992 )
...
* initial rip out
* fix unused imports
* delete tests and fix lint
* fix peers scoring for blobs
2023-12-08 15:42:55 -05:00
Jimmy Chen
67e0569d9b
Fix corrupted DB on networks where the first slot is skipped (Holesky) ( #4985 )
...
* Fix zero block roots on skip slots.
* Remove temporary comment, println code and unused imports.
* Remove `println!` in test.
2023-12-07 15:12:06 +11:00
ethDreamer
52117f43ba
Small Improvements ( #4980 )
...
* initial changes
* use arc<blobsidecar> in vector
* Utilize new pattern for KzgVerifiedBlob
* fmt
* Update beacon_node/beacon_chain/src/blob_verification.rs
Co-authored-by: realbigsean <seananderson33@GMAIL.com>
* forgot to save..
* lint
* fmt.. again
---------
Co-authored-by: realbigsean <seananderson33@GMAIL.com>
2023-12-06 15:51:40 -05:00
Pawan Dhananjay
31044402ee
Sidecar inclusion proof ( #4900 )
...
* Refactor BlobSidecar to new type
* Fix some compile errors
* Gossip verification compiles
* Fix http api types take 1
* Fix another round of compile errors
* Beacon node crate compiles
* EF tests compile
* Remove all blob signing from VC
* fmt
* Tests compile
* Fix some tests
* Fix more http tests
* get compiling
* Fix gossip conditions and tests
* Add basic proof generation and verification
* remove unnecessary ssz decode
* add back build_sidecar
* remove default at fork for blobs
* fix beacon chain tests
* get relase tests compiling
* fix lints
* fix existing spec tests
* add new ef tests
* fix gossip duplicate rule
* lints
* add back sidecar signature check in gossip
* add finalized descendant check to blob sidecar gossip
* fix error conversion
* fix release tests
* sidecar inclusion self review cleanup
* Add proof verification and computation metrics
* Remove accidentally committed file
* Unify some block and blob errors; add slashing conditions for sidecars
* Address review comment
* Clean up re-org tests (#4957 )
* Address more review comments
* Add Comments & Eliminate Unnecessary Clones
* update names
* Update beacon_node/beacon_chain/src/metrics.rs
Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>
* Update beacon_node/network/src/network_beacon_processor/tests.rs
Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>
* pr feedback
* fix test compile
* Sidecar Inclusion proof small refactor and updates (#4967 )
* Update some comments, variables and small cosmetic fixes.
* Couple blobs and proofs into a tuple in `PayloadAndBlobs` for simplicity and safety.
* Update function comment.
* Update testing/ef_tests/src/cases/merkle_proof_validity.rs
Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>
* Rename the block and blob wrapper types used in the beacon API interfaces.
* make sure gossip invalid blobs are passed to the slasher (#4970 )
* Add blob headers to slasher before adding to DA checker
* Replace Vec with HashSet in BlockQueue
* fmt
* Rename gindex -> index
* Simplify gossip condition
---------
Co-authored-by: realbigsean <seananderson33@gmail.com>
Co-authored-by: realbigsean <sean@sigmaprime.io>
Co-authored-by: Michael Sproul <michael@sigmaprime.io>
Co-authored-by: Mark Mackey <mark@sigmaprime.io>
Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>
2023-12-05 11:19:59 -05:00
Michael Sproul
c8b2324880
Enable progressive balances fast mode by default ( #4971 )
...
* Enable progressive balances fast mode by default
* Fix default in chain_config
2023-12-04 14:42:49 +11:00
Michael Sproul
547ed1de63
Clone state ahead of block production ( #4925 )
...
* Clone state ahead of block production
* Add pruning and fix logging
* Don't hold 2 states in mem
2023-11-30 13:49:35 +11:00
GeemoCandama
8a599ec7dc
API for LightClientBootstrap, LightClientFinalityUpdate, LightClientOptimisticUpdate and light client events ( #3954 )
...
* rebase and add comment
* conditional test
* test
* optimistic chould be working now
* finality should be working now
* try again
* try again
* clippy fix
* add lc bootstrap beacon api
* add lc optimistic/finality update to events
* fmt
* That error isn't occuring on my computer but I think this should fix it
* Add missing test file
* Update light client types to comply with Altair light client spec.
* Fix test compilation
* Support deserializing light client structures for the Bellatrix fork
* Move `get_light_client_bootstrap` logic to `BeaconChain`. `LightClientBootstrap` API to return `ForkVersionedResponse`.
* Misc fixes.
- log cleanup
- move http_api config mutation to `config::get_config` for consistency
- fix light client API responses
* Add light client bootstrap API test and fix existing ones.
* Fix test for `light-client-server` http api config.
* Appease clippy
* Efficiency improvement when retrieving beacon state.
---------
Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>
2023-11-28 17:14:29 +11:00
Jimmy Chen
6b63d18420
Fix Rust beta compiler warnings (rustc 1.75.0-beta.1 (782883f60 2023-11-12)) ( #4932 )
2023-11-18 03:55:11 +11:00
Mac L
68e076d60a
Remove legacy database migrations ( #4919 )
...
* Remove legacy db migrations
* Fix tests
* Remove migrations to/from v15
* Remove v16 migrations
2023-11-16 22:09:05 +11:00
