Commit Graph

6102 Commits

Author SHA1 Message Date
Eitan Seri-Levi
5c8c8da8b1
Use blocks v3 endpoint in the VC (#4813)
* block v3 endpoint init

* block v3 flow

* block v3 flow

* continue refactor

* the full flow...

* add api logic

* add api logic

* add new endpoint version

* added v3 endpoint

* some debugging

* merge v2 flow with v3

* debugging

* tests passing

* tests passing

* revert cargo lock

* initial v3 test

* blinded payload test case passing

* fix clippy issues

* cleanup

* cleanup

* remove dead code

* fixed logs

* add block value

* block value fix

* linting

* merge unstable

* refactor

* add consensus block value

* lint

* update header name to consensus block value

* prevent setting the participation flag

* clone get_epoch_participation result

* fmt

* clone epoch participation outside of the loop

* add block v3 to vc

* add v3 logic into vc

* add produce-block-v3

* refactor based on feedback

* update

* remove comments

* refactor

* header bugfix

* fmt

* resolve merge conflicts

* fix merge

* fix merge

* refactor

* refactor

* cleanup

* lint

* changes based on feedback

* revert

* remove block v3 fallback to v2

* publish_block_v3 should return irrecoveerable errors

* comments

* comments

* fixed issues from merge

* merge conflicts

* Don't activate at fork; support builder_proposals

* Update CLI flags & book

* Remove duplicate `current_slot` parameter in `publish_block` function, and remove unnecessary clone.

* Revert changes on making block errors irrecoverable.

---------

Co-authored-by: Michael Sproul <michael@sigmaprime.io>
Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>
2024-01-08 16:12:39 -05:00
realbigsean
f70c32ec70
create unified slashing cache (#5033)
* create unified slashing cache

* add observed slashable file

* fix broadcast validation tests

* revert block seen cache changes

* clean up slashable cache test

* check header signatures for RPC blobs

* don't throw error on RPC signature invalie
2024-01-08 10:30:57 -05:00
Paul Hauner
f62cfc6475
Add hint for solving CLI tests failure (#5041)
* Add hint for solving CLI test failure.

* Add `make cli-local`
2024-01-08 10:30:37 -05:00
realbigsean
db05d37675
update goerli config (#5036) 2024-01-08 11:24:12 +11:00
Eitan Seri-Levi
9c1505d082
Block v3 builder boost factor (#5035)
* builder boost factor

* default boost factor

* revert

* deprecate always_prefer_builder_payload, builder-profit-threshold, ignore_builder_override_suggestion_threshold and builder_comparison_factor flags

* revert

* set deprecated flags to no op, revert should_override_builder

* fix test, calc boosted relay value correctly, dont calculate if none

* Add deprecation warnings and restore CLI docs
2024-01-08 11:10:32 +11:00
Jimmy Chen
0c97762032
Track shared memory in metrics (#5023)
* Track shared memory so we can exclude them form resident memory.

* Bump psutil version to 3.3.0 to get shared memory metrics.
2024-01-05 14:48:11 -05:00
Akihito Nakano
01994c438e
Bump zerocopy (#5024) 2023-12-22 09:57:39 -05:00
Michael Sproul
af11e78ae1
Clean up blockv3 metadata and client (#5015)
* Improve block production v3 client

* Delete wayward line

* Overhaul JSON endpoint as well

* Rename timeout param

* Update tests

* I broke everything

* Ah this is an insane fix

* Remove unnecessary optionals

* Doc fix
2023-12-22 09:39:17 -05:00
Age Manning
a7e5926a1f
Information for network testing (#4961)
* Documentation for network testing

* Update doc
2023-12-22 09:02:58 +11:00
realbigsean
c55608be10
suppress error on duplicate blobs (#4995) 2023-12-18 12:15:12 -05:00
Jimmy Chen
dfc3b3714a
Fix incorrect blob queue metrics (#5014)
* Fix blob queue metrics.

* Update blob metric description
2023-12-18 12:04:57 -05:00
Jimmy Chen
b0c374c1ca
Update dependencies to get rid of the yanked deps (#4994)
* Initial attempt to upgrade hashbrown to latest to get rid of the crate warnings.

* Replace `.expect()` usage with use of `const`.

* Update ahash 0.7 as well

* Remove unsafe code.

* Update `lru` to 0.12 and fix release test errors.

* Set non-blocking socket

* Bump testcontainers to 0.15.

* Fix lint

---------

Co-authored-by: Michael Sproul <michael@sigmaprime.io>
2023-12-15 18:31:59 +11:00
Michael Sproul
f1113540d8
Fix off-by-one bug in missed block detection (#5012)
* Regression test

* Fix the bug
2023-12-15 14:23:54 +11:00
Jimmy Chen
366f0d7ac2
Add flag to disable warning logs for duplicate gossip messages (#5009)
* Add flag to disable warning logs for duplicate gossip messages.

* Update Lighthouse book.
2023-12-15 09:26:51 +11:00
Pawan Dhananjay
ae4a296089
Convert a FullPayload to a BlindedPayload correctly (#5005) 2023-12-14 08:44:14 -05:00
Joel Rousseau
189430a45c
Add attestation simulator (#4880)
* basic scaffold

* remove unnecessary ?

* check if committee cache is init

* typed ValidatorMonitor with ethspecs + store attestations within

* nits

* process unaggregated attestation

* typo

* extract in func

* add tests

* better naming

* better naming 2

* less verbose

* use same naming as validator monitor

* use attestation_simulator

* add metrics

* remove cache

* refacto flag_indices process

* add lag

* remove copying state

* clean and lint

* extract metrics

* nits

* compare prom metrics in tests

* implement lag

* nits

* nits

* add attestation simulator service

* fmt

* return beacon_chain as arc

* nit: debug

* sed s/unaggregated/unagg.//

* fmt

* fmt

* nit: remove unused comments

* increase max unaggregated attestation hashmap to 64

* nit: sed s/clone/copied//

* improve perf: remove unecessary hashmap copy

* fix flag indices comp

* start service in client builder

* remove //

* cargo fmt

* lint

* cloned keys

* fmt

* use Slot value instead of pointer

* Update beacon_node/beacon_chain/src/attestation_simulator.rs

Co-authored-by: Paul Hauner <paul@paulhauner.com>

---------

Co-authored-by: Paul Hauner <paul@paulhauner.com>
2023-12-14 11:44:56 +11:00
Pawan Dhananjay
a3a370302a
Use the block header to compute the canonical_root (#5003) 2023-12-14 09:24:36 +11:00
Gua00va
a3fb27c99b
add forK_choice_read_lock as parameter (#4978) 2023-12-13 16:06:00 +11:00
Jimmy Chen
7f64738085
Update Rust version in lcli Dockerfile. (#5002) 2023-12-12 10:20:49 -05:00
Jimmy Chen
153aaa1679
Remove bors.toml 🫡 (#5001) 2023-12-12 16:54:22 +11:00
Jimmy Chen
4cf4819497
Add mergify merge queue configuration file (#4917)
* Add mergify.yml.

* Abstract out CI jobs to a "success" job so that a change in the CI jobs don't require modification to mergify configuration on stable branch.

* Add new jobs to the `needs` list of  `test-suite-success`.

* Set `batch_max_wait_time` to 60 s.
2023-12-12 16:04:29 +11:00
Age Manning
69f1b7afec
Disable flood publishing (#4383)
* Disable flood publish

* Change default configuration
2023-12-12 09:45:54 +11:00
ethDreamer
78ffa378b4
Batch Verify RPC Blobs (#4934) 2023-12-08 16:48:03 -05:00
realbigsean
46184e5ce4
Remove delayed lookups (#4992)
* initial rip out

* fix unused imports

* delete tests and fix lint

* fix peers scoring for blobs
2023-12-08 15:42:55 -05:00
Michael Sproul
b882519d2f
Implement POST validators/validator_balances APIs (#4872)
* Add POST for fetching validators from state

* Implement POST for balances

* Tests
2023-12-08 12:09:36 +11:00
Michael Sproul
e02adbf7bf
Update docs for v4.6.0 (#4982)
* Update DB migration docs

* Document VC broadcast modes

* Update downgrade example (#6)

* update downgrade example

* Add period

* Add v4.1.0

---------

Co-authored-by: chonghe <44791194+chong-he@users.noreply.github.com>
2023-12-08 10:45:05 +11:00
Divma
6c0c41c7ac
upgrade libp2p to v0.53.* (#4935)
* update libp2p and address compiler errors

* remove bandwidth logging from transport

* use libp2p registry

* make clippy happy

* use rust 1.73

* correct rpc keep alive

* remove comments and obsolte code

* remove libp2p prefix

* make clippy happy

* use quic under facade

* remove fast msg id

* bubble up close statements

* fix wrong comment
2023-12-07 20:39:59 +11:00
Jimmy Chen
67e0569d9b
Fix corrupted DB on networks where the first slot is skipped (Holesky) (#4985)
* Fix zero block roots on skip slots.

* Remove temporary comment, println code and unused imports.

* Remove `println!` in test.
2023-12-07 15:12:06 +11:00
Eitan Seri-Levi
8ba39cbf2c
Implement graffiti management API (#4951)
* implement get graffiti

* add set graffiti

* add set graffiti

* delete graffiti

* set graffiti

* set graffiti

* fmt

* added tests

* add graffiti file check

* update

* fixed delete req

* remove unused code

* changes based on feedback

* changes based on feedback

* invalid auth test plus lint

* fmt

* remove unneeded async
2023-12-07 12:02:46 +11:00
chonghe
d9d84242a7
CLI in Lighthouse Book (#4571)
* Add cli.sh file

* update bash script

* update Makefile

* update

* modified test-suite

* fix path

* Fix cli.sh permissions

* update cmd

* cli_manual

* Revise to update

* Update directory in Github

* Correct cli.txt directory

* test old cli_manual

* change exit 1

* Update cli and makefile

* Move cli.sh

* remove files

* fix permission

* Indentation and revision

* Fixed permission

* Create new cli folder

* remove dummy

* put a dummy file

* Revise cli.sh

* comment

* function

* remove vm.md

* test make cli

* test

* testing

* testing

* update

* update

* test

* test

* add vm

* change back non-debug mode

* add exist and update for future debug

* revise

* remove troubleshooting part

* update

* add summary.md

* test

* test

* Update Makefile

Co-authored-by: Mac L <mjladson@pm.me>

* Update Makefile

Co-authored-by: Mac L <mjladson@pm.me>

* Remove help-cli.md and rearrange

* Remove help-cli.md

* Update scripts/cli.sh

Co-authored-by: Mac L <mjladson@pm.me>

* Update scripts/cli.sh

Co-authored-by: Mac L <mjladson@pm.me>

* remove maxperf

* move then to same line as if

* Fix indent and echo file not found

* To be explicit in replacing the old file

* Add logging when there are changes

* Add local variables

* spacing

* remove cargo fmt

* update .md files

* Edit exit message to avoid confusion

* Remove am and add vm subcommands

* Add cargo-fmt

* Revise test-suite.yml

* Update SUMMARY.md

* Add set -e

* Add vm

* Fix

* Add vm

* set -e

* Remove return 1 and add :

* Small revision

* Fix typo

* Update scripts/cli.sh

Co-authored-by: Mac L <mjladson@pm.me>

* Indent

* Update scripts/cli.sh

Co-authored-by: Mac L <mjladson@pm.me>

* Remove .exe in Windows

* Fix period with \.

* test

* check diff

* linux commit

* Add cli.sh file

* update bash script

* update Makefile

* update

* modified test-suite

* fix path

* Fix cli.sh permissions

* update cmd

* cli_manual

* Revise to update

* Update directory in Github

* Correct cli.txt directory

* test old cli_manual

* change exit 1

* Update cli and makefile

* Move cli.sh

* remove files

* fix permission

* Indentation and revision

* Fixed permission

* Create new cli folder

* remove dummy

* put a dummy file

* Revise cli.sh

* comment

* function

* remove vm.md

* test make cli

* test

* testing

* testing

* update

* update

* test

* test

* add vm

* change back non-debug mode

* add exist and update for future debug

* revise

* remove troubleshooting part

* update

* add summary.md

* test

* test

* Update Makefile

Co-authored-by: Mac L <mjladson@pm.me>

* Update Makefile

Co-authored-by: Mac L <mjladson@pm.me>

* Remove help-cli.md and rearrange

* Remove help-cli.md

* Update scripts/cli.sh

Co-authored-by: Mac L <mjladson@pm.me>

* Update scripts/cli.sh

Co-authored-by: Mac L <mjladson@pm.me>

* remove maxperf

* move then to same line as if

* Fix indent and echo file not found

* To be explicit in replacing the old file

* Add logging when there are changes

* Add local variables

* spacing

* remove cargo fmt

* Edit exit message to avoid confusion

* update .md files

* Remove am and add vm subcommands

* Add cargo-fmt

* Revise test-suite.yml

* Update SUMMARY.md

* Add set -e

* Add vm

* Fix

* Add vm

* set -e

* Remove return 1 and add :

* Small revision

* Fix typo

* Update scripts/cli.sh

Co-authored-by: Mac L <mjladson@pm.me>

* Indent

* Update scripts/cli.sh

Co-authored-by: Mac L <mjladson@pm.me>

* Remove .exe in Windows

* Fix period with \.

* test

* check diff

* Revert "Merge branch 'book-cli' of https://github.com/chong-he/lighthouse into book-cli"

This reverts commit 314005d3f8bc0c13ecfa663ac712b1a2bae17540, reversing
changes made to a007f613786221211051394fad76ee1f5d0fe0f5.

* update

* update

* Remove echo diff

* Dockerize

* Remove `-ti`

* take ownership inside container

* fix mistake

* proper escaping, restore ownership afterwards

* try without taking ownership of repo

* update

* add diff for troubleshooting

* binary

* update using linux

* binary

* make file

* remove diff

* add diff

* update progressive balance help text

* Remove diff

---------

Co-authored-by: Mac L <mjladson@pm.me>
Co-authored-by: antondlr <anton@delaruelle.net>
2023-12-07 10:39:22 +11:00
ethDreamer
52117f43ba
Small Improvements (#4980)
* initial changes

* use arc<blobsidecar> in vector

* Utilize new pattern for KzgVerifiedBlob

* fmt

* Update beacon_node/beacon_chain/src/blob_verification.rs

Co-authored-by: realbigsean <seananderson33@GMAIL.com>

* forgot to save..

* lint

* fmt.. again

---------

Co-authored-by: realbigsean <seananderson33@GMAIL.com>
2023-12-06 15:51:40 -05:00
Pawan Dhananjay
31044402ee
Sidecar inclusion proof (#4900)
* Refactor BlobSidecar to new type

* Fix some compile errors

* Gossip verification compiles

* Fix http api types take 1

* Fix another round of compile errors

* Beacon node crate compiles

* EF tests compile

* Remove all blob signing from VC

* fmt

* Tests compile

* Fix some tests

* Fix more http tests

* get compiling

* Fix gossip conditions and tests

* Add basic proof generation and verification

* remove unnecessary ssz decode

* add back build_sidecar

* remove default at fork for blobs

* fix beacon chain tests

* get relase tests compiling

* fix lints

* fix existing spec tests

* add new ef tests

* fix gossip duplicate rule

* lints

* add back sidecar signature check in gossip

* add finalized descendant check to blob sidecar gossip

* fix error conversion

* fix release tests

* sidecar inclusion self review cleanup

* Add proof verification and computation metrics

* Remove accidentally committed file

* Unify some block and blob errors; add slashing conditions for sidecars

* Address review comment

* Clean up re-org tests (#4957)

* Address more review comments

* Add Comments & Eliminate Unnecessary Clones

* update names

* Update beacon_node/beacon_chain/src/metrics.rs

Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>

* Update beacon_node/network/src/network_beacon_processor/tests.rs

Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>

* pr feedback

* fix test compile

* Sidecar Inclusion proof small refactor and updates (#4967)

* Update some comments, variables and small cosmetic fixes.

* Couple blobs and proofs into a tuple in `PayloadAndBlobs` for simplicity and safety.

* Update function comment.

* Update testing/ef_tests/src/cases/merkle_proof_validity.rs

Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>

* Rename the block and blob wrapper types used in the beacon API interfaces.

* make sure gossip invalid blobs are passed to the slasher (#4970)

* Add blob headers to slasher before adding to DA checker

* Replace Vec with HashSet in BlockQueue

* fmt

* Rename gindex -> index

* Simplify gossip condition

---------

Co-authored-by: realbigsean <seananderson33@gmail.com>
Co-authored-by: realbigsean <sean@sigmaprime.io>
Co-authored-by: Michael Sproul <michael@sigmaprime.io>
Co-authored-by: Mark Mackey <mark@sigmaprime.io>
Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>
2023-12-05 11:19:59 -05:00
Manu NALEPA
ec8edfb89a
EIP-3076 tests - complete database (#4958)
* EIP-3076 interchange tests: Add `should_succeed_complete` boolean.

This new added `should_succeed_complete` boolean means the test should succeed using
complete anti-slashing DB, while the untouched `should_succeed` boolean is
still implicitely reserved for minimal anti-slashing DB.

Note:
This commit only adds the new `should_succeed_complete` boolean, and copy the value from
`should_succeed` without any meaning regarding the value this boolean should
take.

* `TestEIP3076SpecTests`: Modify two tests

Those two tests are modified in a way they both comply with minimal AND
complete anti-slashing DB.

* Disallow false positives and differentiate more minimal vs complete cases

* Fix my own typos

* Update to v5.3.0 tag

---------

Co-authored-by: Michael Sproul <michael@sigmaprime.io>
2023-12-05 10:49:50 +11:00
Jimmy Chen
4250385ae1
Fix stuck linkcheck on CI (#4977)
* Fix linkcheck CI job timeouts. Use linkcheck 3.0.0 without Docker.

* Add sleep to wait for the mdbook server to start serving
2023-12-05 10:32:10 +11:00
Michael Sproul
c8b2324880
Enable progressive balances fast mode by default (#4971)
* Enable progressive balances fast mode by default

* Fix default in chain_config
2023-12-04 14:42:49 +11:00
Gua00va
44aaf13ff0
Standard Liveness Endpoint (#4853)
* Changes to use required Endpoint

* Format

* fixed doppleganger service

* minor fix

* efficiency changes

* fixed tests

* remove commented line

---------

Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>
2023-11-30 17:41:22 +11:00
Michael Sproul
547ed1de63
Clone state ahead of block production (#4925)
* Clone state ahead of block production

* Add pruning and fix logging

* Don't hold 2 states in mem
2023-11-30 13:49:35 +11:00
ethDreamer
43d98153d6
Refactor & Fix Bugs in Payload Selection Logic (#4950)
* Refactor & Fix Bugs in Payload Selection Logic

* Fix lint

* Update beacon_node/execution_layer/src/lib.rs

* Finish renaming function
2023-11-29 15:20:12 +11:00
Paul Hauner
86163d94f2
Add lcli mock-el (#4587)
* Track time to early attester cache

* Add debug log for early attester cache

* Add mock-el command to lcli

* Revert beacon chain changes

* Tidy, fix compilation errors

* Add required flag

* Default to SYNCING response

* Hide flag
2023-11-29 10:04:29 +11:00
GeemoCandama
8a599ec7dc
API for LightClientBootstrap, LightClientFinalityUpdate, LightClientOptimisticUpdate and light client events (#3954)
* rebase and add comment

* conditional test

* test

* optimistic chould be working now

* finality should be working now

* try again

* try again

* clippy fix

* add lc bootstrap beacon api

* add lc optimistic/finality update to events

* fmt

* That error isn't occuring on my computer but I think this should fix it

* Add missing test file

* Update light client types to comply with Altair light client spec.

* Fix test compilation

* Support deserializing light client structures for the Bellatrix fork

* Move `get_light_client_bootstrap` logic to `BeaconChain`. `LightClientBootstrap` API to return `ForkVersionedResponse`.

* Misc fixes.
- log cleanup
- move http_api config mutation to `config::get_config` for consistency
- fix light client API responses

* Add light client bootstrap API test and fix existing ones.

* Fix test for `light-client-server` http api config.

* Appease clippy

* Efficiency improvement when retrieving beacon state.

---------

Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>
2023-11-28 17:14:29 +11:00
Michael Sproul
44c1817c2b
Remove block-delay-ms (#4956) 2023-11-28 16:00:28 +11:00
Jimmy Chen
c88cb371a0
Remove MAX_BLOBS_PER_BLOCK from Holesky config. (#4955) 2023-11-27 23:07:40 +11:00
Alexander Uvizhev
b4556a3d62
Broadcast various requests as per #4684 (#4920)
* multiple broadcast flags

* rewrite with single --broadcast option

* satisfy cargo fmt

* shorten sync-committee-messages

* fix a doc comment and a test

* use strum

* Add broadcast test to simulator

* bring --disable-run-on-all flag back with deprecation notice
2023-11-27 15:39:37 +11:00
realbigsean
e856a904ef
Merge pull request #4939 from eserilev/block-v3-general-tests
Add more block v3 tests
2023-11-21 14:52:52 -05:00
Eitan Seri-Levi
98f159cc18 fmt 2023-11-20 21:37:58 -08:00
Eitan Seri-Levi
24a0a7ffd0 remove cache check 2023-11-20 21:37:04 -08:00
Eitan Seri-Levi
228180bb35 add some more block v3 tests 2023-11-20 20:46:13 -08:00
Jimmy Chen
6b63d18420
Fix Rust beta compiler warnings (rustc 1.75.0-beta.1 (782883f60 2023-11-12)) (#4932) 2023-11-18 03:55:11 +11:00
Mac L
68e076d60a
Remove legacy database migrations (#4919)
* Remove legacy db migrations

* Fix tests

* Remove migrations to/from v15

* Remove v16 migrations
2023-11-16 22:09:05 +11:00
Michael Sproul
d04e361129
Fix missed block logs (#4922) 2023-11-16 22:07:58 +11:00