* move gossipsub into a separate crate
* Merge branch 'unstable' of github.com:sigp/lighthouse into separate-gossipsub
* address review 2
* clippy beta
* update logging to log gossipsub logs
* fix lib.rs and tests.rs
* update decode.rs
* auto-delete in Cargo.lock
* delete milagro in cargo.toml
* remove milagro from makefile
* remove milagro from the name
* delete milagro in comment
* delete milagro in cargo.toml
* delete in /testing/ef_tests/cargo.toml
* delete milagro in the logical OR
* delete milagro in /lighthouse/src/main.rs
* delete milagro in /crypto/bls/tests/tests.rs
* delete milagro in comment
* delete milagro in /testing//ef_test/src//cases/bls_eth_aggregate_pubkeys.rs
* delete milagro
* delete more in lib.rs
* delete more in lib.rs
* delete more in lib.rs
* delete milagro in /crypto/bls/src/lib.rs
* delete milagro in crypto/bls/src/mod.rs
* delete milagro.rs
* remove exit-future usage,
as it is non maintained, and replace with async-channel which is already in the repo.
* Merge branch 'unstable' of https://github.com/sigp/lighthouse into remove-exit-future
* Merge branch 'unstable' of https://github.com/sigp/lighthouse into remove-exit-future
* Deactivate RPC Connection Handler
after goodbye message is sent
* nit: use to_string instead of format
* Merge branch 'unstable' of https://github.com/sigp/lighthouse into rpc-shutdown-improvement
* clippy
* Fix cargo.lock
* Merge latest unstable
* MVP implementation (untested)
* update store checkpoint sync test
* update cli help
* Merge pull request #5253 from realbigsean/checkpoint-blobs-sean
Checkpoint blobs sean
* Warn only if blobs are missing from server
* Merge remote-tracking branch 'origin/unstable' into checkpoint-blobs
* Verify checkpoint blobs
* Move blob verification earlier
* add metrics layer
* add metrics
* simplify getting the target
* make clippy happy
* fix typos
* unify deps under workspace
* make import statement shorter, fix typos
* enable warn by default, mark flag as deprecated
* do not exit on error when initializing logging fails
* revert exit on error
* adjust bootnode logging
* add logging layer
* non blocking file writer
* non blocking file writer
* add tracing visitor
* use target as is by default
* make libp2p events register correctly
* adjust repilcated cli help
* refactor tracing layer
* linting
* filesize
* log gossipsub, dont filter by log level
* turn on debug logs by default, remove deprecation warning
* truncate file, add timestamp, add unit test
* suppress output (#5)
* use tracing appender
* cleanup
* Add a task to remove old log files and upgrade to warn level
* Add the time feature for tokio
* Udeps and fmt
---------
Co-authored-by: Diva M <divma@protonmail.com>
Co-authored-by: Divma <26765164+divagant-martian@users.noreply.github.com>
Co-authored-by: Age Manning <Age@AgeManning.com>
* add metrics layer
* add metrics
* simplify getting the target
* make clippy happy
* fix typos
* unify deps under workspace
* make import statement shorter, fix typos
* enable warn by default, mark flag as deprecated
* do not exit on error when initializing logging fails
* revert exit on error
* adjust bootnode logging
* use target as is by default
* make libp2p events register correctly
* adjust repilcated cli help
* turn on debug logs by default, remove deprecation warning
* suppress output (#5)
---------
Co-authored-by: Age Manning <Age@AgeManning.com>
* switch libp2p source to sigp fork
* Shift the connection closing inside RPC behaviour
* Tag specific commits
* Add slow peer scoring
* Fix test
* Use default yamux config
* Pin discv5 to our libp2p fork and cargo update
* Upgrade libp2p to enable yamux gains
* Add a comment specifying the branch being used
* cleanup build output from within container
(prevents CI warnings related to fs permissions)
* Remove revision tags add branches for testing, will revert back once we're happy
* Update to latest rust-libp2p version
* Pin forks
* Update cargo.lock
* Re-pin to panic-free rust
---------
Co-authored-by: Age Manning <Age@AgeManning.com>
Co-authored-by: Pawan Dhananjay <pawandhananjay@gmail.com>
Co-authored-by: antondlr <anton@delaruelle.net>
Co-authored-by: Michael Sproul <michael@sigmaprime.io>
* Initial attempt to upgrade hashbrown to latest to get rid of the crate warnings.
* Replace `.expect()` usage with use of `const`.
* Update ahash 0.7 as well
* Remove unsafe code.
* Update `lru` to 0.12 and fix release test errors.
* Set non-blocking socket
* Bump testcontainers to 0.15.
* Fix lint
---------
Co-authored-by: Michael Sproul <michael@sigmaprime.io>
* update libp2p and address compiler errors
* remove bandwidth logging from transport
* use libp2p registry
* make clippy happy
* use rust 1.73
* correct rpc keep alive
* remove comments and obsolte code
* remove libp2p prefix
* make clippy happy
* use quic under facade
* remove fast msg id
* bubble up close statements
* fix wrong comment
* Track time to early attester cache
* Add debug log for early attester cache
* Add mock-el command to lcli
* Revert beacon chain changes
* Tidy, fix compilation errors
* Add required flag
* Default to SYNCING response
* Hide flag
* multiple broadcast flags
* rewrite with single --broadcast option
* satisfy cargo fmt
* shorten sync-committee-messages
* fix a doc comment and a test
* use strum
* Add broadcast test to simulator
* bring --disable-run-on-all flag back with deprecation notice
## Issue Addressed
Closes#4817.
## Proposed Changes
- Fill in the linear block roots array between 0 and the slot of the first block (e.g. slots 0 and 1 on Holesky).
- Backport the `--freezer`, `--skip` and `--limit` options for `lighthouse db inspect` from tree-states. This allows us to easily view the database corruption of 4817 using `lighthouse db inspect --network holesky --freezer --column bbr --output values --limit 2`.
- Backport the `iter_column_from` change and `MemoryStore` overhaul from tree-states. These are required to enable `lighthouse db inspect`.
- Rework `freezer_upper_limit` to allow state lookups for slots below the `state_lower_limit`. Currently state lookups will fail until state reconstruction completes entirely.
There is a new regression test for the main bug, but no test for the `freezer_upper_limit` fix because we don't currently support running state reconstruction partially (see #3026). This will be fixed once we merge `tree-states`! In lieu of an automated test, I've tested manually on a Holesky node while it was reconstructing.
## Additional Info
Users who backfilled Holesky to slot 0 (e.g. using `--reconstruct-historic-states`) need to either:
- Re-sync from genesis.
- Re-sync using checkpoint sync and the changes from this PR.
Due to the recency of the Holesky genesis, writing a custom pass to fix up broken databases (which would require its own thorough testing) was deemed unnecessary. This is the primary reason for this PR being marked `backwards-incompat`.
This will create few conflicts with Deneb, which I've already resolved on `tree-states-deneb` and will be happy to backport to Deneb once this PR is merged to unstable.
## Issue Addressed
Makes lighthouse compliant with new kzg changes in https://github.com/ethereum/consensus-specs/releases/tag/v1.4.0-beta.3
## Proposed Changes
1. Adds new official trusted setup
2. Refactors kzg to match upstream changes in https://github.com/ethereum/c-kzg-4844/pull/377
3. Updates pre-generated `BlobBundle` to work with official trusted setup. ~~Using json here instead of ssz to account for different value of `MaxBlobCommitmentsPerBlock` in minimal and mainnet. By using json, we can just use one pre generated bundle for both minimal and mainnet. Size of 2 separate ssz bundles is approximately equal to one json bundle cc @jimmygchen~~
Dunno what I was doing, ssz works without any issues
4. Stores trusted_setup as just bytes in eth2_network_config so that we don't have kzg dependency in that lib and in lcli.
Co-authored-by: realbigsean <seananderson33@gmail.com>
Co-authored-by: realbigsean <seananderson33@GMAIL.com>
## Issue Addressed
updates libp2p to the latest version and uses the new `SwarmBuilder`. Superseeds https://github.com/sigp/lighthouse/pull/4695/
CC @mxinden I don't think we can use both `bandwidth_loggers` with the new syntax right?
## Proposed Changes
Add `compare_fields(as_iter)` as a field attribute to `compare_fields_derive`. This allows any iterable type to be compared in the same as a slice (by index).
This is forwards-compatible with tree-states types like `List` and `Vector` which can not be cast to slices.
## Issue Addressed
Addresses #4778, and potentially fixes the flaky deneb builder test `builder_works_post_deneb`.
The [deneb builder test](c5c84f1213/beacon_node/http_api/tests/tests.rs (L5371)) has been quite flaky on our CI (`release-tests`) since it was introduced. I'm guessing that it might be timing out on the builder `get_header` call (1 second), and therefore the local payload is used, while the test expects builder payload to be used.
On my machine the [`get_header` ](c5c84f1213/beacon_node/execution_layer/src/test_utils/mock_builder.rs (L367)) call takes about 550ms, which could easily go over 1s on slower environments (our windows CI runner is much slower than the ubuntu one).
I did a profile on the test and it showed that `blob_to_kzg_commiment` and `compute_kzg_proof` was taking a large chunk of time, so perhaps pre-generating the blobs could help stablise this test.
## Proposed Changes
Pre-generate blobs bundle for Mainnet and Minimal presets.
Before the change `get_header` took about **550ms**, and it's now reduced to **50-55ms** after the change. If timeout was indeed the cause of the flaky test, this fix should stablise it. This also brings the flaky `builder_works_post_deneb` test time from 50s to 10s. (8s if we only use a single blob)
* use workspace deps in kzg crate
* delete unused blobs dp path field
* full match on fork name in engine api get payload v3
* only accept v3 payloads on get payload v3 endpoint in mock el
* remove FIXMEs related to merge transition tests
* move static tx to test utils
* default max_per_epoch_activation_churn_limit to mainnet value
* remove unnecessary async
* remove comment
* use task executor in `blob_sidecars` endpoint
## Proposed Changes
- only use LH types to avoid build issues
- use warp instead of axum for the server to avoid importing the dep
## Additional Info
- wondering if we can move the `execution_layer/test_utils` to its own crate and import it as a dev dependency
- this would be made easier by separating out our engine API types into their own crate so we can use them in the test crate
- or maybe we can look into using reth types for the engine api if they are in their own crate
Co-authored-by: realbigsean <seananderson33@gmail.com>
* add processing and processed caching to the DA checker
* move processing cache out of critical cache
* get it compiling
* fix lints
* add docs to `AvailabilityView`
* some self review
* fix lints
* fix beacon chain tests
* cargo fmt
* make availability view easier to implement, start on testing
* move child component cache and finish test
* cargo fix
* cargo fix
* cargo fix
* fmt and lint
* make blob commitments not optional, rename some caches, add missing blobs struct
* Update beacon_node/beacon_chain/src/data_availability_checker/processing_cache.rs
Co-authored-by: ethDreamer <37123614+ethDreamer@users.noreply.github.com>
* marks review feedback and other general cleanup
* cargo fix
* improve availability view docs
* some renames
* some renames and docs
* fix should delay lookup logic
* get rid of some wrapper methods
* fix up single lookup changes
* add a couple docs
* add single blob merge method and improve process_... docs
* update some names
* lints
* fix merge
* remove blob indices from lookup creation log
* remove blob indices from lookup creation log
* delayed lookup logging improvement
* check fork choice before doing any blob processing
* remove unused dep
* Update beacon_node/beacon_chain/src/data_availability_checker/availability_view.rs
Co-authored-by: Michael Sproul <micsproul@gmail.com>
* Update beacon_node/beacon_chain/src/data_availability_checker/availability_view.rs
Co-authored-by: Michael Sproul <micsproul@gmail.com>
* Update beacon_node/beacon_chain/src/data_availability_checker/availability_view.rs
Co-authored-by: Michael Sproul <micsproul@gmail.com>
* Update beacon_node/beacon_chain/src/data_availability_checker/availability_view.rs
Co-authored-by: Michael Sproul <micsproul@gmail.com>
* Update beacon_node/network/src/sync/block_lookups/delayed_lookup.rs
Co-authored-by: Michael Sproul <micsproul@gmail.com>
* remove duplicate deps
* use gen range in random blobs geneartor
* rename processing cache fields
* require block root in rpc block construction and check block root consistency
* send peers as vec in single message
* spawn delayed lookup service from network beacon processor
* fix tests
---------
Co-authored-by: ethDreamer <37123614+ethDreamer@users.noreply.github.com>
Co-authored-by: Michael Sproul <micsproul@gmail.com>
