* Add cli.sh file
* update bash script
* update Makefile
* update
* modified test-suite
* fix path
* Fix cli.sh permissions
* update cmd
* cli_manual
* Revise to update
* Update directory in Github
* Correct cli.txt directory
* test old cli_manual
* change exit 1
* Update cli and makefile
* Move cli.sh
* remove files
* fix permission
* Indentation and revision
* Fixed permission
* Create new cli folder
* remove dummy
* put a dummy file
* Revise cli.sh
* comment
* function
* remove vm.md
* test make cli
* test
* testing
* testing
* update
* update
* test
* test
* add vm
* change back non-debug mode
* add exist and update for future debug
* revise
* remove troubleshooting part
* update
* add summary.md
* test
* test
* Update Makefile
Co-authored-by: Mac L <mjladson@pm.me>
* Update Makefile
Co-authored-by: Mac L <mjladson@pm.me>
* Remove help-cli.md and rearrange
* Remove help-cli.md
* Update scripts/cli.sh
Co-authored-by: Mac L <mjladson@pm.me>
* Update scripts/cli.sh
Co-authored-by: Mac L <mjladson@pm.me>
* remove maxperf
* move then to same line as if
* Fix indent and echo file not found
* To be explicit in replacing the old file
* Add logging when there are changes
* Add local variables
* spacing
* remove cargo fmt
* update .md files
* Edit exit message to avoid confusion
* Remove am and add vm subcommands
* Add cargo-fmt
* Revise test-suite.yml
* Update SUMMARY.md
* Add set -e
* Add vm
* Fix
* Add vm
* set -e
* Remove return 1 and add :
* Small revision
* Fix typo
* Update scripts/cli.sh
Co-authored-by: Mac L <mjladson@pm.me>
* Indent
* Update scripts/cli.sh
Co-authored-by: Mac L <mjladson@pm.me>
* Remove .exe in Windows
* Fix period with \.
* test
* check diff
* linux commit
* Add cli.sh file
* update bash script
* update Makefile
* update
* modified test-suite
* fix path
* Fix cli.sh permissions
* update cmd
* cli_manual
* Revise to update
* Update directory in Github
* Correct cli.txt directory
* test old cli_manual
* change exit 1
* Update cli and makefile
* Move cli.sh
* remove files
* fix permission
* Indentation and revision
* Fixed permission
* Create new cli folder
* remove dummy
* put a dummy file
* Revise cli.sh
* comment
* function
* remove vm.md
* test make cli
* test
* testing
* testing
* update
* update
* test
* test
* add vm
* change back non-debug mode
* add exist and update for future debug
* revise
* remove troubleshooting part
* update
* add summary.md
* test
* test
* Update Makefile
Co-authored-by: Mac L <mjladson@pm.me>
* Update Makefile
Co-authored-by: Mac L <mjladson@pm.me>
* Remove help-cli.md and rearrange
* Remove help-cli.md
* Update scripts/cli.sh
Co-authored-by: Mac L <mjladson@pm.me>
* Update scripts/cli.sh
Co-authored-by: Mac L <mjladson@pm.me>
* remove maxperf
* move then to same line as if
* Fix indent and echo file not found
* To be explicit in replacing the old file
* Add logging when there are changes
* Add local variables
* spacing
* remove cargo fmt
* Edit exit message to avoid confusion
* update .md files
* Remove am and add vm subcommands
* Add cargo-fmt
* Revise test-suite.yml
* Update SUMMARY.md
* Add set -e
* Add vm
* Fix
* Add vm
* set -e
* Remove return 1 and add :
* Small revision
* Fix typo
* Update scripts/cli.sh
Co-authored-by: Mac L <mjladson@pm.me>
* Indent
* Update scripts/cli.sh
Co-authored-by: Mac L <mjladson@pm.me>
* Remove .exe in Windows
* Fix period with \.
* test
* check diff
* Revert "Merge branch 'book-cli' of https://github.com/chong-he/lighthouse into book-cli"
This reverts commit 314005d3f8bc0c13ecfa663ac712b1a2bae17540, reversing
changes made to a007f613786221211051394fad76ee1f5d0fe0f5.
* update
* update
* Remove echo diff
* Dockerize
* Remove `-ti`
* take ownership inside container
* fix mistake
* proper escaping, restore ownership afterwards
* try without taking ownership of repo
* update
* add diff for troubleshooting
* binary
* update using linux
* binary
* make file
* remove diff
* add diff
* update progressive balance help text
* Remove diff
---------
Co-authored-by: Mac L <mjladson@pm.me>
Co-authored-by: antondlr <anton@delaruelle.net>
* EIP-3076 interchange tests: Add `should_succeed_complete` boolean.
This new added `should_succeed_complete` boolean means the test should succeed using
complete anti-slashing DB, while the untouched `should_succeed` boolean is
still implicitely reserved for minimal anti-slashing DB.
Note:
This commit only adds the new `should_succeed_complete` boolean, and copy the value from
`should_succeed` without any meaning regarding the value this boolean should
take.
* `TestEIP3076SpecTests`: Modify two tests
Those two tests are modified in a way they both comply with minimal AND
complete anti-slashing DB.
* Disallow false positives and differentiate more minimal vs complete cases
* Fix my own typos
* Update to v5.3.0 tag
---------
Co-authored-by: Michael Sproul <michael@sigmaprime.io>
* Changes to use required Endpoint
* Format
* fixed doppleganger service
* minor fix
* efficiency changes
* fixed tests
* remove commented line
---------
Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>
* Track time to early attester cache
* Add debug log for early attester cache
* Add mock-el command to lcli
* Revert beacon chain changes
* Tidy, fix compilation errors
* Add required flag
* Default to SYNCING response
* Hide flag
* rebase and add comment
* conditional test
* test
* optimistic chould be working now
* finality should be working now
* try again
* try again
* clippy fix
* add lc bootstrap beacon api
* add lc optimistic/finality update to events
* fmt
* That error isn't occuring on my computer but I think this should fix it
* Add missing test file
* Update light client types to comply with Altair light client spec.
* Fix test compilation
* Support deserializing light client structures for the Bellatrix fork
* Move `get_light_client_bootstrap` logic to `BeaconChain`. `LightClientBootstrap` API to return `ForkVersionedResponse`.
* Misc fixes.
- log cleanup
- move http_api config mutation to `config::get_config` for consistency
- fix light client API responses
* Add light client bootstrap API test and fix existing ones.
* Fix test for `light-client-server` http api config.
* Appease clippy
* Efficiency improvement when retrieving beacon state.
---------
Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>
* multiple broadcast flags
* rewrite with single --broadcast option
* satisfy cargo fmt
* shorten sync-committee-messages
* fix a doc comment and a test
* use strum
* Add broadcast test to simulator
* bring --disable-run-on-all flag back with deprecation notice
* Added in process_justification_and_finalization
Added in process_justification_and_finalization to compute_attestation_rewards_altair to take into account justified attestations when coming out of inactivity leak. Also added in test to check for this edge case.
* Added in justification and finalization for compute_attestation_rewards_base
* Added in test for altair rewards without inactivity leak
* Delete BN spec flag and VC beacon-node flag
* Remove warn
* slog
* add warn
* delete eth1-endpoint
* delete server from vc cli.rs
* delete server flag in config.rs
* delete delete-lockfiles in vc
* delete allow-unsynced flag in VC
* delete strict-fee-recipient in VC and warn log
* delete merge flag in bn (hidden)
* delete count-unrealized and count-unrealized-full in bn (hidden)
* delete http-disable-legacy-spec in bn (hidden)
* delete eth1-endpoint in lcli
* delete warn message lcli
* delete eth1-endpoints
* delete minify in slashing protection
* delete minify related
* Remove mut
* add back warn! log
* Indentation
* Delete count-unrealized
* Delete eth1-endpoints
* Delete eth1-endpoint test
* delete eth1-endpints test
* delete allow-unsynced test
* Add back lcli eth1-endpoint
---------
Co-authored-by: Michael Sproul <michael@sigmaprime.io>
## Issue Addressed
-downgrades `Missing components over rpc` to debug because this isn't unusual and just results in a re-try
-removes the result from `Block component processed for lookup` because this prints the full block on an unknown parent error
Co-authored-by: realbigsean <seananderson33@gmail.com>
## Issue Addressed
The comment implies that observations for the given slot would be retained but they are not.
## Proposed Changes
I'm pretty sure the functionality is correct and the comment is slightly incorrect, so just update the comment. The comment needs to say something along the lines of "less than or equal to" rather than just "less than."
## Additional Info
It doesn't make sense to keep finalized observations since those are no longer accepted.
## Issue Addressed
Closes#4481.
(Continuation of #4648)
## Proposed Changes
- [x] Add `lighthouse db prune-states`
- [x] Make it work
- [x] Ensure block roots are handled correctly (to be addressed in 4735)
- [x] Check perf on mainnet/Goerli/Gnosis (takes a few seconds max)
- [x] Run block root healing logic (#4875 ) at the beginning
- [x] Add some tests
- [x] Update docs
- [x] Add `--freezer` flag and other improvements to `lighthouse db inspect`
Co-authored-by: Michael Sproul <michael@sigmaprime.io>
Co-authored-by: Jimmy Chen <jimmy@sigmaprime.io>
Co-authored-by: Michael Sproul <micsproul@gmail.com>
## Issue Addressed
#4582
## Proposed Changes
Add a new v3 block fetching flow that can decide to return a Full OR Blinded payload
## Additional Info
Co-authored-by: Michael Sproul <micsproul@gmail.com>
## Issue Addressed
I observed our forward sync on devnet 9 would stall when we would hit this log:
```
250425:Oct 19 00:54:17.133 WARN Blocks and blobs request for range received invalid data, error: KzgCommitmentMismatch, batch_id: 4338, peer_id: 16Uiu2HAmHbmkEQFDrJfNuy1aYyAfHkNUwSD9FN7EVAqGJ8YTF9Mh, service: sync, module: network::sync::manager:1036
```
## Proposed Changes
`range_sync_block_and_blob_response` [here](1cb02a13a5/beacon_node/network/src/sync/manager.rs (L1013)) removes the request from the sync manager. later, however if there's an error, `inject_error` [here](1cb02a13a5/beacon_node/network/src/sync/manager.rs (L1055)) expects the request to exist so we can handle retry logic. So this PR just re-inserts the request (withthout any accumulated blobs or blocks) when we hit an error here.
The issue is unique to block+blob sync because the error here is only possible from mismatches between blocks + blobs after we've downloaded both, there's no equivalent error in block sync
Co-authored-by: realbigsean <seananderson33@gmail.com>
## Issue Addressed
Closes#4817.
## Proposed Changes
- Fill in the linear block roots array between 0 and the slot of the first block (e.g. slots 0 and 1 on Holesky).
- Backport the `--freezer`, `--skip` and `--limit` options for `lighthouse db inspect` from tree-states. This allows us to easily view the database corruption of 4817 using `lighthouse db inspect --network holesky --freezer --column bbr --output values --limit 2`.
- Backport the `iter_column_from` change and `MemoryStore` overhaul from tree-states. These are required to enable `lighthouse db inspect`.
- Rework `freezer_upper_limit` to allow state lookups for slots below the `state_lower_limit`. Currently state lookups will fail until state reconstruction completes entirely.
There is a new regression test for the main bug, but no test for the `freezer_upper_limit` fix because we don't currently support running state reconstruction partially (see #3026). This will be fixed once we merge `tree-states`! In lieu of an automated test, I've tested manually on a Holesky node while it was reconstructing.
## Additional Info
Users who backfilled Holesky to slot 0 (e.g. using `--reconstruct-historic-states`) need to either:
- Re-sync from genesis.
- Re-sync using checkpoint sync and the changes from this PR.
Due to the recency of the Holesky genesis, writing a custom pass to fix up broken databases (which would require its own thorough testing) was deemed unnecessary. This is the primary reason for this PR being marked `backwards-incompat`.
This will create few conflicts with Deneb, which I've already resolved on `tree-states-deneb` and will be happy to backport to Deneb once this PR is merged to unstable.
Align the document with the [implementation](2841f60686/beacon_node/execution_layer/src/lib.rs (L740)) regarding the `--builder` flag.
The `tokio::join!` macro takes a list of async expressions and evaluates them concurrently on the same task.
1. Add commitments to logs and update the `Display` implementation of `KzgCommitment` to become truncated similarly to block root.
I've been finding it difficult to debug scenarios involving multiple blobs for the same `(index, block_root)`. Logging the commitment will help with this, we can match it to what exists in the block.
Example output:
```
Oct 20 21:13:36.700 DEBG Successfully verified gossip blob commitment: 0xa3c1…1cd8, index: 0, root: 0xf31e…f9de, slot: 154568
Oct 20 21:13:36.785 DEBG Successfully verified gossip block commitments: [0xa3c1…1cd8, 0x8655…02ff, 0x8d6a…955a, 0x84ac…3a1b, 0x9752…629b, 0xb9fc…20fb], root: 0xf31eeb732702e429e89057b15e1c0c631e8452e09e03cb1924353f536ef4f9de, slot: 154568, graffiti: teku/besu, service: beacon
```
Example output in a block with no blobs (this will show up pre-deneb):
```
426734:Oct 20 21:15:24.113 DEBG Successfully verified gossip block, commitments: [], root: 0x619db1360ba0e8d44ae2a0f2450ebca47e167191feecffcfac0e8d7b6c39623c, slot: 154577, graffiti: teku/nethermind, service: beacon, module: beacon_chain::beacon_chain:2765
```
2. Remove `strum::IntoStaticStr` from `AvailabilityCheckError`. This is because `IntoStaticStr` end up dropping information inside the enum. So kzg commitments in this error are dropped, making it more difficult to debug
```
AvailabilityCheckError::KzgCommitmentMismatch {
blob_commitment: KzgCommitment,
block_commitment: KzgCommitment,
},
```
which is output as just `AvailabilityCheckError`
3. Some additional misc sync logs I found useful in debugging https://github.com/sigp/lighthouse/pull/4869
4. This downgrades ”Block returned for single block lookup not present” to debug because I don’t think we can fix the scenario that causes this unless we can cancel inflight rpc requests
Co-authored-by: realbigsean <seananderson33@gmail.com>
## Issue Addressed
Fixes#4697.
This also unblocks the state pruning PR (#4835). Because self healing breaks if state pruning is applied to a database with missing block roots.
## Proposed Changes
- Fill in the missing block roots between last restore point slot and split slot when upgrading to latest database version.
## Issue Addressed
Makes lighthouse compliant with new kzg changes in https://github.com/ethereum/consensus-specs/releases/tag/v1.4.0-beta.3
## Proposed Changes
1. Adds new official trusted setup
2. Refactors kzg to match upstream changes in https://github.com/ethereum/c-kzg-4844/pull/377
3. Updates pre-generated `BlobBundle` to work with official trusted setup. ~~Using json here instead of ssz to account for different value of `MaxBlobCommitmentsPerBlock` in minimal and mainnet. By using json, we can just use one pre generated bundle for both minimal and mainnet. Size of 2 separate ssz bundles is approximately equal to one json bundle cc @jimmygchen~~
Dunno what I was doing, ssz works without any issues
4. Stores trusted_setup as just bytes in eth2_network_config so that we don't have kzg dependency in that lib and in lcli.
Co-authored-by: realbigsean <seananderson33@gmail.com>
Co-authored-by: realbigsean <seananderson33@GMAIL.com>
## Issue Addressed
N/A
## Proposed Changes
Sends blocks and blobs from http_api to the network channel for publishing in a single network channel send. This is to avoid overhead of multiple calls.
Also adds a metric for rpc blob retrieval duration.
## Issue Addressed
Addresses the recent CI failures caused by caching `blst` for the wrong CPU type.
## Proposed Changes
- Use `FEATURES: jemalloc,portable` when building Lighthouse & `lcli` in tests
- Add a new `TEST_FEATURES` and set to `portable` for all CI test jobs.
- Updated Makefiles to read the `TEST_FEATURES` environment variable, and default to none.
## Issue Addressed
We run `clippy` as part of our CI, so it would help new devs if we add the `make lint` command to the dev setup section in the Lighthouse book.
## Issue Addressed
#4512
Which issue # does this PR address?
## Proposed Changes
Add inactivity calculation for Altair
Please list or describe the changes introduced by this PR.
Add inactivity calculation for Altair
## Additional Info
Please provide any additional information. For example, future considerations
or information useful for reviewers.
Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>
## Issue Addressed
updates libp2p to the latest version and uses the new `SwarmBuilder`. Superseeds https://github.com/sigp/lighthouse/pull/4695/
CC @mxinden I don't think we can use both `bandwidth_loggers` with the new syntax right?
## Issue Addressed
Fix a bug in `lcli skip-slots` that resulted in it always writing the pre-state to the output file.
## Proposed Changes
Correctly keep track of the post-state, and write it.
## Issue Addressed
#4827
## Proposed Changes
This PR introduces a new build-arg to the Lighthouse Dockerfile: `CARGO_USE_GIT_CLI`. This arg will be passed into the `CARGO_NET_GIT_FETCH_WITH_CLI` [environment variable](https://doc.rust-lang.org/cargo/reference/config.html#netgit-fetch-with-cli), which instructs `cargo` to use the git CLI during `fetch` operations instead of the git library. Doing so works around [a bug](https://github.com/rust-lang/cargo/issues/10583) with the git library that causes it to go OOM during `fetch` operations on `arm64` platforms.
The default value is `false` so this doesn't affect Lighthouse builds or the CI pipeline. Running a build with `--build-arg CARGO_USE_GIT_CLI=true` will activate it, which is necessary to cross-compile the `arm64` binary when not using `cross` (i.e., when building via the Dockerfile instead of natively if you don't have a rust environment ready to go).
Special thanks to @michaelsproul for helping me repro the initial problem.
Co-authored-by: Michael Sproul <micsproul@gmail.com>
