Commit Graph

46 Commits

Author SHA1 Message Date
Jimmy Chen
98cac2bc6b Deneb review .github (CI cleanup) (#4696)
## Issue Addressed

Related to https://github.com/sigp/lighthouse/issues/4676.

Deneb-specifc CI code to be removed before merging to `unstable`. Dot not merge until we're ready to merge into `unstable`, as we may need to release deneb docker images before merging.

Keep in mind that most of the changes in the below PR (to `unstable`) have already 
been merged to `deneb-free-blobs`, so merging `deneb-free-blobs` into `unstable` would include those changes - it would be ok if the release runners are ready, otherwise we may want to exclude them before merging.
- https://github.com/sigp/lighthouse/pull/4592
2023-10-18 15:23:31 +00:00
Jimmy Chen
c0b6b92f27
Merge unstable 20230925 into deneb-free-blobs. 2023-09-26 10:32:18 +10:00
realbigsean
082bb2d638 Self hosted docker builds (#4592)
## Issue Addressed

We're OOM'ing on Docker builds on the Deneb branch https://github.com/sigp/lighthouse/issues/3929

Are we ok to self host automated docker builds?


Co-authored-by: realbigsean <seananderson33@gmail.com>
Co-authored-by: realbigsean <sean@sigmaprime.io>
Co-authored-by: antondlr <anton@delaruelle.net>
2023-09-21 04:17:24 +00:00
Jimmy Chen
ff792d950c
Merge branch 'unstable' into merge-unstable-to-deneb-20230816
# Conflicts:
#	beacon_node/http_api/src/lib.rs
2023-08-16 14:31:59 +10:00
Jimmy Chen
ca050053bf Use the native concurrency property to cancel workflows (#4572)
I noticed that some of our workflows aren't getting cancelled when a new one has been triggered, so we ended up having a long queue in our CI when multiple changes are triggered in a short period.

Looking at the comment here, I noticed the list of workflow IDs are outdated and no longer exist, and some new ones are missing:
dfcb3363c7/.github/workflows/cancel-previous-runs.yml (L12-L13)

I attempted to update these, and came across this comment on the [`cancel-workflow-action`](https://github.com/styfle/cancel-workflow-action) repo:
> You probably don't need to install this custom action.
>
> Instead, use the native [concurrency](https://github.blog/changelog/2021-04-19-github-actions-limit-workflow-run-or-job-concurrency/) property to cancel workflows, for example:

So I thought instead of updating the workflow and maintaining the workflow IDs, perhaps we can try experimenting the [native `concurrency` property](https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#concurrency).
2023-08-14 03:16:03 +00:00
realbigsean
7f7ad799b3
Fix manifest lists (#4605)
* self hosted docker builds attempted fix

* use imagetools instead of docker manifest
2023-08-10 15:51:44 -04:00
realbigsean
ab37f02ddc
Fix env var references (#4604)
* self hosted docker builds attempted fix

* fix env var references in docker builds
2023-08-10 13:09:18 -04:00
realbigsean
11027e3487
self hosted docker builds attempted fix (#4603) 2023-08-10 12:42:56 -04:00
realbigsean
754ce5ec61
remove self hosted runner check where it might not be needed 2023-08-10 09:42:50 -04:00
realbigsean
8d81f1bee7
self hosted docker builds 2023-08-10 09:42:36 -04:00
realbigsean
ec1b36474b
update automatic docker builds for the correct branch (#4375) 2023-06-05 10:41:11 -04:00
Jimmy Chen
70c4ae35ab
Merge branch 'unstable' into deneb-free-blobs
# Conflicts:
#	.github/workflows/docker.yml
#	.github/workflows/local-testnet.yml
#	.github/workflows/test-suite.yml
#	Cargo.lock
#	Cargo.toml
#	beacon_node/beacon_chain/src/beacon_chain.rs
#	beacon_node/beacon_chain/src/builder.rs
#	beacon_node/beacon_chain/src/test_utils.rs
#	beacon_node/execution_layer/src/engine_api/json_structures.rs
#	beacon_node/network/src/beacon_processor/mod.rs
#	beacon_node/network/src/beacon_processor/worker/gossip_methods.rs
#	beacon_node/network/src/sync/backfill_sync/mod.rs
#	beacon_node/store/src/config.rs
#	beacon_node/store/src/hot_cold_store.rs
#	common/eth2_network_config/Cargo.toml
#	consensus/ssz/src/decode/impls.rs
#	consensus/ssz_derive/src/lib.rs
#	consensus/ssz_derive/tests/tests.rs
#	consensus/ssz_types/src/serde_utils/mod.rs
#	consensus/tree_hash/src/impls.rs
#	consensus/tree_hash/src/lib.rs
#	consensus/types/Cargo.toml
#	consensus/types/src/beacon_state.rs
#	consensus/types/src/chain_spec.rs
#	consensus/types/src/eth_spec.rs
#	consensus/types/src/fork_name.rs
#	lcli/Cargo.toml
#	lcli/src/main.rs
#	lcli/src/new_testnet.rs
#	scripts/local_testnet/el_bootnode.sh
#	scripts/local_testnet/genesis.json
#	scripts/local_testnet/geth.sh
#	scripts/local_testnet/setup.sh
#	scripts/local_testnet/start_local_testnet.sh
#	scripts/local_testnet/vars.env
#	scripts/tests/doppelganger_protection.sh
#	scripts/tests/genesis.json
#	scripts/tests/vars.env
#	testing/ef_tests/Cargo.toml
#	validator_client/src/block_service.rs
2023-05-30 22:44:05 +10:00
Michael Sproul
826e748629
Prepare CI for merge queues (#4252)
* Prepare CI for merge queues

* Fix syntax SNAFUs
2023-05-02 01:59:51 +00:00
Diva M
d93753cc88
Merge branch 'unstable' into off-4844 2023-03-02 15:38:00 -05:00
Michael Sproul
066c27750a
Merge remote-tracking branch 'origin/staging' into capella-update 2023-02-17 12:05:36 +11:00
realbigsean
b805fa6279
merge with upstream 2023-02-15 14:20:12 -05:00
Michael Sproul
2fcfdf1a01 Fix docker and deps (#3978)
## Proposed Changes

- Fix this cargo-audit failure for `sqlite3-sys`: https://github.com/sigp/lighthouse/actions/runs/4179008889/jobs/7238473962
- Prevent the Docker builds from running out of RAM on CI by removing `gnosis` and LMDB support from the `-dev` images (see: https://github.com/sigp/lighthouse/pull/3959#issuecomment-1430531155, successful run on my fork: https://github.com/michaelsproul/lighthouse/actions/runs/4179162480/jobs/7239537947).
2023-02-15 11:51:46 +00:00
Michael Sproul
18c8cab4da
Merge remote-tracking branch 'origin/unstable' into capella-merge 2023-02-14 12:07:27 +11:00
Nazar Hussain
fa1d4c7054 Invalid cross build feature flag (#3959)
## Issue Addressed

The documentation referring to build from source mismatches with the what gitworkflow uses. 

aa5b7ef783/book/src/installation-source.md (L118-L120)

## Proposed Changes

Because the github workflow uses `cross` to build from source and for that build there is different env variable `CROSS_FEATURES` so need pass at the compile time. 

## Additional Info

Verified that existing `-dev` builds does not contains the `minimal` spec enabled. 

```bash
> docker run --rm --name node-5-cl-lighthouse sigp/lighthouse:latest-amd64-unstable-dev lighthouse --version
Lighthouse v3.4.0-aa5b7ef
BLS library: blst-portable
SHA256 hardware acceleration: true
Allocator: jemalloc
Specs: mainnet (true), minimal (false), gnosis (true)
```
2023-02-13 03:32:03 +00:00
Nazar Hussain
c33eb29ee3 Fix the whitespace in docker workflow (#3952)
## Issue Addressed

Fix a whitespace issue that was causing failure in the docker build. 


## Additional Info

https://github.com/sigp/lighthouse/pull/3948
2023-02-08 20:23:21 +00:00
Nazar Hussain
7934485aef Update the docker build to include features based images (#3875)
## Proposed Changes

There are some features that are enabled/disabled with the `FEATURES` env variable. This PR would introduce a pattern to introduce docker images based on those features. This can be useful later on to have specific images for some experimental features in the future.

## Additional Info

We at Lodesart need to have `minimal` spec support for some cross-client network testing. To make it efficient on the CI, we tend to use minimal preset.
2023-02-08 02:18:51 +00:00
antondlr
9f2baced0b fix multiarch docker builds (#3904)
## Issue Addressed

#3902 
Tested and confirmed working [here](https://github.com/antondlr/lighthouse/actions/runs/3970418322)

## Additional Info

buildx v0.10.0 added provenance attestations to images but they are packed in a way that's incompatible with `docker manifest`
https://github.com/docker/buildx/releases
2023-01-25 14:21:54 +01:00
Michael Sproul
bb0e99c097 Merge remote-tracking branch 'origin/unstable' into capella 2023-01-21 10:37:26 +11:00
antondlr
3e67fa3038 fix multiarch docker builds (#3904)
## Issue Addressed

#3902 
Tested and confirmed working [here](https://github.com/antondlr/lighthouse/actions/runs/3970418322)

## Additional Info

buildx v0.10.0 added provenance attestations to images but they are packed in a way that's incompatible with `docker manifest`
https://github.com/docker/buildx/releases
2023-01-20 20:26:32 +00:00
realbigsean
d96d793bfb
fix compilation issues 2023-01-12 14:17:14 -05:00
realbigsean
06f71e8cce
merge capella 2023-01-12 12:51:09 -05:00
ethDreamer
52c1055fdc
Remove withdrawals-processing feature (#3864)
* Use spec to Determine Supported Engine APIs

* Remove `withdrawals-processing` feature

* Fixed Tests

* Missed Some Spots

* Fixed Another Test

* Stupid Clippy
2023-01-12 15:15:08 +11:00
realbigsean
c4a41e8f55
only use withdrawals-processing in docker build on capella 2022-12-29 11:07:22 -05:00
realbigsean
6d85930520
create automatic docker build for eip4844 2022-12-28 11:11:17 -05:00
Mark Mackey
b75ca74222 Removed withdrawals feature flag 2022-12-19 15:38:46 -06:00
Michael Sproul
2c7ebc7278
Enable withdrawals features in Capella docker images (#3805) 2022-12-15 12:25:45 +11:00
Michael Sproul
d48460782b
Publish capella images on push (#3803) 2022-12-15 11:42:35 +11:00
Michael Sproul
9bd6d9ce7a CI gardening maintenance (#3706)
## Issue Addressed

Closes https://github.com/sigp/lighthouse/issues/3656

## Proposed Changes

* Replace `set-output` by `$GITHUB_OUTPUT` usage
* Avoid rate-limits when installing `protoc` by making authenticated requests (continuation of https://github.com/sigp/lighthouse/pull/3621)
* Upgrade all Ubuntu 18.04 usage to 22.04 (18.04 is end of life)
* Upgrade macOS-latest to explicit macOS-12 to silence warning
* Use `actions/checkout@v3` and `actions/cache@v3` to avoid deprecated NodeJS v12

## Additional Info

Can't silence the NodeJS warnings entirely due to https://github.com/sigp/lighthouse/issues/3705. Can fix that in future.
2022-11-13 22:40:44 +00:00
Michael Sproul
f77e3bc0ad Add maxperf build profile (#3608)
## Proposed Changes

Add a new Cargo compilation profile called `maxperf` which enables more aggressive compiler optimisations at the expense of compilation time.

Some rough initial benchmarks show that this can provide up to a 25% reduction to run time for CPU bound tasks like block processing: https://docs.google.com/spreadsheets/d/15jHuZe7lLHhZq9Nw8kc6EL0Qh_N_YAYqkW2NQ_Afmtk/edit

The numbers in that spreadsheet compare the `consensus-context` branch from #3604 to the same branch compiled with the `maxperf` profile using:

```
PROFILE=maxperf make install-lcli
```

## Additional Info

The downsides of the maxperf profile are:

- It increases compile times substantially, which will particularly impact low-spec hardware. Compiling `lcli` is about 3x slower. Compiling Lighthouse is about 5x slower on my 5950X: 17m 38s rather than 3m 28s.

As a result I think we should not enable this everywhere by default.

- **Option 1**: enable by default for our released binaries. This gives the majority of users the fastest version of `lighthouse` possible, at the expense of slowing down our release CI. Source builds will continue to use the default `release` profile unless users opt-in to `maxperf`.
- **Option 2**: enable by default for source builds. This gives users building from source an edge, but makes them pay for it with compilation time. 

I think I would prefer Option 1. I'll try doing some benchmarking to see how long a maxperf build of Lighthouse would take on GitHub actions.

Credit to Nicholas Nethercote for documenting these options in the Rust Performance Book: https://nnethercote.github.io/perf-book/build-configuration.html.
2022-09-29 06:13:33 +00:00
Michael Sproul
139b44342f Optimized Docker images (#2966)
## Issue Addressed

Closes #2938

## Proposed Changes

* Build and publish images with a `-modern` suffix which enable CPU optimizations for modern hardware.
* Add docs for the plethora of available images!
* Unify all the Docker workflows in `docker.yml` (including for tagged releases).

## Additional Info

The `Dockerfile` is no longer used by our Docker Hub builds, as we use `cross` and a generic approach for ARM and x86. There's a new CI job `docker-build-from-source` which tests the `Dockerfile` without publishing anything.
2022-01-31 22:55:03 +00:00
realbigsean
bcf8ba6318 Add lcli Dockerfile and auto-build to CI (#2469)
## Issue Addressed

Resolves: #2087

## Proposed Changes

- Add a `Dockerfile` to the `lcli` directory
- Add a github actions job to build and push and `lcli` docker image on pushes to `unstable` and `stable`

## Additional Info

It's a little awkward but `lcli` requires the full project scope so must be built: 
- from the `lighthouse` dir with: `docker build -f ./lcli/Dockerflie .`
- from the `lcli` dir with: `docker build -f ./Dockerfile ../`

Didn't include `libssl-dev` or `ca-certificates`, `lcli` doesn't need these right?

Co-authored-by: realbigsean <seananderson33@gmail.com>
Co-authored-by: Michael Sproul <micsproul@gmail.com>
Co-authored-by: Michael Sproul <michael@sigmaprime.io>
2021-07-21 03:24:23 +00:00
Michael Sproul
1a748699d9 Rust 1.50 fixes (#2199)
## Issue Addressed

https://rustsec.org/advisories/RUSTSEC-2021-0021.html

## Proposed Changes

* Bump `nb-connect` to avoid the above advisory (and migrate from the yanked crate)
* Add a `rustup update stable` to the Docker build to fix this issue: https://github.com/sigp/lighthouse/actions/runs/567075245
2021-02-15 06:09:54 +00:00
realbigsean
51f7724c76 Automate docker version tag (#2150)
## Issue Addressed

N/A

## Proposed Changes

On any tag formatted `v*`, a full multi-arch docker build will be kicked off and automatically pushed to docker hub with the version tag.

This is a bit repetitive, because the image built will usually be the same as the image built on pushes to `stable`, but it seems like the simplest way to go about it and this will also work if we incorporate a workflow with `vX.X.X-rc` tags. 

## Additional Info

This may also need to wait for env variable updates: https://github.com/sigp/lighthouse/pull/2135#issuecomment-754977433

Co-authored-by: realbigsean <seananderson33@gmail.com>
2021-01-19 03:50:10 +00:00
realbigsean
908c8eadf3 remove protected environment (#2135)
## Issue Addressed

N/A

## Proposed Changes

Remove Github Action environments

## Additional Info

N/A


Co-authored-by: realbigsean <seananderson33@gmail.com>
2021-01-19 01:29:06 +00:00
realbigsean
b5e81eb6b2 add automated release workflow (#2077)
## Issue Addressed

Resolves #1674 

## Proposed Changes

- Whenever a tag is pushed with the prefix `v` this workflow is triggered
- creates portable and non-portable binaries for linux x86_64, linux aarch64, macOS
  - an attempt at using github actions caching
- signs each binary using GPG
- auto-generates full changelog based on commit messages since the last release
- creates a **draft** release
- hot new formatting (preview [here](https://github.com/realbigsean/lighthouse/releases/tag/v0.9.23))
- has been taking around 35 minutes

## Additional Info

TODOs:
- Figure out how we should automate dockerhub's version tag. 
  - It'd be quickest just to tag `latest`, but we'd need to make sure the docker workflow completes before this starts
- we do the same cross-compile in the `docker` workflow, we could try to use the same binary
- integrate a similar flow for unstable binaries (`-rc` tag?)
- improve caching, potentially use sccache
- if we start using a self-hosted runner this'll require some re-working

Need to add the following secrets to Github: 

- `GPG_PASSPHRASE`
- ~~`GPG_PUBLIC_KEY`~~ hard-coded this, because it was tough manage as a secret
- `GPG_SIGNING_KEY` 


Co-authored-by: realbigsean <seananderson33@gmail.com>
2020-12-23 07:53:34 +00:00
realbigsean
ca08fc7831 Revert "add caching to test suite (#2089)" (#2098)
## Issue Addressed

N/A

## Proposed Changes

I didn't realize the `PORTABLE` env variable is only picked up by `install` in the `Makefile` so we are still getting `SIGILL`s:

https://github.com/sigp/lighthouse/runs/1565004525?check_suite_focus=true

## Additional Info



Co-authored-by: realbigsean <seananderson33@gmail.com>
2020-12-16 23:29:07 +00:00
realbigsean
80f47fcfff add caching to test suite (#2089)
## Issue Addressed

N/A

## Proposed Changes

Add some caching to the test suite and to the aarch64 cross-compile in the docker build. 

## Additional Info

Cache hits only occur if the Cargo.lock file is unchanged, Github Actions runner OS matches, and the cache is "in scope". Some documentation on github actions cache scoping is here:

https://docs.github.com/en/free-pro-team@latest/actions/guides/caching-dependencies-to-speed-up-workflows#matching-a-cache-key

I'm not sure how frequently we'll get cache hits, I imagine only on smaller PR's or updates to the same PR.  And there is a cache size limit that we may end up reaching quickly.  But Github actions handles evictions if we go over that limit. 

Not sure how much of an impact this will end up having but I don't really see a downside to trying it out.

Co-authored-by: realbigsean <seananderson33@gmail.com>
2020-12-16 03:44:03 +00:00
realbigsean
c1e27f4c89 Improve docker auto builds (#2078)
## Issue Addressed

N/A

## Proposed Changes

- hardcode `ubuntu-18.04` -- I don't think this was causing us issues, but github actions is in the process of migrating `ubuntu-latest` from Ubuntu 18 -> 20.. so just in case
- different source of emulation dependencies -> https://github.com/tonistiigi/binfmt 
  - this one is explicitly referenced in the `buildx` github docs
- install emulation dependencies and run `docker buildx` in the same `run` command
- enable `buildx` with  `DOCKER_CLI_EXPERIMENTAL: enabled` rather than re-building it

## Additional Info

N/A


Co-authored-by: realbigsean <seananderson33@gmail.com>
2020-12-11 00:19:35 +00:00
realbigsean
adbd49ddc6 Multiarch docker GitHub actions (#2065)
## Issue Addressed

Resolves #1512

## Proposed Changes

- Adds a new docker Github Actions workflow  
- Removes the Dockerhub hook
- Adds a new Dockerfile for use with pre-existing cross-compiled binaries 
- on pushes to `unstable` 
  - builds an ARM64 image and tags it `latest-arm64-unstable`
  - builds an AMD64 image and tags it `latest-amd64-unstable`
  - builds an multiarch image by creating a manifest list referencing the prior two images and tags it `latest-unstable`
- on pushes to `stable` 
  - builds an ARM64 image and tags it `latest-arm64`
  - builds an AMD64 image and tags it `latest-amd64`
  - builds an multiarch image by creating a manifest list referencing the prior two images and tags it `latest`

## Additional Info
- for ARM64, first `cross` is used to cross compile the `lighthouse` and  `lcli` binaries, then `docker buildx` is installed to actually build the docker image for the correct target platform. The image build pretty much just copies the binaries from local into the docker image (thanks @michaelsproul :) )
- The AMD64 and ARM64 builds run in parallel, in total it's been taking around 45mins on a local runner
- This PR does **not** cover version tags on docker images at the moment

Co-authored-by: realbigsean <seananderson33@gmail.com>
2020-12-09 06:06:37 +00:00
Michael Sproul
211109bbc0 Revert "add a github action for build multi-arch docker images (#1574)" (#1591)
This reverts commit 2627463366.

## Issue Addressed

This is a temporary fix for #1589, by reverting #1574. The Docker image needs to be built with `--build-arg PORTABLE=true`, and we could probably integrate that into the multi-arch build, but in the interests of expediting a fix, this PR opts for a revert.
2020-09-06 04:46:25 +00:00
realbigsean
2627463366 add a github action for build multi-arch docker images (#1574)
## Issue Addressed

#1512

## Proposed Changes

Use Github Actions to automate the Docker image build, so that we can make a multi-arch image.  

## Additional Info

This change will require adding the DOCKER_USERNAME and DOCKER_PASSWORD secrets in Github. It will also require disabling the Docker Hub automated build.
2020-09-04 02:43:32 +00:00