Commit Graph

26 Commits

Author SHA1 Message Date
Michael Sproul
066c27750a
Merge remote-tracking branch 'origin/staging' into capella-update 2023-02-17 12:05:36 +11:00
Michael Sproul
2fcfdf1a01 Fix docker and deps (#3978)
## Proposed Changes

- Fix this cargo-audit failure for `sqlite3-sys`: https://github.com/sigp/lighthouse/actions/runs/4179008889/jobs/7238473962
- Prevent the Docker builds from running out of RAM on CI by removing `gnosis` and LMDB support from the `-dev` images (see: https://github.com/sigp/lighthouse/pull/3959#issuecomment-1430531155, successful run on my fork: https://github.com/michaelsproul/lighthouse/actions/runs/4179162480/jobs/7239537947).
2023-02-15 11:51:46 +00:00
Michael Sproul
18c8cab4da
Merge remote-tracking branch 'origin/unstable' into capella-merge 2023-02-14 12:07:27 +11:00
Nazar Hussain
fa1d4c7054 Invalid cross build feature flag (#3959)
## Issue Addressed

The documentation referring to build from source mismatches with the what gitworkflow uses. 

aa5b7ef783/book/src/installation-source.md (L118-L120)

## Proposed Changes

Because the github workflow uses `cross` to build from source and for that build there is different env variable `CROSS_FEATURES` so need pass at the compile time. 

## Additional Info

Verified that existing `-dev` builds does not contains the `minimal` spec enabled. 

```bash
> docker run --rm --name node-5-cl-lighthouse sigp/lighthouse:latest-amd64-unstable-dev lighthouse --version
Lighthouse v3.4.0-aa5b7ef
BLS library: blst-portable
SHA256 hardware acceleration: true
Allocator: jemalloc
Specs: mainnet (true), minimal (false), gnosis (true)
```
2023-02-13 03:32:03 +00:00
Nazar Hussain
c33eb29ee3 Fix the whitespace in docker workflow (#3952)
## Issue Addressed

Fix a whitespace issue that was causing failure in the docker build. 


## Additional Info

https://github.com/sigp/lighthouse/pull/3948
2023-02-08 20:23:21 +00:00
Nazar Hussain
7934485aef Update the docker build to include features based images (#3875)
## Proposed Changes

There are some features that are enabled/disabled with the `FEATURES` env variable. This PR would introduce a pattern to introduce docker images based on those features. This can be useful later on to have specific images for some experimental features in the future.

## Additional Info

We at Lodesart need to have `minimal` spec support for some cross-client network testing. To make it efficient on the CI, we tend to use minimal preset.
2023-02-08 02:18:51 +00:00
Michael Sproul
bb0e99c097 Merge remote-tracking branch 'origin/unstable' into capella 2023-01-21 10:37:26 +11:00
antondlr
3e67fa3038 fix multiarch docker builds (#3904)
## Issue Addressed

#3902 
Tested and confirmed working [here](https://github.com/antondlr/lighthouse/actions/runs/3970418322)

## Additional Info

buildx v0.10.0 added provenance attestations to images but they are packed in a way that's incompatible with `docker manifest`
https://github.com/docker/buildx/releases
2023-01-20 20:26:32 +00:00
ethDreamer
52c1055fdc
Remove withdrawals-processing feature (#3864)
* Use spec to Determine Supported Engine APIs

* Remove `withdrawals-processing` feature

* Fixed Tests

* Missed Some Spots

* Fixed Another Test

* Stupid Clippy
2023-01-12 15:15:08 +11:00
Mark Mackey
b75ca74222 Removed withdrawals feature flag 2022-12-19 15:38:46 -06:00
Michael Sproul
2c7ebc7278
Enable withdrawals features in Capella docker images (#3805) 2022-12-15 12:25:45 +11:00
Michael Sproul
d48460782b
Publish capella images on push (#3803) 2022-12-15 11:42:35 +11:00
Michael Sproul
9bd6d9ce7a CI gardening maintenance (#3706)
## Issue Addressed

Closes https://github.com/sigp/lighthouse/issues/3656

## Proposed Changes

* Replace `set-output` by `$GITHUB_OUTPUT` usage
* Avoid rate-limits when installing `protoc` by making authenticated requests (continuation of https://github.com/sigp/lighthouse/pull/3621)
* Upgrade all Ubuntu 18.04 usage to 22.04 (18.04 is end of life)
* Upgrade macOS-latest to explicit macOS-12 to silence warning
* Use `actions/checkout@v3` and `actions/cache@v3` to avoid deprecated NodeJS v12

## Additional Info

Can't silence the NodeJS warnings entirely due to https://github.com/sigp/lighthouse/issues/3705. Can fix that in future.
2022-11-13 22:40:44 +00:00
Michael Sproul
f77e3bc0ad Add maxperf build profile (#3608)
## Proposed Changes

Add a new Cargo compilation profile called `maxperf` which enables more aggressive compiler optimisations at the expense of compilation time.

Some rough initial benchmarks show that this can provide up to a 25% reduction to run time for CPU bound tasks like block processing: https://docs.google.com/spreadsheets/d/15jHuZe7lLHhZq9Nw8kc6EL0Qh_N_YAYqkW2NQ_Afmtk/edit

The numbers in that spreadsheet compare the `consensus-context` branch from #3604 to the same branch compiled with the `maxperf` profile using:

```
PROFILE=maxperf make install-lcli
```

## Additional Info

The downsides of the maxperf profile are:

- It increases compile times substantially, which will particularly impact low-spec hardware. Compiling `lcli` is about 3x slower. Compiling Lighthouse is about 5x slower on my 5950X: 17m 38s rather than 3m 28s.

As a result I think we should not enable this everywhere by default.

- **Option 1**: enable by default for our released binaries. This gives the majority of users the fastest version of `lighthouse` possible, at the expense of slowing down our release CI. Source builds will continue to use the default `release` profile unless users opt-in to `maxperf`.
- **Option 2**: enable by default for source builds. This gives users building from source an edge, but makes them pay for it with compilation time. 

I think I would prefer Option 1. I'll try doing some benchmarking to see how long a maxperf build of Lighthouse would take on GitHub actions.

Credit to Nicholas Nethercote for documenting these options in the Rust Performance Book: https://nnethercote.github.io/perf-book/build-configuration.html.
2022-09-29 06:13:33 +00:00
Michael Sproul
139b44342f Optimized Docker images (#2966)
## Issue Addressed

Closes #2938

## Proposed Changes

* Build and publish images with a `-modern` suffix which enable CPU optimizations for modern hardware.
* Add docs for the plethora of available images!
* Unify all the Docker workflows in `docker.yml` (including for tagged releases).

## Additional Info

The `Dockerfile` is no longer used by our Docker Hub builds, as we use `cross` and a generic approach for ARM and x86. There's a new CI job `docker-build-from-source` which tests the `Dockerfile` without publishing anything.
2022-01-31 22:55:03 +00:00
realbigsean
bcf8ba6318 Add lcli Dockerfile and auto-build to CI (#2469)
## Issue Addressed

Resolves: #2087

## Proposed Changes

- Add a `Dockerfile` to the `lcli` directory
- Add a github actions job to build and push and `lcli` docker image on pushes to `unstable` and `stable`

## Additional Info

It's a little awkward but `lcli` requires the full project scope so must be built: 
- from the `lighthouse` dir with: `docker build -f ./lcli/Dockerflie .`
- from the `lcli` dir with: `docker build -f ./Dockerfile ../`

Didn't include `libssl-dev` or `ca-certificates`, `lcli` doesn't need these right?

Co-authored-by: realbigsean <seananderson33@gmail.com>
Co-authored-by: Michael Sproul <micsproul@gmail.com>
Co-authored-by: Michael Sproul <michael@sigmaprime.io>
2021-07-21 03:24:23 +00:00
Michael Sproul
1a748699d9 Rust 1.50 fixes (#2199)
## Issue Addressed

https://rustsec.org/advisories/RUSTSEC-2021-0021.html

## Proposed Changes

* Bump `nb-connect` to avoid the above advisory (and migrate from the yanked crate)
* Add a `rustup update stable` to the Docker build to fix this issue: https://github.com/sigp/lighthouse/actions/runs/567075245
2021-02-15 06:09:54 +00:00
realbigsean
51f7724c76 Automate docker version tag (#2150)
## Issue Addressed

N/A

## Proposed Changes

On any tag formatted `v*`, a full multi-arch docker build will be kicked off and automatically pushed to docker hub with the version tag.

This is a bit repetitive, because the image built will usually be the same as the image built on pushes to `stable`, but it seems like the simplest way to go about it and this will also work if we incorporate a workflow with `vX.X.X-rc` tags. 

## Additional Info

This may also need to wait for env variable updates: https://github.com/sigp/lighthouse/pull/2135#issuecomment-754977433

Co-authored-by: realbigsean <seananderson33@gmail.com>
2021-01-19 03:50:10 +00:00
realbigsean
908c8eadf3 remove protected environment (#2135)
## Issue Addressed

N/A

## Proposed Changes

Remove Github Action environments

## Additional Info

N/A


Co-authored-by: realbigsean <seananderson33@gmail.com>
2021-01-19 01:29:06 +00:00
realbigsean
b5e81eb6b2 add automated release workflow (#2077)
## Issue Addressed

Resolves #1674 

## Proposed Changes

- Whenever a tag is pushed with the prefix `v` this workflow is triggered
- creates portable and non-portable binaries for linux x86_64, linux aarch64, macOS
  - an attempt at using github actions caching
- signs each binary using GPG
- auto-generates full changelog based on commit messages since the last release
- creates a **draft** release
- hot new formatting (preview [here](https://github.com/realbigsean/lighthouse/releases/tag/v0.9.23))
- has been taking around 35 minutes

## Additional Info

TODOs:
- Figure out how we should automate dockerhub's version tag. 
  - It'd be quickest just to tag `latest`, but we'd need to make sure the docker workflow completes before this starts
- we do the same cross-compile in the `docker` workflow, we could try to use the same binary
- integrate a similar flow for unstable binaries (`-rc` tag?)
- improve caching, potentially use sccache
- if we start using a self-hosted runner this'll require some re-working

Need to add the following secrets to Github: 

- `GPG_PASSPHRASE`
- ~~`GPG_PUBLIC_KEY`~~ hard-coded this, because it was tough manage as a secret
- `GPG_SIGNING_KEY` 


Co-authored-by: realbigsean <seananderson33@gmail.com>
2020-12-23 07:53:34 +00:00
realbigsean
ca08fc7831 Revert "add caching to test suite (#2089)" (#2098)
## Issue Addressed

N/A

## Proposed Changes

I didn't realize the `PORTABLE` env variable is only picked up by `install` in the `Makefile` so we are still getting `SIGILL`s:

https://github.com/sigp/lighthouse/runs/1565004525?check_suite_focus=true

## Additional Info



Co-authored-by: realbigsean <seananderson33@gmail.com>
2020-12-16 23:29:07 +00:00
realbigsean
80f47fcfff add caching to test suite (#2089)
## Issue Addressed

N/A

## Proposed Changes

Add some caching to the test suite and to the aarch64 cross-compile in the docker build. 

## Additional Info

Cache hits only occur if the Cargo.lock file is unchanged, Github Actions runner OS matches, and the cache is "in scope". Some documentation on github actions cache scoping is here:

https://docs.github.com/en/free-pro-team@latest/actions/guides/caching-dependencies-to-speed-up-workflows#matching-a-cache-key

I'm not sure how frequently we'll get cache hits, I imagine only on smaller PR's or updates to the same PR.  And there is a cache size limit that we may end up reaching quickly.  But Github actions handles evictions if we go over that limit. 

Not sure how much of an impact this will end up having but I don't really see a downside to trying it out.

Co-authored-by: realbigsean <seananderson33@gmail.com>
2020-12-16 03:44:03 +00:00
realbigsean
c1e27f4c89 Improve docker auto builds (#2078)
## Issue Addressed

N/A

## Proposed Changes

- hardcode `ubuntu-18.04` -- I don't think this was causing us issues, but github actions is in the process of migrating `ubuntu-latest` from Ubuntu 18 -> 20.. so just in case
- different source of emulation dependencies -> https://github.com/tonistiigi/binfmt 
  - this one is explicitly referenced in the `buildx` github docs
- install emulation dependencies and run `docker buildx` in the same `run` command
- enable `buildx` with  `DOCKER_CLI_EXPERIMENTAL: enabled` rather than re-building it

## Additional Info

N/A


Co-authored-by: realbigsean <seananderson33@gmail.com>
2020-12-11 00:19:35 +00:00
realbigsean
adbd49ddc6 Multiarch docker GitHub actions (#2065)
## Issue Addressed

Resolves #1512

## Proposed Changes

- Adds a new docker Github Actions workflow  
- Removes the Dockerhub hook
- Adds a new Dockerfile for use with pre-existing cross-compiled binaries 
- on pushes to `unstable` 
  - builds an ARM64 image and tags it `latest-arm64-unstable`
  - builds an AMD64 image and tags it `latest-amd64-unstable`
  - builds an multiarch image by creating a manifest list referencing the prior two images and tags it `latest-unstable`
- on pushes to `stable` 
  - builds an ARM64 image and tags it `latest-arm64`
  - builds an AMD64 image and tags it `latest-amd64`
  - builds an multiarch image by creating a manifest list referencing the prior two images and tags it `latest`

## Additional Info
- for ARM64, first `cross` is used to cross compile the `lighthouse` and  `lcli` binaries, then `docker buildx` is installed to actually build the docker image for the correct target platform. The image build pretty much just copies the binaries from local into the docker image (thanks @michaelsproul :) )
- The AMD64 and ARM64 builds run in parallel, in total it's been taking around 45mins on a local runner
- This PR does **not** cover version tags on docker images at the moment

Co-authored-by: realbigsean <seananderson33@gmail.com>
2020-12-09 06:06:37 +00:00
Michael Sproul
211109bbc0 Revert "add a github action for build multi-arch docker images (#1574)" (#1591)
This reverts commit 2627463366.

## Issue Addressed

This is a temporary fix for #1589, by reverting #1574. The Docker image needs to be built with `--build-arg PORTABLE=true`, and we could probably integrate that into the multi-arch build, but in the interests of expediting a fix, this PR opts for a revert.
2020-09-06 04:46:25 +00:00
realbigsean
2627463366 add a github action for build multi-arch docker images (#1574)
## Issue Addressed

#1512

## Proposed Changes

Use Github Actions to automate the Docker image build, so that we can make a multi-arch image.  

## Additional Info

This change will require adding the DOCKER_USERNAME and DOCKER_PASSWORD secrets in Github. It will also require disabling the Docker Hub automated build.
2020-09-04 02:43:32 +00:00