Optimized Docker images (#2966)

## Issue Addressed

Closes #2938

## Proposed Changes

* Build and publish images with a `-modern` suffix which enable CPU optimizations for modern hardware.
* Add docs for the plethora of available images!
* Unify all the Docker workflows in `docker.yml` (including for tagged releases).

## Additional Info

The `Dockerfile` is no longer used by our Docker Hub builds, as we use `cross` and a generic approach for ARM and x86. There's a new CI job `docker-build-from-source` which tests the `Dockerfile` without publishing anything.
This commit is contained in:
Michael Sproul 2022-01-31 22:55:03 +00:00
parent bdd70d7aef
commit 139b44342f
10 changed files with 187 additions and 186 deletions

View File

@ -5,6 +5,8 @@ on:
branches:
- unstable
- stable
tags:
- v*
env:
DOCKER_PASSWORD: ${{ secrets.DOCKER_PASSWORD }}
@ -13,20 +15,48 @@ env:
LCLI_IMAGE_NAME: ${{ github.repository_owner }}/lcli
jobs:
extract-branch-name:
# Extract the VERSION which is either `latest` or `vX.Y.Z`, and the VERSION_SUFFIX
# which is either empty or `-unstable`.
#
# It would be nice if the arch didn't get spliced into the version between `latest` and
# `unstable`, but for now we keep the two parts of the version separate for backwards
# compatibility.
extract-version:
runs-on: ubuntu-18.04
steps:
- name: Extract branch name
run: echo "::set-output name=BRANCH_NAME::$(echo ${GITHUB_REF#refs/heads/})"
id: extract_branch
- name: Extract version (if stable)
if: github.event.ref == 'refs/heads/stable'
run: |
echo "VERSION=latest" >> $GITHUB_ENV
echo "VERSION_SUFFIX=" >> $GITHUB_ENV
- name: Extract version (if unstable)
if: github.event.ref == 'refs/heads/unstable'
run: |
echo "VERSION=latest" >> $GITHUB_ENV
echo "VERSION_SUFFIX=-unstable" >> $GITHUB_ENV
- name: Extract version (if tagged release)
if: startsWith(github.event.ref, 'refs/tags')
run: |
echo "VERSION=$(echo ${GITHUB_REF#refs/tags/})" >> $GITHUB_ENV
echo "VERSION_SUFFIX=" >> $GITHUB_ENV
outputs:
BRANCH_NAME: ${{ steps.extract_branch.outputs.BRANCH_NAME }}
build-docker-arm64:
VERSION: ${{ env.VERSION }}
VERSION_SUFFIX: ${{ env.VERSION_SUFFIX }}
build-docker-single-arch:
name: build-docker-${{ matrix.binary }}
runs-on: ubuntu-18.04
needs: [extract-branch-name]
# We need to enable experimental docker features in order to use `docker buildx`
strategy:
matrix:
binary: [aarch64,
aarch64-portable,
x86_64,
x86_64-portable]
needs: [extract-version]
env:
# We need to enable experimental docker features in order to use `docker buildx`
DOCKER_CLI_EXPERIMENTAL: enabled
VERSION: ${{ needs.extract-version.outputs.VERSION }}
VERSION_SUFFIX: ${{ needs.extract-version.outputs.VERSION_SUFFIX }}
steps:
- uses: actions/checkout@v2
- name: Update Rust
@ -34,85 +64,76 @@ jobs:
- name: Dockerhub login
run: |
echo "${DOCKER_PASSWORD}" | docker login --username ${DOCKER_USERNAME} --password-stdin
- name: Cross build lighthouse binary
- name: Cross build Lighthouse binary
run: |
cargo install cross
make build-aarch64-portable
- name: Move cross-built ARM binary into Docker scope
make build-${{ matrix.binary }}
- name: Move cross-built binary into Docker scope (if ARM)
if: startsWith(matrix.binary, 'aarch64')
run: |
mkdir ./bin;
mv ./target/aarch64-unknown-linux-gnu/release/lighthouse ./bin;
- name: Set Env
if: needs.extract-branch-name.outputs.BRANCH_NAME == 'unstable'
- name: Move cross-built binary into Docker scope (if x86_64)
if: startsWith(matrix.binary, 'x86_64')
run: |
echo "TAG_SUFFIX=-unstable" >> $GITHUB_ENV;
mkdir ./bin;
mv ./target/x86_64-unknown-linux-gnu/release/lighthouse ./bin;
- name: Map aarch64 to arm64 short arch
if: startsWith(matrix.binary, 'aarch64')
run: echo "SHORT_ARCH=arm64" >> $GITHUB_ENV
- name: Map x86_64 to amd64 short arch
if: startsWith(matrix.binary, 'x86_64')
run: echo "SHORT_ARCH=amd64" >> $GITHUB_ENV;
- name: Set modernity suffix
if: endsWith(matrix.binary, '-portable') != true
run: echo "MODERNITY_SUFFIX=-modern" >> $GITHUB_ENV;
# Install dependencies for emulation. Have to create a new builder to pick up emulation support.
- name: Build ARM64 dockerfile (with push)
- name: Build Dockerfile and push
run: |
docker run --privileged --rm tonistiigi/binfmt --install arm64
docker run --privileged --rm tonistiigi/binfmt --install ${SHORT_ARCH}
docker buildx create --use --name cross-builder
docker buildx build \
--platform=linux/arm64 \
--platform=linux/${SHORT_ARCH} \
--file ./Dockerfile.cross . \
--tag ${IMAGE_NAME}:latest-arm64${TAG_SUFFIX} \
--tag ${IMAGE_NAME}:${VERSION}-${SHORT_ARCH}${VERSION_SUFFIX}${MODERNITY_SUFFIX} \
--push
build-docker-amd64:
runs-on: ubuntu-18.04
needs: [extract-branch-name]
steps:
- uses: actions/checkout@v2
- name: Update Rust
run: rustup update stable
- name: Dockerhub login
run: |
echo "${DOCKER_PASSWORD}" | docker login --username ${DOCKER_USERNAME} --password-stdin
- name: Set Env
if: needs.extract-branch-name.outputs.BRANCH_NAME == 'unstable'
run: |
echo "TAG_SUFFIX=-unstable" >> $GITHUB_ENV;
- name: Build AMD64 dockerfile (with push)
run: |
docker build \
--build-arg PORTABLE=true \
--tag ${IMAGE_NAME}:latest-amd64${TAG_SUFFIX} \
--file ./Dockerfile .
docker push ${IMAGE_NAME}:latest-amd64${TAG_SUFFIX}
build-docker-multiarch:
name: build-docker-multiarch${{ matrix.modernity }}
runs-on: ubuntu-18.04
needs: [build-docker-arm64, build-docker-amd64, extract-branch-name]
# We need to enable experimental docker features in order to use `docker manifest`
needs: [build-docker-single-arch, extract-version]
strategy:
matrix:
modernity: ["", "-modern"]
env:
# We need to enable experimental docker features in order to use `docker manifest`
DOCKER_CLI_EXPERIMENTAL: enabled
VERSION: ${{ needs.extract-version.outputs.VERSION }}
VERSION_SUFFIX: ${{ needs.extract-version.outputs.VERSION_SUFFIX }}
steps:
- name: Dockerhub login
run: |
echo "${DOCKER_PASSWORD}" | docker login --username ${DOCKER_USERNAME} --password-stdin
- name: Set Env
if: needs.extract-branch-name.outputs.BRANCH_NAME == 'unstable'
run: |
echo "TAG_SUFFIX=-unstable" >> $GITHUB_ENV;
- name: Create and push multiarch manifest
run: |
docker manifest create ${IMAGE_NAME}:latest${TAG_SUFFIX} \
--amend ${IMAGE_NAME}:latest-arm64${TAG_SUFFIX} \
--amend ${IMAGE_NAME}:latest-amd64${TAG_SUFFIX};
docker manifest push ${IMAGE_NAME}:latest${TAG_SUFFIX}
docker manifest create ${IMAGE_NAME}:${VERSION}${VERSION_SUFFIX}${{ matrix.modernity }} \
--amend ${IMAGE_NAME}:${VERSION}-arm64${VERSION_SUFFIX}${{ matrix.modernity }} \
--amend ${IMAGE_NAME}:${VERSION}-amd64${VERSION_SUFFIX}${{ matrix.modernity }};
docker manifest push ${IMAGE_NAME}:${VERSION}${VERSION_SUFFIX}${{ matrix.modernity }}
build-docker-lcli:
runs-on: ubuntu-18.04
needs: [extract-branch-name]
needs: [extract-version]
env:
VERSION: ${{ needs.extract-version.outputs.VERSION }}
VERSION_SUFFIX: ${{ needs.extract-version.outputs.VERSION_SUFFIX }}
steps:
- uses: actions/checkout@v2
- name: Dockerhub login
run: |
echo "${DOCKER_PASSWORD}" | docker login --username ${DOCKER_USERNAME} --password-stdin
- name: Set Env
if: needs.extract-branch-name.outputs.BRANCH_NAME == 'unstable'
run: |
echo "TAG_SUFFIX=-unstable" >> $GITHUB_ENV;
- name: Build lcli dockerfile (with push)
run: |
docker build \
--build-arg PORTABLE=true \
--tag ${LCLI_IMAGE_NAME}:latest${TAG_SUFFIX} \
--tag ${LCLI_IMAGE_NAME}:${VERSION}${VERSION_SUFFIX} \
--file ./lcli/Dockerfile .
docker push ${LCLI_IMAGE_NAME}:latest${TAG_SUFFIX}
docker push ${LCLI_IMAGE_NAME}:${VERSION}${VERSION_SUFFIX}

View File

@ -20,74 +20,6 @@ jobs:
id: extract_version
outputs:
VERSION: ${{ steps.extract_version.outputs.VERSION }}
build-docker-arm64:
runs-on: ubuntu-18.04
needs: [extract-version]
# We need to enable experimental docker features in order to use `docker buildx`
env:
DOCKER_CLI_EXPERIMENTAL: enabled
VERSION: ${{ needs.extract-version.outputs.VERSION }}
steps:
- uses: actions/checkout@v2
- name: Update Rust
run: rustup update stable
- name: Dockerhub login
run: |
echo "${DOCKER_PASSWORD}" | docker login --username ${DOCKER_USERNAME} --password-stdin
- name: Cross build lighthouse binary
run: |
cargo install cross
make build-aarch64-portable
- name: Move cross-built ARM binary into Docker scope
run: |
mkdir ./bin;
mv ./target/aarch64-unknown-linux-gnu/release/lighthouse ./bin;
# Install dependencies for emulation. Have to create a new builder to pick up emulation support.
- name: Build ARM64 dockerfile (with push)
run: |
docker run --privileged --rm tonistiigi/binfmt --install arm64
docker buildx create --use --name cross-builder
docker buildx build \
--platform=linux/arm64 \
--file ./Dockerfile.cross . \
--tag ${IMAGE_NAME}:${{ env.VERSION }}-arm64 \
--push
build-docker-amd64:
runs-on: ubuntu-18.04
needs: [extract-version]
env:
DOCKER_CLI_EXPERIMENTAL: enabled
VERSION: ${{ needs.extract-version.outputs.VERSION }}
steps:
- uses: actions/checkout@v2
- name: Dockerhub login
run: |
echo "${DOCKER_PASSWORD}" | docker login --username ${DOCKER_USERNAME} --password-stdin
- name: Build AMD64 dockerfile (with push)
run: |
docker build \
--build-arg PORTABLE=true \
--tag ${IMAGE_NAME}:${{ env.VERSION }}-amd64 \
--file ./Dockerfile .
docker push ${IMAGE_NAME}:${{ env.VERSION }}-amd64
build-docker-multiarch:
runs-on: ubuntu-18.04
needs: [build-docker-arm64, build-docker-amd64, extract-version]
# We need to enable experimental docker features in order to use `docker manifest`
env:
DOCKER_CLI_EXPERIMENTAL: enabled
VERSION: ${{ needs.extract-version.outputs.VERSION }}
steps:
- name: Dockerhub login
run: |
echo "${DOCKER_PASSWORD}" | docker login --username ${DOCKER_USERNAME} --password-stdin
- name: Create and push multiarch manifest
run: |
docker manifest create ${IMAGE_NAME}:${{ env.VERSION }} \
--amend ${IMAGE_NAME}:${{ env.VERSION }}-arm64 \
--amend ${IMAGE_NAME}:${{ env.VERSION }}-amd64;
docker manifest push ${IMAGE_NAME}:${{ env.VERSION }}
build:
name: Build Release
strategy:

View File

@ -123,7 +123,9 @@ jobs:
- name: Get latest version of stable Rust
run: rustup update stable
- name: Build the root Dockerfile
run: docker build .
run: docker build --build-arg FEATURES=portable -t lighthouse:local .
- name: Test the built image
run: docker run -t lighthouse:local lighthouse --version
eth1-simulator-ubuntu:
name: eth1-simulator-ubuntu
runs-on: ubuntu-latest

View File

@ -1,8 +1,8 @@
FROM rust:1.56.1-bullseye AS builder
FROM rust:1.58.1-bullseye AS builder
RUN apt-get update && apt-get -y upgrade && apt-get install -y cmake libclang-dev
COPY . lighthouse
ARG PORTABLE
ENV PORTABLE $PORTABLE
ARG FEATURES
ENV FEATURES $FEATURES
RUN cd lighthouse && make
FROM ubuntu:latest

View File

@ -1,8 +1,6 @@
.PHONY: tests
EF_TESTS = "testing/ef_tests"
BEACON_CHAIN_CRATE = "beacon_node/beacon_chain"
OP_POOL_CRATE = "beacon_node/operation_pool"
STATE_TRANSITION_VECTORS = "testing/state_transition_vectors"
GIT_TAG := $(shell git describe --tags --candidates 1)
BIN_DIR = "bin"
@ -22,19 +20,11 @@ FORKS=phase0 altair
#
# Binaries will most likely be found in `./target/release`
install:
ifeq ($(PORTABLE), true)
cargo install --path lighthouse --force --locked --features portable
else
cargo install --path lighthouse --force --locked
endif
cargo install --path lighthouse --force --locked --features "$(FEATURES)"
# Builds the lcli binary in release (optimized).
install-lcli:
ifeq ($(PORTABLE), true)
cargo install --path lcli --force --locked --features portable
else
cargo install --path lcli --force --locked
endif
cargo install --path lcli --force --locked --features "$(FEATURES)"
# The following commands use `cross` to build a cross-compile.
#
@ -50,13 +40,13 @@ endif
# optimized CPU functions that may not be available on some systems. This
# results in a more portable binary with ~20% slower BLS verification.
build-x86_64:
cross build --release --manifest-path lighthouse/Cargo.toml --target x86_64-unknown-linux-gnu --features modern,gnosis
cross build --release --bin lighthouse --target x86_64-unknown-linux-gnu --features modern,gnosis
build-x86_64-portable:
cross build --release --manifest-path lighthouse/Cargo.toml --target x86_64-unknown-linux-gnu --features portable,gnosis
cross build --release --bin lighthouse --target x86_64-unknown-linux-gnu --features portable,gnosis
build-aarch64:
cross build --release --manifest-path lighthouse/Cargo.toml --target aarch64-unknown-linux-gnu --features gnosis
cross build --release --bin lighthouse --target aarch64-unknown-linux-gnu --features gnosis
build-aarch64-portable:
cross build --release --manifest-path lighthouse/Cargo.toml --target aarch64-unknown-linux-gnu --features portable,gnosis
cross build --release --bin lighthouse --target aarch64-unknown-linux-gnu --features portable,gnosis
# Create a `.tar.gz` containing a binary for a specific target.
define tarball_release_binary
@ -102,21 +92,21 @@ check-benches:
# Typechecks consensus code *without* allowing deprecated legacy arithmetic or metrics.
check-consensus:
cargo check --manifest-path=consensus/state_processing/Cargo.toml --no-default-features
cargo check -p state_processing --no-default-features
# Runs only the ef-test vectors.
run-ef-tests:
rm -rf $(EF_TESTS)/.accessed_file_log.txt
cargo test --release --manifest-path=$(EF_TESTS)/Cargo.toml --features "ef_tests"
cargo test --release --manifest-path=$(EF_TESTS)/Cargo.toml --features "ef_tests,fake_crypto"
cargo test --release --manifest-path=$(EF_TESTS)/Cargo.toml --features "ef_tests,milagro"
cargo test --release -p ef_tests --features "ef_tests"
cargo test --release -p ef_tests --features "ef_tests,fake_crypto"
cargo test --release -p ef_tests --features "ef_tests,milagro"
./$(EF_TESTS)/check_all_files_accessed.py $(EF_TESTS)/.accessed_file_log.txt $(EF_TESTS)/consensus-spec-tests
# Run the tests in the `beacon_chain` crate for all known forks.
test-beacon-chain: $(patsubst %,test-beacon-chain-%,$(FORKS))
test-beacon-chain-%:
env FORK_NAME=$* cargo test --release --features fork_from_env --manifest-path=$(BEACON_CHAIN_CRATE)/Cargo.toml
env FORK_NAME=$* cargo test --release --features fork_from_env -p beacon_chain
# Run the tests in the `operation_pool` crate for all known forks.
test-op-pool: $(patsubst %,test-op-pool-%,$(FORKS))
@ -124,7 +114,7 @@ test-op-pool: $(patsubst %,test-op-pool-%,$(FORKS))
test-op-pool-%:
env FORK_NAME=$* cargo test --release \
--features 'beacon_chain/fork_from_env'\
--manifest-path=$(OP_POOL_CRATE)/Cargo.toml
-p operation_pool
# Runs only the tests/state_transition_vectors tests.
run-state-transition-tests:

View File

@ -19,15 +19,16 @@ project.
The `Makefile` in the project contains four targets for cross-compiling:
- `build-x86_64`: builds an optimized version for x86_64 processors (suitable for most users).
Supports Intel Broadwell (2014) and newer, and AMD Ryzen (2017) and newer.
- `build-x86_64-portable`: builds a version for x86_64 processors which avoids using some modern CPU
instructions that are incompatible with older CPUs. Suitable for pre-Broadwell/Ryzen CPUs.
- `build-aarch64`: builds an optimized version for 64-bit ARM processors
(suitable for Raspberry Pi 4).
instructions that are incompatible with older CPUs.
- `build-aarch64`: builds an optimized version for 64-bit ARM processors (suitable for Raspberry Pi 4).
- `build-aarch64-portable`: builds a version for 64-bit ARM processors which avoids using some
modern CPU instructions. In practice, very few ARM processors lack the instructions necessary to
run the faster non-portable build.
For more information about optimized vs portable builds see
[Portability](./installation-binaries.md#portability).
### Example
```bash

View File

@ -1,20 +1,17 @@
# Docker Guide
This repository has a `Dockerfile` in the root which builds an image with the
`lighthouse` binary installed. A pre-built image is available on Docker Hub.
There are two ways to obtain a Lighthouse Docker image:
## Obtaining the Docker image
1. [Docker Hub](#docker-hub), or
2. By [building a Docker image from source](#building-the-docker-image).
There are two ways to obtain the docker image, either via Docker Hub or
building the image from source. Once you have obtained the docker image via one
of these methods, proceed to [Using the Docker image](#using-the-docker-image).
Once you have obtained the docker image via one of these methods, proceed to [Using the Docker
image](#using-the-docker-image).
### Docker Hub
## Docker Hub
Lighthouse maintains the
[sigp/lighthouse](https://hub.docker.com/repository/docker/sigp/lighthouse/)
Docker Hub repository which provides an easy way to run Lighthouse without
building the image yourself.
Lighthouse maintains the [sigp/lighthouse][docker_hub] Docker Hub repository which provides an easy
way to run Lighthouse without building the image yourself.
Obtain the latest image with:
@ -28,26 +25,69 @@ Download and test the image with:
$ docker run sigp/lighthouse lighthouse --version
```
If you can see the latest [Lighthouse
release](https://github.com/sigp/lighthouse/releases) version (see example
below), then you've
successfully installed Lighthouse via Docker.
If you can see the latest [Lighthouse release](https://github.com/sigp/lighthouse/releases) version
(see example below), then you've successfully installed Lighthouse via Docker.
#### Example Version Output
> Pro tip: try the `latest-modern` image for a 20-30% speed-up! See [Available Docker
> Images](#available-docker-images) below.
### Example Version Output
```
Lighthouse vx.x.xx-xxxxxxxxx
BLS Library: xxxx-xxxxxxx
```
> Note: when you're running the Docker Hub image you're relying upon a
> pre-built binary instead of building from source.
### Available Docker Images
> Note: due to the Docker Hub image being compiled to work on arbitrary machines, it isn't as highly
> optimized as an image built from source. We're working to improve this, but for now if you want
> the absolute best performance, please build the image yourself.
There are several images available on Docker Hub.
### Building the Docker Image
Most users should use the `latest-modern` tag, which corresponds to the latest stable release of
Lighthouse with optimizations enabled. If you are running on older hardware then the default
`latest` image bundles a _portable_ version of Lighthouse which is slower but with better hardware
compatibility (see [Portability](./installation-binaries.md#portability)).
To install a specific tag (in this case `latest-modern`) add the tag name to your `docker` commands
like so:
```
$ docker pull sigp/lighthouse:latest-modern
```
Image tags follow this format:
```
${version}${arch}${stability}${modernity}
```
The `version` is:
* `vX.Y.Z` for a tagged Lighthouse release, e.g. `v2.1.1`
* `latest` for the `stable` branch (latest release) or `unstable` branch
The `stability` is:
* `-unstable` for the `unstable` branch
* empty for a tagged release or the `stable` branch
The `arch` is:
* `-amd64` for x86_64, e.g. Intel, AMD
* `-arm64` for aarch64, e.g. Rasperry Pi 4
* empty for a multi-arch image (works on either `amd64` or `arm64` platforms)
The `modernity` is:
* `-modern` for optimized builds
* empty for a `portable` unoptimized build
Examples:
* `latest-unstable-modern`: most recent `unstable` build for all modern CPUs (x86_64 or ARM)
* `latest-amd64`: most recent Lighthouse release for older x86_64 CPUs
* `latest-amd64-unstable`: most recent `unstable` build for older x86_64 CPUs
## Building the Docker Image
To build the image from source, navigate to
the root of the repository and run:
@ -103,3 +143,5 @@ If you use the `--http` flag you may also want to expose the HTTP port with `-p
```bash
$ docker run -p 9000:9000 -p 127.0.0.1:5052:5052 sigp/lighthouse lighthouse beacon --http --http-address 0.0.0.0
```
[docker_hub]: https://hub.docker.com/repository/docker/sigp/lighthouse/

View File

@ -20,13 +20,13 @@ Additionally there is also a `-portable` suffix which indicates if the `portable
- Without `portable`: uses modern CPU instructions to provide the fastest signature verification times (may cause `Illegal instruction` error on older CPUs)
- With `portable`: approx. 20% slower, but should work on all modern 64-bit processors.
For details, see [Portability](#portability).
## Usage
Each binary is contained in a `.tar.gz` archive. For this example, lets assume the user needs
a portable `x86_64` binary.
> Whilst this example uses `v0.2.13` we recommend always using the latest release.
### Steps
1. Go to the [Releases](https://github.com/sigp/lighthouse/releases) page and
@ -41,6 +41,19 @@ a portable `x86_64` binary.
> Windows users will need to execute the commands in Step 3 from PowerShell.
## Portability
Portable builds of Lighthouse are designed to run on the widest range of hardware possible, but
sacrifice the ability to make use of modern CPU instructions.
If you have a modern CPU then you should try running a non-portable build to get a 20-30% speed up.
* For **x86_64**, any CPU supporting the [ADX](https://en.wikipedia.org/wiki/Intel_ADX) instruction set
extension is compatible with the optimized build. This includes Intel Broadwell (2014)
and newer, and AMD Ryzen (2017) and newer.
* For **ARMv8**, most CPUs are compatible with the optimized build, including the Cortex-A72 used by
the Raspberry Pi 4.
## Troubleshooting
If you get a SIGILL (exit code 132), then your CPU is incompatible with the optimized build

View File

@ -1,7 +1,7 @@
# `lcli` requires the full project to be in scope, so this should be built either:
# - from the `lighthouse` dir with the command: `docker build -f ./lcli/Dockerflie .`
# - from the current directory with the command: `docker build -f ./Dockerfile ../`
FROM rust:1.56.1-bullseye AS builder
FROM rust:1.58.1-bullseye AS builder
RUN apt-get update && apt-get -y upgrade && apt-get install -y cmake
COPY . lighthouse
ARG PORTABLE

View File

@ -1,4 +1,4 @@
FROM rust:1.56.1-bullseye AS builder
FROM rust:1.58.1-bullseye AS builder
RUN apt-get update && apt-get -y upgrade && apt-get install -y cmake libclang-dev
COPY . lighthouse