Cross-compile to vendored x86_84, aarch64 (Raspberry Pi 4) (#1497)

## Issue Addressed

NA

## Proposed Changes

Adds support for using the [`cross`](https://github.com/rust-embedded/cross) project to produce cross-compiled binaries using Docker images.

Provides quite clean and simple cross-compiles cause all the complexity is hidden in Dockerfiles. It does require you to be in the `docker` group though.

## Details

- Adds shortcut commands to `Makefile`
- Ensures `reqwest` and `discv5` use vendored openssl libs (i.e., static not shared).
- Switches to a [commit](284f705964) of blst that has a renamed C function to avoid a collision with openssl (upstream issue: https://github.com/supranational/blst/issues/21).
- Updates `ring` to the latest satisfiable version, since an earlier version was causing issues with `cross`.
- Off-topic, but adds extra message about Windows support as suggested by Discord user.

## Additional Info

- ~~Blocked on #1495~~
- There are no tests in CI for this yet for a few reasons:
  - I'm hesitant to add more long-running tasks.
  - Short-term bitrot should be avoided since we'll use it each release.
  - In the long term I think it would be good to automate binary creation on a release.
- I observed the binaries increase in size from 50mb to 52mb after these changes.
This commit is contained in:
Paul Hauner 2020-08-11 05:16:30 +00:00
parent b83fcd5e5c
commit b063df5bf9
16 changed files with 93 additions and 12 deletions

12
Cargo.lock generated
View File

@ -521,7 +521,7 @@ dependencies = [
[[package]]
name = "blst"
version = "0.1.1"
source = "git+https://github.com/sigp/blst.git?rev=22bfb91721af125d1cb08aa201a18477665a45fe#22bfb91721af125d1cb08aa201a18477665a45fe"
source = "git+https://github.com/sigp/blst.git?rev=284f7059642851c760a09fb1708bcb59c7ca323c#284f7059642851c760a09fb1708bcb59c7ca323c"
dependencies = [
"cc",
"glob",
@ -3468,6 +3468,15 @@ version = "0.1.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "77af24da69f9d9341038eba93a073b1fdaaa1b788221b00a69bce9e762cb32de"
[[package]]
name = "openssl-src"
version = "111.10.2+1.1.1g"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a287fdb22e32b5b60624d4a5a7a02dbe82777f730ec0dbc42a0554326fef5a70"
dependencies = [
"cc",
]
[[package]]
name = "openssl-sys"
version = "0.9.58"
@ -3477,6 +3486,7 @@ dependencies = [
"autocfg 1.0.0",
"cc",
"libc",
"openssl-src",
"pkg-config",
"vcpkg",
]

View File

@ -21,6 +21,28 @@ else
cargo install --path lcli --force --locked
endif
# The following commands use `cross` to build a cross-compile.
#
# These commands require that:
#
# - `cross` is installed (`cargo install cross`).
# - Docker is running.
# - The current user is in the `docker` group.
#
# The resulting binaries will be created in the `target/` directory.
#
# The *-portable options compile the blst library *without* the use of some
# optimized CPU functions that may not be available on some systems. This
# results in a more portable binary with ~20% slower BLS verification.
build-x86_64:
cross build --release --manifest-path lighthouse/Cargo.toml --target x86_64-unknown-linux-gnu
build-x86_64-portable:
cross build --release --manifest-path lighthouse/Cargo.toml --target x86_64-unknown-linux-gnu --features portable
build-aarch64:
cross build --release --manifest-path lighthouse/Cargo.toml --target aarch64-unknown-linux-gnu
build-aarch64-portable:
cross build --release --manifest-path lighthouse/Cargo.toml --target aarch64-unknown-linux-gnu --features portable
# Runs the full workspace tests in **release**, without downloading any additional
# test vectors.
test-release:

View File

@ -31,7 +31,7 @@ slog-async = "2.5.0"
tokio = "0.2.21"
dirs = "2.0.2"
futures = "0.3.5"
reqwest = "0.10.4"
reqwest = { version = "0.10.4", features = ["native-tls-vendored"] }
url = "2.1.1"
eth1 = { path = "../eth1" }
genesis = { path = "../genesis" }

View File

@ -11,7 +11,7 @@ web3 = "0.11.0"
sloggers = "1.0.0"
[dependencies]
reqwest = "0.10.4"
reqwest = { version = "0.10.4", features = ["native-tls-vendored"] }
futures = { version = "0.3.5", features = ["compat"] }
serde_json = "1.0.52"
serde = { version = "1.0.110", features = ["derive"] }

View File

@ -32,7 +32,7 @@ snap = "1.0.0"
void = "1.0.2"
tokio-io-timeout = "0.4.0"
tokio-util = { version = "0.3.1", features = ["codec", "compat"] }
discv5 = { version = "0.1.0-alpha.8", features = ["libp2p"] }
discv5 = { version = "0.1.0-alpha.8", features = ["libp2p", "openssl-vendored"] }
tiny-keccak = "2.0.2"
environment = { path = "../../lighthouse/environment" }
# TODO: Remove rand crate for mainnet

View File

@ -7,6 +7,7 @@
* [Installation](./installation.md)
* [Docker](./docker.md)
* [Raspberry Pi 4](./pi.md)
* [Cross-Compiling](./cross-compiling.md)
* [Key Management](./key-management.md)
* [Create a wallet](./wallet-create.md)
* [Create a validator](./validator-create.md)

View File

@ -3,7 +3,9 @@
## 0. Install Rust
If you don't have Rust installed already, visit [rustup.rs](https://rustup.rs/) to install it.
> Note: if you're not familiar with Rust or you'd like more detailed instructions, see our [installation guide](./installation.md).
> Notes:
> - If you're not familiar with Rust or you'd like more detailed instructions, see our [installation guide](./installation.md).
> - Windows is presently only supported via [WSL](https://docs.microsoft.com/en-us/windows/wsl/about).
## 1. Download and install Lighthouse

View File

@ -0,0 +1,41 @@
# Cross-compiling
Lighthouse supports cross-compiling, allowing users to run a binary on one
platform (e.g., `aarch64`) that was compiled on another platform (e.g.,
`x86_64`).
## Instructions
Cross-compiling requires [`Docker`](https://docs.docker.com/engine/install/),
[`rustembedded/cross`](https://github.com/rust-embedded/cross) and for the
current user to be in the `docker` group.
The binaries will be created in the `target/` directory of the Lighthouse
project.
### Targets
The `Makefile` in the project contains four targets for cross-compiling:
- `build-x86_64`: builds an optimized version for x86_64 processors (suitable
for most users).
- `build-x86_64-portable`: builds a version x86_64 processors which avoids
using some modern CPU instructions that might cause an "illegal
instruction" error on older CPUs.
- `build-aarch64`: builds an optimized version for 64bit ARM processors
(suitable for Raspberry Pi 4).
- `build-aarch64-portable`: builds a version 64 bit ARM processors which avoids
using some modern CPU instructions that might cause an "illegal
instruction" error on older CPUs.
### Example
```bash
cd lighthouse
make build-aarch64
```
The `lighthouse` binary will be compiled inside a Docker container and placed
in `lighthouse/target/aarch64-unknown-linux-gnu/release`.

View File

@ -5,6 +5,11 @@ Tested on:
- Raspberry Pi 4 Model B (4GB)
- `Ubuntu 20.04 LTS (GNU/Linux 5.4.0-1011-raspi aarch64)`
*Note: [Lighthouse supports cross-compiling](./cross-compiling.md) to target a
Raspberry Pi (`aarch64`). Compiling on a faster machine (i.e., `x86_64`
desktop) may be convenient.*
### 1. Install Ubuntu
Follow the [Ubuntu Raspberry Pi installation instructions](https://ubuntu.com/download/raspberry-pi).

View File

@ -7,7 +7,7 @@ edition = "2018"
build = "build.rs"
[build-dependencies]
reqwest = { version = "0.10.4", features = ["blocking", "json"] }
reqwest = { version = "0.10.4", features = ["blocking", "json", "native-tls-vendored"] }
serde_json = "1.0.52"
sha2 = "0.9.1"
hex = "0.4.2"

View File

@ -7,7 +7,7 @@ edition = "2018"
build = "build.rs"
[build-dependencies]
reqwest = { version = "0.10.4", features = ["blocking"] }
reqwest = { version = "0.10.4", features = ["blocking", "native-tls-vendored"] }
eth2_config = { path = "../eth2_config"}
handlebars = "3.3.0"
serde_json = "1.0.56"

View File

@ -7,7 +7,7 @@ edition = "2018"
# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
[dependencies]
reqwest = { version = "0.10.4", features = ["json"] }
reqwest = { version = "0.10.4", features = ["json", "native-tls-vendored"] }
url = "2.1.1"
serde = "1.0.110"
futures = "0.3.5"

View File

@ -17,7 +17,7 @@ eth2_hashing = "0.1.0"
ethereum-types = "0.9.1"
arbitrary = { version = "0.4.4", features = ["derive"], optional = true }
zeroize = { version = "1.0.0", features = ["zeroize_derive"] }
blst = { git = "https://github.com/sigp/blst.git", rev = "22bfb91721af125d1cb08aa201a18477665a45fe" }
blst = { git = "https://github.com/sigp/blst.git", rev = "284f7059642851c760a09fb1708bcb59c7ca323c" }
[features]
default = ["supranational"]

View File

@ -10,7 +10,7 @@ description = "Hashing primitives used in Ethereum 2.0"
lazy_static = { version = "1.4.0", optional = true }
[target.'cfg(not(target_arch = "wasm32"))'.dependencies]
ring = "0.16.9"
ring = "0.16.12"
[target.'cfg(target_arch = "wasm32")'.dependencies]
sha2 = "0.9.1"

View File

@ -21,4 +21,4 @@ slog-json = "2.3.0"
exit-future = "0.2.0"
lazy_static = "1.4.0"
lighthouse_metrics = { path = "../../common/lighthouse_metrics" }
discv5 = "0.1.0-alpha.8"
discv5 = { version = "0.1.0-alpha.8", features = ["libp2p", "openssl-vendored"] }

View File

@ -10,7 +10,7 @@ beacon_node = { path = "../../beacon_node" }
types = { path = "../../consensus/types" }
eth2_config = { path = "../../common/eth2_config" }
tempdir = "0.3.7"
reqwest = "0.10.4"
reqwest = { version = "0.10.4", features = ["native-tls-vendored"] }
url = "2.1.1"
serde = "1.0.110"
futures = "0.3.5"