Go to file
David Boreham 071cefa9cc
Merge pull request #24 from cerc-io/telackey/fixturenet
Add build and compose defintions for Merge 'fixturenet'.
2022-10-14 14:11:56 -06:00
app Clusters 2022-10-07 07:12:43 -06:00
compose Add comment and minor fix. 2022-10-14 14:32:35 -05:00
config Merge branch 'telackey/fixturenet' of github.com:cerc-io/stack-orchestrator into telackey/fixturenet 2022-10-14 13:49:32 -05:00
container-build Review comment 2022-10-14 14:36:49 -05:00
.gitignore trailing newline 2022-09-29 15:21:50 -06:00
cli.py Make all options use dash word delimiter 2022-10-06 07:18:29 -06:00
container-image-list.txt Merge branch 'main' into telackey/fixturenet 2022-10-14 12:57:14 -05:00
docker-compose-test.yaml Initial commit 2022-08-03 20:37:16 -06:00
first_time_setup.sh Merge pull request #16 from cerc-io/dboreham/use-bash 2022-10-06 07:57:45 -06:00
LICENSE Initial version of pip packaging 2022-08-23 11:32:55 -06:00
MANIFEST.in Initial version of pip packaging 2022-08-23 11:32:55 -06:00
pod-list.txt Add build and compose defintions for Merge 'fixturenet'. 2022-10-13 22:18:39 -05:00
README.md Rename cluster -> pod 2022-10-07 06:48:30 -06:00
repository-list.txt Initial rename 2022-10-12 15:28:45 -06:00
requirements.txt Add new module 2022-08-23 20:51:18 -06:00
setup.py Some lint goodness 2022-10-03 19:37:18 -06:00
tox.ini Moar complex 2022-10-03 19:43:43 -06:00

Stack Orchestrator

Stack Orchestrator allows building and deployment of a Laconic stack on a single machine with minimial prerequisites.

Setup

Developer Mode

Developer mode runs the orchestrator from a cloned git repository.

Prerequisites

Stack Orchestrator is a Python3 CLI tool that runs on any OS with Python3 and Docker. Tested on: Ubuntu 20/22.

Ensure that the following are already installed:

  1. Python3 (the version 3.8 available in Ubuntu 20/22 works)
    $ python3 --version
    Python 3.8.10
    
  2. Docker (Install a current version from dockerco, don't use the version from any Linux distro)
    $ docker --version
    Docker version 20.10.17, build 100c701
    
  3. If installed from regular package repository (not Docker Desktop), BE AWARE that the compose plugin may need to be installed, as well.
    DOCKER_CONFIG=${DOCKER_CONFIG:-$HOME/.docker}
    mkdir -p $DOCKER_CONFIG/cli-plugins
    curl -SL https://github.com/docker/compose/releases/download/v2.11.2/docker-compose-linux-x86_64 -o $DOCKER_CONFIG/cli-plugins/docker-compose
    chmod +x ~/.docker/cli-plugins/docker-compose
    
    # see https://docs.docker.com/compose/install/linux/#install-the-plugin-manually for further details
    # or to install for all users.
    

Install

  1. Clone this repository:
    $ git clone (https://github.com/cerc-io/stack-orchestrator.git
    
  2. Optional first time setup for empty dev root and fresh SO checkout:
    ./first_time_setup.sh
    # e.g. /home/USER/workspace1/
    # only contains the stack-orchestrator repo cloned in step 1.
    # this will naively attempt to setup and activate the venv, shiv, generate a LOCAL standalone laconic-so
    # and then setup-repositories in workspace1/
    
  3. Enter the project directory:
    $ cd stack-orchestrator
    
  4. Create and activate a venv:
    $ python3 -m venv venv
    $ source ./venv/bin/activate
    (venv) $
    
  5. Install the cli in edit mode:
    $ pip install --editable .
    
  6. Verify installation:
    (venv) $ laconic-so
    Usage: laconic-so [OPTIONS] COMMAND [ARGS]...
    
     Laconic Stack Orchestrator
    
    Options:
     --quiet
     --verbose
     --dry-run
     -h, --help  Show this message and exit.
    
    Commands:
     build-containers    build the set of containers required for a complete...
     deploy-system       deploy a stack
     setup-repositories  git clone the set of repositories required to build...
    

Build a zipapp (single file distributable script)

Use shiv to build a single file Python executable zip archive of laconic-so:

  1. Install shiv:
    $ (venv) pip install shiv
    
  2. Run shiv to create a zipapp file:
    $ (venv)  shiv -c laconic-so -o laconic-so .
    
    This creates a file ./laconic-so that is executable outside of any venv, and on other machines and OSes and architectures, and requiring only the system Python3:
  3. Verify it works:
    $ cp stack-orchetrator/laconic-so ~/bin
    $ laconic-so
       Usage: python -m laconic-so [OPTIONS] COMMAND [ARGS]...
    
       Laconic Stack Orchestrator
    
    Options:
       --quiet
       --verbose
       --dry-run
       -h, --help  Show this message and exit.
    
    Commands:
       build-containers    build the set of containers required for a complete...
       deploy-system       deploy a stack
       setup-repositories  git clone the set of repositories required to build...
    

CI Mode

write-me

Usage

There are three sub-commands: setup-repositories, build-containers and deploy-system that are generally run in order:

Note: $ laconic-so will run the version installed to ~/bin, while ./laconic-so can be invoked to run locally built version in a checkout

Setup Repositories

Clones the set of git repositories necessary to build a system.

Note: the use of ssh-agent is recommended in order to avoid entering your ssh key passphrase for each repository.

$ laconic-so --verbose setup-repositories #this will default to ~/cerc or CERC_REPO_BASE_DIR from an env file
#$ ./laconic-so --verbose --local-stack setup-repositories #this will use cwd ../ as dev_root_path

Build Containers

Builds the set of docker container images required to run a system. It takes around 10 minutes to build all the containers from cold.

$ laconic-so --verbose build-containers #this will default to ~/cerc or CERC_REPO_BASE_DIR from an env file
#$ ./laconic-so --verbose --local-stack build-containers #this will use cwd ../ as dev_root_path

Deploy System

Uses docker compose to deploy a system.

Use ---include <list of components> to deploy a subset of all containers:

$ laconic-so --verbose deploy-system --include db-sharding,contract,ipld-eth-server,go-ethereum-foundry up
$ laconic-so --verbose deploy-system --include db-sharding,contract,ipld-eth-server,go-ethereum-foundry down

Note: deploy-system command interacts with most recently built container images.

Platform Support

Native aarm64 is not currently supported. x64 emulation on ARM64 macos should work (not yet tested).

Implementation

The orchestrator's operation is driven by files shown below. repository-list.txt container the list of git repositories; container-image-list.txt contains the list of container image names, while clister-list.txt specifies the set of compose components (corresponding to individual docker-compose-xxx.yml files which may in turn specify more than one container). Files required to build each container image are stored under ./container-build/<container-name> Files required at deploy-time are stored under ./config/<component-name>

├── pod-list.txt
├── compose
│   ├── docker-compose-contract.yml
│   ├── docker-compose-db-sharding.yml
│   ├── docker-compose-db.yml
│   ├── docker-compose-eth-statediff-fill-service.yml
│   ├── docker-compose-go-ethereum-foundry.yml
│   ├── docker-compose-ipld-eth-beacon-db.yml
│   ├── docker-compose-ipld-eth-beacon-indexer.yml
│   ├── docker-compose-ipld-eth-server.yml
│   ├── docker-compose-lighthouse.yml
│   └── docker-compose-prometheus-grafana.yml
├── config
│   └── ipld-eth-server
├── container-build
│   ├── cerc-eth-statediff-fill-service
│   ├── cerc-go-ethereum
│   ├── cerc-go-ethereum-foundry
│   ├── cerc-ipld-eth-beacon-db
│   ├── cerc-ipld-eth-beacon-indexer
│   ├── cerc-ipld-eth-db
│   ├── cerc-ipld-eth-server
│   ├── cerc-lighthouse
│   └── cerc-test-contract
├── container-image-list.txt
├── repository-list.txt

write-more-of-me