deep-stack/stack-orchestrator
3
0
Fork 0
forked from cerc-io/stack-orchestrator
Code Pull Requests Activity

update readme

Browse Source
This commit is contained in:
zramsay 2023-01-09 19:44:39 -05:00
parent b13300ca81
commit 70723dd6d4
No known key found for this signature in database
GPG Key ID: FECD2DBFAE9DBE63
6 changed files with 315 additions and 297 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

91
.github/CONTRIBUTING.md vendored Normal file
View File

@ -0,0 +1,91 @@
# Contributing
Thank you for taking the time to make a contribution to Stack Orchestrator.
## Install (developer mode)
Suitable for developers either modifying or debugging the orchestrator Python code:
#### Prerequisites
In addition to the binary install prerequisites listed above, the following are required:
1. Python venv package
This may or may not be already installed depending on the host OS and version. Check by running:
```
$ python3 -m venv
usage: venv [-h] [--system-site-packages] [--symlinks | --copies] [--clear] [--upgrade] [--without-pip] [--prompt PROMPT] ENV_DIR [ENV_DIR ...]
venv: error: the following arguments are required: ENV_DIR
```
If the venv package is missing you should see a message indicating how to install it, for example with:
```
$ apt install python3.10-venv
```
#### Install
1. Clone this repository:
```
$ git clone (https://github.com/cerc-io/stack-orchestrator.git
```
4. Enter the project directory:
```
$ cd stack-orchestrator
```
5. Create and activate a venv:
```
$ python3 -m venv venv
$ source ./venv/bin/activate
(venv) $
```
6. Install the cli in edit mode:
```
$ pip install --editable .
```
7. Verify installation:
```
(venv) $ laconic-so
Usage: laconic-so [OPTIONS] COMMAND [ARGS]...
Laconic Stack Orchestrator
Options:
--quiet
--verbose
--dry-run
-h, --help Show this message and exit.
Commands:
build-containers build the set of containers required for a complete...
deploy-system deploy a stack
setup-repositories git clone the set of repositories required to build...
```
#### Build a zipapp (single file distributable script)
Use shiv to build a single file Python executable zip archive of laconic-so:
1. Install [shiv](https://github.com/linkedin/shiv):
```
$ (venv) pip install shiv
$ (venv) pip install wheel
```
1. Run shiv to create a zipapp file:
```
$ (venv) shiv -c laconic-so -o laconic-so .
```
This creates a file `./laconic-so` that is executable outside of any venv, and on other machines and OSes and architectures, and requiring only the system Python3:
1. Verify it works:
```
$ cp stack-orchetrator/laconic-so ~/bin
$ laconic-so
Usage: python -m laconic-so [OPTIONS] COMMAND [ARGS]...
Laconic Stack Orchestrator
Options:
--quiet
--verbose
--dry-run
-h, --help Show this message and exit.
Commands:
build-containers build the set of containers required for a complete...
deploy-system deploy a stack
setup-repositories git clone the set of repositories required to build...
```

233
README.md
View File

@ -1,49 +1,42 @@
# Stack Orchestrator # Stack Orchestrator
Stack Orchestrator allows building and deployment of a Laconic stack on a single machine with minimial prerequisites. Stack Orchestrator allows building and deployment of a Laconic Stack on a single machine with minimial prerequisites. It is a Python3 CLI tool that runs on any OS with Python3 and Docker. The following diagram summarizes the relevant repositories in the Laconic Stack - and the relationship to Stack Orchestrator.
## Setup ![The Stack](/docs/images/laconic-stack.png)
### Prerequisites
Stack Orchestrator is a Python3 CLI tool that runs on any OS with Python3 and Docker. Tested on: Ubuntu 20/22. ## Install
Ensure that the following are already installed: Ensure that the following are already installed:
1. Python3 (the stock Python3 version available in Ubuntu 20 and 22 is suitable) - [Python3](https://wiki.python.org/moin/BeginnersGuide/Download)
``` - [Docker](https://docs.docker.com/get-docker/)
$ python3 --version - [Docker Compose](https://docs.docker.com/compose/install/)
Python 3.8.10
``` Note: if installing docker-compose via package manager (as opposed to Docker Desktop), you must [install the plugin](https://docs.docker.com/compose/install/linux/#install-the-plugin-manually), e.g., on Linux:
2. Docker (Install a current version from dockerco, don't use the version from any Linux distro)
``` ```bash
$ docker --version mkdir -p ~/.docker/cli-plugins
Docker version 20.10.17, build 100c701 curl -SL https://github.com/docker/compose/releases/download/v2.11.2/docker-compose-linux-x86_64 -o ~/.docker/cli-plugins/docker-compose
```
3. If installed from regular package repository (not Docker Desktop), BE AWARE that the compose plugin may need to be installed, as well.
```
DOCKER_CONFIG=${DOCKER_CONFIG:-$HOME/.docker}
mkdir -p $DOCKER_CONFIG/cli-plugins
curl -SL https://github.com/docker/compose/releases/download/v2.11.2/docker-compose-linux-x86_64 -o $DOCKER_CONFIG/cli-plugins/docker-compose
chmod +x ~/.docker/cli-plugins/docker-compose chmod +x ~/.docker/cli-plugins/docker-compose
# see https://docs.docker.com/compose/install/linux/#install-the-plugin-manually for further details
# or to install for all users.
``` ```
### User Mode Install Next, download the latest release from [this page](https://github.com/cerc-io/stack-orchestrator/tags), into a suitable directory (e.g. `~/bin`):
User mode runs the orchestrator from a "binary" single-file release and does not require special Python environment setup. Use this mode unless you plan to make changes to the orchestrator source code. ```bash
curl -L -o ~/bin/laconic-so https://github.com/cerc-io/stack-orchestrator/releases/latest/download/laconic-so
```
*NOTE: User Mode is currently broken, use "Developer mode" described below for now.* Give it permissions:
```bash
chmod +x ~/bin/laconic-so
```
Ensure `laconic-so` is on the [`PATH`](https://unix.stackexchange.com/a/26059)
Verify operation:
1. Download the latest release from [this page](https://github.com/cerc-io/stack-orchestrator/tags), into a suitable directory (e.g. `~/bin`):
``` ```
$ cd ~/bin laconic-so --help
$ curl -L https://github.com/cerc-io/stack-orchestrator/releases/download/v1.0.3-alpha/laconic-so
```
1. Ensure `laconic-so` is on the `PATH`
1. Verify operation:
```
$ ~/bin/laconic-so --help
Usage: python -m laconic-so [OPTIONS] COMMAND [ARGS]... Usage: python -m laconic-so [OPTIONS] COMMAND [ARGS]...
Laconic Stack Orchestrator Laconic Stack Orchestrator
@ -57,162 +50,56 @@ User mode runs the orchestrator from a "binary" single-file release and does not
Commands: Commands:
build-containers build the set of containers required for a complete... build-containers build the set of containers required for a complete...
build-npms build the set of npm packages required for a...
deploy-system deploy a stack deploy-system deploy a stack
setup-repositories git clone the set of repositories required to build... setup-repositories git clone the set of repositories required to build...
``` ```
### Developer mode Install
Suitable for developers either modifying or debugging the orchestrator Python code:
#### Prerequisites
In addition to the binary install prerequisites listed above, the following are required:
1. Python venv package
This may or may not be already installed depending on the host OS and version. Check by running:
```
$ python3 -m venv
usage: venv [-h] [--system-site-packages] [--symlinks | --copies] [--clear] [--upgrade] [--without-pip] [--prompt PROMPT] ENV_DIR [ENV_DIR ...]
venv: error: the following arguments are required: ENV_DIR
```
If the venv package is missing you should see a message indicating how to install it, for example with:
```
$ apt install python3.8-venv
```
#### Install
1. Clone this repository:
```
$ git clone (https://github.com/cerc-io/stack-orchestrator.git
```
4. Enter the project directory:
```
$ cd stack-orchestrator
```
5. Create and activate a venv:
```
$ python3 -m venv venv
$ source ./venv/bin/activate
(venv) $
```
6. Install the cli in edit mode:
```
$ pip install --editable .
```
7. Verify installation:
```
(venv) $ laconic-so
Usage: laconic-so [OPTIONS] COMMAND [ARGS]...
Laconic Stack Orchestrator
Options:
--quiet
--verbose
--dry-run
-h, --help Show this message and exit.
Commands:
build-containers build the set of containers required for a complete...
deploy-system deploy a stack
setup-repositories git clone the set of repositories required to build...
```
#### Build a zipapp (single file distributable script)
Use shiv to build a single file Python executable zip archive of laconic-so:
1. Install [shiv](https://github.com/linkedin/shiv):
```
$ (venv) pip install shiv
$ (venv) pip install wheel
```
1. Run shiv to create a zipapp file:
```
$ (venv) shiv -c laconic-so -o laconic-so .
```
This creates a file `./laconic-so` that is executable outside of any venv, and on other machines and OSes and architectures, and requiring only the system Python3:
1. Verify it works:
```
$ cp stack-orchetrator/laconic-so ~/bin
$ laconic-so
Usage: python -m laconic-so [OPTIONS] COMMAND [ARGS]...
Laconic Stack Orchestrator
Options:
--quiet
--verbose
--dry-run
-h, --help Show this message and exit.
Commands:
build-containers build the set of containers required for a complete...
deploy-system deploy a stack
setup-repositories git clone the set of repositories required to build...
```
### CI Mode
_write-me_
## Usage ## Usage
There are three sub-commands: `setup-repositories`, `build-containers` and `deploy-system` that are generally run in order:
Note: $ laconic-so will run the version installed to ~/bin, while ./laconic-so can be invoked to run locally built Three sub-commands: `setup-repositories`, `build-containers` and `deploy-system` are generally run in order. The following is a slim example for standing up the `erc20-watcher`. Go further with the [erc20 watcher demo](/app/data/stacks/erc20) and other pieces of the stack, within the [`stacks` directory](/app/data/stacks).
version in a checkout
### Setup Repositories ### Setup Repositories
Clones the set of git repositories necessary to build a system.
Note: the use of `ssh-agent` is recommended in order to avoid entering your ssh key passphrase for each repository. Clone the set of git repositories necessary to build a system:
```
$ laconic-so --verbose setup-repositories #this will default to ~/cerc or CERC_REPO_BASE_DIR from an env file ```bash
#$ ./laconic-so --verbose --local-stack setup-repositories #this will use cwd ../ as dev_root_path laconic-so --verbose setup-repositories --include cerc-io/go-ethereum,cerc-io/ipld-eth-db,cerc-io/ipld-eth-server,cerc-io/watcher-ts
``` ```
This will default to `~/cerc` or - if set - the environment variable `CERC_REPO_BASE_DIR`
### Build Containers ### Build Containers
Builds the set of docker container images required to run a system. It takes around 10 minutes to build all the containers from cold.
```
$ laconic-so --verbose build-containers #this will default to ~/cerc or CERC_REPO_BASE_DIR from an env file
#$ ./laconic-so --verbose --local-stack build-containers #this will use cwd ../ as dev_root_path
Build the set of docker container images required to run a system. It takes around 10 minutes to build all the containers from scratch.
```bash
laconic-so --verbose build-containers --include cerc/go-ethereum,cerc/go-ethereum-foundry,cerc/ipld-eth-db,cerc/ipld-eth-server,cerc/watcher-erc20
``` ```
### Deploy System ### Deploy System
Uses `docker compose` to deploy a system.
Use `---include <list of components>` to deploy a subset of all containers: Uses `docker-compose` to deploy a system (with most recently built container images).
```bash
laconic-so --verbose deploy-system --include ipld-eth-db,go-ethereum-foundry,ipld-eth-server,watcher-erc20 up
``` ```
$ laconic-so --verbose deploy-system --include db-sharding,contract,ipld-eth-server,go-ethereum-foundry up
Check out he GraphQL playground here: [http://localhost:3002/graphql](http://localhost:3002/graphql)
See the [erc20 watcher demo](/app/data/stacks/erc20) to continue further.
### Cleanup
```bash
laconic-so --verbose deploy-system --include ipld-eth-db,go-ethereum-foundry,ipld-eth-server,watcher-erc20 down
``` ```
```
$ laconic-so --verbose deploy-system --include db-sharding,contract,ipld-eth-server,go-ethereum-foundry down ## Contributing
```
Note: deploy-system command interacts with most recently built container images. See the [CONTRIBUTING.md](.github/CONTRIBUTING.md) for developer mode install.
## Platform Support ## Platform Support
Native aarm64 is _not_ currently supported. x64 emulation on ARM64 macos should work (not yet tested).
## Implementation
The orchestrator's operation is driven by files shown below. `repository-list.txt` container the list of git repositories; `container-image-list.txt` contains
the list of container image names, while `clister-list.txt` specifies the set of compose components (corresponding to individual docker-compose-xxx.yml files which may in turn specify more than one container).
Files required to build each container image are stored under `./container-build/<container-name>`
Files required at deploy-time are stored under `./config/<component-name>`
```
├── pod-list.txt
├── compose
│   ├── docker-compose-contract.yml
│   ├── docker-compose-db-sharding.yml
│   ├── docker-compose-db.yml
│   ├── docker-compose-eth-statediff-fill-service.yml
│   ├── docker-compose-go-ethereum-foundry.yml
│   ├── docker-compose-ipld-eth-beacon-db.yml
│   ├── docker-compose-ipld-eth-beacon-indexer.yml
│   ├── docker-compose-ipld-eth-server.yml
│   ├── docker-compose-lighthouse.yml
│   └── docker-compose-prometheus-grafana.yml
├── config
│   └── ipld-eth-server
├── container-build
│   ├── cerc-eth-statediff-fill-service
│   ├── cerc-go-ethereum
│   ├── cerc-go-ethereum-foundry
│   ├── cerc-ipld-eth-beacon-db
│   ├── cerc-ipld-eth-beacon-indexer
│   ├── cerc-ipld-eth-db
│   ├── cerc-ipld-eth-server
│   ├── cerc-lighthouse
│   └── cerc-test-contract
├── container-image-list.txt
├── repository-list.txt
```
_write-more-of-me_ Native aarm64 is _not_ currently supported. x64 emulation on ARM64 macos should work (not yet tested).

66
app/data/stacks/erc20/README.md
View File

@ -1,74 +1,74 @@
# ERC20 Watcher # ERC20 Watcher
Instructions to deploy a local ERC20 watcher stack (core + watcher) for demonstration and testing purposes using [laconic-stack-orchestrator](../../README.md#setup) Instructions to deploy a local ERC20 watcher stack (core + watcher) for demonstration and testing purposes using [stack orchestrator](/README.md#install)
## Setup ## Setup
* Clone / pull required repositories: Clone required repositories:
```bash ```bash
$ laconic-so setup-repositories --include cerc-io/go-ethereum,cerc-io/ipld-eth-db,cerc-io/ipld-eth-server,cerc-io/watcher-ts --pull laconic-so setup-repositories --include cerc-io/go-ethereum,cerc-io/ipld-eth-db,cerc-io/ipld-eth-server,cerc-io/watcher-ts
``` ```
* Build the core and watcher container images: Build the core and watcher container images:
```bash ```bash
$ laconic-so build-containers --include cerc/go-ethereum,cerc/go-ethereum-foundry,cerc/ipld-eth-db,cerc/ipld-eth-server,cerc/watcher-erc20 laconic-so build-containers --include cerc/go-ethereum,cerc/go-ethereum-foundry,cerc/ipld-eth-db,cerc/ipld-eth-server,cerc/watcher-erc20
``` ```
This should create the required docker images in the local image registry. This should create the required docker images in the local image registry.
* Deploy the stack: Deploy the stack:
```bash ```bash
$ laconic-so deploy-system --include db,go-ethereum-foundry,ipld-eth-server,watcher-erc20 up laconic-so deploy-system --include ipld-eth-db,go-ethereum-foundry,ipld-eth-server,watcher-erc20 up
``` ```
## Demo ## Demo
* Find the watcher container's id using `docker ps` and export it for later use: Find the watcher container's id using `docker ps` and export it for later use:
```bash ```bash
$ export CONTAINER_ID=<CONTAINER_ID> export CONTAINER_ID=<CONTAINER_ID>
``` ```
* Deploy an ERC20 token: Deploy an ERC20 token:
```bash ```bash
$ docker exec $CONTAINER_ID yarn token:deploy:docker docker exec $CONTAINER_ID yarn token:deploy:docker
``` ```
Export the address of the deployed token to a shell variable for later use: Export the address of the deployed token to a shell variable for later use:
```bash ```bash
$ export TOKEN_ADDRESS=<TOKEN_ADDRESS> export TOKEN_ADDRESS=<TOKEN_ADDRESS>
``` ```
* Open `http://localhost:3002/graphql` (GraphQL Playground) in a browser window Open `http://localhost:3002/graphql` (GraphQL Playground) in a browser window
* Connect MetaMask to `http://localhost:8545` (with chain ID `99`) Connect MetaMask to `http://localhost:8545` (with chain ID `99`)
* Add the deployed token as an asset in MetaMask and check that the initial balance is zero Add the deployed token as an asset in MetaMask and check that the initial balance is zero
* Export your MetaMask account (second account) address to a shell variable for later use: Export your MetaMask account (second account) address to a shell variable for later use:
```bash ```bash
$ export RECIPIENT_ADDRESS=<RECIPIENT_ADDRESS> export RECIPIENT_ADDRESS=<RECIPIENT_ADDRESS>
``` ```
* To get the primary account's address, run: To get the primary account's address, run:
```bash ```bash
$ docker exec $CONTAINER_ID yarn account:docker docker exec $CONTAINER_ID yarn account:docker
``` ```
* To get the current block hash at any time, run: To get the current block hash at any time, run:
```bash ```bash
$ docker exec $CONTAINER_ID yarn block:latest:docker docker exec $CONTAINER_ID yarn block:latest:docker
``` ```
* Fire a GQL query in the playground to get the name, symbol and total supply of the deployed token: Fire a GQL query in the playground to get the name, symbol and total supply of the deployed token:
```graphql ```graphql
query { query {
@ -104,7 +104,7 @@ Instructions to deploy a local ERC20 watcher stack (core + watcher) for demonstr
} }
``` ```
* Fire the following query to get balances for the primary and the recipient account at the latest block hash: Fire the following query to get balances for the primary and the recipient account at the latest block hash:
```graphql ```graphql
query { query {
@ -132,26 +132,26 @@ Instructions to deploy a local ERC20 watcher stack (core + watcher) for demonstr
} }
``` ```
* The initial balance for the primary account should be `1000000000000000000000` - The initial balance for the primary account should be `1000000000000000000000`
* The initial balance for the recipient should be `0` - The initial balance for the recipient should be `0`
* Transfer tokens to the recipient account: Transfer tokens to the recipient account:
```bash ```bash
$ docker exec $CONTAINER_ID yarn token:transfer:docker --token $TOKEN_ADDRESS --to $RECIPIENT_ADDRESS --amount 100 docker exec $CONTAINER_ID yarn token:transfer:docker --token $TOKEN_ADDRESS --to $RECIPIENT_ADDRESS --amount 100
``` ```
* Fire the above GQL query again with the latest block hash to get updated balances for the primary (`from`) and the recipient (`to`) account: Fire the above GQL query again with the latest block hash to get updated balances for the primary (`from`) and the recipient (`to`) account:
* The balance for the primary account should be reduced by the transfer amount (`100`) - The balance for the primary account should be reduced by the transfer amount (`100`)
* The balance for the recipient account should be equal to the transfer amount (`100`) - The balance for the recipient account should be equal to the transfer amount (`100`)
* Transfer funds between different accounts using MetaMask and use the playground to query the balance before and after the transfer. Transfer funds between different accounts using MetaMask and use the playground to query the balance before and after the transfer.
## Clean up ## Clean up
* To stop all the services running in background run: To stop all the services running in background run:
```bash ```bash
$ laconic-so deploy-system --include db,go-ethereum-foundry,ipld-eth-server,watcher-erc20 down laconic-so deploy-system --include ipld-eth-db,go-ethereum-foundry,ipld-eth-server,watcher-erc20 down
``` ```

3
docs/cli.md Normal file
View File

@ -0,0 +1,3 @@
# laconic-so
Sub-commands and flags

BIN
docs/images/laconic-stack.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 34 MiB

37
docs/spec.md Normal file
View File

@ -0,0 +1,37 @@
# Spec
TODO: update
## Implementation
The orchestrator's operation is driven by files shown below. `repository-list.txt` container the list of git repositories; `container-image-list.txt` contains
the list of container image names, while `clister-list.txt` specifies the set of compose components (corresponding to individual docker-compose-xxx.yml files which may in turn specify more than one container).
Files required to build each container image are stored under `./container-build/<container-name>`
Files required at deploy-time are stored under `./config/<component-name>`
```
├── pod-list.txt
├── compose
│   ├── docker-compose-contract.yml
│   ├── docker-compose-db-sharding.yml
│   ├── docker-compose-db.yml
│   ├── docker-compose-eth-statediff-fill-service.yml
│   ├── docker-compose-go-ethereum-foundry.yml
│   ├── docker-compose-ipld-eth-beacon-db.yml
│   ├── docker-compose-ipld-eth-beacon-indexer.yml
│   ├── docker-compose-ipld-eth-server.yml
│   ├── docker-compose-lighthouse.yml
│   └── docker-compose-prometheus-grafana.yml
├── config
│   └── ipld-eth-server
├── container-build
│   ├── cerc-eth-statediff-fill-service
│   ├── cerc-go-ethereum
│   ├── cerc-go-ethereum-foundry
│   ├── cerc-ipld-eth-beacon-db
│   ├── cerc-ipld-eth-beacon-indexer
│   ├── cerc-ipld-eth-db
│   ├── cerc-ipld-eth-server
│   ├── cerc-lighthouse
│   └── cerc-test-contract
├── container-image-list.txt
├── repository-list.txt
```