Compare commits

..
Author SHA1 Message Date
telackey d4c4cc1ed0 Add comments.
Former-commit-id: 46f00e1d485661b6328b8ca6911a81455a346f6a
2023-01-30 11:24:56 -06:00
telackey 2e5eee0223 Add comments.
Former-commit-id: a3a60b76959056ba9cc40f0de3a8bd9a2ee2e17e
2023-01-30 11:23:45 -06:00
telackey c0455e3e0b Simplify build.
Former-commit-id: 3a80606d6c20751fa12c911fc965380b49086710
2023-01-30 11:20:23 -06:00
telackey cecc7a1274 EOL
Former-commit-id: 5f09e42422e5c458919925de751be93e028d05ba
2023-01-30 11:09:05 -06:00
telackey 97592b0fca Remove unneeded env
Former-commit-id: a3880de4cde7b5fdd43cd953a7c4b198cc445573
2023-01-30 11:08:27 -06:00
telackey da3636bc17 Fix typo
Former-commit-id: 34413ea2607ead60df6905992b29808828d60448
2023-01-28 00:54:48 -06:00
telackey 22077dedc8 Remove generated files.
Former-commit-id: d0c506af881fee53f174f6d8380c1e2b36cdc3d6
2023-01-28 00:51:11 -06:00
telackey ce43e3061c Add a pod for generating geth data as quickly as possible.
Former-commit-id: e8bdb205c7faa89c583b0652fac1801d0177a3c1
2023-01-28 00:49:46 -06:00
77 changed files with 316 additions and 1444 deletions

No files matched your search

+29 -17
View File
@@ -8,11 +8,11 @@ Stack Orchestrator allows building and deployment of a Laconic Stack on a single
Ensure that the following are already installed: Ensure that the following are already installed:
- [Python3](https://wiki.python.org/moin/BeginnersGuide/Download): `python3 --version` >= `3.8.10` (the Python3 shipped in Ubuntu 20+ is good to go) - [Python3](https://wiki.python.org/moin/BeginnersGuide/Download): `python3 --version` >= `3.10.8`
- [Docker](https://docs.docker.com/get-docker/): `docker --version` >= `20.10.21` - [Docker](https://docs.docker.com/get-docker/): `docker --version` >= `20.10.21`
- [jq](https://stedolan.github.io/jq/download/): `jq --version` >= `1.5` - [Docker Compose](https://docs.docker.com/compose/install/): `docker-compose --version` >= `2.13.0`
Note: if installing docker-compose via package manager on Linux (as opposed to Docker Desktop), you must [install the plugin](https://docs.docker.com/compose/install/linux/#install-the-plugin-manually), e.g. : Note: if installing docker-compose via package manager (as opposed to Docker Desktop), you must [install the plugin](https://docs.docker.com/compose/install/linux/#install-the-plugin-manually), e.g., on Linux:
```bash ```bash
mkdir -p ~/.docker/cli-plugins mkdir -p ~/.docker/cli-plugins
@@ -20,27 +20,39 @@ curl -SL https://github.com/docker/compose/releases/download/v2.11.2/docker-comp
chmod +x ~/.docker/cli-plugins/docker-compose chmod +x ~/.docker/cli-plugins/docker-compose
``` ```
Next decide on a directory where you would like to put the stack-orchestrator program. Typically this would be Next, download the latest release from [this page](https://github.com/cerc-io/stack-orchestrator/tags), into a suitable directory (e.g. `~/bin`):
a "user" binary directory such as `~/bin` or perhaps `/usr/local/laconic` or possibly just the current working directory.
Now, having selected that directory, download the latest release from [this page](https://github.com/cerc-io/stack-orchestrator/tags) into it (we're using `~/bin` below for concreteness but edit to suit if you selected a different directory). Also be sure that the destination directory exists and is writable:
```bash ```bash
curl -L -o ~/bin/laconic-so https://github.com/cerc-io/stack-orchestrator/releases/latest/download/laconic-so curl -L -o ~/bin/laconic-so https://github.com/cerc-io/stack-orchestrator/releases/latest/download/laconic-so
``` ```
Give it execute permissions: Give it permissions:
```bash ```bash
chmod +x ~/bin/laconic-so chmod +x ~/bin/laconic-so
``` ```
Ensure `laconic-so` is on the [`PATH`](https://unix.stackexchange.com/a/26059) Ensure `laconic-so` is on the [`PATH`](https://unix.stackexchange.com/a/26059)
Verify operation (your version will probably be different, just check here that you see some version outut and not an error): Verify operation:
``` ```
laconic-so version laconic-so --help
Version: v1.0.27-7831078 Usage: python -m laconic-so [OPTIONS] COMMAND [ARGS]...
Laconic Stack Orchestrator
Options:
--quiet
--verbose
--dry-run
--local-stack
-h, --help Show this message and exit.
Commands:
build-containers build the set of containers required for a complete...
build-npms build the set of npm packages required for a...
deploy-system deploy a stack
setup-repositories git clone the set of repositories required to build...
``` ```
## Usage ## Usage
@@ -52,25 +64,25 @@ Three sub-commands: `setup-repositories`, `build-containers` and `deploy-system`
Clone the set of git repositories necessary to build a system: Clone the set of git repositories necessary to build a system:
```bash ```bash
laconic-so --stack erc20 setup-repositories laconic-so --verbose setup-repositories --include cerc-io/go-ethereum,cerc-io/ipld-eth-db,cerc-io/ipld-eth-server,cerc-io/watcher-ts
``` ```
This will default to cloning git reposiories into: `~/cerc` or - if set - the environment variable `CERC_REPO_BASE_DIR` This will default to `~/cerc` or - if set - the environment variable `CERC_REPO_BASE_DIR`
### Build Containers ### Build Containers
Build the set of docker container images required to run a system. It takes around 10 minutes to build all the containers from scratch. Build the set of docker container images required to run a system. It takes around 10 minutes to build all the containers from scratch.
```bash ```bash
laconic-so --stack erc20 build-containers laconic-so --verbose build-containers --include cerc/go-ethereum,cerc/go-ethereum-foundry,cerc/ipld-eth-db,cerc/ipld-eth-server,cerc/watcher-erc20
``` ```
### Deploy System ### Deploy System
Uses `docker compose` to deploy a system (with most recently built container images). Uses `docker-compose` to deploy a system (with most recently built container images).
```bash ```bash
laconic-so --stack erc20 deploy-system up laconic-so --verbose deploy-system --include ipld-eth-db,go-ethereum-foundry,ipld-eth-server,watcher-erc20 up
``` ```
Check out he GraphQL playground here: [http://localhost:3002/graphql](http://localhost:3002/graphql) Check out he GraphQL playground here: [http://localhost:3002/graphql](http://localhost:3002/graphql)
@@ -80,7 +92,7 @@ See the [erc20 watcher demo](/app/data/stacks/erc20) to continue further.
### Cleanup ### Cleanup
```bash ```bash
laconic-so --stack erc20 deploy-system down laconic-so --verbose deploy-system --include ipld-eth-db,go-ethereum-foundry,ipld-eth-server,watcher-erc20 down
``` ```
## Contributing ## Contributing
-71
View File
@@ -1,71 +0,0 @@
# Copyright © 2022, 2023 Cerc
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU Affero General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU Affero General Public License for more details.
# You should have received a copy of the GNU Affero General Public License
# along with this program. If not, see <http:#www.gnu.org/licenses/>.
import os
from abc import ABC, abstractmethod
from .deploy_system import get_stack_status
def get_stack(config, stack):
if stack == "package-registry":
return package_registry_stack(config, stack)
else:
return base_stack(config, stack)
class base_stack(ABC):
def __init__(self, config, stack):
self.config = config
self.stack = stack
@abstractmethod
def ensure_available(self):
pass
@abstractmethod
def get_url(self):
pass
class package_registry_stack(base_stack):
def ensure_available(self):
self.url = "<no registry url set>"
# Check if we were given an external registry URL
url_from_environment = os.environ.get("CERC_NPM_REGISTRY_URL")
if url_from_environment:
if self.config.verbose:
print(f"Using package registry url from CERC_NPM_REGISTRY_URL: {url_from_environment}")
self.url = url_from_environment
else:
# Otherwise we expect to use the local package-registry stack
# First check if the stack is up
registry_running = get_stack_status(self.config, "package-registry")
if registry_running:
# If it is available, get its mapped port and construct its URL
if self.config.debug:
print("Found local package registry stack is up")
# TODO: get url from deploy-stack
self.url = "http://gitea.local:3000/api/packages/cerc-io/npm/"
else:
# If not, print a message about how to start it and return fail to the caller
print("ERROR: The package-registry stack is not running, and no external registry specified with CERC_NPM_REGISTRY_URL")
print("ERROR: Start the local package registry with: laconic-so --stack package-registry deploy-system up")
return False
return True
def get_url(self):
return self.url
+3 -6
View File
@@ -83,10 +83,7 @@ def command(ctx, include, exclude):
container_build_env = { container_build_env = {
"CERC_NPM_URL": "http://gitea.local:3000/api/packages/cerc-io/npm/", "CERC_NPM_URL": "http://gitea.local:3000/api/packages/cerc-io/npm/",
"CERC_NPM_AUTH_TOKEN": config("CERC_NPM_AUTH_TOKEN", default="<token-not-supplied>"), "CERC_NPM_AUTH_TOKEN": config("CERC_NPM_AUTH_TOKEN", default="<token-not-supplied>"),
"CERC_REPO_BASE_DIR": dev_root_path, "CERC_REPO_BASE_DIR": dev_root_path
"CERC_HOST_UID": f"{os.getuid()}",
"CERC_HOST_GID": f"{os.getgid()}",
"DOCKER_BUILDKIT": "0"
} }
def process_container(container): def process_container(container):
@@ -106,10 +103,10 @@ def command(ctx, include, exclude):
# Check if we have a repo for this container. If not, set the context dir to the container-build subdir # Check if we have a repo for this container. If not, set the context dir to the container-build subdir
repo_full_path = os.path.join(dev_root_path, repo_dir) repo_full_path = os.path.join(dev_root_path, repo_dir)
repo_dir_or_build_dir = repo_dir if os.path.exists(repo_full_path) else build_dir repo_dir_or_build_dir = repo_dir if os.path.exists(repo_full_path) else build_dir
build_command = os.path.join(container_build_dir, "default-build.sh") + f" {container}:local {repo_dir_or_build_dir}" build_command = os.path.join(container_build_dir, "default-build.sh") + f" {container} {repo_dir_or_build_dir}"
if not dry_run: if not dry_run:
if verbose: if verbose:
print(f"Executing: {build_command} with environment: {container_build_env}") print(f"Executing: {build_command}")
build_result = subprocess.run(build_command, shell=True, env=container_build_env) build_result = subprocess.run(build_command, shell=True, env=container_build_env)
if verbose: if verbose:
print(f"Return code is: {build_result.returncode}") print(f"Return code is: {build_result.returncode}")
+6 -58
View File
@@ -20,16 +20,12 @@
import os import os
import sys import sys
from shutil import rmtree, copytree
from decouple import config from decouple import config
import click import click
import importlib.resources import importlib.resources
from python_on_whales import docker, DockerException from python_on_whales import docker, DockerException
from .base import get_stack
from .util import include_exclude_check, get_parsed_stack_config from .util import include_exclude_check, get_parsed_stack_config
builder_js_image_name = "cerc/builder-js:local"
@click.command() @click.command()
@click.option('--include', help="only build these packages") @click.option('--include', help="only build these packages")
@click.option('--exclude', help="don\'t build these packages") @click.option('--exclude', help="don\'t build these packages")
@@ -45,38 +41,17 @@ def command(ctx, include, exclude):
stack = ctx.obj.stack stack = ctx.obj.stack
continue_on_error = ctx.obj.continue_on_error continue_on_error = ctx.obj.continue_on_error
_ensure_prerequisites()
# build-npms depends on having access to a writable package registry
# so we check here that it is available
package_registry_stack = get_stack(ctx.obj, "package-registry")
registry_available = package_registry_stack.ensure_available()
if not registry_available:
print("FATAL: no npm registry available for build-npms command")
sys.exit(1)
npm_registry_url = package_registry_stack.get_url()
npm_registry_url_token = config("CERC_NPM_AUTH_TOKEN", default=None)
if not npm_registry_url_token:
print("FATAL: CERC_NPM_AUTH_TOKEN is not defined")
sys.exit(1)
if local_stack: if local_stack:
dev_root_path = os.getcwd()[0:os.getcwd().rindex("stack-orchestrator")] dev_root_path = os.getcwd()[0:os.getcwd().rindex("stack-orchestrator")]
print(f'Local stack dev_root_path (CERC_REPO_BASE_DIR) overridden to: {dev_root_path}') print(f'Local stack dev_root_path (CERC_REPO_BASE_DIR) overridden to: {dev_root_path}')
else: else:
dev_root_path = os.path.expanduser(config("CERC_REPO_BASE_DIR", default="~/cerc")) dev_root_path = os.path.expanduser(config("CERC_REPO_BASE_DIR", default="~/cerc"))
build_root_path = os.path.join(dev_root_path, "build-trees") if not quiet:
if verbose:
print(f'Dev Root is: {dev_root_path}') print(f'Dev Root is: {dev_root_path}')
if not os.path.isdir(dev_root_path): if not os.path.isdir(dev_root_path):
print('Dev root directory doesn\'t exist, creating') print('Dev root directory doesn\'t exist, creating')
os.makedirs(dev_root_path)
if not os.path.isdir(dev_root_path):
print('Build root directory doesn\'t exist, creating')
os.makedirs(build_root_path)
# See: https://stackoverflow.com/a/20885799/1701505 # See: https://stackoverflow.com/a/20885799/1701505
from . import data from . import data
@@ -99,38 +74,21 @@ def command(ctx, include, exclude):
print(f"Building npm package: {package}") print(f"Building npm package: {package}")
repo_dir = package repo_dir = package
repo_full_path = os.path.join(dev_root_path, repo_dir) repo_full_path = os.path.join(dev_root_path, repo_dir)
# Copy the repo and build that to avoid propagating JS tooling file changes back into the cloned repo # TODO: make the npm registry url configurable.
repo_copy_path = os.path.join(build_root_path, repo_dir) build_command = ["sh", "-c", "cd /workspace && build-npm-package-local-dependencies.sh http://gitea.local:3000/api/packages/cerc-io/npm/"]
# First delete any old build tree
if os.path.isdir(repo_copy_path):
if verbose:
print(f"Deleting old build tree: {repo_copy_path}")
if not dry_run:
rmtree(repo_copy_path)
# Now copy the repo into the build tree location
if verbose:
print(f"Copying build tree from: {repo_full_path} to: {repo_copy_path}")
if not dry_run:
copytree(repo_full_path, repo_copy_path)
build_command = ["sh", "-c", f"cd /workspace && build-npm-package-local-dependencies.sh {npm_registry_url}"]
if not dry_run: if not dry_run:
if verbose: if verbose:
print(f"Executing: {build_command}") print(f"Executing: {build_command}")
# Originally we used the PEP 584 merge operator: envs = {"CERC_NPM_AUTH_TOKEN": os.environ["CERC_NPM_AUTH_TOKEN"]} | ({"CERC_SCRIPT_DEBUG": "true"} if debug else {})
# envs = {"CERC_NPM_AUTH_TOKEN": npm_registry_url_token} | ({"CERC_SCRIPT_DEBUG": "true"} if debug else {})
# but that isn't available in Python 3.8 (default in Ubuntu 20) so for now we use dict.update:
envs = {"CERC_NPM_AUTH_TOKEN": npm_registry_url_token}
envs.update({"CERC_SCRIPT_DEBUG": "true"} if debug else {})
try: try:
docker.run(builder_js_image_name, docker.run("cerc/builder-js",
remove=True, remove=True,
interactive=True, interactive=True,
tty=True, tty=True,
user=f"{os.getuid()}:{os.getgid()}", user=f"{os.getuid()}:{os.getgid()}",
envs=envs, envs=envs,
# TODO: detect this host name in npm_registry_url rather than hard-wiring it
add_hosts=[("gitea.local", "host-gateway")], add_hosts=[("gitea.local", "host-gateway")],
volumes=[(repo_copy_path, "/workspace")], volumes=[(repo_full_path, "/workspace")],
command=build_command command=build_command
) )
# Note that although the docs say that build_result should contain # Note that although the docs say that build_result should contain
@@ -153,13 +111,3 @@ def command(ctx, include, exclude):
else: else:
if verbose: if verbose:
print(f"Excluding: {package}") print(f"Excluding: {package}")
def _ensure_prerequisites():
# Check that the builder-js container is available and
# Tell the user how to build it if not
images = docker.image.list(builder_js_image_name)
if len(images) == 0:
print(f"FATAL: builder image: {builder_js_image_name} is required but was not found")
print("Please run this command to create it: laconic-so --stack build-support build-containers")
sys.exit(1)
@@ -0,0 +1,67 @@
version: '3.7'
services:
datanet-eth-bootnode-geth:
hostname: datanet-eth-bootnode-geth
env_file:
- ../config/datanet-eth/datanet-eth.env
environment:
RUN_BOOTNODE: "true"
image: cerc/datanet-eth-geth:local
ports:
- "9898"
- "30303"
datanet-eth-geth-1:
hostname: datanet-eth-geth-1
cap_add:
- SYS_PTRACE
environment:
CERC_REMOTE_DEBUG: "false"
CERC_RUN_STATEDIFF: "false"
CERC_STATEDIFF_DB_NODE_ID: 1
env_file:
- ../config/datanet-eth/datanet-eth.env
image: cerc/datanet-eth-geth:local
healthcheck:
test: ["CMD", "nc", "-v", "localhost", "8545"]
interval: 30s
timeout: 10s
retries: 10
start_period: 3s
depends_on:
- datanet-eth-bootnode-geth
ports:
- "8545"
- "40000"
- "6060"
datanet-eth-geth-2:
hostname: datanet-eth-geth-2
healthcheck:
test: ["CMD", "nc", "-v", "localhost", "8545"]
interval: 30s
timeout: 10s
retries: 10
start_period: 3s
env_file:
- ../config/datanet-eth/datanet-eth.env
image: cerc/datanet-eth-geth:local
depends_on:
- datanet-eth-bootnode-geth
datanet-tx-spammer:
restart: always
image: cerc/tx-spammer:local
environment:
ACCOUNTS_CSV_URL: http://datanet-eth-bootnode-geth:9898/accounts.csv
ETH_HTTP_PATH: http://datanet-eth-geth-1:8545
LOG_LEVEL: info
SPAMMER_COMMAND: autoSend
ETH_CALL_FREQ: 0
ETH_SEND_FREQ: 0
depends_on:
datanet-eth-bootnode-geth:
condition: service_started
datanet-eth-geth-1:
condition: service_healthy
@@ -13,8 +13,6 @@ services:
grafana: grafana:
restart: always restart: always
image: grafana/grafana image: grafana/grafana
environment:
- GF_SECURITY_ADMIN_PASSWORD=changeme6325
volumes: volumes:
- ../config/fixturenet-eth-metrics/grafana/etc/provisioning/dashboards:/etc/grafana/provisioning/dashboards - ../config/fixturenet-eth-metrics/grafana/etc/provisioning/dashboards:/etc/grafana/provisioning/dashboards
- ../config/fixturenet-eth-metrics/grafana/etc/provisioning/datasources:/etc/grafana/provisioning/datasources - ../config/fixturenet-eth-metrics/grafana/etc/provisioning/datasources:/etc/grafana/provisioning/datasources
@@ -5,9 +5,8 @@ services:
image: cerc/laconicd:local image: cerc/laconicd:local
command: ["sh", "/docker-entrypoint-scripts.d/create-fixturenet.sh"] command: ["sh", "/docker-entrypoint-scripts.d/create-fixturenet.sh"]
volumes: volumes:
# TODO: look at folding these scripts into the container # TODO: look at folding this script into the container
- ../config/fixturenet-laconicd/create-fixturenet.sh:/docker-entrypoint-scripts.d/create-fixturenet.sh - ../config/fixturenet-laconicd/create-fixturenet.sh:/docker-entrypoint-scripts.d/create-fixturenet.sh
- ../config/fixturenet-laconicd/export-mykey.sh:/docker-entrypoint-scripts.d/export-mykey.sh
# TODO: determine which of the ports below is really needed # TODO: determine which of the ports below is really needed
ports: ports:
- "6060" - "6060"
@@ -19,7 +18,4 @@ services:
- "9090" - "9090"
- "9091" - "9091"
- "1317" - "1317"
cli:
image: cerc/laconic-registry-cli:local
volumes:
- ../config/fixturenet-laconicd/registry-cli-config-template.yml:/registry-cli-config-template.yml
@@ -1,8 +0,0 @@
# Add-on pod to include foundry tooling within a fixturenet
services:
foundry:
image: cerc/foundry:local
command: ["while :; do sleep 600; done"]
volumes:
- ../config/foundry/foundry.toml:/foundry.toml
- ./foundry/workspace:/workspace
-13
View File
@@ -1,13 +0,0 @@
version: "3.2"
# See: https://docs.ipfs.tech/install/run-ipfs-inside-docker/#set-up
services:
ipfs:
image: ipfs/kubo:master-2023-02-20-714a968
restart: always
volumes:
- ./ipfs/import:/import
- ./ipfs/data:/data/ipfs
ports:
- "8080"
- "4001"
- "5001"
@@ -1,25 +0,0 @@
version: "3.2"
services:
laconicd:
restart: unless-stopped
image: cerc/laconicd:local
command: ["sh", "/docker-entrypoint-scripts.d/create-fixturenet.sh"]
volumes:
- ../config/fixturenet-laconicd/create-fixturenet.sh:/docker-entrypoint-scripts.d/create-fixturenet.sh
ports:
- "9473"
- "8545"
- "8546"
- "1317"
healthcheck:
test: ["CMD", "nc", "-v", "localhost", "8545"]
interval: 20s
timeout: 5s
retries: 15
start_period: 10s
networks:
# https://docs.docker.com/compose/networking/#configure-the-default-network
default:
name: mobymask-v2-network
@@ -1,130 +0,0 @@
version: '3.2'
services:
mobymask-watcher-db:
restart: unless-stopped
image: postgres:14-alpine
environment:
- POSTGRES_USER=vdbm
- POSTGRES_MULTIPLE_DATABASES=mobymask-watcher,mobymask-watcher-job-queue
- POSTGRES_EXTENSION=mobymask-watcher-job-queue:pgcrypto
- POSTGRES_PASSWORD=password
volumes:
- ../config/postgresql/multiple-postgressql-databases.sh:/docker-entrypoint-initdb.d/multiple-postgressql-databases.sh
- mobymask_watcher_db_data:/var/lib/postgresql/data
ports:
- "0.0.0.0:15432:5432"
healthcheck:
test: ["CMD", "nc", "-v", "localhost", "5432"]
interval: 20s
timeout: 5s
retries: 15
start_period: 10s
mobymask:
restart: unless-stopped
depends_on:
mobymask-watcher-db:
condition: service_healthy
image: cerc/mobymask:local
working_dir: /app/packages/server
environment:
- ENV=PROD
command: ["sh", "-c", "npm start"]
volumes:
- ../config/watcher-mobymask-v2/secrets.json:/app/packages/server/secrets.json
- moby_data_server:/app/packages/server
healthcheck:
test: ["CMD", "nc", "-v", "localhost", "3330"]
interval: 20s
timeout: 5s
retries: 15
start_period: 10s
mobymask-watcher-server:
restart: unless-stopped
depends_on:
mobymask-watcher-db:
condition: service_healthy
image: cerc/watcher-mobymask-v2:local
command: ["sh", "-c", "yarn server"]
volumes:
- ../config/watcher-mobymask-v2/watcher.toml:/app/packages/mobymask-v2-watcher/environments/local.toml
- ../config/watcher-mobymask-v2/peer.env:/app/packages/peer/.env
- ../config/watcher-mobymask-v2/relay-id.json:/app/packages/mobymask-v2-watcher/relay-id.json
ports:
- "0.0.0.0:3001:3001"
- "0.0.0.0:9001:9001"
- "0.0.0.0:9090:9090"
healthcheck:
test: ["CMD", "busybox", "nc", "localhost", "9090"]
interval: 20s
timeout: 5s
retries: 15
start_period: 5s
mobymask-watcher-peer:
restart: unless-stopped
depends_on:
mobymask:
condition: service_healthy
image: cerc/watcher-mobymask-v2:local
command: ["sh", "peer-start.sh"]
volumes:
- ../config/watcher-mobymask-v2/watcher.toml:/app/packages/mobymask-v2-watcher/environments/local.toml
- ../config/watcher-mobymask-v2/peer-id.json:/app/packages/mobymask-v2-watcher/peer-id.json
- ../config/watcher-mobymask-v2/peer-start.sh:/app/packages/mobymask-v2-watcher/peer-start.sh
- moby_data_server:/server
mobymask-app:
depends_on:
mobymask-watcher-server:
condition: service_healthy
mobymask:
condition: service_healthy
image: cerc/mobymask-ui:local
command: ["sh", "mobymask-app-start.sh"]
volumes:
- ../config/watcher-mobymask-v2/mobymask-app.env:/app/.env
- ../config/watcher-mobymask-v2/mobymask-app-config.json:/app/src/mobymask-app-config.json
- ../config/watcher-mobymask-v2/mobymask-app-start.sh:/app/mobymask-app-start.sh
- moby_data_server:/server
ports:
- "0.0.0.0:3002:3000"
healthcheck:
test: ["CMD", "nc", "-v", "localhost", "3000"]
interval: 20s
timeout: 5s
retries: 15
start_period: 10s
shm_size: '1GB'
peer-test-app:
depends_on:
mobymask-watcher-server:
condition: service_healthy
image: cerc/react-peer:local
working_dir: /app/packages/test-app
command: ["sh", "-c", "yarn build && serve -s build"]
volumes:
- ../config/watcher-mobymask-v2/test-app-config.json:/app/packages/test-app/src/config.json
ports:
- "0.0.0.0:3003:3000"
healthcheck:
test: ["CMD", "nc", "-v", "localhost", "3000"]
interval: 20s
timeout: 5s
retries: 15
start_period: 10s
volumes:
mobymask_watcher_db_data:
moby_data_server:
networks:
# https://docs.docker.com/compose/networking/#configure-the-default-network
default:
# https://docs.docker.com/compose/networking/#use-a-pre-existing-network
name: mobymask-v2-network
external: true
@@ -0,0 +1,9 @@
# The password used to access test accounts (eg, via personal_unlockAccount). The password is the same for all accounts.
ACCOUNT_PASSWORD=secret1212
# ENODE of the geth bootnode.
BOOTNODE_KEY="b0ac22adcad37213c7c565810a50f1772291e7b0ce53fb73e7ec2a3c75bc13b5"
ENODE="enode://af22c29c316ad069cf48a09a4ad5cf04a251b411e45098888d114c6dd7f489a13786620d5953738762afa13711d4ffb3b19aa5de772d8af72f851f7e9c5b164a@datanet-eth-bootnode-geth:30303"
# JWT is required by the startup script, but won't be used without lighthouse.
JWT="0x6cdcac3501046a08e186730dd8bd136cfaf0fdc1fc955f6e15ad3068c0ff2af0"
@@ -1,8 +1,8 @@
#!/bin/bash #!/bin/sh
# Originally from: https://github.com/cerc-io/laconicd/blob/main/init.sh
# TODO: fold this back into the laconicd repo
# TODO: this file is now an unmodified copy of cerc-io/laconicd/init.sh
# so we should have a mechanism to bundle it inside the container rather than link from here
# at deploy time.
KEY="mykey" KEY="mykey"
CHAINID="laconic_9000-1" CHAINID="laconic_9000-1"
@@ -10,7 +10,7 @@ MONIKER="localtestnet"
KEYRING="test" KEYRING="test"
KEYALGO="eth_secp256k1" KEYALGO="eth_secp256k1"
LOGLEVEL="info" LOGLEVEL="info"
# trace evm # to trace evm
TRACE="--trace" TRACE="--trace"
# TRACE="" # TRACE=""
@@ -28,7 +28,7 @@ laconicd config chain-id $CHAINID
# if $KEY exists it should be deleted # if $KEY exists it should be deleted
laconicd keys add $KEY --keyring-backend $KEYRING --algo $KEYALGO laconicd keys add $KEY --keyring-backend $KEYRING --algo $KEYALGO
# Set moniker and chain-id for Ethermint (Moniker can be anything, chain-id must be an integer) # Set moniker and chain-id for laconic (Moniker can be anything, chain-id must be an integer)
laconicd init $MONIKER --chain-id $CHAINID laconicd init $MONIKER --chain-id $CHAINID
# Change parameter token denominations to aphoton # Change parameter token denominations to aphoton
@@ -37,28 +37,28 @@ cat $HOME/.laconicd/config/genesis.json | jq '.app_state["crisis"]["constant_fee
cat $HOME/.laconicd/config/genesis.json | jq '.app_state["gov"]["deposit_params"]["min_deposit"][0]["denom"]="aphoton"' > $HOME/.laconicd/config/tmp_genesis.json && mv $HOME/.laconicd/config/tmp_genesis.json $HOME/.laconicd/config/genesis.json cat $HOME/.laconicd/config/genesis.json | jq '.app_state["gov"]["deposit_params"]["min_deposit"][0]["denom"]="aphoton"' > $HOME/.laconicd/config/tmp_genesis.json && mv $HOME/.laconicd/config/tmp_genesis.json $HOME/.laconicd/config/genesis.json
cat $HOME/.laconicd/config/genesis.json | jq '.app_state["mint"]["params"]["mint_denom"]="aphoton"' > $HOME/.laconicd/config/tmp_genesis.json && mv $HOME/.laconicd/config/tmp_genesis.json $HOME/.laconicd/config/genesis.json cat $HOME/.laconicd/config/genesis.json | jq '.app_state["mint"]["params"]["mint_denom"]="aphoton"' > $HOME/.laconicd/config/tmp_genesis.json && mv $HOME/.laconicd/config/tmp_genesis.json $HOME/.laconicd/config/genesis.json
# Custom modules # Custom modules
cat $HOME/.laconicd/config/genesis.json | jq '.app_state["registry"]["params"]["record_rent"]["denom"]="aphoton"' > $HOME/.laconicd/config/tmp_genesis.json && mv $HOME/.laconicd/config/tmp_genesis.json $HOME/.laconicd/config/genesis.json cat $HOME/.laconicd/config/genesis.json | jq '.app_state["nameservice"]["params"]["record_rent"]["denom"]="aphoton"' > $HOME/.laconicd/config/tmp_genesis.json && mv $HOME/.laconicd/config/tmp_genesis.json $HOME/.laconicd/config/genesis.json
cat $HOME/.laconicd/config/genesis.json | jq '.app_state["registry"]["params"]["authority_rent"]["denom"]="aphoton"' > $HOME/.laconicd/config/tmp_genesis.json && mv $HOME/.laconicd/config/tmp_genesis.json $HOME/.laconicd/config/genesis.json cat $HOME/.laconicd/config/genesis.json | jq '.app_state["nameservice"]["params"]["authority_rent"]["denom"]="aphoton"' > $HOME/.laconicd/config/tmp_genesis.json && mv $HOME/.laconicd/config/tmp_genesis.json $HOME/.laconicd/config/genesis.json
cat $HOME/.laconicd/config/genesis.json | jq '.app_state["registry"]["params"]["authority_auction_commit_fee"]["denom"]="aphoton"' > $HOME/.laconicd/config/tmp_genesis.json && mv $HOME/.laconicd/config/tmp_genesis.json $HOME/.laconicd/config/genesis.json cat $HOME/.laconicd/config/genesis.json | jq '.app_state["nameservice"]["params"]["authority_auction_commit_fee"]["denom"]="aphoton"' > $HOME/.laconicd/config/tmp_genesis.json && mv $HOME/.laconicd/config/tmp_genesis.json $HOME/.laconicd/config/genesis.json
cat $HOME/.laconicd/config/genesis.json | jq '.app_state["registry"]["params"]["authority_auction_reveal_fee"]["denom"]="aphoton"' > $HOME/.laconicd/config/tmp_genesis.json && mv $HOME/.laconicd/config/tmp_genesis.json $HOME/.laconicd/config/genesis.json cat $HOME/.laconicd/config/genesis.json | jq '.app_state["nameservice"]["params"]["authority_auction_reveal_fee"]["denom"]="aphoton"' > $HOME/.laconicd/config/tmp_genesis.json && mv $HOME/.laconicd/config/tmp_genesis.json $HOME/.laconicd/config/genesis.json
cat $HOME/.laconicd/config/genesis.json | jq '.app_state["registry"]["params"]["authority_auction_minimum_bid"]["denom"]="aphoton"' > $HOME/.laconicd/config/tmp_genesis.json && mv $HOME/.laconicd/config/tmp_genesis.json $HOME/.laconicd/config/genesis.json cat $HOME/.laconicd/config/genesis.json | jq '.app_state["nameservice"]["params"]["authority_auction_minimum_bid"]["denom"]="aphoton"' > $HOME/.laconicd/config/tmp_genesis.json && mv $HOME/.laconicd/config/tmp_genesis.json $HOME/.laconicd/config/genesis.json
if [[ "$TEST_REGISTRY_EXPIRY" == "true" ]]; then if [[ "$TEST_NAMESERVICE_EXPIRY" == "true" ]]; then
echo "Setting timers for expiry tests." echo "Setting timers for expiry tests."
cat $HOME/.laconicd/config/genesis.json | jq '.app_state["registry"]["params"]["record_rent_duration"]="60s"' > $HOME/.laconicd/config/tmp_genesis.json && mv $HOME/.laconicd/config/tmp_genesis.json $HOME/.laconicd/config/genesis.json cat $HOME/.laconicd/config/genesis.json | jq '.app_state["nameservice"]["params"]["record_rent_duration"]="60s"' > $HOME/.laconicd/config/tmp_genesis.json && mv $HOME/.laconicd/config/tmp_genesis.json $HOME/.laconicd/config/genesis.json
cat $HOME/.laconicd/config/genesis.json | jq '.app_state["registry"]["params"]["authority_grace_period"]="60s"' > $HOME/.laconicd/config/tmp_genesis.json && mv $HOME/.laconicd/config/tmp_genesis.json $HOME/.laconicd/config/genesis.json cat $HOME/.laconicd/config/genesis.json | jq '.app_state["nameservice"]["params"]["authority_grace_period"]="60s"' > $HOME/.laconicd/config/tmp_genesis.json && mv $HOME/.laconicd/config/tmp_genesis.json $HOME/.laconicd/config/genesis.json
cat $HOME/.laconicd/config/genesis.json | jq '.app_state["registry"]["params"]["authority_rent_duration"]="60s"' > $HOME/.laconicd/config/tmp_genesis.json && mv $HOME/.laconicd/config/tmp_genesis.json $HOME/.laconicd/config/genesis.json cat $HOME/.laconicd/config/genesis.json | jq '.app_state["nameservice"]["params"]["authority_rent_duration"]="60s"' > $HOME/.laconicd/config/tmp_genesis.json && mv $HOME/.laconicd/config/tmp_genesis.json $HOME/.laconicd/config/genesis.json
fi fi
if [[ "$TEST_AUCTION_ENABLED" == "true" ]]; then if [[ "$TEST_AUCTION_ENABLED" == "true" ]]; then
echo "Enabling auction and setting timers." echo "Enabling auction and setting timers."
cat $HOME/.laconicd/config/genesis.json | jq '.app_state["registry"]["params"]["authority_auction_enabled"]=true' > $HOME/.laconicd/config/tmp_genesis.json && mv $HOME/.laconicd/config/tmp_genesis.json $HOME/.laconicd/config/genesis.json cat $HOME/.laconicd/config/genesis.json | jq '.app_state["nameservice"]["params"]["authority_auction_enabled"]=true' > $HOME/.laconicd/config/tmp_genesis.json && mv $HOME/.laconicd/config/tmp_genesis.json $HOME/.laconicd/config/genesis.json
cat $HOME/.laconicd/config/genesis.json | jq '.app_state["registry"]["params"]["authority_rent_duration"]="60s"' > $HOME/.laconicd/config/tmp_genesis.json && mv $HOME/.laconicd/config/tmp_genesis.json $HOME/.laconicd/config/genesis.json cat $HOME/.laconicd/config/genesis.json | jq '.app_state["nameservice"]["params"]["authority_rent_duration"]="60s"' > $HOME/.laconicd/config/tmp_genesis.json && mv $HOME/.laconicd/config/tmp_genesis.json $HOME/.laconicd/config/genesis.json
cat $HOME/.laconicd/config/genesis.json | jq '.app_state["registry"]["params"]["authority_grace_period"]="300s"' > $HOME/.laconicd/config/tmp_genesis.json && mv $HOME/.laconicd/config/tmp_genesis.json $HOME/.laconicd/config/genesis.json cat $HOME/.laconicd/config/genesis.json | jq '.app_state["nameservice"]["params"]["authority_grace_period"]="300s"' > $HOME/.laconicd/config/tmp_genesis.json && mv $HOME/.laconicd/config/tmp_genesis.json $HOME/.laconicd/config/genesis.json
cat $HOME/.laconicd/config/genesis.json | jq '.app_state["registry"]["params"]["authority_auction_commits_duration"]="60s"' > $HOME/.laconicd/config/tmp_genesis.json && mv $HOME/.laconicd/config/tmp_genesis.json $HOME/.laconicd/config/genesis.json cat $HOME/.laconicd/config/genesis.json | jq '.app_state["nameservice"]["params"]["authority_auction_commits_duration"]="60s"' > $HOME/.laconicd/config/tmp_genesis.json && mv $HOME/.laconicd/config/tmp_genesis.json $HOME/.laconicd/config/genesis.json
cat $HOME/.laconicd/config/genesis.json | jq '.app_state["registry"]["params"]["authority_auction_reveals_duration"]="60s"' > $HOME/.laconicd/config/tmp_genesis.json && mv $HOME/.laconicd/config/tmp_genesis.json $HOME/.laconicd/config/genesis.json cat $HOME/.laconicd/config/genesis.json | jq '.app_state["nameservice"]["params"]["authority_auction_reveals_duration"]="60s"' > $HOME/.laconicd/config/tmp_genesis.json && mv $HOME/.laconicd/config/tmp_genesis.json $HOME/.laconicd/config/genesis.json
fi fi
# increase block time (?) # increase block time (?)
@@ -1,2 +0,0 @@
#!/bin/sh
echo y | laconicd keys export mykey --unarmored-hex --unsafe
@@ -1,7 +0,0 @@
services:
cns:
restEndpoint: 'http://laconicd:1317'
gqlEndpoint: 'http://laconicd:9473/api'
userKey: REPLACE_WITH_MYKEY
bondId:
chainId: laconic_9000-1
-2
View File
@@ -1,2 +0,0 @@
[profile.default]
eth-rpc-url = "http://fixturenet-eth-geth-1:8545"
@@ -1,8 +0,0 @@
{
"relayNodes": [
"/ip4/127.0.0.1/tcp/9090/ws/p2p/12D3KooWSPCsVkHVyLQoCqhu2YRPvvM7o6r6NRYyLM5zeA6Uig5t"
],
"peer": {
"enableDebugInfo": true
}
}
@@ -1,9 +0,0 @@
#!/bin/sh
set -e
# Merging config files to get deployed contract address
jq -s '.[0] * .[1]' /app/src/mobymask-app-config.json /server/config.json > /app/src/config.json
npm run build
serve -s build
@@ -1 +0,0 @@
REACT_APP_WATCHER_URI=http://localhost:3001/graphql
@@ -1,5 +0,0 @@
{
"id": "12D3KooWK6myjc8r1KBnfP9igp31qJkPaVfsKDjKrjoSefV5SDEo",
"privKey": "CAESQJMHbMaH+UEOtjGOzXYtoPO/cdHakCtN1hcnknIWzx/6ie1lxb+8kfzBjwt7apfj8fHlTCYSIVK8Q2AWu9a2h3g=",
"pubKey": "CAESIIntZcW/vJH8wY8Le2qX4/Hx5UwmEiFSvENgFrvWtod4"
}
@@ -1,10 +0,0 @@
#!/bin/sh
# Private key of account with balance
PRIVATE_KEY=
# Assign deployed contract address from server config
CONTRACT_ADDRESS=`jq '.address' /server/config.json`
echo 'yarn peer-listener --contract-address <CONTRACT_ADDRESS> --private-key <PRIVATE_KEY>'
yarn peer-listener --contract-address $CONTRACT_ADDRESS --private-key $PRIVATE_KEY
@@ -1 +0,0 @@
RELAY="/ip4/127.0.0.1/tcp/9090/ws/p2p/12D3KooWSPCsVkHVyLQoCqhu2YRPvvM7o6r6NRYyLM5zeA6Uig5t"
@@ -1,5 +0,0 @@
{
"id": "12D3KooWSPCsVkHVyLQoCqhu2YRPvvM7o6r6NRYyLM5zeA6Uig5t",
"privKey": "CAESQGsqG5o4VlWJZM9XlA3MjabyZOXWQ2MLZU5AhBQsjXGt9iSlGtTuNOrHX5xSRgLBxLuMoqWsjGxE/dDB9c46RI8=",
"pubKey": "CAESIPYkpRrU7jTqx1+cUkYCwcS7jKKlrIxsRP3QwfXOOkSP"
}
@@ -1,5 +0,0 @@
{
"rpcUrl": "http://laconicd:8545",
"privateKey": "GENESIS_ACCOUNT_PRIVATE_KEY",
"baseURI": "http://127.0.0.1:3002/#"
}
@@ -1,8 +0,0 @@
{
"relayNodes": [
"/ip4/127.0.0.1/tcp/9090/ws/p2p/12D3KooWSPCsVkHVyLQoCqhu2YRPvvM7o6r6NRYyLM5zeA6Uig5t"
],
"peer": {
"enableDebugInfo": true
}
}
@@ -1,70 +0,0 @@
[server]
host = "0.0.0.0"
port = 3001
kind = "lazy"
# Checkpointing state.
checkpointing = true
# Checkpoint interval in number of blocks.
checkpointInterval = 2000
# Enable state creation
enableState = true
# Boolean to filter logs by contract.
filterLogs = true
# Max block range for which to return events in eventsInRange GQL query.
# Use -1 for skipping check on block range.
maxEventsBlockRange = -1
[server.p2p]
enableRelay = true
enablePeer = false
[server.p2p.relay]
host = "0.0.0.0"
port = 9090
relayPeers = []
peerIdFile = './relay-id.json'
enableDebugInfo = true
[server.p2p.peer]
relayMultiaddr = '/dns4/mobymask-watcher-server/tcp/9090/ws/p2p/12D3KooWSPCsVkHVyLQoCqhu2YRPvvM7o6r6NRYyLM5zeA6Uig5t'
pubSubTopic = 'mobymask'
peerIdFile = './peer-id.json'
enableDebugInfo = true
[metrics]
host = "0.0.0.0"
port = 9000
[metrics.gql]
port = 9001
[database]
type = "postgres"
host = "mobymask-watcher-db"
port = 5432
database = "mobymask-watcher"
username = "vdbm"
password = "password"
synchronize = true
logging = false
[upstream]
[upstream.ethServer]
gqlApiEndpoint = "http://ipld-eth-server:8083/graphql"
rpcProviderEndpoint = "http://laconicd:8545"
blockDelayInMilliSecs = 60000
[upstream.cache]
name = "requests"
enabled = false
deleteOnStart = false
[jobQueue]
dbConnectionString = "postgres://vdbm:password@mobymask-watcher-db/mobymask-watcher-job-queue"
maxCompletionLagInSecs = 300
jobDelayInMilliSecs = 100
eventsInBatch = 50
@@ -1,3 +0,0 @@
#!/usr/bin/env bash
# Build a local version of the task executor for act-runner
docker build -t cerc/act-runner-task-executor:local -f ${CERC_REPO_BASE_DIR}/act_runner/Dockerfile.task-executor ${CERC_REPO_BASE_DIR}/act_runner
@@ -1,3 +0,0 @@
#!/usr/bin/env bash
# Build a local version of the act-runner image
docker build -t cerc/act-runner:local -f ${CERC_REPO_BASE_DIR}/act_runner/Dockerfile ${CERC_REPO_BASE_DIR}/act_runner
@@ -16,12 +16,6 @@ RUN apt-get update && export DEBIAN_FRONTEND=noninteractive && export DEBCONF_NO
RUN mkdir /scripts RUN mkdir /scripts
COPY install-dependencies.sh /scripts COPY install-dependencies.sh /scripts
# Override the definition of GERBIL_PATH in the base image, but
# is safe because (at present) no gerbil packages are installed in the base image
# We do this in order to allow a set of pre-installed packages from the container
# to be used with an arbitrary, potentially different set of projects bind mounted
# at /src
ENV GERBIL_PATH=/.gerbil
RUN bash /scripts/install-dependencies.sh RUN bash /scripts/install-dependencies.sh
# Needed to prevent git from raging about /src # Needed to prevent git from raging about /src
@@ -10,7 +10,6 @@ DEPS=(github.com/fare/gerbil-utils
github.com/vyzo/gerbil-libp2p github.com/vyzo/gerbil-libp2p
) ; ) ;
for i in ${DEPS[@]} ; do for i in ${DEPS[@]} ; do
echo "Installing gerbil package: $i"
gxpkg install $i && gxpkg install $i &&
gxpkg build $i gxpkg build $i
done done
@@ -1,30 +1,14 @@
# Originally from: https://github.com/devcontainers/images/blob/main/src/javascript-node/.devcontainer/Dockerfile # Originally from: https://github.com/devcontainers/images/blob/main/src/javascript-node/.devcontainer/Dockerfile
# Which depends on: https://github.com/nodejs/docker-node/blob/main/Dockerfile-debian.template
# [Choice] Node.js version (use -bullseye variants on local arm64/Apple Silicon): 18, 16, 14, 18-bullseye, 16-bullseye, 14-bullseye, 18-buster, 16-buster, 14-buster # [Choice] Node.js version (use -bullseye variants on local arm64/Apple Silicon): 18, 16, 14, 18-bullseye, 16-bullseye, 14-bullseye, 18-buster, 16-buster, 14-buster
ARG VARIANT=18-bullseye ARG VARIANT=16-bullseye
FROM node:${VARIANT} FROM node:${VARIANT}
# Set these args to change the uid/gid for the base container's "node" user to match that of the host user (so bind mounts work as expected).
ARG CERC_HOST_UID=1000
ARG CERC_HOST_GID=1000
# Make these values available at runtime to allow a consistency check.
ENV HOST_UID=${CERC_HOST_UID}
ENV HOST_GID=${CERC_HOST_GID}
ARG USERNAME=node ARG USERNAME=node
ARG NPM_GLOBAL=/usr/local/share/npm-global ARG NPM_GLOBAL=/usr/local/share/npm-global
# Add NPM global to PATH. # Add NPM global to PATH.
ENV PATH=${NPM_GLOBAL}/bin:${PATH} ENV PATH=${NPM_GLOBAL}/bin:${PATH}
RUN \
if [ ${CERC_HOST_GID} -ne 1000 ] ; then \
groupmod -g ${CERC_HOST_GID} ${USERNAME} ; \
fi \
&& if [ ${CERC_HOST_UID} -ne 1000 ] ; then \
usermod -u ${CERC_HOST_UID} -g ${CERC_HOST_GID} ${USERNAME} && chown ${CERC_HOST_UID}:${CERC_HOST_GID} /home/${USERNAME} ; \
fi
RUN \ RUN \
# Configure global npm install location, use group to adapt to UID/GID changes # Configure global npm install location, use group to adapt to UID/GID changes
if ! cat /etc/group | grep -e "^npm:" > /dev/null 2>&1; then groupadd -r npm; fi \ if ! cat /etc/group | grep -e "^npm:" > /dev/null 2>&1; then groupadd -r npm; fi \
@@ -55,7 +39,6 @@ RUN mkdir /scripts
COPY build-npm-package.sh /scripts COPY build-npm-package.sh /scripts
COPY yarn-local-registry-fixup.sh /scripts COPY yarn-local-registry-fixup.sh /scripts
COPY build-npm-package-local-dependencies.sh /scripts COPY build-npm-package-local-dependencies.sh /scripts
COPY check-uid.sh /scripts
ENV PATH="${PATH}:/scripts" ENV PATH="${PATH}:/scripts"
COPY entrypoint.sh . COPY entrypoint.sh .
@@ -1,7 +1,7 @@
#!/bin/bash #!/bin/bash
# Usage: build-npm-package-local-dependencies.sh <registry-url> <publish-with-this-version> # Usage: build-npm-package-local-dependencies.sh <registry-url> <publish-with-this-version>
# Runs build-npm-package.sh after first fixing up yarn.lock to use a local # Runs build-npm-package.sh after first fixing up yarn.lock to use a local
# npm registry for all packages in a spcific scope (currently @cerc-io and @lirewine) # npm registry for all packages in a spcific scope (currently @cerc-io)
if [ -n "$CERC_SCRIPT_DEBUG" ]; then if [ -n "$CERC_SCRIPT_DEBUG" ]; then
set -x set -x
fi fi
@@ -17,21 +17,18 @@ fi
set -e set -e
local_npm_registry_url=$1 local_npm_registry_url=$1
package_publish_version=$2 package_publish_version=$2
# If we need to handle an additional scope, add it to the list below: # TODO: make this a paramater and allow a list of scopes
npm_scopes_to_handle=("@cerc-io" "@lirewine") npm_scope_for_local="@cerc-io"
for npm_scope_for_local in ${npm_scopes_to_handle[@]} # We need to configure the local registry
npm config set ${npm_scope_for_local}:registry ${local_npm_registry_url}
npm config set -- ${local_npm_registry_url}:_authToken ${CERC_NPM_AUTH_TOKEN}
# Find the set of dependencies from the specified scope
mapfile -t dependencies_from_scope < <(cat package.json | jq -r '.dependencies | with_entries(if (.key|test("^'${npm_scope_for_local}'/.*$")) then ( {key: .key, value: .value } ) else empty end ) | keys[]')
echo "Fixing up dependencies"
for package in "${dependencies_from_scope[@]}"
do do
# We need to configure the local registry echo "Fixing up package ${package}"
npm config set ${npm_scope_for_local}:registry ${local_npm_registry_url} yarn-local-registry-fixup.sh $package ${local_npm_registry_url}
npm config set -- ${local_npm_registry_url}:_authToken ${CERC_NPM_AUTH_TOKEN}
# Find the set of dependencies from the specified scope
mapfile -t dependencies_from_scope < <(cat package.json | jq -r '.dependencies | with_entries(if (.key|test("^'${npm_scope_for_local}'/.*$")) then ( {key: .key, value: .value } ) else empty end ) | keys[]')
echo "Fixing up dependencies in scope ${npm_scope_for_local}"
for package in "${dependencies_from_scope[@]}"
do
echo "Fixing up package ${package}"
yarn-local-registry-fixup.sh $package ${local_npm_registry_url}
done
done done
echo "Running build" echo "Running build"
build-npm-package.sh ${local_npm_registry_url} ${package_publish_version} build-npm-package.sh ${local_npm_registry_url} ${package_publish_version}
@@ -22,8 +22,8 @@ set -e
# Get the name of this package from package.json since we weren't passed that # Get the name of this package from package.json since we weren't passed that
package_name=$( cat package.json | jq -r .name ) package_name=$( cat package.json | jq -r .name )
local_npm_registry_url=$1 local_npm_registry_url=$1
npm config set @cerc-io:registry ${local_npm_registry_url}
npm config set @lirewine:registry ${local_npm_registry_url} npm config set @lirewine:registry ${local_npm_registry_url}
npm config set @cerc-io:registry ${local_npm_registry_url}
npm config set -- ${local_npm_registry_url}:_authToken ${CERC_NPM_AUTH_TOKEN} npm config set -- ${local_npm_registry_url}:_authToken ${CERC_NPM_AUTH_TOKEN}
# First check if the version of this package we're trying to build already exists in the registry # First check if the version of this package we're trying to build already exists in the registry
package_exists=$( yarn info --json ${package_name}@${package_publish_version} 2>/dev/null | jq -r .data.dist.tarball ) package_exists=$( yarn info --json ${package_name}@${package_publish_version} 2>/dev/null | jq -r .data.dist.tarball )
@@ -1,21 +0,0 @@
#!/bin/bash
# Make the container usable for uid/gid != 1000
if [[ -n "$CERC_SCRIPT_DEBUG" ]]; then
set -x
fi
current_uid=$(id -u)
current_gid=$(id -g)
# Don't check if running as root
if [[ ${current_uid} == 0 ]]; then
exit 0
fi
# Check the current uid/gid vs the uid/gid used to build the container.
# We do this because both bind mounts and npm tooling require the uid/gid to match.
if [[ ${current_gid} != ${HOST_GID} ]]; then
echo "Warning: running with gid: ${current_gid} which is not the gid for which this container was built (${HOST_GID})"
exit 0
fi
if [[ ${current_uid} != ${HOST_UID} ]]; then
echo "Warning: running with gid: ${current_uid} which is not the uid for which this container was built (${HOST_UID})"
exit 0
fi
@@ -1,3 +1,2 @@
#!/bin/sh #!/bin/sh
/scripts/check-uid.sh
exec "$@" exec "$@"
+25
View File
@@ -0,0 +1,25 @@
#!/usr/bin/env bash
#
#Build cerc/datanet-eth-geth
set -e
SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
# Make sure the "build" directory is empty.
rm -rf $SCRIPT_DIR/build
# Copy the fixture-net scripts and config.
cp -rp $SCRIPT_DIR/../cerc-fixturenet-eth-geth $SCRIPT_DIR/build
# Then remove terminal_total_difficulty and replace it with capped_maximum_difficulty.
# This has two effects:
# (1) Disables the Merge (so all we need is geth, not lighthouse).
# (2) Maintains a fast block rate, since the difficulty will never exceed the capped value.
sed -i '' 's/^terminal_total_difficulty:.*$/capped_maximum_difficulty: 1/' $SCRIPT_DIR/build/genesis/el/el-config.yaml
# Build the image.
docker build -t cerc/datanet-eth-geth:local ${SCRIPT_DIR}/build
# Clean up the "build" directory.
rm -rf $SCRIPT_DIR/build
@@ -21,9 +21,11 @@ istanbul_block=`cat $tmp_dir/genesis-config.yaml | grep istanbul_block | awk '{
berlin_block=`cat $tmp_dir/genesis-config.yaml | grep berlin_block | awk '{ print $2 }'` berlin_block=`cat $tmp_dir/genesis-config.yaml | grep berlin_block | awk '{ print $2 }'`
london_block=`cat $tmp_dir/genesis-config.yaml | grep london_block | awk '{ print $2 }'` london_block=`cat $tmp_dir/genesis-config.yaml | grep london_block | awk '{ print $2 }'`
merge_fork_block=`cat $tmp_dir/genesis-config.yaml | grep merge_fork_block | awk '{ print $2 }'` merge_fork_block=`cat $tmp_dir/genesis-config.yaml | grep merge_fork_block | awk '{ print $2 }'`
capped_maximum_difficulty=`cat $tmp_dir/genesis-config.yaml | grep capped_maximum_difficulty | awk '{ print $2 }'`
python3 ../accounts/mnemonic_to_csv.py $tmp_dir/genesis-config.yaml > ../build/el/accounts.csv
python3 /apps/el-gen/genesis_geth.py $tmp_dir/genesis-config.yaml | \ python3 /apps/el-gen/genesis_geth.py $tmp_dir/genesis-config.yaml | \
jq ".config.terminalTotalDifficulty=$ttd" | \
jq ".config.homesteadBlock=$homestead_block" | \ jq ".config.homesteadBlock=$homestead_block" | \
jq ".config.eip150Block=$eip150_block" | \ jq ".config.eip150Block=$eip150_block" | \
jq ".config.eip155Block=$eip155_block" | \ jq ".config.eip155Block=$eip155_block" | \
@@ -35,4 +37,19 @@ python3 /apps/el-gen/genesis_geth.py $tmp_dir/genesis-config.yaml | \
jq ".config.berlinBlock=$berlin_block" | \ jq ".config.berlinBlock=$berlin_block" | \
jq ".config.londonBlock=$london_block" | \ jq ".config.londonBlock=$london_block" | \
jq ".config.mergeForkBlock=$merge_fork_block" > ../build/el/geth.json jq ".config.mergeForkBlock=$merge_fork_block" > ../build/el/geth.json
python3 ../accounts/mnemonic_to_csv.py $tmp_dir/genesis-config.yaml > ../build/el/accounts.csv
if [ -n "$ttd" ]; then
cat ../build/el/geth.json | jq ".config.terminalTotalDifficulty=$ttd" > ../build/el/geth.json.jq
mv ../build/el/geth.json.jq ../build/el/geth.json
else
cat ../build/el/geth.json | jq "del(.config.terminalTotalDifficulty)" > ../build/el/geth.json.jq
mv ../build/el/geth.json.jq ../build/el/geth.json
fi
if [ -n "$capped_maximum_difficulty" ]; then
cat ../build/el/geth.json | jq ".config.cappedMaximumDifficulty=$capped_maximum_difficulty" > ../build/el/geth.json.jq
mv ../build/el/geth.json.jq ../build/el/geth.json
else
cat ../build/el/geth.json | jq "del(.config.cappedMaximumDifficulty)" > ../build/el/geth.json.jq
mv ../build/el/geth.json.jq ../build/el/geth.json
fi
@@ -24,7 +24,7 @@ if [ -z "$GETH_BASE_URL" ]; then
fi fi
function inc_status() { function inc_status() {
echo " done" echo " DONE!"
STATUS=$((STATUS + 1)) STATUS=$((STATUS + 1))
if [ $STATUS -lt ${#STATUSES[@]} ]; then if [ $STATUS -lt ${#STATUSES[@]} ]; then
echo -n "Waiting for ${STATUSES[$STATUS]}..." echo -n "Waiting for ${STATUSES[$STATUS]}..."
@@ -1,3 +0,0 @@
#!/usr/bin/env bash
# Build a local version of the foundry-rs/foundry image
docker build -t cerc/foundry:local -f ${CERC_REPO_BASE_DIR}/foundry/Dockerfile-debian ${CERC_REPO_BASE_DIR}/foundry
@@ -1,4 +1,6 @@
FROM cerc/foundry:local # Use locally build foundry base image to work around there being
# no aarm64 image published.
FROM foundry-rs/foundry:local
RUN apk update ; apk add --no-cache --allow-untrusted ca-certificates curl bash git jq RUN apk update ; apk add --no-cache --allow-untrusted ca-certificates curl bash git jq
RUN apk add --no-cache --upgrade grep RUN apk add --no-cache --upgrade grep
@@ -1,4 +1,4 @@
FROM quay.io/keycloak/keycloak:20.0 FROM quay.io/keycloak/keycloak:20.0
WORKDIR /opt/keycloak/providers WORKDIR /opt/keycloak/providers
RUN curl -L https://github.com/aerogear/keycloak-metrics-spi/releases/download/2.5.3/keycloak-metrics-spi-2.5.3.jar --output keycloak-metrics-spi.jar RUN curl -L https://github.com/aerogear/keycloak-metrics-spi/releases/download/2.5.3/keycloak-metrics-spi-2.5.3.jar --output keycloak-metrics-spi.jar
RUN curl -L https://github.com/cerc-io/keycloak-api-key-demo/releases/download/v0.3/api-key-module-0.3.jar --output api-key-module.jar RUN curl -L https://github.com/cerc-io/keycloak-api-key-demo/releases/download/v0.2/api-key-module-0.2.jar --output api-key-module.jar
@@ -1,6 +1,6 @@
# Originally from: https://github.com/devcontainers/images/blob/main/src/javascript-node/.devcontainer/Dockerfile # Originally from: https://github.com/devcontainers/images/blob/main/src/javascript-node/.devcontainer/Dockerfile
# [Choice] Node.js version (use -bullseye variants on local arm64/Apple Silicon): 18, 16, 14, 18-bullseye, 16-bullseye, 14-bullseye, 18-buster, 16-buster, 14-buster # [Choice] Node.js version (use -bullseye variants on local arm64/Apple Silicon): 18, 16, 14, 18-bullseye, 16-bullseye, 14-bullseye, 18-buster, 16-buster, 14-buster
ARG VARIANT=18-bullseye ARG VARIANT=16-bullseye
FROM node:${VARIANT} FROM node:${VARIANT}
ARG USERNAME=node ARG USERNAME=node
@@ -40,20 +40,18 @@ RUN apt-get update && export DEBIAN_FRONTEND=noninteractive \
# RUN su node -c "npm install -g <your-package-list-here>" # RUN su node -c "npm install -g <your-package-list-here>"
# Configure the local npm registry # Configure the local npm registry
RUN npm config set @cerc-io:registry ${CERC_NPM_URL} \ RUN npm config set @lirewine:registry ${CERC_NPM_URL} \
&& npm config set @cerc-io:registry ${CERC_NPM_URL} \
&& npm config set @lirewine:registry ${CERC_NPM_URL} \ && npm config set @lirewine:registry ${CERC_NPM_URL} \
&& npm config set -- ${CERC_NPM_URL}:_authToken ${CERC_NPM_AUTH_TOKEN} && npm config set -- ${CERC_NPM_URL}:_authToken ${CERC_NPM_AUTH_TOKEN}
# TODO: the image at this point could be made a base image for several different CLI images # TODO: the image at this point could be made a base image for several different CLI images
# that install different Node-based CLI commands # that install different Node-based CLI commands
# DEBUG, remove
RUN yarn info @cerc-io/laconic-registry-cli
# Globally install the cli package # Globally install the cli package
RUN yarn global add @cerc-io/laconic-registry-cli RUN yarn global add @cerc-io/laconic-registry-cli
# Add scripts ENTRYPOINT ["laconic"]
RUN mkdir /scripts
ENV PATH="${PATH}:/scripts"
COPY ./import-key.sh /scripts
# Default command sleeps forever so docker doesn't kill it
CMD ["sh", "-c", "while :; do sleep 600; done"]
@@ -1,2 +0,0 @@
#!/bin/sh
sed 's/REPLACE_WITH_MYKEY/'${1}'/' registry-cli-config-template.yml > config.yml
@@ -1,14 +0,0 @@
FROM node:18.15.0-alpine3.16
RUN apk --update --no-cache add make git jq
WORKDIR /app
COPY . .
RUN npm install -g serve
RUN echo "Building mobymask-ui" && \
npm install
WORKDIR /app
@@ -1,7 +0,0 @@
#!/usr/bin/env bash
# Build cerc/mobymask-ui
# See: https://stackoverflow.com/a/246128/1701505
SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
docker build -t cerc/mobymask-ui:local -f ${SCRIPT_DIR}/Dockerfile ${CERC_REPO_BASE_DIR}/mobymask-ui
@@ -1,13 +0,0 @@
FROM node:16.17.1-alpine3.16
RUN apk --update --no-cache add python3 alpine-sdk
WORKDIR /app
COPY . .
RUN yarn
# Add scripts
RUN mkdir /scripts
ENV PATH="${PATH}:/scripts"
@@ -1,7 +0,0 @@
#!/usr/bin/env bash
# Build cerc/mobymask
# See: https://stackoverflow.com/a/246128/1701505
SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
docker build -t cerc/mobymask:local -f ${SCRIPT_DIR}/Dockerfile ${CERC_REPO_BASE_DIR}/MobyMask
@@ -1,14 +0,0 @@
FROM node:18.15.0-alpine3.16
RUN apk --update --no-cache add make git python3
WORKDIR /app
COPY . .
RUN yarn global add serve
RUN echo "Building react-peer" && \
yarn install --ignore-scripts && yarn build --ignore @cerc-io/test-app
WORKDIR /app
@@ -1,7 +0,0 @@
#!/usr/bin/env bash
# Build cerc/react-peer
# See: https://stackoverflow.com/a/246128/1701505
SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
docker build -t cerc/react-peer:local -f ${SCRIPT_DIR}/Dockerfile ${CERC_REPO_BASE_DIR}/react-peer
@@ -1,20 +0,0 @@
FROM ubuntu:22.04
RUN apt-get update \
&& apt-get install -y curl gnupg build-essential \
&& curl --silent --location https://deb.nodesource.com/setup_18.x | bash - \
&& apt-get update \
&& apt-get install -y nodejs git busybox jq \
&& node -v
RUN corepack enable \
&& yarn --version
WORKDIR /app
COPY . .
RUN echo "Building watcher-ts" && \
yarn && yarn build
WORKDIR /app/packages/mobymask-v2-watcher
@@ -1,7 +0,0 @@
#!/usr/bin/env bash
# Build cerc/watcher-mobymask-v2
# See: https://stackoverflow.com/a/246128/1701505
SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
docker build -t cerc/watcher-mobymask-v2:local -f ${SCRIPT_DIR}/Dockerfile ${CERC_REPO_BASE_DIR}/watcher-ts
+1 -4
View File
@@ -1,9 +1,6 @@
#!/usr/bin/env bash #!/usr/bin/env bash
# Usage: default-build.sh <image-tag> [<repo-relative-path>] # Usage: default-build.sh <image-tag> [<repo-relative-path>]
# if <repo-relative-path> is not supplied, the context is the directory where the Dockerfile lives # if <repo-relative-path> is not supplied, the context is the directory where the Dockerfile lives
if [[ -n "$CERC_SCRIPT_DEBUG" ]]; then
set -x
fi
if [[ $# -ne 2 ]]; then if [[ $# -ne 2 ]]; then
echo "Illegal number of parameters" >&2 echo "Illegal number of parameters" >&2
exit 1 exit 1
@@ -11,4 +8,4 @@ fi
image_tag=$1 image_tag=$1
build_dir=$2 build_dir=$2
echo "Building ${image_tag} in ${build_dir}" echo "Building ${image_tag} in ${build_dir}"
docker build -t ${image_tag} --build-arg CERC_HOST_UID=${CERC_HOST_UID} --build-arg CERC_HOST_GID=${CERC_HOST_GID} ${build_dir} docker build -t ${image_tag} ${build_dir}
+4
View File
@@ -0,0 +1,4 @@
#!/usr/bin/env bash
# Build foundry-rs/foundry
# HACK below : TARGETARCH needs to be derived from the host environment
docker build -t foundry-rs/foundry:local ${CERC_REPO_BASE_DIR}/foundry
+2 -7
View File
@@ -1,4 +1,4 @@
cerc/foundry foundry-rs/foundry
cerc/test-contract cerc/test-contract
cerc/eth-statediff-fill-service cerc/eth-statediff-fill-service
cerc/eth-statediff-service cerc/eth-statediff-service
@@ -11,6 +11,7 @@ cerc/ipld-eth-beacon-indexer
cerc/ipld-eth-server cerc/ipld-eth-server
cerc/laconicd cerc/laconicd
cerc/laconic-registry-cli cerc/laconic-registry-cli
cerc/datanet-eth-geth
cerc/fixturenet-eth-geth cerc/fixturenet-eth-geth
cerc/fixturenet-eth-lighthouse cerc/fixturenet-eth-lighthouse
cerc/watcher-mobymask cerc/watcher-mobymask
@@ -18,15 +19,9 @@ cerc/watcher-erc20
cerc/watcher-erc721 cerc/watcher-erc721
cerc/watcher-uniswap-v3 cerc/watcher-uniswap-v3
cerc/uniswap-v3-info cerc/uniswap-v3-info
cerc/watcher-mobymask-v2
cerc/react-peer
cerc/mobymask-ui
cerc/mobymask
cerc/test-container cerc/test-container
cerc/eth-probe cerc/eth-probe
cerc/builder-js cerc/builder-js
cerc/keycloak cerc/keycloak
cerc/tx-spammer cerc/tx-spammer
cerc/builder-gerbil cerc/builder-gerbil
cerc/act-runner
cerc/act-runner-task-executor
+2 -3
View File
@@ -1,5 +1,4 @@
laconic-sdk
laconic-registry-cli
gem gem
laconic-sdk
debug debug
crypto laconic-registry-cli
+1 -4
View File
@@ -8,17 +8,14 @@ ipld-eth-server
lighthouse lighthouse
laconicd laconicd
fixturenet-laconicd fixturenet-laconicd
datanet-eth
fixturenet-eth fixturenet-eth
fixturenet-eth-metrics fixturenet-eth-metrics
watcher-mobymask watcher-mobymask
watcher-erc20 watcher-erc20
watcher-erc721 watcher-erc721
watcher-uniswap-v3 watcher-uniswap-v3
watcher-mobymask-v2
mobymask-laconicd
test test
eth-probe eth-probe
keycloak keycloak
tx-spammer tx-spammer
kubo
foundry
+1 -8
View File
@@ -11,16 +11,9 @@ cerc-io/laconic-sdk
cerc-io/laconic-registry-cli cerc-io/laconic-registry-cli
cerc-io/mobymask-watcher cerc-io/mobymask-watcher
cerc-io/watcher-ts cerc-io/watcher-ts
cerc-io/react-peer
cerc-io/mobymask-ui
cerc-io/MobyMask
vulcanize/uniswap-watcher-ts vulcanize/uniswap-watcher-ts
vulcanize/uniswap-v3-info vulcanize/uniswap-v3-info
vulcanize/assemblyscript vulcanize/assemblyscript
cerc-io/eth-probe cerc-io/eth-probe
cerc-io/tx-spammer cerc-io/tx-spammer
dboreham/foundry foundry-rs/foundry
lirewine/gem
lirewine/debug
lirewine/crypto
telackey/act_runner
-59
View File
@@ -1,59 +0,0 @@
# Build Support Stack
## Instructions
JS/TS/NPM builds need an npm registry to store intermediate package artifacts.
This can be supplied by the user (e.g. using a hosted registry or even npmjs.com), or a local registry using gitea can be deployed by stack orchestrator.
To use a user-supplied registry set these environment variables:
`CERC_NPM_REGISTRY_URL` and
`CERC_NPM_AUTH_TOKEN`
Leave `CERC_NPM_REGISTRY_URL` un-set to use the local gitea registry.
### 1. Build support containers
```
$ laconic-so --stack build-support build-containers
```
Note that the scheme/gerbil builder container can take a while to build so if you aren't going to build scheme projects it can be skipped with:
```
$ laconic-so --stack build-support build-containers --exclude cerc/builder-gerbil
```
### 2. Deploy Gitea Package Registry
```
$ laconic-so --stack package-registry setup-repositories
$ laconic-so --stack package-registry deploy up
[+] Running 3/3
⠿ Network laconic-aecc4a21d3a502b14522db97d427e850_gitea Created 0.0s
⠿ Container laconic-aecc4a21d3a502b14522db97d427e850-db-1 Started 1.2s
⠿ Container laconic-aecc4a21d3a502b14522db97d427e850-server-1 Started 1.9s
New user 'gitea_admin' has been successfully created!
This is your gitea access token: 84fe66a73698bf11edbdccd0a338236b7d1d5c45. Keep it safe and secure, it can not be fetched again from gitea.
To use with laconic-so set this environment variable: export CERC_NPM_AUTH_TOKEN=3e493e77b3e83fe9e882f7e3a79dd4d5441c308b
Created the organization cerc-io
Gitea was configured to use host name: gitea.local, ensure that this resolves to localhost, e.g. with sudo vi /etc/hosts
Success, gitea is properly initialized
$
```
### 3. Configure the hostname gitea.local
How to do this is OS-dependent but usually involves editing a `hosts` file. For example on Linux add this line to the file `/etc/hosts` (needs sudo):
```
127.0.0.1 gitea.local
```
Test with:
```
$ ping gitea.local
PING gitea.local (127.0.0.1) 56(84) bytes of data.
64 bytes from localhost (127.0.0.1): icmp_seq=1 ttl=64 time=0.147 ms
64 bytes from localhost (127.0.0.1): icmp_seq=2 ttl=64 time=0.033 ms
```
Although not necessary in order to build and publish packages, you can now access the Gitea web interface at: [http://gitea.local:3000](http://gitea.local:3000) using these credentials: gitea_admin/admin1234 (Note: please properly secure Gitea if public internet access is allowed).
Now npm packages can be built:
### Build npm Packages
Ensure that `CERC_NPM_AUTH_TOKEN` is set with the token printed above when the package-registry stack was deployed (the actual token value will be different than shown in this example):
```
$ export CERC_NPM_AUTH_TOKEN=84fe66a73698bf11edbdccd0a338236b7d1d5c45
$ laconic-so build-npms --include laconic-sdk
```
-6
View File
@@ -1,6 +0,0 @@
version: "1.1"
name: build-support
decription: "Build Support Components"
containers:
- cerc/builder-js
- cerc/builder-gerbil
+4 -4
View File
@@ -7,13 +7,13 @@ Instructions to deploy a local ERC20 watcher stack (core + watcher) for demonstr
Clone required repositories: Clone required repositories:
```bash ```bash
laconic-so --stack erc20 setup-repositories laconic-so setup-repositories --include cerc-io/go-ethereum,cerc-io/ipld-eth-db,cerc-io/ipld-eth-server,cerc-io/watcher-ts
``` ```
Build the core and watcher container images: Build the core and watcher container images:
```bash ```bash
laconic-so --stack erc20 build-containers laconic-so build-containers --include cerc/go-ethereum,cerc/go-ethereum-foundry,cerc/ipld-eth-db,cerc/ipld-eth-server,cerc/watcher-erc20
``` ```
This should create the required docker images in the local image registry. This should create the required docker images in the local image registry.
@@ -21,7 +21,7 @@ This should create the required docker images in the local image registry.
Deploy the stack: Deploy the stack:
```bash ```bash
laconic-so --stack erc20 deploy-system up laconic-so deploy-system --include ipld-eth-db,go-ethereum-foundry,ipld-eth-server,watcher-erc20 up
``` ```
## Demo ## Demo
@@ -153,5 +153,5 @@ Transfer funds between different accounts using MetaMask and use the playground
To stop all the services running in background run: To stop all the services running in background run:
```bash ```bash
laconic-so --stack erc20 deploy-system down laconic-so deploy-system --include ipld-eth-db,go-ethereum-foundry,ipld-eth-server,watcher-erc20 down
``` ```
+12 -12
View File
@@ -6,23 +6,23 @@ Instructions to deploy a local ERC721 watcher stack (core + watcher) for demonst
* Clone / pull required repositories: * Clone / pull required repositories:
```bash ```bash
laconic-so --stack erc721 setup-repositories $ laconic-so setup-repositories --include cerc-io/go-ethereum,cerc-io/ipld-eth-db,cerc-io/ipld-eth-server,cerc-io/watcher-ts --pull
``` ```
* Build the core and watcher container images: * Build the core and watcher container images:
```bash ```bash
laconic-so --stack erc721 build-containers $ laconic-so build-containers --include cerc/go-ethereum,cerc/go-ethereum-foundry,cerc/ipld-eth-db,cerc/ipld-eth-server,cerc/watcher-erc721
``` ```
This should create the required docker images in the local image registry. This should create the required docker images in the local image registry.
* Deploy the stack: * Deploy the stack:
```bash ```bash
laconic-so --stack erc721 deploy-system up $ laconic-so deploy-system --include db,go-ethereum-foundry,ipld-eth-server,watcher-erc721 up
``` ```
## Demo ## Demo
@@ -209,6 +209,6 @@ laconic-so --stack erc721 deploy-system up
* To stop all the services running in background: * To stop all the services running in background:
```bash ```bash
laconic-so --stack erc721 deploy-system down $ laconic-so deploy-system --include db,go-ethereum-foundry,ipld-eth-server,watcher-erc721 down
``` ```
@@ -1,6 +0,0 @@
# fixturenet-eth
A "loaded" version of fixturenet-eth, with all the bells and whistles enabled.
TODO: write me
@@ -1,24 +0,0 @@
version: "1.0"
name: fixturenet-eth-loaded
decription: "Loaded Ethereum Fixturenet"
repos:
- cerc-io/go-ethereum
- cerc-io/tx-spammer
- cerc-io/ipld-eth-server
- cerc-io/ipld-eth-db
- cerc/go-ethereum
containers:
- cerc/lighthouse
- cerc/fixturenet-eth-geth
- cerc/fixturenet-eth-lighthouse
- cerc/ipld-eth-server
- cerc/ipld-eth-db
- cerc/keycloak
- cerc/tx-spammer
pods:
- fixturenet-eth
- tx-spammer
- fixturenet-eth-metrics
- keycloak
- ipld-eth-server
- ipld-eth-db
+20 -24
View File
@@ -20,31 +20,27 @@ This should create several container images in the local image registry:
## Deploy the stack ## Deploy the stack
``` ```
$ laconic-so --stack fixturenet-eth deploy up $ laconic-so --stack fixturenet-eth deploy-system up
``` ```
## Check status ## Check status
``` ```
$ laconic-so --stack fixturenet-eth deploy exec fixturenet-eth-bootnode-lighthouse /scripts/status-internal.sh $ container-build/cerc-fixturenet-eth-lighthouse/scripts/status.sh
Waiting for geth to generate DAG.... done Waiting for geth to generate DAG ..................................................... DONE!
Waiting for beacon phase0.... done Waiting for beacon phase0 .... DONE!
Waiting for beacon altair.... done Waiting for beacon altair .... DONE!
Waiting for beacon bellatrix pre-merge.... done Waiting for beacon bellatrix pre-merge .... DONE!
Waiting for beacon bellatrix merge.... done Waiting for beacon bellatrix merge .... DONE!
$ laconic-so --stack fixturenet-eth deploy ps $ docker ps -f 'name=laconic' --format 'table {{.Names}}\t{{.Ports}}' | cut -d'-' -f3- | sort
Running containers: NAMES PORTS
id: c6538b60c0328dadfa2c5585c4d09674a6a13e6d712ff1cd82a26849e4e5679b, name: laconic-b12fa16e999821562937781f8ab0b1e8-fixturenet-eth-bootnode-geth-1, ports: 0.0.0.0:58909->30303/tcp, 0.0.0.0:58910->9898/tcp fixturenet-eth-bootnode-geth-1 8545-8546/tcp, 30303/udp, 0.0.0.0:55847->9898/tcp, 0.0.0.0:55848->30303/tcp
id: 5b70597a8211bc7e78d33e50486cb565a7f4a9ce581ce150b3bb450e342bdeda, name: laconic-b12fa16e999821562937781f8ab0b1e8-fixturenet-eth-bootnode-lighthouse-1, ports: fixturenet-eth-bootnode-lighthouse-1
id: 19ed78867b6c534d893835cdeb1e89a9ea553b8e8c02ab02468e4bd1563a340f, name: laconic-b12fa16e999821562937781f8ab0b1e8-fixturenet-eth-geth-1-1, ports: 0.0.0.0:58911->40000/tcp, 0.0.0.0:58912->6060/tcp, 0.0.0.0:58913->8545/tcp fixturenet-eth-geth-1-1 8546/tcp, 30303/tcp, 30303/udp, 0.0.0.0:55851->8545/tcp
id: 8da0e30a1ce33122d8fd2225e4d26c7f30eb4bfbfa743f2af04d9db5d0bf7fa6, name: laconic-b12fa16e999821562937781f8ab0b1e8-fixturenet-eth-geth-2-1, ports: fixturenet-eth-geth-2-1 8545-8546/tcp, 30303/tcp, 30303/udp
id: 387a42a14971034588ba9aeb9b9e2ca7fc0cc61b96f8fe8c2ab770c9d6fb1e0f, name: laconic-b12fa16e999821562937781f8ab0b1e8-fixturenet-eth-lighthouse-1-1, ports: 0.0.0.0:58917->8001/tcp fixturenet-eth-lighthouse-1-1 0.0.0.0:55858->8001/tcp
id: de5115bf89087bae03b291664a73ffe3554fe23e79e4b8345e088b040d5580ac, name: laconic-b12fa16e999821562937781f8ab0b1e8-fixturenet-eth-lighthouse-2-1, ports: fixturenet-eth-lighthouse-2-1
id: 2a7e5a0fb2be7fc9261a7b725a40818facbbe6d0cb2497d82c0e02de0a8e959b, name: laconic-b12fa16e999821562937781f8ab0b1e8-foundry-1, ports:
$ laconic-so --stack fixturenet-eth deploy exec foundry "cast block-number"
3
``` ```
## Additional pieces ## Additional pieces
@@ -73,11 +69,11 @@ $ laconic-so deploy-system --include db,fixturenet-eth,ipld-eth-server,ipld-eth-
# Status # Status
$ container-build/cerc-fixturenet-eth-lighthouse/scripts/status.sh $ container-build/cerc-fixturenet-eth-lighthouse/scripts/status.sh
Waiting for geth to generate DAG.... done Waiting for geth to generate DAG.... DONE!
Waiting for beacon phase0.... done Waiting for beacon phase0.... DONE!
Waiting for beacon altair.... done Waiting for beacon altair.... DONE!
Waiting for beacon bellatrix pre-merge.... done Waiting for beacon bellatrix pre-merge.... DONE!
Waiting for beacon bellatrix merge.... done Waiting for beacon bellatrix merge.... DONE!
$ docker ps -f 'name=laconic' --format 'table {{.Names}}\t{{.Ports}}' | cut -d'-' -f3- | sort $ docker ps -f 'name=laconic' --format 'table {{.Names}}\t{{.Ports}}' | cut -d'-' -f3- | sort
NAMES PORTS NAMES PORTS
-3
View File
@@ -4,13 +4,10 @@ decription: "Ethereum Fixturenet"
repos: repos:
- cerc-io/go-ethereum - cerc-io/go-ethereum
- cerc-io/tx-spammer - cerc-io/tx-spammer
- dboreham/foundry
containers: containers:
- cerc/go-ethereum - cerc/go-ethereum
- cerc/lighthouse - cerc/lighthouse
- cerc/fixturenet-eth-geth - cerc/fixturenet-eth-geth
- cerc/fixturenet-eth-lighthouse - cerc/fixturenet-eth-lighthouse
- cerc/foundry
pods: pods:
- fixturenet-eth - fixturenet-eth
- foundry
+9 -39
View File
@@ -1,48 +1,18 @@
# Laconicd Fixturenet # Laconicd Fixturenet
Instructions for deploying a local Laconic blockchain "fixturenet" for development and testing purposes using laconic-stack-orchestrator. Instructions for deploying a local Laconic blockchain "fixturenet" for development and testing purposes using laconic-stack-orchestrator (the installation of which is covered [here](https://github.com/cerc-io/stack-orchestrator#user-mode)):
## 1. Install Laconic Stack Orchestrator ## Clone required repositories
Installation is covered in detail [here](https://github.com/cerc-io/stack-orchestrator#user-mode) but if you're on Linux and already have docker installed it should be as simple as:
``` ```
$ mkdir my-working-dir $ laconic-so setup-repositories --include cerc-io/laconicd,cerc-io/laconic-sdk,cerc-io/laconic-registry-cli
$ cd my-working-dir
$ curl -L -o ./laconic-so https://github.com/cerc-io/stack-orchestrator/releases/latest/download/laconic-so
$ chmod +x ./laconic-so
$ export PATH=$PATH:$(pwd) # Or move laconic-so to ~/bin or your favorite on-path directory
``` ```
## 2. Prepare the local build environment ## Build the laconicd container
Note that this step needs only to be done once on a new machine.
Detailed instructions can be found [here](../build-support/README.md). For the impatient run these commands:
``` ```
$ laconic-so --stack build-support build-containers --exclude cerc/builder-gerbil $ laconic-so build-containers --include cerc/laconicd
$ laconic-so --stack package-registry setup-repositories
$ laconic-so --stack package-registry deploy-system up
``` ```
Then add the localhost alias `gitea.local` and set `CERC_NPM_AUTH_TOKEN` to the token printed when the package-registry stack was deployed above: This should create a container with tag `cerc/laconicd` in the local image registry.
## Deploy the stack
``` ```
$ sudo vi /etc/hosts $ laconic-so deploy-system --include fixturenet-laconicd up
$ export CERC_NPM_AUTH_TOKEN=<my-token>
```
## 3. Clone required repositories
```
$ laconic-so --stack fixturenet-laconicd setup-repositories
```
## 4. Build the stack's packages and containers
```
$ laconic-so --stack fixturenet-laconicd build-npms
$ laconic-so --stack fixturenet-laconicd build-containers
```
## 5. Deploy the stack
```
$ laconic-so --stack fixturenet-laconicd deploy up
```
Correct operation should be verified by checking the laconicd container's logs with:
```
$ laconic-so --stack fixturenet-laconicd deploy logs
```
## 6. Test with the Registry CLI
```
$ laconic-so --stack fixturenet-laconicd deploy exec cli "laconic cns status"
``` ```
Correct operation should be verified by checking the laconicd container's log.
@@ -13,6 +13,3 @@ containers:
- cerc/laconic-registry-cli - cerc/laconic-registry-cli
pods: pods:
- fixturenet-laconicd - fixturenet-laconicd
config:
cli:
key: laconicd.mykey
-179
View File
@@ -1,179 +0,0 @@
# MobyMask v2 watcher
Instructions to deploy MobyMask v2 watcher stack using [laconic-stack-orchestrator](/README.md#install)
## Setup
Clone required repositories:
```bash
laconic-so --stack mobymask-v2 setup-repositories
```
NOTE: If repositories already exist and are checked out to different versions, `setup-repositories` command will throw an error.
For getting around this, the repositories mentioned below can be removed and then run the command.
Checkout to the required versions and branches in repos
```bash
# watcher-ts
cd ~/cerc/watcher-ts
git checkout v0.2.31
# react-peer
cd ~/cerc/react-peer
git checkout v0.2.29
# mobymask-ui
cd ~/cerc/mobymask-ui
git checkout laconic
# laconicd
cd ~/cerc/laconicd
git checkout v0.8.0
# MobyMask
cd ~/cerc/MobyMask
git checkout v0.1.1
```
Build the container images:
```bash
laconic-so --stack mobymask-v2 build-containers
```
This should create the required docker images in the local image registry.
Deploy the stack:
* Deploy the laconic chain
```bash
laconic-so --stack mobymask-v2 deploy-system --include mobymask-laconicd up
```
* Check that laconic chain status is healthy
```bash
docker ps
```
* Export the private key from laconicd
```bash
laconic-so --stack mobymask-v2 deploy-system --include mobymask-laconicd exec laconicd "echo y | laconicd keys export mykey --unarmored-hex --unsafe"
```
* Set the private key in [secrets.json](../../config/watcher-mobymask-v2/secrets.json) file that will be used by mobymask container to deploy contract
* Create a new account
```bash
laconic-so --stack mobymask-v2 deploy-system --include mobymask-laconicd exec laconicd "laconicd keys add alice"
```
* Transfer balance to new account
```bash
laconic-so --stack mobymask-v2 deploy-system --include mobymask-laconicd exec laconicd 'laconicd tx bank send $(laconicd keys show mykey -a) $(laconicd keys show alice -a) 1000000000000000000000000aphoton --fees 2000aphoton'
```
* Export the private key of new account from laconicd
```bash
laconic-so --stack mobymask-v2 deploy-system --include mobymask-laconicd exec laconicd "echo y | laconicd keys export alice --unarmored-hex --unsafe"
```
* Set the private key (`PRIVATE_KEY`) in [peer-start.sh](../../config/watcher-mobymask-v2/peer-start.sh) file that will be used to start the peer that sends txs to L2 chain
* Deploy the other containers
```bash
laconic-so --stack mobymask-v2 deploy-system --include watcher-mobymask-v2 up
```
* Check that all containers are healthy using `docker ps`
NOTE: The `mobymask-ui` container might not start. If mobymask-app is not running at http://localhost:3002, run command again to start the container
```bash
laconic-so --stack mobymask-v2 deploy-system --include watcher-mobymask-v2 up
```
## Tests
Find the watcher container's id:
```bash
laconic-so --stack mobymask-v2 deploy-system --include watcher-mobymask-v2 ps | grep "mobymask-watcher-server"
```
Example output
```
id: 5d3aae4b22039fcd1c9b18feeb91318ede1100581e75bb5ac54f9e436066b02c, name: laconic-bfb01caf98b1b8f7c8db4d33f11b905a-mobymask-watcher-server-1, ports: 0.0.0.0:3001->3001/tcp, 0.0.0.0:9001->9001/tcp, 0.0.0.0:9090->9090/tcp
```
In above output the container ID is `5d3aae4b22039fcd1c9b18feeb91318ede1100581e75bb5ac54f9e436066b02c`
Export it for later use:
```bash
export CONTAINER_ID=<CONTAINER_ID>
```
Run the peer tests:
```bash
docker exec -w /app/packages/peer $CONTAINER_ID yarn test
```
## Web Apps
Check that the status for web-app containers are healthy by using `docker ps`
### mobymask-app
The mobymask-app should be running at http://localhost:3002
### peer-test-app
The peer-test-app should be running at http://localhost:3003
## Details
* The relay node for p2p network is running at http://localhost:9090
* The [peer package](https://github.com/cerc-io/watcher-ts/tree/main/packages/peer) (published in [gitea](https://git.vdb.to/cerc-io/-/packages/npm/@cerc-io%2Fpeer)) can be used in client code for connecting to the network
* The [react-peer package](https://github.com/cerc-io/react-peer/tree/main/packages/react-peer) (published in [gitea](https://git.vdb.to/cerc-io/-/packages/npm/@cerc-io%2Freact-peer)) which uses the peer package can be used in react app for connecting to the network
## Demo
Follow the [demo](./demo.md) to try out the MobyMask app with L2 chain
## Clean up
Stop all the services running in background run:
```bash
laconic-so --stack mobymask-v2 deploy-system --include watcher-mobymask-v2 down
laconic-so --stack mobymask-v2 deploy-system --include mobymask-laconicd down
```
Clear volumes:
* List all volumes
```bash
docker volume ls
```
* Remove volumes created by this stack
Example:
```bash
docker volume rm laconic-bfb01caf98b1b8f7c8db4d33f11b905a_moby_data_server
```
-132
View File
@@ -1,132 +0,0 @@
# Demo
* Get the root invite link URL for mobymask-app
```
laconic-so --stack mobymask-v2 deploy-system --include watcher-mobymask-v2 logs mobymask
```
The invite link is seen at the end of the logs
Example:
```
laconic-bfb01caf98b1b8f7c8db4d33f11b905a-mobymask-1 | http://127.0.0.1:3002/#/members?invitation=%7B%22v%22%3A1%2C%22signedDelegations%22%3A%5B%7B%22signature%22%3A%220x7559bd412f02677d60820e38243acf61547f79339395a34f7d4e1630e645aeb30535fc219f79b6fbd3af0ce3bd05132ad46d2b274a9fbc4c36bc71edd09850891b%22%2C%22delegation%22%3A%7B%22delegate%22%3A%220xc0838c92B2b71756E0eAD5B3C1e1F186baeEAAac%22%2C%22authority%22%3A%220x0000000000000000000000000000000000000000000000000000000000000000%22%2C%22caveats%22%3A%5B%7B%22enforcer%22%3A%220x558024C7d593B840E1BfD83E9B287a5CDad4db15%22%2C%22terms%22%3A%220x0000000000000000000000000000000000000000000000000000000000000000%22%7D%5D%7D%7D%5D%2C%22key%22%3A%220x98da9805821f1802196443e578fd32af567bababa0a249c07c82df01ecaa7d8d%22%7D
```
* Open the invite link in browser to use the mobymask-app.
NOTE: Before opening the invite link, clear the browser cache (local storage) for http://127.0.0.1:3002 to remove old invitations
* In the debug panel, check if it is connected to the p2p network (It should be connected to atleast one other peer for pubsub to work).
* Create an invite link in the app by clicking on `Create new invite link` button.
* Switch to the `MESSAGES` tab in debug panel for viewing incoming messages later.
* Open the invite link in a new browser with different profile (to simulate remote browser)
* Check that it is connected to any other peer in the network.
* In `Report a phishing attempt` section, report multiple phishers using the `submit` button. Click on the `Submit batch to p2p network` button. This broadcasts signed invocations to the connected peers.
* In the `MESSAGES` tab of other browsers, a message can be seen with the signed invocations.
* In a terminal check logs from the watcher peer container.
* Get the container id
```bash
laconic-so --stack mobymask-v2 deploy-system --include watcher-mobymask-v2 ps | grep mobymask-watcher-peer
```
* Check logs
```bash
docker logs -f CONTAINER_ID
```
* It should have received the message, sent transaction to L2 chain and received a transaction receipt with block details.
Example log:
```
2023-03-23T10:25:19.771Z vulcanize:peer-listener [10:25:19] Received a message on mobymask P2P network from peer: PeerId(12D3KooWAVNswtcrX12iDYukEoxdQwD34kJyRWcQTfZ4unGg2xjd)
2023-03-23T10:25:24.143Z laconic:libp2p-utils Transaction receipt for invoke message {
to: '0x558024C7d593B840E1BfD83E9B287a5CDad4db15',
blockNumber: 1996,
blockHash: '0xebef19c21269654804b2ef2d4bb5cb6c88743b37ed77e82222dc5671debf3afb',
transactionHash: '0xf8c5a093a93f793012196073a7d0cb3ed6fbd2846126c066cb31c72100960cb1',
effectiveGasPrice: '1500000007',
gasUsed: '250000'
}
```
* Check the phisher in watcher GQL: http://localhost:3001/graphql
* Use the blockHash from transaction receipt details or query for latest block
```gql
query {
latestBlock {
hash
number
}
}
```
* Get the deployed contract address
```bash
laconic-so --stack mobymask-v2 deploy-system --include watcher-mobymask-v2 exec mobymask-app "cat src/config.json"
```
The value of `address` field is the deployed contract address
* Check for phisher value
```gql
query {
isPhisher(
blockHash: "TX_OR_LATEST_BLOCK_HASH",
contractAddress: "CONTRACT_ADDRESS",
# If reported phisher name was "test" then key0 value is "TWT:test"
key0: "TWT:PHISHER_NAME"
) {
value
}
}
```
It should return true for reported phisher names.
* Watcher internally is using L2 chain `eth_getStorageAt` method.
* Check the phisher name in mobymask app in `Check Phisher Status` section.
* Watcher GQL API is used for checking phisher.
* Manage the invitations by clicking on the `Outstanding Invitations in p2p network`.
* Revoke the created invitation by clicking on `Revoke (p2p network)`
* Revocation messages can be seen in the debug panel `MESSAGES` tab of other browsers.
* Check the watcher peer logs. It should receive a message and log the transaction receipt for revoke message.
* Try reporting a phisher from the revoked invitee's browser.
* The invocation message for reporting phisher would be broadcasted to all peers.
* Check the watcher peer logs. A transaction failed error should be logged.
* Check the reported phisher in [watcher GQL](https://localhost:3001/graphql)
```gql
query {
isPhisher(
blockHash: "LATEST_BLOCK_HASH",
contractAddress: "CONTRACT_ADDRESS",
key0: "TWT:PHISHER_NAME"
) {
value
}
}
```
It should return false as the invitation/delegation used for reporting phishers has been revoked.
-17
View File
@@ -1,17 +0,0 @@
version: "1.0"
name: mobymask-v2
repos:
- cerc-io/watcher-ts
- cerc-io/react-peer
- cerc-io/mobymask-ui
- cerc-io/laconicd
- cerc-io/MobyMask
containers:
- cerc/watcher-mobymask-v2
- cerc/react-peer
- cerc/mobymask-ui
- cerc/laconicd
- cerc/mobymask
pods:
- mobymask-laconicd
- watcher-mobymask-v2
@@ -1,5 +0,0 @@
# Package Registry Stack
The Package Registry Stack supports a build environment that requires a package registry (initially for NPM packages only).
Setup instructions can be found [here](../build-support/README.md).
@@ -1,15 +0,0 @@
version: "1.1"
name: package-registry
decription: "Local Package Registry"
repos:
- cerc-io/hosting
- telackey/act_runner
containers:
- cerc/act-runner
- cerc/act-runner-task-executor
pods:
- name: gitea
repository: cerc-io/hosting
path: gitea
pre_start_command: "run-this-first.sh"
post_start_command: "initialize-gitea.sh"
+1 -1
View File
@@ -1,2 +1,2 @@
# This file should be re-generated running: scripts/update-version-file.sh script # This file should be re-generated running: scripts/update-version-file.sh script
v1.0.32-1aeb44d v1.0.16-ecf943b
+52 -214
View File
@@ -16,13 +16,9 @@
# Deploys the system components using docker-compose # Deploys the system components using docker-compose
import hashlib import hashlib
import copy
import os import os
import sys import sys
from dataclasses import dataclass from python_on_whales import DockerClient
from decouple import config
import subprocess
from python_on_whales import DockerClient, DockerException
import click import click
import importlib.resources import importlib.resources
from pathlib import Path from pathlib import Path
@@ -44,32 +40,63 @@ def command(ctx, include, exclude, cluster, command, extra_args):
debug = ctx.obj.debug debug = ctx.obj.debug
quiet = ctx.obj.quiet quiet = ctx.obj.quiet
verbose = ctx.obj.verbose verbose = ctx.obj.verbose
local_stack = ctx.obj.local_stack
dry_run = ctx.obj.dry_run dry_run = ctx.obj.dry_run
stack = ctx.obj.stack stack = ctx.obj.stack
cluster_context = _make_cluster_context(ctx.obj, include, exclude, cluster) # See: https://stackoverflow.com/questions/25389095/python-get-path-of-root-project-structure
compose_dir = Path(__file__).absolute().parent.joinpath("data", "compose")
if cluster is None:
# Create default unique, stable cluster name from confile file path
# TODO: change this to the config file path
path = os.path.realpath(sys.argv[0])
hash = hashlib.md5(path.encode()).hexdigest()
cluster = f"laconic-{hash}"
if verbose:
print(f"Using cluster name: {cluster}")
# See: https://stackoverflow.com/a/20885799/1701505
from . import data
with importlib.resources.open_text(data, "pod-list.txt") as pod_list_file:
all_pods = pod_list_file.read().splitlines()
pods_in_scope = []
if stack:
stack_config = get_parsed_stack_config(stack)
# TODO: syntax check the input here
pods_in_scope = stack_config['pods']
else:
pods_in_scope = all_pods
if verbose:
print(f"Pods: {pods_in_scope}")
# Construct a docker compose command suitable for our purpose
compose_files = []
for pod in pods_in_scope:
if include_exclude_check(pod, include, exclude):
compose_file_name = os.path.join(compose_dir, f"docker-compose-{pod}.yml")
compose_files.append(compose_file_name)
else:
if verbose:
print(f"Excluding: {pod}")
if verbose:
print(f"files: {compose_files}")
# See: https://gabrieldemarmiesse.github.io/python-on-whales/sub-commands/compose/ # See: https://gabrieldemarmiesse.github.io/python-on-whales/sub-commands/compose/
docker = DockerClient(compose_files=cluster_context.compose_files, compose_project_name=cluster_context.cluster) docker = DockerClient(compose_files=compose_files, compose_project_name=cluster)
extra_args_list = list(extra_args) or None extra_args_list = list(extra_args) or None
if not dry_run: if not dry_run:
if command == "up": if command == "up":
container_exec_env = _make_runtime_env(ctx.obj) if debug:
for attr, value in container_exec_env.items(): os.environ["CERC_SCRIPT_DEBUG"] = "true"
os.environ[attr] = value
if verbose: if verbose:
print(f"Running compose up with container_exec_env: {container_exec_env}, extra_args: {extra_args_list}") print(f"Running compose up for extra_args: {extra_args_list}")
for pre_start_command in cluster_context.pre_start_commands:
_run_command(ctx.obj, cluster_context.cluster, pre_start_command)
docker.compose.up(detach=True, services=extra_args_list) docker.compose.up(detach=True, services=extra_args_list)
for post_start_command in cluster_context.post_start_commands:
_run_command(ctx.obj, cluster_context.cluster, post_start_command)
_orchestrate_cluster_config(ctx.obj, cluster_context.config, docker, container_exec_env)
elif command == "down": elif command == "down":
if verbose: if verbose:
print("Running compose down") print("Running compose down")
@@ -79,14 +106,13 @@ def command(ctx, include, exclude, cluster, command, extra_args):
print("Usage: exec <service> <cmd>") print("Usage: exec <service> <cmd>")
sys.exit(1) sys.exit(1)
service_name = extra_args_list[0] service_name = extra_args_list[0]
command_to_exec = ["sh", "-c"] + extra_args_list[1:] command_to_exec = extra_args_list[1:]
container_exec_env = _make_runtime_env(ctx.obj) container_exec_env = {
"CERC_SCRIPT_DEBUG": "true"
} if debug else {}
if verbose: if verbose:
print(f"Running compose exec {service_name} {command_to_exec}") print(f"Running compose exec {service_name} {command_to_exec}")
try: docker.compose.execute(service_name, command_to_exec, envs=container_exec_env)
docker.compose.execute(service_name, command_to_exec, envs=container_exec_env)
except DockerException as error:
print(f"container command returned error exit status")
elif command == "port": elif command == "port":
if extra_args_list is None or len(extra_args_list) < 2: if extra_args_list is None or len(extra_args_list) < 2:
print("Usage: port <service> <exposed-port>") print("Usage: port <service> <exposed-port>")
@@ -121,192 +147,4 @@ def command(ctx, include, exclude, cluster, command, extra_args):
elif command == "logs": elif command == "logs":
if verbose: if verbose:
print("Running compose logs") print("Running compose logs")
logs_output = docker.compose.logs(services=extra_args_list if extra_args_list is not None else []) docker.compose.logs()
print(logs_output)
def get_stack_status(ctx, stack):
ctx_copy = copy.copy(ctx)
ctx_copy.stack = stack
cluster_context = _make_cluster_context(ctx_copy, None, None, None)
docker = DockerClient(compose_files=cluster_context.compose_files, compose_project_name=cluster_context.cluster)
# TODO: refactor to avoid duplicating this code above
if ctx.verbose:
print("Running compose ps")
container_list = docker.compose.ps()
if len(container_list) > 0:
if ctx.debug:
print(f"Container list from compose ps: {container_list}")
return True
else:
if ctx.debug:
print("No containers found from compose ps")
False
def _make_runtime_env(ctx):
container_exec_env = {
"CERC_HOST_UID": f"{os.getuid()}",
"CERC_HOST_GID": f"{os.getgid()}"
}
container_exec_env.update({"CERC_SCRIPT_DEBUG": "true"} if ctx.debug else {})
return container_exec_env
def _make_cluster_context(ctx, include, exclude, cluster):
if ctx.local_stack:
dev_root_path = os.getcwd()[0:os.getcwd().rindex("stack-orchestrator")]
print(f'Local stack dev_root_path (CERC_REPO_BASE_DIR) overridden to: {dev_root_path}')
else:
dev_root_path = os.path.expanduser(config("CERC_REPO_BASE_DIR", default="~/cerc"))
# See: https://stackoverflow.com/questions/25389095/python-get-path-of-root-project-structure
compose_dir = Path(__file__).absolute().parent.joinpath("data", "compose")
if cluster is None:
# Create default unique, stable cluster name from confile file path and stack name if provided
# TODO: change this to the config file path
path = os.path.realpath(sys.argv[0])
unique_cluster_descriptor = f"{path},{ctx.stack},{include},{exclude}"
if ctx.debug:
print(f"pre-hash descriptor: {unique_cluster_descriptor}")
hash = hashlib.md5(unique_cluster_descriptor.encode()).hexdigest()
cluster = f"laconic-{hash}"
if ctx.verbose:
print(f"Using cluster name: {cluster}")
# See: https://stackoverflow.com/a/20885799/1701505
from . import data
with importlib.resources.open_text(data, "pod-list.txt") as pod_list_file:
all_pods = pod_list_file.read().splitlines()
pods_in_scope = []
if ctx.stack:
stack_config = get_parsed_stack_config(ctx.stack)
# TODO: syntax check the input here
pods_in_scope = stack_config['pods']
cluster_config = stack_config['config'] if 'config' in stack_config else None
else:
pods_in_scope = all_pods
cluster_config = None
# Convert all pod definitions to v1.1 format
pods_in_scope = _convert_to_new_format(pods_in_scope)
if ctx.verbose:
print(f"Pods: {pods_in_scope}")
# Construct a docker compose command suitable for our purpose
compose_files = []
pre_start_commands = []
post_start_commands = []
for pod in pods_in_scope:
pod_name = pod["name"]
pod_repository = pod["repository"]
pod_path = pod["path"]
if include_exclude_check(pod_name, include, exclude):
if pod_repository is None or pod_repository == "internal":
compose_file_name = os.path.join(compose_dir, f"docker-compose-{pod_path}.yml")
else:
pod_root_dir = os.path.join(dev_root_path, pod_repository.split("/")[-1], pod["path"])
compose_file_name = os.path.join(pod_root_dir, "docker-compose.yml")
pod_pre_start_command = pod["pre_start_command"]
pod_post_start_command = pod["post_start_command"]
if pod_pre_start_command is not None:
pre_start_commands.append(os.path.join(pod_root_dir, pod_pre_start_command))
if pod_post_start_command is not None:
post_start_commands.append(os.path.join(pod_root_dir, pod_post_start_command))
compose_files.append(compose_file_name)
else:
if ctx.verbose:
print(f"Excluding: {pod_name}")
if ctx.verbose:
print(f"files: {compose_files}")
return cluster_context(cluster, compose_files, pre_start_commands, post_start_commands, cluster_config)
class cluster_context:
def __init__(self, cluster, compose_files, pre_start_commands, post_start_commands, config) -> None:
self.cluster = cluster
self.compose_files = compose_files
self.pre_start_commands = pre_start_commands
self.post_start_commands = post_start_commands
self.config = config
def _convert_to_new_format(old_pod_array):
new_pod_array = []
for old_pod in old_pod_array:
if isinstance(old_pod, dict):
new_pod_array.append(old_pod)
else:
new_pod = {
"name": old_pod,
"repository": "internal",
"path": old_pod
}
new_pod_array.append(new_pod)
return new_pod_array
def _run_command(ctx, cluster_name, command):
if ctx.verbose:
print(f"Running command: {command}")
command_dir = os.path.dirname(command)
command_file = os.path.join(".", os.path.basename(command))
command_env = os.environ.copy()
command_env["CERC_SO_COMPOSE_PROJECT"] = cluster_name
if ctx.debug:
command_env["CERC_SCRIPT_DEBUG"] = "true"
command_result = subprocess.run(command_file, shell=True, env=command_env, cwd=command_dir)
if command_result.returncode != 0:
print(f"FATAL Error running command: {command}")
sys.exit(1)
def _orchestrate_cluster_config(ctx, cluster_config, docker, container_exec_env):
@dataclass
class ConfigDirective:
source_container: str
source_variable: str
destination_container: str
destination_variable: str
if cluster_config is not None:
for container in cluster_config:
container_config = cluster_config[container]
if ctx.verbose:
print(f"{container} config: {container_config}")
for directive in container_config:
pd = ConfigDirective(
container_config[directive].split(".")[0],
container_config[directive].split(".")[1],
container,
directive
)
if ctx.verbose:
print(f"Setting {pd.destination_container}.{pd.destination_variable} = {pd.source_container}.{pd.source_variable}")
# TODO: fix the script paths so they're consistent between containers
source_value = docker.compose.execute(pd.source_container,
["sh", "-c",
f"sh /docker-entrypoint-scripts.d/export-{pd.source_variable}.sh"],
tty=False,
envs=container_exec_env)
# TODO: handle the case that the value is not yet available
if ctx.debug:
print(f"fetched source value: {source_value}")
destination_output = docker.compose.execute(pd.destination_container,
["sh", "-c",
f"sh /scripts/import-{pd.destination_variable}.sh {source_value}"],
tty=False,
envs=container_exec_env)
if ctx.debug:
print(f"destination output: {destination_output}")
# TODO: detect errors here
-1
View File
@@ -55,6 +55,5 @@ def cli(ctx, stack, quiet, verbose, dry_run, local_stack, debug, continue_on_err
cli.add_command(setup_repositories.command, "setup-repositories") cli.add_command(setup_repositories.command, "setup-repositories")
cli.add_command(build_containers.command, "build-containers") cli.add_command(build_containers.command, "build-containers")
cli.add_command(build_npms.command, "build-npms") cli.add_command(build_npms.command, "build-npms")
cli.add_command(deploy_system.command, "deploy") # deploy is an alias for deploy-system
cli.add_command(deploy_system.command, "deploy-system") cli.add_command(deploy_system.command, "deploy-system")
cli.add_command(version.command, "version") cli.add_command(version.command, "version")
+1 -1
View File
@@ -1,6 +1,6 @@
python-decouple>=3.6 python-decouple>=3.6
GitPython>=3.1.27 GitPython>=3.1.27
tqdm>=4.64.0 tqdm>=4.64.0
python-on-whales>=0.58.0 python-on-whales>=0.52.0
click>=8.1.3 click>=8.1.3
pyyaml>=6.0 pyyaml>=6.0
-7
View File
@@ -24,11 +24,4 @@ $TEST_TARGET_SO --stack test deploy-system up
# TODO: test that we can use the deployed container somehow # TODO: test that we can use the deployed container somehow
# Clean up # Clean up
$TEST_TARGET_SO --stack test deploy-system down $TEST_TARGET_SO --stack test deploy-system down
# Run same test but not using the stack definition
# Test building the a stack container
$TEST_TARGET_SO build-containers --include cerc/test-container
# Deploy the test container
$TEST_TARGET_SO deploy-system --include test up
# Clean up
$TEST_TARGET_SO deploy-system --include test down
echo "Test passed" echo "Test passed"