Merge pull request #253 from cerc-io/dboreham/wait-for-export-cluster-config

Detect transient errors exporting variables and re-try Former-commit-id: a4a9607ea8
Detect transient errors exporting variables and re-try
2023-03-28 11:44:54 -06:00 · 2023-03-28 11:44:02 -06:00 · 2023-03-28 10:47:07 -06:00 · 2023-03-28 10:46:26 -06:00 · 2023-03-28 09:53:22 -06:00 · 2023-03-28 09:52:34 -06:00
104 changed files with 3332 additions and 274 deletions
@@ -8,11 +8,11 @@ Stack Orchestrator allows building and deployment of a Laconic Stack on a single

 Ensure that the following are already installed:

- [Python3](https://wiki.python.org/moin/BeginnersGuide/Download)
- [Docker](https://docs.docker.com/get-docker/) 
- [Docker Compose](https://docs.docker.com/compose/install/)
+- [Python3](https://wiki.python.org/moin/BeginnersGuide/Download): `python3 --version` >= `3.8.10` (the Python3 shipped in Ubuntu 20+ is good to go)
+- [Docker](https://docs.docker.com/get-docker/): `docker --version` >= `20.10.21`
+- [jq](https://stedolan.github.io/jq/download/): `jq --version` >= `1.5`

-Note: if installing docker-compose via package manager (as opposed to Docker Desktop), you must [install the plugin](https://docs.docker.com/compose/install/linux/#install-the-plugin-manually), e.g., on Linux:
+Note: if installing docker-compose via package manager on Linux (as opposed to Docker Desktop), you must [install the plugin](https://docs.docker.com/compose/install/linux/#install-the-plugin-manually), e.g. :

 ```bash
 mkdir -p ~/.docker/cli-plugins
@@ -20,39 +20,27 @@ curl -SL https://github.com/docker/compose/releases/download/v2.11.2/docker-comp
 chmod +x ~/.docker/cli-plugins/docker-compose
 ```

-Next, download the latest release from [this page](https://github.com/cerc-io/stack-orchestrator/tags), into a suitable directory (e.g. `~/bin`):
+Next decide on a directory where you would like to put the stack-orchestrator program. Typically this would be 
+a "user" binary directory such as `~/bin` or perhaps `/usr/local/laconic` or possibly just the current working directory.
+
+Now, having selected that directory, download the latest release from [this page](https://github.com/cerc-io/stack-orchestrator/tags) into it (we're using `~/bin` below for concreteness but edit to suit if you selected a different directory). Also be sure that the destination directory exists and is writable:

 ```bash
 curl -L -o ~/bin/laconic-so https://github.com/cerc-io/stack-orchestrator/releases/latest/download/laconic-so
 ```

-Give it permissions:
+Give it execute permissions:
 ```bash
 chmod +x ~/bin/laconic-so
 ```

 Ensure `laconic-so` is on the [`PATH`](https://unix.stackexchange.com/a/26059)

-Verify operation:
+Verify operation (your version will probably be different, just check here that you see some version outut and not an error):

 ```
-laconic-so --help
-Usage: python -m laconic-so [OPTIONS] COMMAND [ARGS]...
-
-  Laconic Stack Orchestrator
-
-Options:
-  --quiet
-  --verbose
-  --dry-run
-  --local-stack
-  -h, --help     Show this message and exit.
-
-Commands:
-  build-containers    build the set of containers required for a complete...
-  build-npms          build the set of npm packages required for a...  
-  deploy-system       deploy a stack
-  setup-repositories  git clone the set of repositories required to build...
+laconic-so version
+Version: v1.0.27-7831078
 ```

 ## Usage
@@ -64,25 +52,25 @@ Three sub-commands: `setup-repositories`, `build-containers` and `deploy-system`
 Clone the set of git repositories necessary to build a system:

 ```bash
-laconic-so --verbose setup-repositories --include cerc-io/go-ethereum,cerc-io/ipld-eth-db,cerc-io/ipld-eth-server,cerc-io/watcher-ts
+laconic-so --stack erc20 setup-repositories
 ```

-This will default to `~/cerc` or - if set - the environment variable `CERC_REPO_BASE_DIR`
+This will default to cloning git reposiories into: `~/cerc` or - if set - the environment variable `CERC_REPO_BASE_DIR`

 ### Build Containers

 Build the set of docker container images required to run a system. It takes around 10 minutes to build all the containers from scratch.

 ```bash
-laconic-so --verbose build-containers --include cerc/go-ethereum,cerc/go-ethereum-foundry,cerc/ipld-eth-db,cerc/ipld-eth-server,cerc/watcher-erc20
+laconic-so --stack erc20 build-containers
 ```

 ### Deploy System

-Uses `docker-compose` to deploy a system (with most recently built container images).
+Uses `docker compose` to deploy a system (with most recently built container images).

 ```bash
-laconic-so --verbose deploy-system --include ipld-eth-db,go-ethereum-foundry,ipld-eth-server,watcher-erc20 up
+laconic-so --stack erc20 deploy-system up
 ```

 Check out he GraphQL playground here: [http://localhost:3002/graphql](http://localhost:3002/graphql)
@@ -92,12 +80,12 @@ See the [erc20 watcher demo](/app/data/stacks/erc20) to continue further.
 ### Cleanup

 ```bash
-laconic-so --verbose deploy-system --include ipld-eth-db,go-ethereum-foundry,ipld-eth-server,watcher-erc20 down
+laconic-so --stack erc20 deploy-system down
 ```

 ## Contributing

-See the [CONTRIBUTING.md](.github/CONTRIBUTING.md) for developer mode install.
+See the [CONTRIBUTING.md](/docs/CONTRIBUTING.md) for developer mode install.

 ## Platform Support

@@ -0,0 +1,71 @@
+# Copyright © 2022, 2023 Cerc
+
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http:#www.gnu.org/licenses/>.
+
+import os
+from abc import ABC, abstractmethod
+from .deploy_system import get_stack_status
+
+
+def get_stack(config, stack):
+    if stack == "package-registry":
+        return package_registry_stack(config, stack)
+    else:
+        return base_stack(config, stack)
+
+
+class base_stack(ABC):
+
+    def __init__(self, config, stack):
+        self.config = config
+        self.stack = stack
+
+    @abstractmethod
+    def ensure_available(self):
+        pass
+
+    @abstractmethod
+    def get_url(self):
+        pass
+
+
+class package_registry_stack(base_stack):
+
+    def ensure_available(self):
+        self.url = "<no registry url set>"
+        # Check if we were given an external registry URL
+        url_from_environment = os.environ.get("CERC_NPM_REGISTRY_URL")
+        if url_from_environment:
+            if self.config.verbose:
+                print(f"Using package registry url from CERC_NPM_REGISTRY_URL: {url_from_environment}")
+            self.url = url_from_environment
+        else:
+            # Otherwise we expect to use the local package-registry stack
+            # First check if the stack is up
+            registry_running = get_stack_status(self.config, "package-registry")
+            if registry_running:
+                # If it is available, get its mapped port and construct its URL
+                if self.config.debug:
+                    print("Found local package registry stack is up")
+                # TODO: get url from deploy-stack
+                self.url = "http://gitea.local:3000/api/packages/cerc-io/npm/"
+            else:
+                # If not, print a message about how to start it and return fail to the caller
+                print("ERROR: The package-registry stack is not running, and no external registry specified with CERC_NPM_REGISTRY_URL")
+                print("ERROR: Start the local package registry with: laconic-so --stack package-registry deploy-system up")
+                return False
+        return True
+
+    def get_url(self):
+        return self.url
@@ -83,7 +83,10 @@ def command(ctx, include, exclude):
    container_build_env = {
        "CERC_NPM_URL": "http://gitea.local:3000/api/packages/cerc-io/npm/",
        "CERC_NPM_AUTH_TOKEN": config("CERC_NPM_AUTH_TOKEN", default="<token-not-supplied>"),
-        "CERC_REPO_BASE_DIR": dev_root_path
+        "CERC_REPO_BASE_DIR": dev_root_path,
+        "CERC_HOST_UID": f"{os.getuid()}",
+        "CERC_HOST_GID": f"{os.getgid()}",
+        "DOCKER_BUILDKIT": "0"
    }

    def process_container(container):
@@ -103,10 +106,10 @@ def command(ctx, include, exclude):
            # Check if we have a repo for this container. If not, set the context dir to the container-build subdir
            repo_full_path = os.path.join(dev_root_path, repo_dir)
            repo_dir_or_build_dir = repo_dir if os.path.exists(repo_full_path) else build_dir
-            build_command = os.path.join(container_build_dir, "default-build.sh") + f" {container} {repo_dir_or_build_dir}"
+            build_command = os.path.join(container_build_dir, "default-build.sh") + f" {container}:local {repo_dir_or_build_dir}"
        if not dry_run:
            if verbose:
-                print(f"Executing: {build_command}")
+                print(f"Executing: {build_command} with environment: {container_build_env}")
            build_result = subprocess.run(build_command, shell=True, env=container_build_env)
            if verbose:
                print(f"Return code is: {build_result.returncode}")
@@ -20,12 +20,16 @@

 import os
 import sys
+from shutil import rmtree, copytree
 from decouple import config
 import click
 import importlib.resources
 from python_on_whales import docker, DockerException
+from .base import get_stack
 from .util import include_exclude_check, get_parsed_stack_config

+builder_js_image_name = "cerc/builder-js:local"
+
@click.command()
@click.option('--include', help="only build these packages")
@click.option('--exclude', help="don\'t build these packages")
@@ -41,17 +45,38 @@ def command(ctx, include, exclude):
    stack = ctx.obj.stack
    continue_on_error = ctx.obj.continue_on_error

+    _ensure_prerequisites()
+
+    # build-npms depends on having access to a writable package registry
+    # so we check here that it is available
+    package_registry_stack = get_stack(ctx.obj, "package-registry")
+    registry_available = package_registry_stack.ensure_available()
+    if not registry_available:
+        print("FATAL: no npm registry available for build-npms command")
+        sys.exit(1)
+    npm_registry_url = package_registry_stack.get_url()
+    npm_registry_url_token = config("CERC_NPM_AUTH_TOKEN", default=None)
+    if not npm_registry_url_token:
+        print("FATAL: CERC_NPM_AUTH_TOKEN is not defined")
+        sys.exit(1)
+
    if local_stack:
        dev_root_path = os.getcwd()[0:os.getcwd().rindex("stack-orchestrator")]
        print(f'Local stack dev_root_path (CERC_REPO_BASE_DIR) overridden to: {dev_root_path}')
    else:
        dev_root_path = os.path.expanduser(config("CERC_REPO_BASE_DIR", default="~/cerc"))

-    if not quiet:
+    build_root_path = os.path.join(dev_root_path, "build-trees")
+
+    if verbose:
        print(f'Dev Root is: {dev_root_path}')

    if not os.path.isdir(dev_root_path):
        print('Dev root directory doesn\'t exist, creating')
+        os.makedirs(dev_root_path)
+    if not os.path.isdir(dev_root_path):
+        print('Build root directory doesn\'t exist, creating')
+        os.makedirs(build_root_path)

    # See: https://stackoverflow.com/a/20885799/1701505
    from . import data
@@ -74,21 +99,38 @@ def command(ctx, include, exclude):
            print(f"Building npm package: {package}")
        repo_dir = package
        repo_full_path = os.path.join(dev_root_path, repo_dir)
-        # TODO: make the npm registry url configurable.
-        build_command = ["sh", "-c", "cd /workspace && build-npm-package-local-dependencies.sh http://gitea.local:3000/api/packages/cerc-io/npm/"]
+        # Copy the repo and build that to avoid propagating JS tooling file changes back into the cloned repo
+        repo_copy_path = os.path.join(build_root_path, repo_dir)
+        # First delete any old build tree
+        if os.path.isdir(repo_copy_path):
+            if verbose:
+                print(f"Deleting old build tree: {repo_copy_path}")
+            if not dry_run:
+                rmtree(repo_copy_path)
+        # Now copy the repo into the build tree location
+        if verbose:
+            print(f"Copying build tree from: {repo_full_path} to: {repo_copy_path}")
+        if not dry_run:
+            copytree(repo_full_path, repo_copy_path)
+        build_command = ["sh", "-c", f"cd /workspace && build-npm-package-local-dependencies.sh {npm_registry_url}"]
        if not dry_run:
            if verbose:
                print(f"Executing: {build_command}")
-            envs = {"CERC_NPM_AUTH_TOKEN": os.environ["CERC_NPM_AUTH_TOKEN"]} | ({"CERC_SCRIPT_DEBUG": "true"} if debug else {})
+            # Originally we used the PEP 584 merge operator:
+            # envs = {"CERC_NPM_AUTH_TOKEN": npm_registry_url_token} | ({"CERC_SCRIPT_DEBUG": "true"} if debug else {})
+            # but that isn't available in Python 3.8 (default in Ubuntu 20) so for now we use dict.update:
+            envs = {"CERC_NPM_AUTH_TOKEN": npm_registry_url_token}
+            envs.update({"CERC_SCRIPT_DEBUG": "true"} if debug else {})
            try:
-                docker.run("cerc/builder-js",
+                docker.run(builder_js_image_name,
                           remove=True,
                           interactive=True,
                           tty=True,
                           user=f"{os.getuid()}:{os.getgid()}",
                           envs=envs,
+                           # TODO: detect this host name in npm_registry_url rather than hard-wiring it
                           add_hosts=[("gitea.local", "host-gateway")],
-                           volumes=[(repo_full_path, "/workspace")],
+                           volumes=[(repo_copy_path, "/workspace")],
                           command=build_command
                           )
                # Note that although the docs say that build_result should contain
@@ -111,3 +153,13 @@ def command(ctx, include, exclude):
        else:
            if verbose:
                print(f"Excluding: {package}")
+
+
+def _ensure_prerequisites():
+    # Check that the builder-js container is available and
+    # Tell the user how to build it if not
+    images = docker.image.list(builder_js_image_name)
+    if len(images) == 0:
+        print(f"FATAL: builder image: {builder_js_image_name} is required but was not found")
+        print("Please run this command to create it: laconic-so --stack build-support build-containers")
+        sys.exit(1)
@@ -0,0 +1,23 @@
+version: "3.2"
+services:
+  prometheus:
+    restart: always
+    image: prom/prometheus
+    depends_on:
+      fixturenet-eth-geth-1:
+        condition: service_healthy
+    volumes:
+      - ../config/fixturenet-eth-metrics/prometheus/etc:/etc/prometheus
+    ports:
+      - "9090"
+  grafana:
+    restart: always
+    image: grafana/grafana
+    environment:
+      - GF_SECURITY_ADMIN_PASSWORD=changeme6325
+    volumes:
+      - ../config/fixturenet-eth-metrics/grafana/etc/provisioning/dashboards:/etc/grafana/provisioning/dashboards
+      - ../config/fixturenet-eth-metrics/grafana/etc/provisioning/datasources:/etc/grafana/provisioning/datasources
+      - ../config/fixturenet-eth-metrics/grafana/etc/dashboards:/etc/grafana/dashboards
+    ports:
+      - "3000"
@@ -20,6 +20,7 @@ services:
      CERC_REMOTE_DEBUG: "true"
      CERC_RUN_STATEDIFF: "detect"
      CERC_STATEDIFF_DB_NODE_ID: 1
+      CERC_SCRIPT_DEBUG: ${CERC_SCRIPT_DEBUG}
    env_file:
      - ../config/fixturenet-eth/fixturenet-eth.env
    image: cerc/fixturenet-eth-geth:local
@@ -5,8 +5,9 @@ services:
    image: cerc/laconicd:local
    command: ["sh", "/docker-entrypoint-scripts.d/create-fixturenet.sh"]
    volumes:
-      # TODO: look at folding this script into the container
+      # TODO: look at folding these scripts into the container
      - ../config/fixturenet-laconicd/create-fixturenet.sh:/docker-entrypoint-scripts.d/create-fixturenet.sh
+      - ../config/fixturenet-laconicd/export-mykey.sh:/docker-entrypoint-scripts.d/export-mykey.sh
    # TODO: determine which of the ports below is really needed
    ports:
      - "6060"
@@ -18,4 +19,7 @@ services:
      - "9090"
      - "9091"
      - "1317"
-
+  cli:
+    image: cerc/laconic-registry-cli:local
+    volumes:
+      - ../config/fixturenet-laconicd/registry-cli-config-template.yml:/registry-cli-config-template.yml
@@ -0,0 +1,8 @@
+# Add-on pod to include foundry tooling within a fixturenet
+services:
+  foundry:
+    image: cerc/foundry:local
+    command: ["while :; do sleep 600; done"]
+    volumes:
+      - ../config/foundry/foundry.toml:/foundry.toml
+      - ./foundry/workspace:/workspace
@@ -8,7 +8,7 @@ services:
        condition: service_healthy
    image: cerc/go-ethereum-foundry:local
    healthcheck:
-      test: ["CMD", "nc", "-v", "localhost", "8545"]
+      test: ["CMD", "nc", "-vz", "localhost", "8545"]
      interval: 30s
      timeout: 3s
      retries: 10
@@ -20,16 +20,24 @@ services:
      DATABASE_USER: "vdbm"
      DATABASE_PASSWORD: "password"
      ETH_CHAIN_ID: 99
-      ETH_FORWARD_ETH_CALLS: $eth_forward_eth_calls
-      ETH_PROXY_ON_ERROR: $eth_proxy_on_error
-      ETH_HTTP_PATH: $eth_http_path
+      ETH_FORWARD_ETH_CALLS: "false"
+      ETH_FORWARD_GET_STORAGE_AT: "false"
+      ETH_PROXY_ON_ERROR: "false"
+      METRICS: "true"
+      PROM_HTTP: "true"
+      PROM_HTTP_ADDR: "0.0.0.0"
+      PROM_HTTP_PORT: "8090"
+      LOGRUS_LEVEL: "debug"
+      CERC_REMOTE_DEBUG: "true"
    volumes:
      - type: bind
        source: ../config/ipld-eth-server/chain.json
        target: /tmp/chain.json
    ports:
-      - "127.0.0.1:8081:8081"
-      - "127.0.0.1:8082:8082"
+      - "8081"
+      - "8082"
+      - "8090"
+      - "40000"
    healthcheck:
      test: ["CMD", "nc", "-v", "localhost", "8081"]
      interval: 20s
@@ -29,9 +29,17 @@ services:
        condition: service_healthy
  keycloak-nginx:
    image: nginx:1.23-alpine
+    restart: always
    volumes:
      - ../config/keycloak/nginx:/etc/nginx/conf.d
    ports:
      - 80
    depends_on:
      - keycloak
+  keycloak-nginx-prometheus-exporter:
+    image: nginx/nginx-prometheus-exporter
+    restart: always
+    environment:
+      - SCRAPE_URI=http://keycloak-nginx:80/stub_status
+    depends_on:
+      - keycloak-nginx
@@ -0,0 +1,13 @@
+version: "3.2"
+# See: https://docs.ipfs.tech/install/run-ipfs-inside-docker/#set-up
+services:
+  ipfs:
+    image: ipfs/kubo:master-2023-02-20-714a968
+    restart: always
+    volumes:
+      - ./ipfs/import:/import
+      - ./ipfs/data:/data/ipfs
+    ports:
+      - "8080"
+      - "4001"
+      - "5001"
@@ -0,0 +1,25 @@
+version: "3.2"
+
+services:
+  laconicd:
+    restart: unless-stopped
+    image: cerc/laconicd:local
+    command: ["sh", "/docker-entrypoint-scripts.d/create-fixturenet.sh"]
+    volumes:
+      - ../config/fixturenet-laconicd/create-fixturenet.sh:/docker-entrypoint-scripts.d/create-fixturenet.sh
+    ports:
+      - "9473"
+      - "8545"
+      - "8546"
+      - "1317"
+    healthcheck:
+      test: ["CMD", "nc", "-v", "localhost", "8545"]
+      interval: 20s
+      timeout: 5s
+      retries: 15
+      start_period: 10s
+
+networks:
+  # https://docs.docker.com/compose/networking/#configure-the-default-network
+  default:
+    name: mobymask-v2-network
@@ -1,21 +0,0 @@
-version: "3.2"
-services:
-  # If you want prometheus to work, you must update the following file in the ops repo locally.
-  # localhost:6060 --> go-ethereum:6060
-  prometheus:
-    restart: always
-    user: "987"
-    image: prom/prometheus
-    volumes:
-      - ${cerc_ops}/metrics/etc:/etc/prometheus
-      - ./prometheus-data:/prometheus
-    ports:
-      - "127.0.0.1:9090:9090"
-  grafana:
-    restart: always
-    user: "472"
-    image: grafana/grafana
-    volumes:
-      - ./grafana-data:/var/lib/grafana
-    ports:
-      - "127.0.0.1:3000:3000"
@@ -3,3 +3,5 @@ services:
  test:
    image: cerc/test-container:local
    restart: always
+    ports:
+      - "80"
@@ -39,7 +39,7 @@ services:
      - "0.0.0.0:3002:3001"
      - "0.0.0.0:9002:9001"
    healthcheck:
-      test: ["CMD", "nc", "-v", "localhost", "3002"]
+      test: ["CMD", "nc", "-vz", "localhost", "3001"]
      interval: 20s
      timeout: 5s
      retries: 15
@@ -0,0 +1,130 @@
+version: '3.2'
+
+services:
+  mobymask-watcher-db:
+    restart: unless-stopped
+    image: postgres:14-alpine
+    environment:
+      - POSTGRES_USER=vdbm
+      - POSTGRES_MULTIPLE_DATABASES=mobymask-watcher,mobymask-watcher-job-queue
+      - POSTGRES_EXTENSION=mobymask-watcher-job-queue:pgcrypto
+      - POSTGRES_PASSWORD=password
+    volumes:
+      - ../config/postgresql/multiple-postgressql-databases.sh:/docker-entrypoint-initdb.d/multiple-postgressql-databases.sh
+      - mobymask_watcher_db_data:/var/lib/postgresql/data
+    ports:
+      - "0.0.0.0:15432:5432"
+    healthcheck:
+      test: ["CMD", "nc", "-v", "localhost", "5432"]
+      interval: 20s
+      timeout: 5s
+      retries: 15
+      start_period: 10s
+
+  mobymask:
+    restart: unless-stopped
+    depends_on:
+      mobymask-watcher-db:
+        condition: service_healthy
+    image: cerc/mobymask:local
+    working_dir: /app/packages/server
+    environment:
+      - ENV=PROD
+    command: ["sh", "-c", "npm start"]
+    volumes:
+      - ../config/watcher-mobymask-v2/secrets.json:/app/packages/server/secrets.json
+      - moby_data_server:/app/packages/server
+    healthcheck:
+      test: ["CMD", "nc", "-v", "localhost", "3330"]
+      interval: 20s
+      timeout: 5s
+      retries: 15
+      start_period: 10s
+
+  mobymask-watcher-server:
+    restart: unless-stopped
+    depends_on:
+      mobymask-watcher-db:
+        condition: service_healthy
+    image: cerc/watcher-mobymask-v2:local
+    command: ["sh", "-c", "yarn server"]
+    volumes:
+      - ../config/watcher-mobymask-v2/watcher.toml:/app/packages/mobymask-v2-watcher/environments/local.toml
+      - ../config/watcher-mobymask-v2/peer.env:/app/packages/peer/.env
+      - ../config/watcher-mobymask-v2/relay-id.json:/app/packages/mobymask-v2-watcher/relay-id.json
+    ports:
+      - "0.0.0.0:3001:3001"
+      - "0.0.0.0:9001:9001"
+      - "0.0.0.0:9090:9090"
+    healthcheck:
+      test: ["CMD", "busybox", "nc", "localhost", "9090"]
+      interval: 20s
+      timeout: 5s
+      retries: 15
+      start_period: 5s
+
+  mobymask-watcher-peer:
+    restart: unless-stopped
+    depends_on:
+      mobymask:
+        condition: service_healthy
+    image: cerc/watcher-mobymask-v2:local
+    command: ["sh", "peer-start.sh"]
+    volumes:
+      - ../config/watcher-mobymask-v2/watcher.toml:/app/packages/mobymask-v2-watcher/environments/local.toml
+      - ../config/watcher-mobymask-v2/peer-id.json:/app/packages/mobymask-v2-watcher/peer-id.json
+      - ../config/watcher-mobymask-v2/peer-start.sh:/app/packages/mobymask-v2-watcher/peer-start.sh
+      - moby_data_server:/server
+
+  mobymask-app:
+    depends_on:
+      mobymask-watcher-server:
+        condition: service_healthy
+      mobymask:
+        condition: service_healthy
+    image: cerc/mobymask-ui:local
+    command: ["sh", "mobymask-app-start.sh"]
+    volumes:
+      - ../config/watcher-mobymask-v2/mobymask-app.env:/app/.env
+      - ../config/watcher-mobymask-v2/mobymask-app-config.json:/app/src/mobymask-app-config.json
+      - ../config/watcher-mobymask-v2/mobymask-app-start.sh:/app/mobymask-app-start.sh
+      - moby_data_server:/server
+    ports:
+      - "0.0.0.0:3002:3000"
+    healthcheck:
+      test: ["CMD", "nc", "-v", "localhost", "3000"]
+      interval: 20s
+      timeout: 5s
+      retries: 15
+      start_period: 10s
+    shm_size: '1GB'
+
+  peer-test-app:
+    depends_on:
+      mobymask-watcher-server:
+        condition: service_healthy
+    image: cerc/react-peer:local
+    working_dir: /app/packages/test-app
+    command: ["sh", "-c", "yarn build && serve -s build"]
+    volumes:
+      - ../config/watcher-mobymask-v2/test-app-config.json:/app/packages/test-app/src/config.json
+    ports:
+      - "0.0.0.0:3003:3000"
+    healthcheck:
+      test: ["CMD", "nc", "-v", "localhost", "3000"]
+      interval: 20s
+      timeout: 5s
+      retries: 15
+      start_period: 10s
+
+volumes:
+  mobymask_watcher_db_data:
+  moby_data_server:
+
+networks:
+  # https://docs.docker.com/compose/networking/#configure-the-default-network
+  default:
+    # https://docs.docker.com/compose/networking/#use-a-pre-existing-network
+    name: mobymask-v2-network
+    external: true
+
@@ -0,0 +1,9 @@
+apiVersion: 1
+
+providers:
+  - name: dashboards
+    type: file
+    updateIntervalSeconds: 10
+    options:
+      path: /etc/grafana/dashboards
+      foldersFromFilesStructure: true
@@ -0,0 +1,19 @@
+apiVersion: 1
+
+datasources:
+  - id: 1
+    uid: jZUuGao4k
+    orgId: 1
+    name: Prometheus
+    type: prometheus
+    typeName: Prometheus
+    typeLogoUrl: public/app/plugins/datasource/prometheus/img/prometheus_logo.svg
+    access: proxy
+    url: http://prometheus:9090
+    user: ""
+    database: ""
+    basicAuth: false
+    isDefault: true
+    jsonData:
+      httpMethod: POST
+    readOnly: false
@@ -0,0 +1,34 @@
+global:
+  scrape_interval: 5s
+  evaluation_interval: 15s
+
+scrape_configs:
+  # ipld-eth-server
+  - job_name: 'ipld-eth-server'
+    metrics_path: /metrics
+    scrape_interval: 5s
+    static_configs:
+    - targets: ['ipld-eth-server:8090']
+
+  # geth
+  - job_name: 'geth'
+    metrics_path: /debug/metrics/prometheus
+    scheme: http
+    static_configs:
+    - targets: ['fixturenet-eth-geth-1:6060']
+
+  # nginx
+  - job_name: 'nginx'
+    scrape_interval: 5s
+    metrics_path: /metrics
+    scheme: http
+    static_configs:
+      - targets: ['keycloak-nginx-prometheus-exporter:9113']
+
+  # keycloak
+  - job_name: 'keycloak'
+    scrape_interval: 5s
+    metrics_path: /auth/realms/cerc/metrics
+    scheme: http
+    static_configs:
+      - targets: ['keycloak:8080']
@@ -19,3 +19,5 @@ CERC_STATEDIFF_DB_USER="vdbm"
 CERC_STATEDIFF_DB_PASSWORD="password"
 CERC_STATEDIFF_DB_GOOSE_MIN_VER=23
 CERC_STATEDIFF_DB_LOG_STATEMENTS="false"
+
+CERC_GETH_VMODULE="statediff/*=5,rpc/*=5"
@@ -1,8 +1,8 @@
-#!/bin/sh
-
-# Originally from: https://github.com/cerc-io/laconicd/blob/main/init.sh
-# TODO: fold this back into the laconicd repo
+#!/bin/bash

+# TODO: this file is now an unmodified copy of cerc-io/laconicd/init.sh
+# so we should have a mechanism to bundle it inside the container rather than link from here
+# at deploy time.

 KEY="mykey"
 CHAINID="laconic_9000-1"
@@ -10,7 +10,7 @@ MONIKER="localtestnet"
 KEYRING="test"
 KEYALGO="eth_secp256k1"
 LOGLEVEL="info"
-# to trace evm
+# trace evm
 TRACE="--trace"
 # TRACE=""

@@ -28,7 +28,7 @@ laconicd config chain-id $CHAINID
 # if $KEY exists it should be deleted
 laconicd keys add $KEY --keyring-backend $KEYRING --algo $KEYALGO

-# Set moniker and chain-id for laconic (Moniker can be anything, chain-id must be an integer)
+# Set moniker and chain-id for Ethermint (Moniker can be anything, chain-id must be an integer)
 laconicd init $MONIKER --chain-id $CHAINID

 # Change parameter token denominations to aphoton
@@ -37,28 +37,28 @@ cat $HOME/.laconicd/config/genesis.json | jq '.app_state["crisis"]["constant_fee
 cat $HOME/.laconicd/config/genesis.json | jq '.app_state["gov"]["deposit_params"]["min_deposit"][0]["denom"]="aphoton"' > $HOME/.laconicd/config/tmp_genesis.json && mv $HOME/.laconicd/config/tmp_genesis.json $HOME/.laconicd/config/genesis.json
 cat $HOME/.laconicd/config/genesis.json | jq '.app_state["mint"]["params"]["mint_denom"]="aphoton"' > $HOME/.laconicd/config/tmp_genesis.json && mv $HOME/.laconicd/config/tmp_genesis.json $HOME/.laconicd/config/genesis.json
 # Custom modules
-cat $HOME/.laconicd/config/genesis.json | jq '.app_state["nameservice"]["params"]["record_rent"]["denom"]="aphoton"' > $HOME/.laconicd/config/tmp_genesis.json && mv $HOME/.laconicd/config/tmp_genesis.json $HOME/.laconicd/config/genesis.json
-cat $HOME/.laconicd/config/genesis.json | jq '.app_state["nameservice"]["params"]["authority_rent"]["denom"]="aphoton"' > $HOME/.laconicd/config/tmp_genesis.json && mv $HOME/.laconicd/config/tmp_genesis.json $HOME/.laconicd/config/genesis.json
-cat $HOME/.laconicd/config/genesis.json | jq '.app_state["nameservice"]["params"]["authority_auction_commit_fee"]["denom"]="aphoton"' > $HOME/.laconicd/config/tmp_genesis.json && mv $HOME/.laconicd/config/tmp_genesis.json $HOME/.laconicd/config/genesis.json
-cat $HOME/.laconicd/config/genesis.json | jq '.app_state["nameservice"]["params"]["authority_auction_reveal_fee"]["denom"]="aphoton"' > $HOME/.laconicd/config/tmp_genesis.json && mv $HOME/.laconicd/config/tmp_genesis.json $HOME/.laconicd/config/genesis.json
-cat $HOME/.laconicd/config/genesis.json | jq '.app_state["nameservice"]["params"]["authority_auction_minimum_bid"]["denom"]="aphoton"' > $HOME/.laconicd/config/tmp_genesis.json && mv $HOME/.laconicd/config/tmp_genesis.json $HOME/.laconicd/config/genesis.json
+cat $HOME/.laconicd/config/genesis.json | jq '.app_state["registry"]["params"]["record_rent"]["denom"]="aphoton"' > $HOME/.laconicd/config/tmp_genesis.json && mv $HOME/.laconicd/config/tmp_genesis.json $HOME/.laconicd/config/genesis.json
+cat $HOME/.laconicd/config/genesis.json | jq '.app_state["registry"]["params"]["authority_rent"]["denom"]="aphoton"' > $HOME/.laconicd/config/tmp_genesis.json && mv $HOME/.laconicd/config/tmp_genesis.json $HOME/.laconicd/config/genesis.json
+cat $HOME/.laconicd/config/genesis.json | jq '.app_state["registry"]["params"]["authority_auction_commit_fee"]["denom"]="aphoton"' > $HOME/.laconicd/config/tmp_genesis.json && mv $HOME/.laconicd/config/tmp_genesis.json $HOME/.laconicd/config/genesis.json
+cat $HOME/.laconicd/config/genesis.json | jq '.app_state["registry"]["params"]["authority_auction_reveal_fee"]["denom"]="aphoton"' > $HOME/.laconicd/config/tmp_genesis.json && mv $HOME/.laconicd/config/tmp_genesis.json $HOME/.laconicd/config/genesis.json
+cat $HOME/.laconicd/config/genesis.json | jq '.app_state["registry"]["params"]["authority_auction_minimum_bid"]["denom"]="aphoton"' > $HOME/.laconicd/config/tmp_genesis.json && mv $HOME/.laconicd/config/tmp_genesis.json $HOME/.laconicd/config/genesis.json

-if [[ "$TEST_NAMESERVICE_EXPIRY" == "true" ]]; then
+if [[ "$TEST_REGISTRY_EXPIRY" == "true" ]]; then
  echo "Setting timers for expiry tests."

-  cat $HOME/.laconicd/config/genesis.json | jq '.app_state["nameservice"]["params"]["record_rent_duration"]="60s"' > $HOME/.laconicd/config/tmp_genesis.json && mv $HOME/.laconicd/config/tmp_genesis.json $HOME/.laconicd/config/genesis.json
-  cat $HOME/.laconicd/config/genesis.json | jq '.app_state["nameservice"]["params"]["authority_grace_period"]="60s"' > $HOME/.laconicd/config/tmp_genesis.json && mv $HOME/.laconicd/config/tmp_genesis.json $HOME/.laconicd/config/genesis.json
-  cat $HOME/.laconicd/config/genesis.json | jq '.app_state["nameservice"]["params"]["authority_rent_duration"]="60s"' > $HOME/.laconicd/config/tmp_genesis.json && mv $HOME/.laconicd/config/tmp_genesis.json $HOME/.laconicd/config/genesis.json
+  cat $HOME/.laconicd/config/genesis.json | jq '.app_state["registry"]["params"]["record_rent_duration"]="60s"' > $HOME/.laconicd/config/tmp_genesis.json && mv $HOME/.laconicd/config/tmp_genesis.json $HOME/.laconicd/config/genesis.json
+  cat $HOME/.laconicd/config/genesis.json | jq '.app_state["registry"]["params"]["authority_grace_period"]="60s"' > $HOME/.laconicd/config/tmp_genesis.json && mv $HOME/.laconicd/config/tmp_genesis.json $HOME/.laconicd/config/genesis.json
+  cat $HOME/.laconicd/config/genesis.json | jq '.app_state["registry"]["params"]["authority_rent_duration"]="60s"' > $HOME/.laconicd/config/tmp_genesis.json && mv $HOME/.laconicd/config/tmp_genesis.json $HOME/.laconicd/config/genesis.json
 fi

 if [[ "$TEST_AUCTION_ENABLED" == "true" ]]; then
  echo "Enabling auction and setting timers."

-  cat $HOME/.laconicd/config/genesis.json | jq '.app_state["nameservice"]["params"]["authority_auction_enabled"]=true' > $HOME/.laconicd/config/tmp_genesis.json && mv $HOME/.laconicd/config/tmp_genesis.json $HOME/.laconicd/config/genesis.json
-  cat $HOME/.laconicd/config/genesis.json | jq '.app_state["nameservice"]["params"]["authority_rent_duration"]="60s"' > $HOME/.laconicd/config/tmp_genesis.json && mv $HOME/.laconicd/config/tmp_genesis.json $HOME/.laconicd/config/genesis.json
-  cat $HOME/.laconicd/config/genesis.json | jq '.app_state["nameservice"]["params"]["authority_grace_period"]="300s"' > $HOME/.laconicd/config/tmp_genesis.json && mv $HOME/.laconicd/config/tmp_genesis.json $HOME/.laconicd/config/genesis.json
-  cat $HOME/.laconicd/config/genesis.json | jq '.app_state["nameservice"]["params"]["authority_auction_commits_duration"]="60s"' > $HOME/.laconicd/config/tmp_genesis.json && mv $HOME/.laconicd/config/tmp_genesis.json $HOME/.laconicd/config/genesis.json
-  cat $HOME/.laconicd/config/genesis.json | jq '.app_state["nameservice"]["params"]["authority_auction_reveals_duration"]="60s"' > $HOME/.laconicd/config/tmp_genesis.json && mv $HOME/.laconicd/config/tmp_genesis.json $HOME/.laconicd/config/genesis.json
+  cat $HOME/.laconicd/config/genesis.json | jq '.app_state["registry"]["params"]["authority_auction_enabled"]=true' > $HOME/.laconicd/config/tmp_genesis.json && mv $HOME/.laconicd/config/tmp_genesis.json $HOME/.laconicd/config/genesis.json
+  cat $HOME/.laconicd/config/genesis.json | jq '.app_state["registry"]["params"]["authority_rent_duration"]="60s"' > $HOME/.laconicd/config/tmp_genesis.json && mv $HOME/.laconicd/config/tmp_genesis.json $HOME/.laconicd/config/genesis.json
+  cat $HOME/.laconicd/config/genesis.json | jq '.app_state["registry"]["params"]["authority_grace_period"]="300s"' > $HOME/.laconicd/config/tmp_genesis.json && mv $HOME/.laconicd/config/tmp_genesis.json $HOME/.laconicd/config/genesis.json
+  cat $HOME/.laconicd/config/genesis.json | jq '.app_state["registry"]["params"]["authority_auction_commits_duration"]="60s"' > $HOME/.laconicd/config/tmp_genesis.json && mv $HOME/.laconicd/config/tmp_genesis.json $HOME/.laconicd/config/genesis.json
+  cat $HOME/.laconicd/config/genesis.json | jq '.app_state["registry"]["params"]["authority_auction_reveals_duration"]="60s"' > $HOME/.laconicd/config/tmp_genesis.json && mv $HOME/.laconicd/config/tmp_genesis.json $HOME/.laconicd/config/genesis.json
 fi

 # increase block time (?)
@@ -0,0 +1,2 @@
+#!/bin/sh
+echo y | laconicd keys export mykey --unarmored-hex --unsafe
@@ -0,0 +1,7 @@
+services:
+  cns:
+    restEndpoint: 'http://laconicd:1317'
+    gqlEndpoint: 'http://laconicd:9473/api'
+    userKey: REPLACE_WITH_MYKEY
+    bondId:
+    chainId: laconic_9000-1
@@ -0,0 +1,2 @@
+[profile.default]
+eth-rpc-url = "http://fixturenet-eth-geth-1:8545"
@@ -20,16 +20,19 @@ server {
        proxy_pass  http://fixturenet-eth-geth-1:8545;
    }

-### ipld-eth-server
+## ipld-eth-server
 #    location ~ ^/ipld/eth/([^/]*)$ {
 #        set $apiKey $1;
 #        if ($apiKey = '') {
 #          set $apiKey $http_X_API_KEY;
 #        }
 #        auth_request /auth;
+#        auth_request_set $user_id $sent_http_x_user_id;
 #        proxy_buffering off;
 #        rewrite /.*$ / break;
 #        proxy_pass  http://ipld-eth-server:8081;
+#        proxy_set_header X-Original-Remote-Addr $remote_addr;
+#        proxy_set_header X-User-Id $user_id;
 #    }
 #
 #    location ~ ^/ipld/gql/([^/]*)$ {
@@ -42,14 +45,14 @@ server {
 #        rewrite /.*$ / break;
 #        proxy_pass  http://ipld-eth-server:8082;
 #    }
-#
-### lighthouse
-#    location /beacon/ {
-#        set $apiKey $http_X_API_KEY;
-#        auth_request /auth;
-#        proxy_buffering off;
-#        proxy_pass  http://fixturenet-eth-lighthouse-1:8001/;
-#    }
+
+## lighthouse
+    location /beacon/ {
+        set $apiKey $http_X_API_KEY;
+        auth_request /auth;
+        proxy_buffering off;
+        proxy_pass  http://fixturenet-eth-lighthouse-1:8001/;
+    }

    location = /auth {
        internal;
@@ -63,7 +66,7 @@ server {
        proxy_set_header X-Original-Host $host;
    }

-#    location = /basic_status {
-#        stub_status;
-#    }
+    location = /stub_status {
+        stub_status;
+    }
 }
@@ -0,0 +1,8 @@
+{
+  "relayNodes": [
+    "/ip4/127.0.0.1/tcp/9090/ws/p2p/12D3KooWSPCsVkHVyLQoCqhu2YRPvvM7o6r6NRYyLM5zeA6Uig5t"
+  ],
+  "peer": {
+    "enableDebugInfo": true
+  }
+}
@@ -0,0 +1,9 @@
+#!/bin/sh
+set -e
+
+# Merging config files to get deployed contract address
+jq -s '.[0] * .[1]' /app/src/mobymask-app-config.json /server/config.json > /app/src/config.json
+
+npm run build
+
+serve -s build
@@ -0,0 +1 @@
+REACT_APP_WATCHER_URI=http://localhost:3001/graphql
@@ -0,0 +1,5 @@
+{
+  "id": "12D3KooWK6myjc8r1KBnfP9igp31qJkPaVfsKDjKrjoSefV5SDEo",
+  "privKey": "CAESQJMHbMaH+UEOtjGOzXYtoPO/cdHakCtN1hcnknIWzx/6ie1lxb+8kfzBjwt7apfj8fHlTCYSIVK8Q2AWu9a2h3g=",
+  "pubKey": "CAESIIntZcW/vJH8wY8Le2qX4/Hx5UwmEiFSvENgFrvWtod4"
+}
@@ -0,0 +1,10 @@
+#!/bin/sh
+
+# Private key of account with balance
+PRIVATE_KEY=
+
+# Assign deployed contract address from server config
+CONTRACT_ADDRESS=`jq '.address' /server/config.json`
+
+echo 'yarn peer-listener --contract-address <CONTRACT_ADDRESS> --private-key <PRIVATE_KEY>'
+yarn peer-listener --contract-address $CONTRACT_ADDRESS --private-key $PRIVATE_KEY
@@ -0,0 +1 @@
+RELAY="/ip4/127.0.0.1/tcp/9090/ws/p2p/12D3KooWSPCsVkHVyLQoCqhu2YRPvvM7o6r6NRYyLM5zeA6Uig5t"
@@ -0,0 +1,5 @@
+{
+  "id": "12D3KooWSPCsVkHVyLQoCqhu2YRPvvM7o6r6NRYyLM5zeA6Uig5t",
+  "privKey": "CAESQGsqG5o4VlWJZM9XlA3MjabyZOXWQ2MLZU5AhBQsjXGt9iSlGtTuNOrHX5xSRgLBxLuMoqWsjGxE/dDB9c46RI8=",
+  "pubKey": "CAESIPYkpRrU7jTqx1+cUkYCwcS7jKKlrIxsRP3QwfXOOkSP"
+}
@@ -0,0 +1,5 @@
+{
+    "rpcUrl": "http://laconicd:8545",
+    "privateKey": "GENESIS_ACCOUNT_PRIVATE_KEY",
+    "baseURI": "http://127.0.0.1:3002/#"
+}
@@ -0,0 +1,8 @@
+{
+  "relayNodes": [
+    "/ip4/127.0.0.1/tcp/9090/ws/p2p/12D3KooWSPCsVkHVyLQoCqhu2YRPvvM7o6r6NRYyLM5zeA6Uig5t"
+  ],
+  "peer": {
+    "enableDebugInfo": true
+  }
+}
@@ -0,0 +1,70 @@
+[server]
+  host = "0.0.0.0"
+  port = 3001
+  kind = "lazy"
+
+  # Checkpointing state.
+  checkpointing = true
+
+  # Checkpoint interval in number of blocks.
+  checkpointInterval = 2000
+
+  # Enable state creation
+  enableState = true
+
+  # Boolean to filter logs by contract.
+  filterLogs = true
+
+  # Max block range for which to return events in eventsInRange GQL query.
+  # Use -1 for skipping check on block range.
+  maxEventsBlockRange = -1
+
+  [server.p2p]
+    enableRelay = true
+    enablePeer = false
+
+    [server.p2p.relay]
+      host = "0.0.0.0"
+      port = 9090
+      relayPeers = []
+      peerIdFile = './relay-id.json'
+      enableDebugInfo = true
+
+    [server.p2p.peer]
+      relayMultiaddr = '/dns4/mobymask-watcher-server/tcp/9090/ws/p2p/12D3KooWSPCsVkHVyLQoCqhu2YRPvvM7o6r6NRYyLM5zeA6Uig5t'
+      pubSubTopic = 'mobymask'
+      peerIdFile = './peer-id.json'
+      enableDebugInfo = true
+
+[metrics]
+  host = "0.0.0.0"
+  port = 9000
+  [metrics.gql]
+    port = 9001
+
+[database]
+  type = "postgres"
+  host = "mobymask-watcher-db"
+  port = 5432
+  database = "mobymask-watcher"
+  username = "vdbm"
+  password = "password"
+  synchronize = true
+  logging = false
+
+[upstream]
+  [upstream.ethServer]
+    gqlApiEndpoint = "http://ipld-eth-server:8083/graphql"
+    rpcProviderEndpoint = "http://laconicd:8545"
+    blockDelayInMilliSecs = 60000
+
+  [upstream.cache]
+    name = "requests"
+    enabled = false
+    deleteOnStart = false
+
+[jobQueue]
+  dbConnectionString = "postgres://vdbm:password@mobymask-watcher-db/mobymask-watcher-job-queue"
+  maxCompletionLagInSecs = 300
+  jobDelayInMilliSecs = 100
+  eventsInBatch = 50
@@ -0,0 +1,3 @@
+#!/usr/bin/env bash
+# Build a local version of the task executor for act-runner
+docker build -t cerc/act-runner-task-executor:local -f ${CERC_REPO_BASE_DIR}/act_runner/Dockerfile.task-executor ${CERC_REPO_BASE_DIR}/act_runner
@@ -0,0 +1,3 @@
+#!/usr/bin/env bash
+# Build a local version of the act-runner image
+docker build -t cerc/act-runner:local -f ${CERC_REPO_BASE_DIR}/act_runner/Dockerfile ${CERC_REPO_BASE_DIR}/act_runner
@@ -0,0 +1,31 @@
+# From: https://github.com/vyzo/gerbil/blob/master/docker/Dockerfile
+FROM gerbil/ubuntu
+
+# Install the Solidity compiler (latest stable version)
+# and guile
+# and libsecp256k1-dev
+RUN apt-get update && export DEBIAN_FRONTEND=noninteractive && export DEBCONF_NOWARNINGS="yes" && \
+    apt-get install -y software-properties-common && \
+    add-apt-repository ppa:ethereum/ethereum && \
+    apt-get update && \
+    apt-get install -y solc && \
+    apt-get install -y guile-3.0 && \
+    apt-get install -y libsecp256k1-dev && \
+    apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
+
+RUN mkdir /scripts
+COPY install-dependencies.sh /scripts
+
+# Override the definition of GERBIL_PATH in the base image, but
+# is safe because (at present) no gerbil packages are installed in the base image
+# We do this in order to allow a set of pre-installed packages from the container
+# to be used with an arbitrary, potentially different set of projects bind mounted
+# at /src
+ENV GERBIL_PATH=/.gerbil
+RUN bash /scripts/install-dependencies.sh
+
+# Needed to prevent git from raging about /src
+RUN git config --global --add safe.directory /src
+
+COPY entrypoint.sh /scripts
+ENTRYPOINT ["/scripts/entrypoint.sh"]
@@ -0,0 +1,21 @@
+## Gerbil Scheme Builder
+
+This container is designed to be used as a simple "build runner" environment for building and running Scheme projects using Gerbil and gerbil-ethereum. Its primary purpose is to allow build/test/run of gerbil code without the need to install and configure all the necessary prerequisites and dependencies on the host system.
+
+### Usage
+
+First build the container with:
+
+```
+$ laconic-so build-containers --include cerc/builder-gerbil
+```
+
+Now, assuming a gerbil project located at `~/projects/my-project`, run bash in the container mounting the project with:
+
+```
+$ docker run -it -v $HOME/projects/my-project:/src cerc/builder-gerbil:latest bash
+root@7c4124bb09e3:/src#
+```
+
+Now gerbil commands can be run.
+
@@ -0,0 +1,2 @@
+#!/bin/sh
+exec "$@"
@@ -0,0 +1,16 @@
+DEPS=(github.com/fare/gerbil-utils
+      github.com/fare/gerbil-poo
+      github.com/fare/gerbil-crypto
+      github.com/fare/gerbil-persist
+      github.com/fare/gerbil-ethereum
+      github.com/drewc/gerbil-swank
+      github.com/drewc/drewc-r7rs-swank
+      github.com/drewc/smug-gerbil
+      github.com/drewc/ftw
+      github.com/vyzo/gerbil-libp2p
+      ) ;
+for i in ${DEPS[@]} ; do
+  echo "Installing gerbil package: $i"
+  gxpkg install $i &&
+  gxpkg build $i
+done
@@ -1,14 +1,30 @@
 # Originally from: https://github.com/devcontainers/images/blob/main/src/javascript-node/.devcontainer/Dockerfile
+# Which depends on: https://github.com/nodejs/docker-node/blob/main/Dockerfile-debian.template
 # [Choice] Node.js version (use -bullseye variants on local arm64/Apple Silicon): 18, 16, 14, 18-bullseye, 16-bullseye, 14-bullseye, 18-buster, 16-buster, 14-buster
-ARG VARIANT=16-bullseye
+ARG VARIANT=18-bullseye
 FROM node:${VARIANT}

+# Set these args to change the uid/gid for the base container's "node" user to match that of the host user (so bind mounts work as expected).
+ARG CERC_HOST_UID=1000
+ARG CERC_HOST_GID=1000
+# Make these values available at runtime to allow a consistency check.
+ENV HOST_UID=${CERC_HOST_UID}
+ENV HOST_GID=${CERC_HOST_GID}
+
 ARG USERNAME=node
 ARG NPM_GLOBAL=/usr/local/share/npm-global

 # Add NPM global to PATH.
 ENV PATH=${NPM_GLOBAL}/bin:${PATH}

+RUN \
+    if [ ${CERC_HOST_GID} -ne 1000 ] ; then \
+        groupmod -g ${CERC_HOST_GID} ${USERNAME} ; \
+    fi \
+    && if [ ${CERC_HOST_UID} -ne 1000 ] ; then \
+        usermod -u ${CERC_HOST_UID} -g ${CERC_HOST_GID} ${USERNAME} && chown ${CERC_HOST_UID}:${CERC_HOST_GID} /home/${USERNAME} ; \
+    fi
+
 RUN \
    # Configure global npm install location, use group to adapt to UID/GID changes
    if ! cat /etc/group | grep -e "^npm:" > /dev/null 2>&1; then groupadd -r npm; fi \
@@ -39,6 +55,7 @@ RUN mkdir /scripts
 COPY build-npm-package.sh /scripts
 COPY yarn-local-registry-fixup.sh /scripts
 COPY build-npm-package-local-dependencies.sh /scripts
+COPY check-uid.sh /scripts
 ENV PATH="${PATH}:/scripts"

 COPY entrypoint.sh .
@@ -1,7 +1,7 @@
 #!/bin/bash
 # Usage: build-npm-package-local-dependencies.sh <registry-url> <publish-with-this-version>
 # Runs build-npm-package.sh after first fixing up yarn.lock to use a local
-# npm registry for all packages in a spcific scope (currently @cerc-io)
+# npm registry for all packages in a spcific scope (currently @cerc-io and @lirewine)
 if [ -n "$CERC_SCRIPT_DEBUG" ]; then
    set -x
 fi
@@ -17,18 +17,21 @@ fi
 set -e
 local_npm_registry_url=$1
 package_publish_version=$2
-# TODO: make this a paramater and allow a list of scopes
-npm_scope_for_local="@cerc-io"
-# We need to configure the local registry 
-npm config set ${npm_scope_for_local}:registry ${local_npm_registry_url}
-npm config set -- ${local_npm_registry_url}:_authToken ${CERC_NPM_AUTH_TOKEN}
-# Find the set of dependencies from the specified scope
-mapfile -t dependencies_from_scope < <(cat package.json | jq -r '.dependencies | with_entries(if (.key|test("^'${npm_scope_for_local}'/.*$")) then ( {key: .key, value: .value } ) else empty end ) | keys[]')
-echo "Fixing up dependencies"
-for package in "${dependencies_from_scope[@]}"
+# If we need to handle an additional scope, add it to the list below:
+npm_scopes_to_handle=("@cerc-io" "@lirewine")
+for npm_scope_for_local in ${npm_scopes_to_handle[@]}
 do
-    echo "Fixing up package ${package}"
-    yarn-local-registry-fixup.sh $package ${local_npm_registry_url}
+    # We need to configure the local registry
+    npm config set ${npm_scope_for_local}:registry ${local_npm_registry_url}
+    npm config set -- ${local_npm_registry_url}:_authToken ${CERC_NPM_AUTH_TOKEN}
+    # Find the set of dependencies from the specified scope
+    mapfile -t dependencies_from_scope < <(cat package.json | jq -r '.dependencies | with_entries(if (.key|test("^'${npm_scope_for_local}'/.*$")) then ( {key: .key, value: .value } ) else empty end ) | keys[]')
+    echo "Fixing up dependencies in scope ${npm_scope_for_local}"
+    for package in "${dependencies_from_scope[@]}"
+    do
+        echo "Fixing up package ${package}"
+        yarn-local-registry-fixup.sh $package ${local_npm_registry_url}
+    done
 done
 echo "Running build"
 build-npm-package.sh ${local_npm_registry_url} ${package_publish_version}
@@ -22,8 +22,8 @@ set -e
 # Get the name of this package from package.json since we weren't passed that
 package_name=$( cat package.json | jq -r .name )
 local_npm_registry_url=$1
-npm config set @lirewine:registry ${local_npm_registry_url}
 npm config set @cerc-io:registry ${local_npm_registry_url}
+npm config set @lirewine:registry ${local_npm_registry_url}
 npm config set -- ${local_npm_registry_url}:_authToken ${CERC_NPM_AUTH_TOKEN}
 # First check if the version of this package we're trying to build already exists in the registry
 package_exists=$( yarn info --json ${package_name}@${package_publish_version} 2>/dev/null | jq -r .data.dist.tarball )
@@ -0,0 +1,21 @@
+#!/bin/bash
+# Make the container usable for uid/gid != 1000
+if [[ -n "$CERC_SCRIPT_DEBUG" ]]; then
+    set -x
+fi
+current_uid=$(id -u)
+current_gid=$(id -g)
+# Don't check if running as root
+if [[ ${current_uid} == 0 ]]; then
+    exit 0
+fi
+# Check the current uid/gid vs the uid/gid used to build the container.
+# We do this because both bind mounts and npm tooling require the uid/gid to match.
+if [[ ${current_gid} != ${HOST_GID} ]]; then
+    echo "Warning: running with gid: ${current_gid} which is not the gid for which this container was built (${HOST_GID})"
+    exit 0
+fi
+if [[ ${current_uid} != ${HOST_UID} ]]; then
+    echo "Warning: running with gid: ${current_uid} which is not the uid for which this container was built (${HOST_UID})"
+    exit 0
+fi
@@ -1,2 +1,3 @@
 #!/bin/sh
+/scripts/check-uid.sh
 exec "$@"
@@ -1,5 +1,9 @@
 #!/bin/bash

+if [ -n "$CERC_SCRIPT_DEBUG" ]; then
+    set -x
+fi
+
 ETHERBASE=`cat /opt/testnet/build/el/accounts.csv | head -1 | cut -d',' -f2`
 NETWORK_ID=`cat /opt/testnet/el/el-config.yaml | grep 'chain_id' | awk '{ print $2 }'`
 NETRESTRICT=`ip addr | grep inet | grep -v '127.0' | awk '{print $2}'`
@@ -28,7 +32,9 @@ else
    echo -n "$JWT" > /opt/testnet/build/el/jwtsecret

    if [ "$CERC_RUN_STATEDIFF" == "detect" ] && [ -n "$CERC_STATEDIFF_DB_HOST" ]; then
-      if [ -n "$(dig $CERC_STATEDIFF_DB_HOST +short)" ]; then
+      dig_result=$(dig $CERC_STATEDIFF_DB_HOST +short)
+      dig_status_code=$?
+      if [[ $dig_status_code = 0 && -n $dig_result ]]; then
        echo "Statediff DB at $CERC_STATEDIFF_DB_HOST"
        CERC_RUN_STATEDIFF="true"
      else
@@ -27,4 +27,8 @@ RUN cd /opt/testnet && make genesis-cl
 # Work around some bugs in lcli where the default path is always used.
 RUN mkdir -p /root/.lighthouse && cd /root/.lighthouse && ln -s /opt/testnet/build/cl/testnet

+RUN mkdir -p /scripts
+COPY scripts/status-internal.sh /scripts
+COPY scripts/status.sh /scripts
+
 ENTRYPOINT ["/opt/testnet/run.sh"]
@@ -0,0 +1,10 @@
+#!/usr/bin/env bash
+# Wrapper to facilitate using status.sh inside the container
+if [ -n "$CERC_SCRIPT_DEBUG" ]; then
+    set -x
+fi
+export LIGHTHOUSE_BASE_URL="http://fixturenet-eth-lighthouse-1:8001"
+export GETH_BASE_URL="http://fixturenet-eth-geth-1:8545"
+# See: https://stackoverflow.com/a/246128/1701505
+SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
+$SCRIPT_DIR/status.sh
@@ -1,5 +1,7 @@
-#!/bin/bash
-
+#!/usr/bin/env bash
+if [ -n "$CERC_SCRIPT_DEBUG" ]; then
+    set -x
+fi
 STATUSES=("geth to generate DAG" "beacon phase0" "beacon altair" "beacon bellatrix pre-merge" "beacon bellatrix merge")
 STATUS=0

@@ -7,18 +9,22 @@ STATUS=0
 LIGHTHOUSE_BASE_URL=${LIGHTHOUSE_BASE_URL}
 GETH_BASE_URL=${GETH_BASE_URL}

+# TODO: Docker commands below should be replaced by some interface into stack orchestrator
+# or some execution environment-neutral mechanism.
 if [ -z "$LIGHTHOUSE_BASE_URL" ]; then
-  LIGHTHOUSE_PORT=`docker ps -f "name=fixturenet-eth-lighthouse-1-1" --format "{{.Ports}}" | head -1 | cut -d':' -f2 | cut -d'-' -f1`
+  LIGHTHOUSE_CONTAINER=`docker ps -q -f "name=fixturenet-eth-lighthouse-1-1"`
+  LIGHTHOUSE_PORT=`docker port $LIGHTHOUSE_CONTAINER 8001 | cut -d':' -f2`
  LIGHTHOUSE_BASE_URL="http://localhost:${LIGHTHOUSE_PORT}"
 fi

 if [ -z "$GETH_BASE_URL" ]; then
-  GETH_PORT=`docker ps -f "name=fixturenet-eth-geth-1-1" --format "{{.Ports}}" | head -1 | cut -d':' -f2 | cut -d'-' -f1`
+  GETH_CONTAINER=`docker ps -q -f "name=fixturenet-eth-geth-1-1"`
+  GETH_PORT=`docker port $GETH_CONTAINER 8545 | cut -d':' -f2`
  GETH_BASE_URL="http://localhost:${GETH_PORT}"
 fi

 function inc_status() {
-  echo " DONE!"
+  echo " done"
  STATUS=$((STATUS + 1))
  if [ $STATUS -lt ${#STATUSES[@]} ]; then
    echo -n "Waiting for ${STATUSES[$STATUS]}..."
@@ -0,0 +1,3 @@
+#!/usr/bin/env bash
+# Build a local version of the foundry-rs/foundry image
+docker build -t cerc/foundry:local -f ${CERC_REPO_BASE_DIR}/foundry/Dockerfile-debian ${CERC_REPO_BASE_DIR}/foundry
@@ -1,7 +1,9 @@
-FROM ghcr.io/foundry-rs/foundry
+# Note: cerc/foundry is Debian based
+FROM cerc/foundry:local
+
+RUN apt-get update && export DEBIAN_FRONTEND=noninteractive \
+    && apt-get -y install --no-install-recommends jq curl netcat

-RUN apk update ; apk add --no-cache --allow-untrusted ca-certificates curl bash git jq
-RUN apk add --no-cache --upgrade grep
 WORKDIR /root

 ARG GENESIS_FILE_PATH=genesis.json
@@ -1,4 +1,4 @@
 FROM quay.io/keycloak/keycloak:20.0
 WORKDIR /opt/keycloak/providers
 RUN curl -L https://github.com/aerogear/keycloak-metrics-spi/releases/download/2.5.3/keycloak-metrics-spi-2.5.3.jar --output keycloak-metrics-spi.jar
-RUN curl -L https://github.com/cerc-io/keycloak-api-key-demo/releases/download/v0.1/api-key-module-0.1.jar --output api-key-module.jar
+RUN curl -L https://github.com/cerc-io/keycloak-api-key-demo/releases/download/v0.3/api-key-module-0.3.jar --output api-key-module.jar
@@ -1,6 +1,6 @@
 # Originally from: https://github.com/devcontainers/images/blob/main/src/javascript-node/.devcontainer/Dockerfile
 # [Choice] Node.js version (use -bullseye variants on local arm64/Apple Silicon): 18, 16, 14, 18-bullseye, 16-bullseye, 14-bullseye, 18-buster, 16-buster, 14-buster
-ARG VARIANT=16-bullseye
+ARG VARIANT=18-bullseye
 FROM node:${VARIANT}

 ARG USERNAME=node
@@ -40,18 +40,20 @@ RUN apt-get update && export DEBIAN_FRONTEND=noninteractive \
 # RUN su node -c "npm install -g <your-package-list-here>"

 # Configure the local npm registry
-RUN npm config set @lirewine:registry ${CERC_NPM_URL} \
-    && npm config set @cerc-io:registry ${CERC_NPM_URL} \
+RUN npm config set @cerc-io:registry ${CERC_NPM_URL} \
    && npm config set @lirewine:registry ${CERC_NPM_URL} \
    && npm config set -- ${CERC_NPM_URL}:_authToken ${CERC_NPM_AUTH_TOKEN}

 # TODO: the image at this point could be made a base image for several different CLI images
 # that install different Node-based CLI commands

-# DEBUG, remove
-RUN yarn info @cerc-io/laconic-registry-cli
-
 # Globally install the cli package
 RUN yarn global add @cerc-io/laconic-registry-cli

-ENTRYPOINT ["laconic"]
+# Add scripts
+RUN mkdir /scripts
+ENV PATH="${PATH}:/scripts"
+COPY ./import-key.sh /scripts
+
+# Default command sleeps forever so docker doesn't kill it
+CMD ["sh", "-c", "while :; do sleep 600; done"]
@@ -0,0 +1,2 @@
+#!/bin/sh
+sed 's/REPLACE_WITH_MYKEY/'${1}'/' registry-cli-config-template.yml > config.yml
@@ -0,0 +1,14 @@
+FROM node:18.15.0-alpine3.16
+
+RUN apk --update --no-cache add make git jq
+
+WORKDIR /app
+
+COPY . .
+
+RUN npm install -g serve
+
+RUN echo "Building mobymask-ui" && \
+    npm install
+
+WORKDIR /app
@@ -0,0 +1,7 @@
+#!/usr/bin/env bash
+# Build cerc/mobymask-ui
+
+# See: https://stackoverflow.com/a/246128/1701505
+SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
+
+docker build -t cerc/mobymask-ui:local -f ${SCRIPT_DIR}/Dockerfile ${CERC_REPO_BASE_DIR}/mobymask-ui
@@ -0,0 +1,13 @@
+FROM node:16.17.1-alpine3.16
+
+RUN apk --update --no-cache add python3 alpine-sdk
+
+WORKDIR /app
+
+COPY . .
+
+RUN yarn
+
+# Add scripts
+RUN mkdir /scripts
+ENV PATH="${PATH}:/scripts"
@@ -0,0 +1,7 @@
+#!/usr/bin/env bash
+# Build cerc/mobymask
+
+# See: https://stackoverflow.com/a/246128/1701505
+SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
+
+docker build -t cerc/mobymask:local -f ${SCRIPT_DIR}/Dockerfile ${CERC_REPO_BASE_DIR}/MobyMask
@@ -0,0 +1,14 @@
+FROM node:18.15.0-alpine3.16
+
+RUN apk --update --no-cache add make git python3
+
+WORKDIR /app
+
+COPY . .
+
+RUN yarn global add serve
+
+RUN echo "Building react-peer" && \
+    yarn install --ignore-scripts && yarn build --ignore @cerc-io/test-app
+
+WORKDIR /app
@@ -0,0 +1,7 @@
+#!/usr/bin/env bash
+# Build cerc/react-peer
+
+# See: https://stackoverflow.com/a/246128/1701505
+SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
+
+docker build -t cerc/react-peer:local -f ${SCRIPT_DIR}/Dockerfile ${CERC_REPO_BASE_DIR}/react-peer
@@ -1,4 +1,11 @@
-FROM alpine:latest
+FROM ubuntu:latest
+
+RUN apt-get update && export DEBIAN_FRONTEND=noninteractive && export DEBCONF_NOWARNINGS="yes" && \
+    apt-get install -y software-properties-common && \
+    apt-get install -y nginx && \
+    apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
+
+EXPOSE 80

 COPY run.sh /app/run.sh

@@ -12,5 +12,5 @@ else
    echo `date` > $EXISTSFILENAME
 fi

-# Sleep forever to keep docker happy
-while true; do sleep 10; done
+# Run nginx which will block here forever
+/usr/sbin/nginx -g "daemon off;"
@@ -0,0 +1,20 @@
+FROM ubuntu:22.04
+
+RUN apt-get update \
+  && apt-get install -y curl gnupg build-essential \
+  && curl --silent --location https://deb.nodesource.com/setup_18.x | bash - \
+  && apt-get update \
+  && apt-get install -y nodejs git busybox jq \
+  && node -v
+
+RUN corepack enable \
+  && yarn --version
+
+WORKDIR /app
+
+COPY . .
+
+RUN echo "Building watcher-ts" && \
+    yarn && yarn build
+
+WORKDIR /app/packages/mobymask-v2-watcher
@@ -0,0 +1,7 @@
+#!/usr/bin/env bash
+# Build cerc/watcher-mobymask-v2
+
+# See: https://stackoverflow.com/a/246128/1701505
+SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
+
+docker build -t cerc/watcher-mobymask-v2:local -f ${SCRIPT_DIR}/Dockerfile ${CERC_REPO_BASE_DIR}/watcher-ts
@@ -1,6 +1,9 @@
 #!/usr/bin/env bash
 # Usage: default-build.sh <image-tag> [<repo-relative-path>]
 # if <repo-relative-path> is not supplied, the context is the directory where the Dockerfile lives
+if [[ -n "$CERC_SCRIPT_DEBUG" ]]; then
+    set -x
+fi
 if [[ $# -ne 2 ]]; then
    echo "Illegal number of parameters" >&2
    exit 1
@@ -8,4 +11,4 @@ fi
 image_tag=$1
 build_dir=$2
 echo "Building ${image_tag} in ${build_dir}"
-docker build -t ${image_tag} ${build_dir}
+docker build -t ${image_tag} --build-arg CERC_HOST_UID=${CERC_HOST_UID} --build-arg CERC_HOST_GID=${CERC_HOST_GID} ${build_dir}
@@ -1,3 +1,4 @@
+cerc/foundry
 cerc/test-contract
 cerc/eth-statediff-fill-service
 cerc/eth-statediff-service
@@ -17,8 +18,15 @@ cerc/watcher-erc20
 cerc/watcher-erc721
 cerc/watcher-uniswap-v3
 cerc/uniswap-v3-info
+cerc/watcher-mobymask-v2
+cerc/react-peer
+cerc/mobymask-ui
+cerc/mobymask
 cerc/test-container
 cerc/eth-probe
 cerc/builder-js
 cerc/keycloak
 cerc/tx-spammer
+cerc/builder-gerbil
+cerc/act-runner
+cerc/act-runner-task-executor
@@ -1,4 +1,5 @@
-gem
 laconic-sdk
-debug
 laconic-registry-cli
+gem
+debug
+crypto
@@ -6,15 +6,19 @@ ipld-eth-beacon-db
 ipld-eth-beacon-indexer
 ipld-eth-server
 lighthouse
-prometheus-grafana
 laconicd
 fixturenet-laconicd
 fixturenet-eth
+fixturenet-eth-metrics
 watcher-mobymask
 watcher-erc20
 watcher-erc721
 watcher-uniswap-v3
+watcher-mobymask-v2
+mobymask-laconicd
 test
 eth-probe
 keycloak
 tx-spammer
+kubo
+foundry
@@ -11,8 +11,16 @@ cerc-io/laconic-sdk
 cerc-io/laconic-registry-cli
 cerc-io/mobymask-watcher
 cerc-io/watcher-ts
+cerc-io/react-peer
+cerc-io/mobymask-ui
+cerc-io/MobyMask
 vulcanize/uniswap-watcher-ts
 vulcanize/uniswap-v3-info
 vulcanize/assemblyscript
 cerc-io/eth-probe
 cerc-io/tx-spammer
+dboreham/foundry
+lirewine/gem
+lirewine/debug
+lirewine/crypto
+telackey/act_runner
@@ -0,0 +1,59 @@
+# Build Support Stack
+
+## Instructions
+
+JS/TS/NPM builds need an npm registry to store intermediate package artifacts.
+This can be supplied by the user (e.g. using a hosted registry or even npmjs.com), or a local registry using gitea can be deployed by stack orchestrator.
+To use a user-supplied registry set these environment variables:
+
+`CERC_NPM_REGISTRY_URL` and 
+`CERC_NPM_AUTH_TOKEN`
+
+Leave `CERC_NPM_REGISTRY_URL` un-set to use the local gitea registry.
+
+### 1. Build support containers
+```
+$ laconic-so --stack build-support build-containers
+```
+Note that the scheme/gerbil builder container can take a while to build so if you aren't going to build scheme projects it can be skipped with:
+```
+$ laconic-so --stack build-support build-containers --exclude cerc/builder-gerbil
+```
+### 2. Deploy Gitea Package Registry
+
+```
+$ laconic-so --stack package-registry setup-repositories
+$ laconic-so --stack package-registry deploy up
+[+] Running 3/3
+ ⠿ Network laconic-aecc4a21d3a502b14522db97d427e850_gitea       Created                                                                                    0.0s
+ ⠿ Container laconic-aecc4a21d3a502b14522db97d427e850-db-1      Started                                                                                    1.2s
+ ⠿ Container laconic-aecc4a21d3a502b14522db97d427e850-server-1  Started                                                                                    1.9s
+New user 'gitea_admin' has been successfully created!
+This is your gitea access token: 84fe66a73698bf11edbdccd0a338236b7d1d5c45. Keep it safe and secure, it can not be fetched again from gitea.
+To use with laconic-so set this environment variable: export CERC_NPM_AUTH_TOKEN=3e493e77b3e83fe9e882f7e3a79dd4d5441c308b
+Created the organization cerc-io
+Gitea was configured to use host name: gitea.local, ensure that this resolves to localhost, e.g. with sudo vi /etc/hosts
+Success, gitea is properly initialized
+$
+```
+### 3. Configure the hostname gitea.local
+How to do this is OS-dependent but usually involves editing a `hosts` file. For example on Linux add this line to the file `/etc/hosts` (needs sudo):
+```
+127.0.0.1       gitea.local
+```
+Test with:
+```
+$ ping gitea.local
+PING gitea.local (127.0.0.1) 56(84) bytes of data.
+64 bytes from localhost (127.0.0.1): icmp_seq=1 ttl=64 time=0.147 ms
+64 bytes from localhost (127.0.0.1): icmp_seq=2 ttl=64 time=0.033 ms
+```
+Although not necessary in order to build and publish packages, you can now access the Gitea web interface at: [http://gitea.local:3000](http://gitea.local:3000) using these credentials: gitea_admin/admin1234 (Note: please properly secure Gitea if public internet access is allowed).
+
+Now npm packages can be built:
+### Build npm Packages
+Ensure that `CERC_NPM_AUTH_TOKEN` is set with the token printed above when the package-registry stack was deployed (the actual token value will be different than shown in this example):
+```
+$ export CERC_NPM_AUTH_TOKEN=84fe66a73698bf11edbdccd0a338236b7d1d5c45
+$ laconic-so build-npms --include laconic-sdk
+```
@@ -0,0 +1,6 @@
+version: "1.1"
+name: build-support
+decription: "Build Support Components"
+containers:
+  - cerc/builder-js
+  - cerc/builder-gerbil
@@ -7,13 +7,13 @@ Instructions to deploy a local ERC20 watcher stack (core + watcher) for demonstr
 Clone required repositories:

 ```bash
-laconic-so setup-repositories --include cerc-io/go-ethereum,cerc-io/ipld-eth-db,cerc-io/ipld-eth-server,cerc-io/watcher-ts 
+laconic-so --stack erc20 setup-repositories
 ```

 Build the core and watcher container images:

 ```bash
-laconic-so build-containers --include cerc/go-ethereum,cerc/go-ethereum-foundry,cerc/ipld-eth-db,cerc/ipld-eth-server,cerc/watcher-erc20
+laconic-so --stack erc20 build-containers
 ```

 This should create the required docker images in the local image registry.
@@ -21,7 +21,7 @@ This should create the required docker images in the local image registry.
 Deploy the stack:

 ```bash
-laconic-so deploy-system --include ipld-eth-db,go-ethereum-foundry,ipld-eth-server,watcher-erc20 up
+laconic-so --stack erc20 deploy-system up
 ```

 ## Demo
@@ -153,5 +153,5 @@ Transfer funds between different accounts using MetaMask and use the playground
 To stop all the services running in background run:

 ```bash
-laconic-so deploy-system --include ipld-eth-db,go-ethereum-foundry,ipld-eth-server,watcher-erc20 down
+laconic-so --stack erc20 deploy-system down
 ```
@@ -13,6 +13,6 @@ containers:
  - cerc/watcher-erc20
 pods:
  - go-ethereum-foundry
-  - db
+  - ipld-eth-db
  - ipld-eth-server
  - watcher-erc20
@@ -6,23 +6,23 @@ Instructions to deploy a local ERC721 watcher stack (core + watcher) for demonst

 * Clone / pull required repositories:

-  ```bash
-  $ laconic-so setup-repositories --include cerc-io/go-ethereum,cerc-io/ipld-eth-db,cerc-io/ipld-eth-server,cerc-io/watcher-ts --pull
-  ```
+```bash
+laconic-so --stack erc721 setup-repositories
+```

 * Build the core and watcher container images:

-  ```bash
-  $ laconic-so build-containers --include cerc/go-ethereum,cerc/go-ethereum-foundry,cerc/ipld-eth-db,cerc/ipld-eth-server,cerc/watcher-erc721
-  ```
+```bash
+laconic-so --stack erc721 build-containers
+```

  This should create the required docker images in the local image registry.

 * Deploy the stack:

-  ```bash
-  $ laconic-so deploy-system --include db,go-ethereum-foundry,ipld-eth-server,watcher-erc721 up
-  ```
+```bash
+laconic-so --stack erc721 deploy-system up
+```

 ## Demo

@@ -209,6 +209,6 @@ Instructions to deploy a local ERC721 watcher stack (core + watcher) for demonst

 * To stop all the services running in background:

-  ```bash
-  $ laconic-so deploy-system --include db,go-ethereum-foundry,ipld-eth-server,watcher-erc721 down
-  ```
+```bash
+laconic-so --stack erc721 deploy-system down
+```
@@ -13,6 +13,6 @@ containers:
  - cerc/watcher-erc721
 pods:
  - go-ethereum-foundry
-  - db
+  - ipld-eth-db
  - ipld-eth-server
  - watcher-erc721
@@ -0,0 +1,6 @@
+# fixturenet-eth
+
+A "loaded" version of fixturenet-eth, with all the bells and whistles enabled.
+
+TODO: write me
+
@@ -0,0 +1,24 @@
+version: "1.0"
+name: fixturenet-eth-loaded
+decription: "Loaded Ethereum Fixturenet"
+repos:
+  - cerc-io/go-ethereum
+  - cerc-io/tx-spammer
+  - cerc-io/ipld-eth-server
+  - cerc-io/ipld-eth-db
+  - cerc/go-ethereum
+containers:
+  - cerc/lighthouse
+  - cerc/fixturenet-eth-geth
+  - cerc/fixturenet-eth-lighthouse
+  - cerc/ipld-eth-server
+  - cerc/ipld-eth-db
+  - cerc/keycloak
+  - cerc/tx-spammer
+pods:
+  - fixturenet-eth
+  - tx-spammer
+  - fixturenet-eth-metrics
+  - keycloak
+  - ipld-eth-server
+  - ipld-eth-db
@@ -4,12 +4,12 @@ Instructions for deploying a local a geth + lighthouse blockchain "fixturenet" f

 ## Clone required repositories
 ```
-$ laconic-so setup-repositories --include cerc-io/go-ethereum
+$ laconic-so --stack fixturenet-eth setup-repositories
 ```

 ## Build the fixturenet-eth containers
 ```
-$ laconic-so build-containers --include cerc/go-ethereum,cerc/lighthouse,cerc/fixturenet-eth-geth,cerc/fixturenet-eth-lighthouse
+$ laconic-so --stack fixturenet-eth build-containers
 ```
 This should create several container images in the local image registry: 

@@ -20,27 +20,31 @@ This should create several container images in the local image registry:

 ## Deploy the stack
 ```
-$ laconic-so deploy-system --include fixturenet-eth up
+$ laconic-so --stack fixturenet-eth deploy up
 ```

 ## Check status

 ```
-$ container-build/cerc-fixturenet-eth-lighthouse/scripts/status.sh
-Waiting for geth to generate DAG ..................................................... DONE!
-Waiting for beacon phase0 .... DONE!
-Waiting for beacon altair .... DONE!
-Waiting for beacon bellatrix pre-merge .... DONE!
-Waiting for beacon bellatrix merge .... DONE!
+$ laconic-so --stack fixturenet-eth deploy exec fixturenet-eth-bootnode-lighthouse /scripts/status-internal.sh
+Waiting for geth to generate DAG.... done
+Waiting for beacon phase0.... done
+Waiting for beacon altair.... done
+Waiting for beacon bellatrix pre-merge.... done
+Waiting for beacon bellatrix merge.... done

-$ docker ps -f 'name=laconic' --format 'table {{.Names}}\t{{.Ports}}'  | cut -d'-' -f3- | sort
-NAMES                                                                           PORTS
-fixturenet-eth-bootnode-geth-1         8545-8546/tcp, 30303/udp, 0.0.0.0:55847->9898/tcp, 0.0.0.0:55848->30303/tcp
-fixturenet-eth-bootnode-lighthouse-1
-fixturenet-eth-geth-1-1                8546/tcp, 30303/tcp, 30303/udp, 0.0.0.0:55851->8545/tcp
-fixturenet-eth-geth-2-1                8545-8546/tcp, 30303/tcp, 30303/udp
-fixturenet-eth-lighthouse-1-1          0.0.0.0:55858->8001/tcp
-fixturenet-eth-lighthouse-2-1
+$ laconic-so --stack fixturenet-eth deploy ps
+Running containers:
+id: c6538b60c0328dadfa2c5585c4d09674a6a13e6d712ff1cd82a26849e4e5679b, name: laconic-b12fa16e999821562937781f8ab0b1e8-fixturenet-eth-bootnode-geth-1, ports: 0.0.0.0:58909->30303/tcp, 0.0.0.0:58910->9898/tcp
+id: 5b70597a8211bc7e78d33e50486cb565a7f4a9ce581ce150b3bb450e342bdeda, name: laconic-b12fa16e999821562937781f8ab0b1e8-fixturenet-eth-bootnode-lighthouse-1, ports:
+id: 19ed78867b6c534d893835cdeb1e89a9ea553b8e8c02ab02468e4bd1563a340f, name: laconic-b12fa16e999821562937781f8ab0b1e8-fixturenet-eth-geth-1-1, ports: 0.0.0.0:58911->40000/tcp, 0.0.0.0:58912->6060/tcp, 0.0.0.0:58913->8545/tcp
+id: 8da0e30a1ce33122d8fd2225e4d26c7f30eb4bfbfa743f2af04d9db5d0bf7fa6, name: laconic-b12fa16e999821562937781f8ab0b1e8-fixturenet-eth-geth-2-1, ports:
+id: 387a42a14971034588ba9aeb9b9e2ca7fc0cc61b96f8fe8c2ab770c9d6fb1e0f, name: laconic-b12fa16e999821562937781f8ab0b1e8-fixturenet-eth-lighthouse-1-1, ports: 0.0.0.0:58917->8001/tcp
+id: de5115bf89087bae03b291664a73ffe3554fe23e79e4b8345e088b040d5580ac, name: laconic-b12fa16e999821562937781f8ab0b1e8-fixturenet-eth-lighthouse-2-1, ports:
+id: 2a7e5a0fb2be7fc9261a7b725a40818facbbe6d0cb2497d82c0e02de0a8e959b, name: laconic-b12fa16e999821562937781f8ab0b1e8-foundry-1, ports:
+
+$ laconic-so --stack fixturenet-eth deploy exec foundry "cast block-number"
+3
 ```

 ## Additional pieces
@@ -69,11 +73,11 @@ $ laconic-so deploy-system --include db,fixturenet-eth,ipld-eth-server,ipld-eth-
 # Status

 $ container-build/cerc-fixturenet-eth-lighthouse/scripts/status.sh
-Waiting for geth to generate DAG.... DONE!
-Waiting for beacon phase0.... DONE!
-Waiting for beacon altair.... DONE!
-Waiting for beacon bellatrix pre-merge.... DONE!
-Waiting for beacon bellatrix merge.... DONE!
+Waiting for geth to generate DAG.... done
+Waiting for beacon phase0.... done
+Waiting for beacon altair.... done
+Waiting for beacon bellatrix pre-merge.... done
+Waiting for beacon bellatrix merge.... done

 $ docker ps -f 'name=laconic' --format 'table {{.Names}}\t{{.Ports}}'  | cut -d'-' -f3- | sort
 NAMES                                                                           PORTS
@@ -1,12 +1,16 @@
 version: "1.0"
 name: fixturenet-eth
+decription: "Ethereum Fixturenet"
 repos:
  - cerc-io/go-ethereum
-  - cerc-io/lighthouse
+  - cerc-io/tx-spammer
+  - dboreham/foundry
 containers:
  - cerc/go-ethereum
  - cerc/lighthouse
  - cerc/fixturenet-eth-geth
  - cerc/fixturenet-eth-lighthouse
+  - cerc/foundry
 pods:
-  - fixturenet-eth
+  - fixturenet-eth
+  - foundry
@@ -1,18 +1,48 @@
 # Laconicd Fixturenet

-Instructions for deploying a local Laconic blockchain "fixturenet" for development and testing purposes using laconic-stack-orchestrator (the installation of which is covered [here](https://github.com/cerc-io/stack-orchestrator#user-mode)):
+Instructions for deploying a local Laconic blockchain "fixturenet" for development and testing purposes using laconic-stack-orchestrator.

-## Clone required repositories
+## 1. Install Laconic Stack Orchestrator
+Installation is covered in detail [here](https://github.com/cerc-io/stack-orchestrator#user-mode) but if you're on Linux and already have docker installed it should be as simple as:
 ```
-$ laconic-so setup-repositories --include cerc-io/laconicd,cerc-io/laconic-sdk,cerc-io/laconic-registry-cli
+$ mkdir my-working-dir
+$ cd my-working-dir
+$ curl -L -o ./laconic-so https://github.com/cerc-io/stack-orchestrator/releases/latest/download/laconic-so
+$ chmod +x ./laconic-so
+$ export PATH=$PATH:$(pwd)  # Or move laconic-so to ~/bin or your favorite on-path directory
 ```
-## Build the laconicd container
+## 2. Prepare the local build environment
+Note that this step needs only to be done once on a new machine. 
+Detailed instructions can be found [here](../build-support/README.md). For the impatient run these commands:
 ```
-$ laconic-so build-containers --include cerc/laconicd
+$ laconic-so --stack build-support build-containers --exclude cerc/builder-gerbil
+$ laconic-so --stack package-registry setup-repositories
+$ laconic-so --stack package-registry deploy-system up
 ```
-This should create a container with tag `cerc/laconicd` in the local image registry.
-## Deploy the stack
+Then add the localhost alias `gitea.local` and set `CERC_NPM_AUTH_TOKEN` to the token printed when the package-registry stack was deployed above:
 ```
-$ laconic-so deploy-system --include fixturenet-laconicd up
+$ sudo vi /etc/hosts
+$ export CERC_NPM_AUTH_TOKEN=<my-token>
+```
+
+## 3. Clone required repositories
+```
+$ laconic-so --stack fixturenet-laconicd setup-repositories
+```
+## 4. Build the stack's packages and containers
+```
+$ laconic-so --stack fixturenet-laconicd build-npms
+$ laconic-so --stack fixturenet-laconicd build-containers
+```
+## 5. Deploy the stack
+```
+$ laconic-so --stack fixturenet-laconicd deploy up
+```
+Correct operation should be verified by checking the laconicd container's logs with:
+```
+$ laconic-so --stack fixturenet-laconicd deploy logs
+```
+## 6. Test with the Registry CLI
+```
+$ laconic-so --stack fixturenet-laconicd deploy exec cli "laconic cns status"
 ```
-Correct operation should be verified by checking the laconicd container's log.
@@ -13,3 +13,6 @@ containers:
  - cerc/laconic-registry-cli
 pods:
  - fixturenet-laconicd
+config:
+  cli:
+    key: laconicd.mykey
@@ -0,0 +1,179 @@
+# MobyMask v2 watcher
+
+Instructions to deploy MobyMask v2 watcher stack using [laconic-stack-orchestrator](/README.md#install)
+
+## Setup
+
+Clone required repositories:
+
+```bash
+laconic-so --stack mobymask-v2 setup-repositories
+```
+
+NOTE: If repositories already exist and are checked out to different versions, `setup-repositories` command will throw an error.
+For getting around this, the repositories mentioned below can be removed and then run the command.
+
+Checkout to the required versions and branches in repos
+
+```bash
+# watcher-ts
+cd ~/cerc/watcher-ts
+git checkout v0.2.31
+
+# react-peer
+cd ~/cerc/react-peer
+git checkout v0.2.29
+
+# mobymask-ui
+cd ~/cerc/mobymask-ui
+git checkout laconic
+
+# laconicd
+cd ~/cerc/laconicd
+git checkout v0.8.0
+
+# MobyMask
+cd ~/cerc/MobyMask
+git checkout v0.1.1
+```
+
+Build the container images:
+
+```bash
+laconic-so --stack mobymask-v2 build-containers
+```
+
+This should create the required docker images in the local image registry.
+
+Deploy the stack:
+
+* Deploy the laconic chain
+
+  ```bash
+  laconic-so --stack mobymask-v2 deploy-system --include mobymask-laconicd up
+  ```
+
+* Check that laconic chain status is healthy
+
+  ```bash
+  docker ps
+  ```
+
+* Export the private key from laconicd
+
+  ```bash
+  laconic-so --stack mobymask-v2 deploy-system --include mobymask-laconicd exec laconicd "echo y | laconicd keys export mykey --unarmored-hex --unsafe"
+  ```
+
+* Set the private key in [secrets.json](../../config/watcher-mobymask-v2/secrets.json) file that will be used by mobymask container to deploy contract
+
+* Create a new account
+
+  ```bash
+  laconic-so --stack mobymask-v2 deploy-system --include mobymask-laconicd exec laconicd "laconicd keys add alice"
+  ```
+
+* Transfer balance to new account
+
+  ```bash
+  laconic-so --stack mobymask-v2 deploy-system --include mobymask-laconicd exec laconicd 'laconicd tx bank send $(laconicd keys show mykey -a) $(laconicd keys show alice -a) 1000000000000000000000000aphoton --fees 2000aphoton'
+  ```
+
+* Export the private key of new account from laconicd
+
+  ```bash
+  laconic-so --stack mobymask-v2 deploy-system --include mobymask-laconicd exec laconicd "echo y | laconicd keys export alice --unarmored-hex --unsafe"
+  ```
+
+* Set the private key (`PRIVATE_KEY`) in [peer-start.sh](../../config/watcher-mobymask-v2/peer-start.sh) file that will be used to start the peer that sends txs to L2 chain
+
+* Deploy the other containers
+
+  ```bash
+  laconic-so --stack mobymask-v2 deploy-system --include watcher-mobymask-v2 up
+  ```
+
+* Check that all containers are healthy using `docker ps`
+
+  NOTE: The `mobymask-ui` container might not start. If mobymask-app is not running at http://localhost:3002, run command again to start the container
+
+  ```bash
+  laconic-so --stack mobymask-v2 deploy-system --include watcher-mobymask-v2 up
+  ```
+
+## Tests
+
+Find the watcher container's id:
+
+```bash
+laconic-so --stack mobymask-v2 deploy-system --include watcher-mobymask-v2 ps | grep "mobymask-watcher-server"
+```
+
+Example output
+
+```
+id: 5d3aae4b22039fcd1c9b18feeb91318ede1100581e75bb5ac54f9e436066b02c, name: laconic-bfb01caf98b1b8f7c8db4d33f11b905a-mobymask-watcher-server-1, ports: 0.0.0.0:3001->3001/tcp, 0.0.0.0:9001->9001/tcp, 0.0.0.0:9090->9090/tcp
+```
+
+In above output the container ID is `5d3aae4b22039fcd1c9b18feeb91318ede1100581e75bb5ac54f9e436066b02c`
+
+Export it for later use:
+
+```bash
+export CONTAINER_ID=<CONTAINER_ID>
+```
+
+Run the peer tests:
+
+```bash
+docker exec -w /app/packages/peer $CONTAINER_ID yarn test
+```
+
+## Web Apps
+
+Check that the status for web-app containers are healthy by using `docker ps`
+
+### mobymask-app
+
+The mobymask-app should be running at http://localhost:3002
+
+### peer-test-app
+
+The peer-test-app should be running at http://localhost:3003
+
+## Details
+
+* The relay node for p2p network is running at http://localhost:9090
+
+* The [peer package](https://github.com/cerc-io/watcher-ts/tree/main/packages/peer) (published in [gitea](https://git.vdb.to/cerc-io/-/packages/npm/@cerc-io%2Fpeer)) can be used in client code for connecting to the network
+
+* The [react-peer package](https://github.com/cerc-io/react-peer/tree/main/packages/react-peer) (published in [gitea](https://git.vdb.to/cerc-io/-/packages/npm/@cerc-io%2Freact-peer)) which uses the peer package can be used in react app for connecting to the network
+
+## Demo
+
+Follow the [demo](./demo.md) to try out the MobyMask app with L2 chain
+
+## Clean up
+
+Stop all the services running in background run:
+
+```bash
+laconic-so --stack mobymask-v2 deploy-system --include watcher-mobymask-v2 down
+
+laconic-so --stack mobymask-v2 deploy-system --include mobymask-laconicd down
+```
+
+Clear volumes:
+
+* List all volumes
+
+  ```bash
+  docker volume ls
+  ```
+
+* Remove volumes created by this stack
+
+  Example:
+  ```bash
+  docker volume rm laconic-bfb01caf98b1b8f7c8db4d33f11b905a_moby_data_server
+  ```
@@ -0,0 +1,132 @@
+# Demo
+
+* Get the root invite link URL for mobymask-app
+
+  ```
+  laconic-so --stack mobymask-v2 deploy-system --include watcher-mobymask-v2 logs mobymask
+  ```
+
+  The invite link is seen at the end of the logs
+  Example:
+  ```
+  laconic-bfb01caf98b1b8f7c8db4d33f11b905a-mobymask-1  | http://127.0.0.1:3002/#/members?invitation=%7B%22v%22%3A1%2C%22signedDelegations%22%3A%5B%7B%22signature%22%3A%220x7559bd412f02677d60820e38243acf61547f79339395a34f7d4e1630e645aeb30535fc219f79b6fbd3af0ce3bd05132ad46d2b274a9fbc4c36bc71edd09850891b%22%2C%22delegation%22%3A%7B%22delegate%22%3A%220xc0838c92B2b71756E0eAD5B3C1e1F186baeEAAac%22%2C%22authority%22%3A%220x0000000000000000000000000000000000000000000000000000000000000000%22%2C%22caveats%22%3A%5B%7B%22enforcer%22%3A%220x558024C7d593B840E1BfD83E9B287a5CDad4db15%22%2C%22terms%22%3A%220x0000000000000000000000000000000000000000000000000000000000000000%22%7D%5D%7D%7D%5D%2C%22key%22%3A%220x98da9805821f1802196443e578fd32af567bababa0a249c07c82df01ecaa7d8d%22%7D
+  ```
+
+* Open the invite link in browser to use the mobymask-app.
+
+  NOTE: Before opening the invite link, clear the browser cache (local storage) for http://127.0.0.1:3002 to remove old invitations
+
+* In the debug panel, check if it is connected to the p2p network (It should be connected to atleast one other peer for pubsub to work).
+
+* Create an invite link in the app by clicking on `Create new invite link` button.
+
+* Switch to the `MESSAGES` tab in debug panel for viewing incoming messages later.
+
+* Open the invite link in a new browser with different profile (to simulate remote browser)
+  * Check that it is connected to any other peer in the network.
+
+* In `Report a phishing attempt` section, report multiple phishers using the `submit` button. Click on the `Submit batch to p2p network` button. This broadcasts signed invocations to the connected peers.
+
+* In the `MESSAGES` tab of other browsers, a message can be seen with the signed invocations.
+
+* In a terminal check logs from the watcher peer container.
+
+  * Get the container id
+
+    ```bash
+    laconic-so --stack mobymask-v2 deploy-system --include watcher-mobymask-v2 ps | grep mobymask-watcher-peer
+    ```
+
+  * Check logs
+
+    ```bash
+    docker logs -f CONTAINER_ID
+    ```
+
+* It should have received the message, sent transaction to L2 chain and received a transaction receipt with block details.
+
+  Example log:
+
+  ```
+  2023-03-23T10:25:19.771Z vulcanize:peer-listener [10:25:19] Received a message on mobymask P2P network from peer: PeerId(12D3KooWAVNswtcrX12iDYukEoxdQwD34kJyRWcQTfZ4unGg2xjd)
+  2023-03-23T10:25:24.143Z laconic:libp2p-utils Transaction receipt for invoke message {
+    to: '0x558024C7d593B840E1BfD83E9B287a5CDad4db15',
+    blockNumber: 1996,
+    blockHash: '0xebef19c21269654804b2ef2d4bb5cb6c88743b37ed77e82222dc5671debf3afb',
+    transactionHash: '0xf8c5a093a93f793012196073a7d0cb3ed6fbd2846126c066cb31c72100960cb1',
+    effectiveGasPrice: '1500000007',
+    gasUsed: '250000'
+  }
+  ```
+
+* Check the phisher in watcher GQL: http://localhost:3001/graphql
+  * Use the blockHash from transaction receipt details or query for latest block
+
+    ```gql
+    query {
+      latestBlock {
+        hash
+        number
+      }
+    }
+    ```
+
+  * Get the deployed contract address
+
+    ```bash
+    laconic-so --stack mobymask-v2 deploy-system --include watcher-mobymask-v2 exec mobymask-app "cat src/config.json"
+    ```
+
+    The value of `address` field is the deployed contract address
+
+  * Check for phisher value
+
+    ```gql
+    query {
+      isPhisher(
+        blockHash: "TX_OR_LATEST_BLOCK_HASH",
+        contractAddress: "CONTRACT_ADDRESS",
+        # If reported phisher name was "test" then key0 value is "TWT:test"
+        key0: "TWT:PHISHER_NAME"
+      ) {
+        value
+      }
+    }
+    ```
+
+    It should return true for reported phisher names.
+
+  * Watcher internally is using L2 chain `eth_getStorageAt` method.
+
+* Check the phisher name in mobymask app in `Check Phisher Status` section.
+    * Watcher GQL API is used for checking phisher.
+
+* Manage the invitations by clicking on the `Outstanding Invitations in p2p network`.
+
+* Revoke the created invitation by clicking on `Revoke (p2p network)`
+
+* Revocation messages can be seen in the debug panel `MESSAGES` tab of other browsers.
+
+* Check the watcher peer logs. It should receive a message and log the transaction receipt for revoke message.
+
+* Try reporting a phisher from the revoked invitee's browser.
+
+  * The invocation message for reporting phisher would be broadcasted to all peers.
+
+  * Check the watcher peer logs. A transaction failed error should be logged.
+
+  * Check the reported phisher in [watcher GQL](https://localhost:3001/graphql)
+
+    ```gql
+    query {
+      isPhisher(
+        blockHash: "LATEST_BLOCK_HASH",
+        contractAddress: "CONTRACT_ADDRESS",
+        key0: "TWT:PHISHER_NAME"
+      ) {
+        value
+      }
+    }
+    ```
+
+    It should return false as the invitation/delegation used for reporting phishers has been revoked.
@@ -0,0 +1,17 @@
+version: "1.0"
+name: mobymask-v2
+repos:
+  - cerc-io/watcher-ts
+  - cerc-io/react-peer
+  - cerc-io/mobymask-ui
+  - cerc-io/laconicd
+  - cerc-io/MobyMask
+containers:
+  - cerc/watcher-mobymask-v2
+  - cerc/react-peer
+  - cerc/mobymask-ui
+  - cerc/laconicd
+  - cerc/mobymask
+pods:
+  - mobymask-laconicd
+  - watcher-mobymask-v2
@@ -0,0 +1,5 @@
+# Package Registry Stack
+
+The Package Registry Stack supports a build environment that requires a package registry (initially for NPM packages only).
+
+Setup instructions can be found [here](../build-support/README.md).
@@ -0,0 +1,15 @@
+version: "1.1"
+name: package-registry
+decription: "Local Package Registry"
+repos:
+  - cerc-io/hosting
+  - telackey/act_runner
+containers:
+  - cerc/act-runner
+  - cerc/act-runner-task-executor
+pods:
+  - name: gitea
+    repository: cerc-io/hosting
+    path: gitea
+    pre_start_command: "run-this-first.sh"
+    post_start_command: "initialize-gitea.sh"
@@ -0,0 +1,3 @@
+# Test Stack
+
+A stack for test/demo purposes.
@@ -0,0 +1,9 @@
+version: "1.0"
+name: test
+description: "A test stack"
+repos:
+  - cerc-io/laconicd
+containers:
+  - cerc/test-container
+pods:
+  - test
@@ -1,2 +1,2 @@
 # This file should be re-generated running: scripts/update-version-file.sh script
-v1.0.11-alpha-edbf674
+v1.0.33-7e1137f
@@ -16,9 +16,13 @@
 # Deploys the system components using docker-compose

 import hashlib
+import copy
 import os
 import sys
-from python_on_whales import DockerClient
+from dataclasses import dataclass
+from decouple import config
+import subprocess
+from python_on_whales import DockerClient, DockerException
 import click
 import importlib.resources
 from pathlib import Path
@@ -30,74 +34,69 @@ from .util import include_exclude_check, get_parsed_stack_config
@click.option("--exclude", help="don\'t start these components")
@click.option("--cluster", help="specify a non-default cluster name")
@click.argument('command', required=True)  # help: command: up|down|ps
-@click.argument('services', nargs=-1)  # help: command: up|down|ps <service1> <service2>
+@click.argument('extra_args', nargs=-1)  # help: command: up|down|ps <service1> <service2>
@click.pass_context
-def command(ctx, include, exclude, cluster, command, services):
+def command(ctx, include, exclude, cluster, command, extra_args):
    '''deploy a stack'''

    # TODO: implement option exclusion and command value constraint lost with the move from argparse to click

+    debug = ctx.obj.debug
    quiet = ctx.obj.quiet
    verbose = ctx.obj.verbose
+    local_stack = ctx.obj.local_stack
    dry_run = ctx.obj.dry_run
    stack = ctx.obj.stack

-    # See: https://stackoverflow.com/questions/25389095/python-get-path-of-root-project-structure
-    compose_dir = Path(__file__).absolute().parent.joinpath("data", "compose")
-
-    if cluster is None:
-        # Create default unique, stable cluster name from confile file path
-        # TODO: change this to the config file path
-        path = os.path.realpath(sys.argv[0])
-        hash = hashlib.md5(path.encode()).hexdigest()
-        cluster = f"laconic-{hash}"
-        if verbose:
-            print(f"Using cluster name: {cluster}")
-
-    # See: https://stackoverflow.com/a/20885799/1701505
-    from . import data
-    with importlib.resources.open_text(data, "pod-list.txt") as pod_list_file:
-        all_pods = pod_list_file.read().splitlines()
-
-    pods_in_scope = []
-    if stack:
-        stack_config = get_parsed_stack_config(stack)
-        # TODO: syntax check the input here
-        pods_in_scope = stack_config['pods']
-    else:
-        pods_in_scope = all_pods
-
-    if verbose:
-        print(f"Pods: {pods_in_scope}")
-
-    # Construct a docker compose command suitable for our purpose
-
-    compose_files = []
-    for pod in pods_in_scope:
-        if include_exclude_check(pod, include, exclude):
-            compose_file_name = os.path.join(compose_dir, f"docker-compose-{pod}.yml")
-            compose_files.append(compose_file_name)
-        else:
-            if verbose:
-                print(f"Excluding: {pod}")
-
-    if verbose:
-        print(f"files: {compose_files}")
+    cluster_context = _make_cluster_context(ctx.obj, include, exclude, cluster)

    # See: https://gabrieldemarmiesse.github.io/python-on-whales/sub-commands/compose/
-    docker = DockerClient(compose_files=compose_files, compose_project_name=cluster)
+    docker = DockerClient(compose_files=cluster_context.compose_files, compose_project_name=cluster_context.cluster)

-    services_list = list(services) or None
+    extra_args_list = list(extra_args) or None

    if not dry_run:
        if command == "up":
+            container_exec_env = _make_runtime_env(ctx.obj)
+            for attr, value in container_exec_env.items():
+                os.environ[attr] = value
            if verbose:
-                print(f"Running compose up for services: {services_list}")
-            docker.compose.up(detach=True, services=services_list)
+                print(f"Running compose up with container_exec_env: {container_exec_env}, extra_args: {extra_args_list}")
+            for pre_start_command in cluster_context.pre_start_commands:
+                _run_command(ctx.obj, cluster_context.cluster, pre_start_command)
+            docker.compose.up(detach=True, services=extra_args_list)
+            for post_start_command in cluster_context.post_start_commands:
+                _run_command(ctx.obj, cluster_context.cluster, post_start_command)
+
+            _orchestrate_cluster_config(ctx.obj, cluster_context.config, docker, container_exec_env)
+
        elif command == "down":
            if verbose:
                print("Running compose down")
            docker.compose.down()
+        elif command == "exec":
+            if extra_args_list is None or len(extra_args_list) < 2:
+                print("Usage: exec <service> <cmd>")
+                sys.exit(1)
+            service_name = extra_args_list[0]
+            command_to_exec = ["sh", "-c"] + extra_args_list[1:]
+            container_exec_env = _make_runtime_env(ctx.obj)
+            if verbose:
+                print(f"Running compose exec {service_name} {command_to_exec}")
+            try:
+                docker.compose.execute(service_name, command_to_exec, envs=container_exec_env)
+            except DockerException as error:
+                print(f"container command returned error exit status")
+        elif command == "port":
+            if extra_args_list is None or len(extra_args_list) < 2:
+                print("Usage: port <service> <exposed-port>")
+                sys.exit(1)
+            service_name = extra_args_list[0]
+            exposed_port = extra_args_list[1]
+            if verbose:
+                print(f"Running compose port {service_name} {exposed_port}")
+            mapped_port_data = docker.compose.port(service_name, exposed_port)
+            print(f"{mapped_port_data[0]}:{mapped_port_data[1]}")
        elif command == "ps":
            if verbose:
                print("Running compose ps")
@@ -122,4 +121,212 @@ def command(ctx, include, exclude, cluster, command, services):
        elif command == "logs":
            if verbose:
                print("Running compose logs")
-            docker.compose.logs()
+            logs_output = docker.compose.logs(services=extra_args_list if extra_args_list is not None else [])
+            print(logs_output)
+
+
+def get_stack_status(ctx, stack):
+
+    ctx_copy = copy.copy(ctx)
+    ctx_copy.stack = stack
+
+    cluster_context = _make_cluster_context(ctx_copy, None, None, None)
+    docker = DockerClient(compose_files=cluster_context.compose_files, compose_project_name=cluster_context.cluster)
+    # TODO: refactor to avoid duplicating this code above
+    if ctx.verbose:
+        print("Running compose ps")
+    container_list = docker.compose.ps()
+    if len(container_list) > 0:
+        if ctx.debug:
+            print(f"Container list from compose ps: {container_list}")
+        return True
+    else:
+        if ctx.debug:
+            print("No containers found from compose ps")
+        False
+
+
+def _make_runtime_env(ctx):
+    container_exec_env = {
+        "CERC_HOST_UID": f"{os.getuid()}",
+        "CERC_HOST_GID": f"{os.getgid()}"
+    }
+    container_exec_env.update({"CERC_SCRIPT_DEBUG": "true"} if ctx.debug else {})
+    return container_exec_env
+
+
+def _make_cluster_context(ctx, include, exclude, cluster):
+
+    if ctx.local_stack:
+        dev_root_path = os.getcwd()[0:os.getcwd().rindex("stack-orchestrator")]
+        print(f'Local stack dev_root_path (CERC_REPO_BASE_DIR) overridden to: {dev_root_path}')
+    else:
+        dev_root_path = os.path.expanduser(config("CERC_REPO_BASE_DIR", default="~/cerc"))
+
+    # See: https://stackoverflow.com/questions/25389095/python-get-path-of-root-project-structure
+    compose_dir = Path(__file__).absolute().parent.joinpath("data", "compose")
+
+    if cluster is None:
+        # Create default unique, stable cluster name from confile file path and stack name if provided
+        # TODO: change this to the config file path
+        path = os.path.realpath(sys.argv[0])
+        unique_cluster_descriptor = f"{path},{ctx.stack},{include},{exclude}"
+        if ctx.debug:
+            print(f"pre-hash descriptor: {unique_cluster_descriptor}")
+        hash = hashlib.md5(unique_cluster_descriptor.encode()).hexdigest()
+        cluster = f"laconic-{hash}"
+        if ctx.verbose:
+            print(f"Using cluster name: {cluster}")
+
+    # See: https://stackoverflow.com/a/20885799/1701505
+    from . import data
+    with importlib.resources.open_text(data, "pod-list.txt") as pod_list_file:
+        all_pods = pod_list_file.read().splitlines()
+
+    pods_in_scope = []
+    if ctx.stack:
+        stack_config = get_parsed_stack_config(ctx.stack)
+        # TODO: syntax check the input here
+        pods_in_scope = stack_config['pods']
+        cluster_config = stack_config['config'] if 'config' in stack_config else None
+    else:
+        pods_in_scope = all_pods
+        cluster_config = None
+
+    # Convert all pod definitions to v1.1 format
+    pods_in_scope = _convert_to_new_format(pods_in_scope)
+
+    if ctx.verbose:
+        print(f"Pods: {pods_in_scope}")
+
+    # Construct a docker compose command suitable for our purpose
+
+    compose_files = []
+    pre_start_commands = []
+    post_start_commands = []
+    for pod in pods_in_scope:
+        pod_name = pod["name"]
+        pod_repository = pod["repository"]
+        pod_path = pod["path"]
+        if include_exclude_check(pod_name, include, exclude):
+            if pod_repository is None or pod_repository == "internal":
+                compose_file_name = os.path.join(compose_dir, f"docker-compose-{pod_path}.yml")
+            else:
+                pod_root_dir = os.path.join(dev_root_path, pod_repository.split("/")[-1], pod["path"])
+                compose_file_name = os.path.join(pod_root_dir, "docker-compose.yml")
+                pod_pre_start_command = pod["pre_start_command"]
+                pod_post_start_command = pod["post_start_command"]
+                if pod_pre_start_command is not None:
+                    pre_start_commands.append(os.path.join(pod_root_dir, pod_pre_start_command))
+                if pod_post_start_command is not None:
+                    post_start_commands.append(os.path.join(pod_root_dir, pod_post_start_command))
+            compose_files.append(compose_file_name)
+        else:
+            if ctx.verbose:
+                print(f"Excluding: {pod_name}")
+
+    if ctx.verbose:
+        print(f"files: {compose_files}")
+
+    return cluster_context(cluster, compose_files, pre_start_commands, post_start_commands, cluster_config)
+
+
+class cluster_context:
+    def __init__(self, cluster, compose_files, pre_start_commands, post_start_commands, config) -> None:
+        self.cluster = cluster
+        self.compose_files = compose_files
+        self.pre_start_commands = pre_start_commands
+        self.post_start_commands = post_start_commands
+        self.config = config
+
+
+def _convert_to_new_format(old_pod_array):
+    new_pod_array = []
+    for old_pod in old_pod_array:
+        if isinstance(old_pod, dict):
+            new_pod_array.append(old_pod)
+        else:
+            new_pod = {
+                "name": old_pod,
+                "repository": "internal",
+                "path": old_pod
+            }
+            new_pod_array.append(new_pod)
+    return new_pod_array
+
+
+def _run_command(ctx, cluster_name, command):
+    if ctx.verbose:
+        print(f"Running command: {command}")
+    command_dir = os.path.dirname(command)
+    command_file = os.path.join(".", os.path.basename(command))
+    command_env = os.environ.copy()
+    command_env["CERC_SO_COMPOSE_PROJECT"] = cluster_name
+    if ctx.debug:
+        command_env["CERC_SCRIPT_DEBUG"] = "true"
+    command_result = subprocess.run(command_file, shell=True, env=command_env, cwd=command_dir)
+    if command_result.returncode != 0:
+        print(f"FATAL Error running command: {command}")
+        sys.exit(1)
+
+
+def _orchestrate_cluster_config(ctx, cluster_config, docker, container_exec_env):
+
+    @dataclass
+    class ConfigDirective:
+        source_container: str
+        source_variable: str
+        destination_container: str
+        destination_variable: str
+
+    if cluster_config is not None:
+        for container in cluster_config:
+            container_config = cluster_config[container]
+            if ctx.verbose:
+                print(f"{container} config: {container_config}")
+            for directive in container_config:
+                pd = ConfigDirective(
+                    container_config[directive].split(".")[0],
+                    container_config[directive].split(".")[1],
+                    container,
+                    directive
+                )
+                if ctx.verbose:
+                    print(f"Setting {pd.destination_container}.{pd.destination_variable}"
+                          f" = {pd.source_container}.{pd.source_variable}")
+                # TODO: add a timeout
+                waiting_for_data = True
+                while waiting_for_data:
+                    # TODO: fix the script paths so they're consistent between containers
+                    source_value = None
+                    try:
+                        source_value = docker.compose.execute(pd.source_container,
+                                                              ["sh", "-c",
+                                                                  "sh /docker-entrypoint-scripts.d/export-"
+                                                                  f"{pd.source_variable}.sh"],
+                                                              tty=False,
+                                                              envs=container_exec_env)
+                    except DockerException as error:
+                        if ctx.debug:
+                            print(f"Docker exception reading config source: {error}")
+                        # If the script executed failed for some reason, we get:
+                        # "It returned with code 1"
+                        if "It returned with code 1" in str(error):
+                            if ctx.verbose:
+                                print("Config export script returned an error, re-trying")
+                        # If the script failed to execute (e.g. the file is not there) then we get:
+                        # "It returned with code 2"
+                        if "It returned with code 2" in str(error):
+                            print(f"Fatal error reading config source: {error}")
+                    if source_value:
+                        if ctx.debug:
+                            print(f"fetched source value: {source_value}")
+                        destination_output = docker.compose.execute(pd.destination_container,
+                                                                    ["sh", "-c",
+                                                                        f"sh /scripts/import-{pd.destination_variable}.sh"
+                                                                        f" {source_value}"],
+                                                                    tty=False,
+                                                                    envs=container_exec_env)
+                        waiting_for_data = False
+                    if ctx.debug:
+                        print(f"destination output: {destination_output}")
@@ -55,5 +55,6 @@ def cli(ctx, stack, quiet, verbose, dry_run, local_stack, debug, continue_on_err
 cli.add_command(setup_repositories.command, "setup-repositories")
 cli.add_command(build_containers.command, "build-containers")
 cli.add_command(build_npms.command, "build-npms")
+cli.add_command(deploy_system.command, "deploy")  # deploy is an alias for deploy-system
 cli.add_command(deploy_system.command, "deploy-system")
 cli.add_command(version.command, "version")
@@ -5,8 +5,11 @@ Thank you for taking the time to make a contribution to Stack Orchestrator.
 ## Install (developer mode)

 Suitable for developers either modifying or debugging the orchestrator Python code:
-#### Prerequisites
-In addition to the binary install prerequisites listed above, the following are required:
+
+### Prerequisites
+
+In addition to the pre-requisites listed in the [README](/README.md), the following are required:
+
 1. Python venv package
   This may or may not be already installed depending on the host OS and version. Check by running:
   ```
@@ -18,26 +21,32 @@ In addition to the binary install prerequisites listed above, the following are
   ```
   $ apt install python3.10-venv
   ```
-#### Install
+
+### Install
+
 1. Clone this repository:
   ```
   $ git clone (https://github.com/cerc-io/stack-orchestrator.git
   ```
-4. Enter the project directory:
+
+2. Enter the project directory:
   ```
   $ cd stack-orchestrator
   ```
-5. Create and activate a venv:
+
+3. Create and activate a venv:
   ```
   $ python3 -m venv venv
   $ source ./venv/bin/activate
   (venv) $
   ```
-6. Install the cli in edit mode:
+
+4. Install the cli in edit mode:
   ```
   $ pip install --editable .
   ```
-7. Verify installation:
+
+5. Verify installation:
   ```
   (venv) $ laconic-so
   Usage: laconic-so [OPTIONS] COMMAND [ARGS]...
@@ -56,19 +65,23 @@ In addition to the binary install prerequisites listed above, the following are
    setup-repositories  git clone the set of repositories required to build...
   ```

-#### Build a zipapp (single file distributable script)
+## Build a zipapp (single file distributable script)
+
 Use shiv to build a single file Python executable zip archive of laconic-so:
+
 1. Install [shiv](https://github.com/linkedin/shiv):
   ```
   $ (venv) pip install shiv
   $ (venv) pip install wheel
   ```
-1. Run shiv to create a zipapp file:
+
+2. Run shiv to create a zipapp file:
   ```
   $ (venv)  shiv -c laconic-so -o laconic-so .
   ```
   This creates a file `./laconic-so` that is executable outside of any venv, and on other machines and OSes and architectures, and requiring only the system Python3:
-1. Verify it works:
+
+3. Verify it works:
   ```
   $ cp stack-orchetrator/laconic-so ~/bin
   $ laconic-so
@@ -88,4 +101,4 @@ Use shiv to build a single file Python executable zip archive of laconic-so:
      setup-repositories  git clone the set of repositories required to build...
   ```

-
+For cutting releases, use the [shiv build script](/scripts/build_shiv_package.sh).
@@ -1 +0,0 @@
-8f6452ed13e85a447103a7fff7cf3fb8ff5ea51a
@@ -0,0 +1,29 @@
+# Release Process
+
+## Manually publish to github releases
+
+In order to build, the shiv and wheel packages must be installed:
+```
+$ pip install shiv
+$ pip install wheel
+```
+
+Then:
+
+1. Define `CERC_GH_RELEASE_SCRIPTS_DIR`
+1. Define `CERC_PACKAGE_RELEASE_GITHUB_TOKEN`
+1. Run `./scripts/tag_new_release.sh <major> <minor> <patch>`
+1. Run `./scripts/build_shiv_package.sh`
+1. Run `./scripts/publish_shiv_package_github.sh <major> <minor> <patch>`
+1. Commit the new version file.
+
+e.g.
+
+```
+$ export CERC_GH_RELEASE_SCRIPTS_DIR=~/projects/cerc/github-release-api/
+$ export CERC_PACKAGE_RELEASE_GITHUB_TOKEN=github_pat_xxxxxx
+$ ./scripts/tag_new_release.sh 1 0 17
+$ ./scripts/build_shiv_package.sh
+$ ./scripts/publish_shiv_package_github.sh 1 0 17
+```
+
@@ -1,37 +1,82 @@
-# Spec
+# Specification

-TODO: update

 ## Implementation
-The orchestrator's operation is driven by files shown below. `repository-list.txt` container the list of git repositories; `container-image-list.txt` contains
-the list of container image names, while `clister-list.txt` specifies the set of compose components (corresponding to individual docker-compose-xxx.yml files which may in turn specify more than one container).
-Files required to build each container image are stored under `./container-build/<container-name>`
-Files required at deploy-time are stored under `./config/<component-name>`
+
+The orchestrator's operation is driven by files shown below. 
+
+- `repository-list.txt` contains the list of git repositories; 
+- `container-image-list.txt` contains the list of container image names
+- `pod-list.txt` specifies the set of compose components (corresponding to individual docker-compose-xxx.yml files which may in turn specify more than one container).
+- `container-build/` contains the files required to build each container image
+- `config/` contains the files required at deploy time
+
 ```
+├── container-image-list.txt
 ├── pod-list.txt
+├── repository-list.txt
 ├── compose
 │   ├── docker-compose-contract.yml
-│   ├── docker-compose-db-sharding.yml
-│   ├── docker-compose-db.yml
+│   ├── docker-compose-eth-probe.yml
 │   ├── docker-compose-eth-statediff-fill-service.yml
+│   ├── docker-compose-fixturenet-eth.yml
+│   ├── docker-compose-fixturenet-laconicd.yml
 │   ├── docker-compose-go-ethereum-foundry.yml
 │   ├── docker-compose-ipld-eth-beacon-db.yml
 │   ├── docker-compose-ipld-eth-beacon-indexer.yml
+│   ├── docker-compose-ipld-eth-db.yml
 │   ├── docker-compose-ipld-eth-server.yml
-│   ├── docker-compose-lighthouse.yml
-│   └── docker-compose-prometheus-grafana.yml
+│   ├── docker-compose-keycloak.yml
+│   ├── docker-compose-laconicd.yml
+│   ├── docker-compose-prometheus-grafana.yml
+│   ├── docker-compose-test.yml
+│   ├── docker-compose-tx-spammer.yml
+│   ├── docker-compose-watcher-erc20.yml
+│   ├── docker-compose-watcher-erc721.yml
+│   ├── docker-compose-watcher-mobymask.yml
+│   └── docker-compose-watcher-uniswap-v3.yml
 ├── config
-│   └── ipld-eth-server
+│   ├── fixturenet-eth
+│   ├── fixturenet-laconicd
+│   ├── ipld-eth-beacon-indexer
+│   ├── ipld-eth-server
+│   ├── keycloak
+│   ├── postgresql
+│   ├── tx-spammer
+│   ├── watcher-erc20
+│   ├── watcher-erc721
+│   ├── watcher-mobymask
+│   └── watcher-uniswap-v3
 ├── container-build
+│   ├── cerc-builder-js
+│   ├── cerc-eth-probe
 │   ├── cerc-eth-statediff-fill-service
+│   ├── cerc-eth-statediff-service
+│   ├── cerc-fixturenet-eth-geth
+│   ├── cerc-fixturenet-eth-lighthouse
 │   ├── cerc-go-ethereum
 │   ├── cerc-go-ethereum-foundry
 │   ├── cerc-ipld-eth-beacon-db
 │   ├── cerc-ipld-eth-beacon-indexer
 │   ├── cerc-ipld-eth-db
 │   ├── cerc-ipld-eth-server
+│   ├── cerc-keycloak
+│   ├── cerc-laconic-registry-cli
+│   ├── cerc-laconicd
 │   ├── cerc-lighthouse
-│   └── cerc-test-contract
-├── container-image-list.txt
-├── repository-list.txt
+│   ├── cerc-test-container
+│   ├── cerc-test-contract
+│   ├── cerc-tx-spammer
+│   ├── cerc-uniswap-v3-info
+│   ├── cerc-watcher-erc20
+│   ├── cerc-watcher-erc721
+│   ├── cerc-watcher-mobymask
+│   ├── cerc-watcher-uniswap-v3
+└── stacks
+    ├── erc20
+    ├── erc721
+    ├── fixturenet-eth
+    ├── fixturenet-laconicd
+    ├── mobymask
+    └── uniswap-v3
 ```
@@ -1,6 +1,6 @@
 python-decouple>=3.6
 GitPython>=3.1.27
 tqdm>=4.64.0
-python-on-whales>=0.52.0
+python-on-whales>=0.58.0
 click>=8.1.3
 pyyaml>=6.0
--- a/Show More
+++ b/Show More
				`@@ -0,0 +1 @@`
				`REACT_APP_WATCHER_URI=http://localhost:3001/graphql`
				`@@ -0,0 +1 @@`
				`RELAY="/ip4/127.0.0.1/tcp/9090/ws/p2p/12D3KooWSPCsVkHVyLQoCqhu2YRPvvM7o6r6NRYyLM5zeA6Uig5t"`