update pod name

* Move cluster name generation into a function

* Generate a unique deployment id for each deployment

app/data/stacks/keycloak/README.md

@@ -0,0 +1,41 @@
+# keycloak
+
+Deploys a stand alone [keycloak](https://www.keycloak.org)
+
+## Clone required repositories
+
+```
+$ laconic-so --stack keycloak setup-repositories
+```
+
+## Build containers
+
+```
+$ laconic-so --stack keycloak build-containers
+```
+
+## Create a deployment
+
+```
+$ laconic-so --stack keycloak deploy init --map-ports-to-host any-same --output keycloak-spec.yml
+$ laconic-so deploy create --spec-file keycloak-spec.yml --deployment-dir keycloak-deployment
+```
+
+## Start the stack
+
+```
+$ laconic-so deployment --dir keycloak-deployment start
+```
+
+Display stack status:
+
+```
+$ laconic-so deployment --dir keycloak-deployment ps
+Running containers:
+```
+
+See stack logs:
+
+```
+$ laconic-so deployment --dir keycloak-deployment logs
+```

app/data/stacks/keycloak/deploy/commands.py

@@ -0,0 +1,14 @@
+# Copyright © 2023 Vulcanize
+
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http:#www.gnu.org/licenses/>.

app/data/stacks/keycloak/stack.yml

@@ -0,0 +1,13 @@
+version: "1.0"
+name: keycloak
+description: "Keycloak"
+repos:
+  - git.vdb.to/cerc-io/keycloak-reg-api
+  - git.vdb.to/cerc-io/keycloak-reg-ui
+containers:
+  - cerc/keycloak
+  - cerc/keycloak-reg-api
+  - cerc/keycloak-reg-ui
+  - cerc/webapp-base
+pods:
+  - keycloak

docs/webapp.md

@@ -0,0 +1,64 @@
+### Building and Running Webapps
+
+It is possible to build and run Next.js webapps using the `build-webapp` and `run-webapp` subcommands.
+
+To make it easier to build once and deploy into different environments and with different configuration,
+compilation and static page generation are separated in the `build-webapp` and `run-webapp` steps.
+
+This offers much more flexibilty than standard Next.js build methods, since any environment variables accessed
+via `process.env`, whether for pages or for API, will have values drawn from their runtime deployment environment,
+not their build environment. 
+
+## Building
+
+Building usually requires no additional configuration.  By default, the Next.js version specified in `package.json`
+is used, and either `yarn` or `npm` will be used automatically depending on which lock files are present.  These
+can be overidden with the build arguments `CERC_NEXT_VERSION` and `CERC_BUILD_TOOL` respectively.  For example: `--extra-build-args "--build-arg CERC_NEXT_VERSION=13.4.12"`
+
+**Example**:
+```
+$ cd ~/cerc
+$ git clone git@git.vdb.to:cerc-io/test-progressive-web-app.git
+$ laconic-so build-webapp --source-repo ~/cerc/test-progressive-web-app
+...
+
+Built host container for ~/cerc/test-progressive-web-app with tag:
+
+    cerc/test-progressive-web-app:local
+
+To test locally run:
+
+    laconic-so run-webapp --image cerc/test-progressive-web-app:local --env-file /path/to/environment.env
+
+```
+
+## Running
+
+With `run-webapp` a new container will be launched on the local machine, with runtime configuration provided by `--env-file` (if specified) and published on an available port.  Multiple instances can be launched with different configuration.
+
+**Example**:
+```
+# Production env
+$ laconic-so run-webapp --image cerc/test-progressive-web-app:local --env-file /path/to/environment/production.env
+
+Image: cerc/test-progressive-web-app:local
+ID: 4c6e893bf436b3e91a2b92ce37e30e499685131705700bd92a90d2eb14eefd05
+URL: http://localhost:32768
+
+# Dev env
+$ laconic-so run-webapp --image cerc/test-progressive-web-app:local --env-file /path/to/environment/dev.env
+
+Image: cerc/test-progressive-web-app:local
+ID: 9ab96494f563aafb6c057d88df58f9eca81b90f8721a4e068493a289a976051c
+URL: http://localhost:32769
+```
+
+## Deploying
+
+Use the subcommand `deploy-webapp create` to make a deployment directory that can be subsequently deployed to a Kubernetes cluster.
+Example commands are shown below, assuming that the webapp container image `cerc/test-progressive-web-app:local` has already been built:
+```
+$ laconic-so deploy-webapp create --kube-config ~/kubectl/k8s-kubeconfig.yaml --image-registry registry.digitalocean.com/laconic-registry --deployment-dir webapp-k8s-deployment --image cerc/test-progressive-web-app:local --url https://test-pwa-app.hosting.laconic.com/ --env-file test-webapp.env
+$ laconic-so deployment --dir webapp-k8s-deployment push-images
+$ laconic-so deployment --dir webapp-k8s-deployment start
+```

stack_orchestrator/build/build_containers.py

@@ -27,7 +27,7 @@ import subprocess
 import click
 import importlib.resources
 from pathlib import Path
-from stack_orchestrator.util import include_exclude_check, get_parsed_stack_config
+from stack_orchestrator.util import include_exclude_check, get_parsed_stack_config, stack_is_external
 from stack_orchestrator.base import get_npm_registry_url
 
 # TODO: find a place for this
@@ -58,7 +58,8 @@ def make_container_build_env(dev_root_path: str,
     return container_build_env
 
 
-def process_container(container,
+def process_container(stack: str,
+                      container,
                       container_build_dir: str,
                       container_build_env: dict,
                       dev_root_path: str,
@@ -69,12 +70,29 @@ def process_container(container,
                       ):
     if not quiet:
         print(f"Building: {container}")
-    build_dir = os.path.join(container_build_dir, container.replace("/", "-"))
+
-    build_script_filename = os.path.join(build_dir, "build.sh")
+    default_container_tag = f"{container}:local"
+    container_build_env.update({"CERC_DEFAULT_CONTAINER_IMAGE_TAG": default_container_tag})
+
+    # Check if this is in an external stack
+    if stack_is_external(stack):
+        container_parent_dir = Path(stack).joinpath("container-build")
+        temp_build_dir = container_parent_dir.joinpath(container.replace("/", "-"))
+        temp_build_script_filename = temp_build_dir.joinpath("build.sh")
+        # Now check if the container exists in the external stack.
+        if not temp_build_script_filename.exists():
+            # If not, revert to building an internal container
+            container_parent_dir = container_build_dir
+    else:
+        container_parent_dir = container_build_dir
+
+    build_dir = container_parent_dir.joinpath(container.replace("/", "-"))
+    build_script_filename = build_dir.joinpath("build.sh")
+
     if verbose:
         print(f"Build script filename: {build_script_filename}")
     if os.path.exists(build_script_filename):
-        build_command = build_script_filename
+        build_command = build_script_filename.as_posix()
     else:
         if verbose:
             print(f"No script file found: {build_script_filename}, using default build script")
@@ -84,7 +102,7 @@ def process_container(container,
         repo_full_path = os.path.join(dev_root_path, repo_dir)
         repo_dir_or_build_dir = repo_full_path if os.path.exists(repo_full_path) else build_dir
         build_command = os.path.join(container_build_dir,
-                                     "default-build.sh") + f" {container}:local {repo_dir_or_build_dir}"
+                                     "default-build.sh") + f" {default_container_tag} {repo_dir_or_build_dir}"
     if not dry_run:
         if verbose:
             print(f"Executing: {build_command} with environment: {container_build_env}")
@@ -158,7 +176,7 @@ def command(ctx, include, exclude, force_rebuild, extra_build_args):
 
     for container in containers_in_scope:
         if include_exclude_check(container, include, exclude):
-            process_container(container, container_build_dir, container_build_env,
+            process_container(stack, container, container_build_dir, container_build_env,
                               dev_root_path, quiet, verbose, dry_run, continue_on_error)
         else:
             if verbose:

stack_orchestrator/build/build_webapp.py

@@ -32,8 +32,9 @@ from stack_orchestrator.build import build_containers
 @click.option('--source-repo', help="directory containing the webapp to build", required=True)
 @click.option("--force-rebuild", is_flag=True, default=False, help="Override dependency checking -- always rebuild")
 @click.option("--extra-build-args", help="Supply extra arguments to build")
+@click.option("--tag", help="Container tag (default: cerc/<app_name>:local)")
 @click.pass_context
-def command(ctx, base_container, source_repo, force_rebuild, extra_build_args):
+def command(ctx, base_container, source_repo, force_rebuild, extra_build_args, tag):
     '''build the specified webapp container'''
 
     quiet = ctx.obj.quiet
@@ -60,7 +61,7 @@ def command(ctx, base_container, source_repo, force_rebuild, extra_build_args):
     container_build_env = build_containers.make_container_build_env(dev_root_path, container_build_dir, debug,
                                                                     force_rebuild, extra_build_args)
 
-    build_containers.process_container(base_container, container_build_dir, container_build_env, dev_root_path, quiet,
+    build_containers.process_container(None, base_container, container_build_dir, container_build_env, dev_root_path, quiet,
                                        verbose, dry_run, continue_on_error)
 
 
@@ -70,8 +71,11 @@ def command(ctx, base_container, source_repo, force_rebuild, extra_build_args):
     container_build_env["CERC_CONTAINER_BUILD_DOCKERFILE"] = os.path.join(container_build_dir,
                                                                     base_container.replace("/", "-"),
                                                                     "Dockerfile.webapp")
+    if not tag:
         webapp_name = os.path.abspath(source_repo).split(os.path.sep)[-1]
         container_build_env["CERC_CONTAINER_BUILD_TAG"] = f"cerc/{webapp_name}:local"
+    else:
+        container_build_env["CERC_CONTAINER_BUILD_TAG"] = tag
 
-    build_containers.process_container(base_container, container_build_dir, container_build_env, dev_root_path, quiet,
+    build_containers.process_container(None, base_container, container_build_dir, container_build_env, dev_root_path, quiet,
                                        verbose, dry_run, continue_on_error)

stack_orchestrator/constants.py

@@ -0,0 +1,32 @@
+# Copyright © 2023 Vulcanize
+
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http:#www.gnu.org/licenses/>.
+
+cluster_name_prefix = "laconic-"
+stack_file_name = "stack.yml"
+spec_file_name = "spec.yml"
+config_file_name = "config.env"
+deployment_file_name = "deployment.yml"
+compose_dir_name = "compose"
+compose_deploy_type = "compose"
+k8s_kind_deploy_type = "k8s-kind"
+k8s_deploy_type = "k8s"
+cluster_id_key = "cluster-id"
+kube_config_key = "kube-config"
+deploy_to_key = "deploy-to"
+network_key = "network"
+http_proxy_key = "http-proxy"
+image_resigtry_key = "image-registry"
+kind_config_filename = "kind-config.yml"
+kube_config_filename = "kubeconfig.yml"

stack_orchestrator/data/compose/docker-compose-fixturenet-laconic-console.yml

@@ -4,6 +4,6 @@ services:
     image: cerc/laconic-console-host:local
     environment:
       - CERC_WEBAPP_FILES_DIR=${CERC_WEBAPP_FILES_DIR:-/usr/local/share/.config/yarn/global/node_modules/@cerc-io/console-app/dist/production}
-      - LACONIC_HOSTED_ENDPOINT=${LACONIC_HOSTED_ENDPOINT:-http://localhost}
+      - LACONIC_HOSTED_ENDPOINT=${LACONIC_HOSTED_ENDPOINT:-http://localhost:9473}
     ports:
       - "80"

stack_orchestrator/data/compose/docker-compose-fixturenet-laconicd.yml

@@ -5,7 +5,7 @@ services:
     command: ["sh", "/docker-entrypoint-scripts.d/create-fixturenet.sh"]
     volumes:
       # The cosmos-sdk node's database directory:
-      - laconicd-data:/root/.laconicd/data
+      - laconicd-data:/root/.laconicd
       # TODO: look at folding these scripts into the container
       - ../config/fixturenet-laconicd/create-fixturenet.sh:/docker-entrypoint-scripts.d/create-fixturenet.sh
       - ../config/fixturenet-laconicd/export-mykey.sh:/docker-entrypoint-scripts.d/export-mykey.sh

stack_orchestrator/data/compose/docker-compose-proxy-server.yml

@@ -0,0 +1,22 @@
+version: "3.2"
+
+services:
+  proxy-server:
+    image: cerc/watcher-ts:local
+    restart: on-failure
+    working_dir: /app/packages/cli
+    environment:
+      ENABLE_PROXY: ${ENABLE_PROXY:-true}
+      PROXY_UPSTREAM: ${CERC_PROXY_UPSTREAM}
+      PROXY_ORIGIN_HEADER: ${CERC_PROXY_ORIGIN_HEADER}
+    command: ["sh", "-c", "./run.sh"]
+    volumes:
+      - ../config/proxy-server/run.sh:/app/packages/cli/run.sh
+    ports:
+      - "4000"
+    healthcheck:
+      test: ["CMD", "nc", "-v", "localhost", "4000"]
+      interval: 20s
+      timeout: 5s
+      retries: 15
+      start_period: 10s

stack_orchestrator/data/compose/docker-compose-uniswap-interface.yml

@@ -0,0 +1,17 @@
+version: "3.2"
+
+services:
+  uniswap-interface:
+    image: cerc/uniswap-interface:local
+    restart: on-failure
+    environment:
+      - REACT_APP_INFURA_KEY=${CERC_INFURA_KEY}
+      - REACT_APP_AWS_API_ENDPOINT=${CERC_UNISWAP_GQL}
+    command: ["./build-app.sh"]
+    volumes:
+      - app_builds:/app-builds
+      - ../config/uniswap-interface/build-app.sh:/app/build-app.sh
+
+volumes:
+  app_builds:
+  app_globs:

stack_orchestrator/data/compose/docker-compose-uniswap-urbit.yml

@@ -0,0 +1,46 @@
+version: '3.7'
+
+services:
+  urbit-fake-ship:
+    restart: unless-stopped
+    image: tloncorp/vere
+    environment:
+      CERC_IPFS_GLOB_HOST_ENDPOINT: ${CERC_IPFS_GLOB_HOST_ENDPOINT:-http://ipfs-glob-host:5001}
+      CERC_IPFS_SERVER_ENDPOINT: ${CERC_IPFS_SERVER_ENDPOINT:-http://ipfs-glob-host:8080}
+    entrypoint: ["bash", "-c", "./run-urbit-ship.sh && ./deploy-uniswap-app.sh && tail -f /dev/null"]
+    volumes:
+      - urbit_data:/urbit
+      - app_builds:/app-builds
+      - app_globs:/app-globs
+      - ../config/urbit/run-urbit-ship.sh:/urbit/run-urbit-ship.sh
+      - ../config/uniswap-interface/deploy-uniswap-app.sh:/urbit/deploy-uniswap-app.sh
+    ports:
+      - "80"
+    healthcheck:
+      test: ["CMD", "nc", "-v", "localhost", "80"]
+      interval: 20s
+      timeout: 5s
+      retries: 15
+      start_period: 10s
+
+  ipfs-glob-host:
+    image: ipfs/kubo:master-2023-02-20-714a968
+    volumes:
+      - ipfs-import:/import
+      - ipfs-data:/data/ipfs
+    ports:
+      - "8080"
+      - "5001"
+    healthcheck:
+      test: ["CMD", "nc", "-v", "localhost", "5001"]
+      interval: 20s
+      timeout: 5s
+      retries: 15
+      start_period: 10s
+
+volumes:
+  urbit_data:
+  app_builds:
+  app_globs:
+  ipfs-import:
+  ipfs-data:

stack_orchestrator/data/compose/docker-compose-watcher-azimuth.yml

@@ -10,6 +10,7 @@ services:
       - POSTGRES_MULTIPLE_DATABASES=azimuth-watcher,azimuth-watcher-job-queue,censures-watcher,censures-watcher-job-queue,claims-watcher,claims-watcher-job-queue,conditional-star-release-watcher,conditional-star-release-watcher-job-queue,delegated-sending-watcher,delegated-sending-watcher-job-queue,ecliptic-watcher,ecliptic-watcher-job-queue,linear-star-release-watcher,linear-star-release-watcher-job-queue,polls-watcher,polls-watcher-job-queue
       - POSTGRES_EXTENSION=azimuth-watcher-job-queue:pgcrypto,censures-watcher-job-queue:pgcrypto,claims-watcher-job-queue:pgcrypto,conditional-star-release-watcher-job-queue:pgcrypto,delegated-sending-watcher-job-queue:pgcrypto,ecliptic-watcher-job-queue:pgcrypto,linear-star-release-watcher-job-queue:pgcrypto,polls-watcher-job-queue:pgcrypto,
       - POSTGRES_PASSWORD=password
+    command: ["postgres", "-c", "max_connections=200"]
     volumes:
       - ../config/postgresql/multiple-postgressql-databases.sh:/docker-entrypoint-initdb.d/multiple-postgressql-databases.sh
       - watcher_db_data:/var/lib/postgresql/data
@@ -22,6 +23,38 @@ services:
       retries: 15
       start_period: 10s
 
+  # Starts the azimuth-watcher job runner
+  azimuth-watcher-job-runner:
+    image: cerc/watcher-azimuth:local
+    restart: unless-stopped
+    depends_on:
+      watcher-db:
+        condition: service_healthy
+    environment:
+      CERC_SCRIPT_DEBUG: ${CERC_SCRIPT_DEBUG}
+      CERC_IPLD_ETH_RPC: ${CERC_IPLD_ETH_RPC}
+      CERC_IPLD_ETH_GQL: ${CERC_IPLD_ETH_GQL}
+      CERC_HISTORICAL_BLOCK_RANGE: 500
+      CONTRACT_ADDRESS: 0x223c067F8CF28ae173EE5CafEa60cA44C335fecB
+      CONTRACT_NAME: Azimuth
+      STARTING_BLOCK: 6784880
+    working_dir: /app/packages/azimuth-watcher
+    command: "./start-job-runner.sh"
+    volumes:
+      - ../config/watcher-azimuth/watcher-config-template.toml:/app/packages/azimuth-watcher/environments/watcher-config-template.toml
+      - ../config/watcher-azimuth/merge-toml.js:/app/packages/azimuth-watcher/merge-toml.js
+      - ../config/watcher-azimuth/start-job-runner.sh:/app/packages/azimuth-watcher/start-job-runner.sh
+    ports:
+      - "9000"
+    healthcheck:
+      test: ["CMD", "nc", "-vz", "localhost", "9000"]
+      interval: 20s
+      timeout: 5s
+      retries: 15
+      start_period: 5s
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
+
   # Starts the azimuth-watcher server
   azimuth-watcher-server:
     image: cerc/watcher-azimuth:local
@@ -29,8 +62,8 @@ services:
     depends_on:
       watcher-db:
         condition: service_healthy
-    env_file:
+      azimuth-watcher-job-runner:
-      - ../config/watcher-azimuth/watcher-params.env
+        condition: service_healthy
     environment:
       CERC_SCRIPT_DEBUG: ${CERC_SCRIPT_DEBUG}
       CERC_IPLD_ETH_RPC: ${CERC_IPLD_ETH_RPC}
@@ -52,6 +85,37 @@ services:
     extra_hosts:
       - "host.docker.internal:host-gateway"
 
+  # Starts the censures-watcher job runner
+  censures-watcher-job-runner:
+    image: cerc/watcher-azimuth:local
+    restart: unless-stopped
+    depends_on:
+      watcher-db:
+        condition: service_healthy
+    environment:
+      CERC_SCRIPT_DEBUG: ${CERC_SCRIPT_DEBUG}
+      CERC_IPLD_ETH_RPC: ${CERC_IPLD_ETH_RPC}
+      CERC_IPLD_ETH_GQL: ${CERC_IPLD_ETH_GQL}
+      CONTRACT_ADDRESS: 0x325f68d32BdEe6Ed86E7235ff2480e2A433D6189
+      CONTRACT_NAME: Censures
+      STARTING_BLOCK: 6784954
+    working_dir: /app/packages/censures-watcher
+    command: "./start-job-runner.sh"
+    volumes:
+      - ../config/watcher-azimuth/watcher-config-template.toml:/app/packages/censures-watcher/environments/watcher-config-template.toml
+      - ../config/watcher-azimuth/merge-toml.js:/app/packages/censures-watcher/merge-toml.js
+      - ../config/watcher-azimuth/start-job-runner.sh:/app/packages/censures-watcher/start-job-runner.sh
+    ports:
+      - "9002"
+    healthcheck:
+      test: ["CMD", "nc", "-vz", "localhost", "9002"]
+      interval: 20s
+      timeout: 5s
+      retries: 15
+      start_period: 5s
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
+
   # Starts the censures-watcher server
   censures-watcher-server:
     image: cerc/watcher-azimuth:local
@@ -59,8 +123,8 @@ services:
     depends_on:
       watcher-db:
         condition: service_healthy
-    env_file:
+      censures-watcher-job-runner:
-      - ../config/watcher-azimuth/watcher-params.env
+        condition: service_healthy
     environment:
       CERC_SCRIPT_DEBUG: ${CERC_SCRIPT_DEBUG}
       CERC_IPLD_ETH_RPC: ${CERC_IPLD_ETH_RPC}
@@ -82,6 +146,37 @@ services:
     extra_hosts:
       - "host.docker.internal:host-gateway"
 
+  # Starts the claims-watcher job runner
+  claims-watcher-job-runner:
+    image: cerc/watcher-azimuth:local
+    restart: unless-stopped
+    depends_on:
+      watcher-db:
+        condition: service_healthy
+    environment:
+      CERC_SCRIPT_DEBUG: ${CERC_SCRIPT_DEBUG}
+      CERC_IPLD_ETH_RPC: ${CERC_IPLD_ETH_RPC}
+      CERC_IPLD_ETH_GQL: ${CERC_IPLD_ETH_GQL}
+      CONTRACT_ADDRESS: 0xe7e7f69b34D7d9Bd8d61Fb22C33b22708947971A
+      CONTRACT_NAME: Claims
+      STARTING_BLOCK: 6784941
+    working_dir: /app/packages/claims-watcher
+    command: "./start-job-runner.sh"
+    volumes:
+      - ../config/watcher-azimuth/watcher-config-template.toml:/app/packages/claims-watcher/environments/watcher-config-template.toml
+      - ../config/watcher-azimuth/merge-toml.js:/app/packages/claims-watcher/merge-toml.js
+      - ../config/watcher-azimuth/start-job-runner.sh:/app/packages/claims-watcher/start-job-runner.sh
+    ports:
+      - "9004"
+    healthcheck:
+      test: ["CMD", "nc", "-vz", "localhost", "9004"]
+      interval: 20s
+      timeout: 5s
+      retries: 15
+      start_period: 5s
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
+
   # Starts the claims-watcher server
   claims-watcher-server:
     image: cerc/watcher-azimuth:local
@@ -89,8 +184,8 @@ services:
     depends_on:
       watcher-db:
         condition: service_healthy
-    env_file:
+      claims-watcher-job-runner:
-      - ../config/watcher-azimuth/watcher-params.env
+        condition: service_healthy
     environment:
       CERC_SCRIPT_DEBUG: ${CERC_SCRIPT_DEBUG}
       CERC_IPLD_ETH_RPC: ${CERC_IPLD_ETH_RPC}
@@ -112,6 +207,37 @@ services:
     extra_hosts:
       - "host.docker.internal:host-gateway"
 
+  # Starts the conditional-star-release-watcher job runner
+  conditional-star-release-watcher-job-runner:
+    image: cerc/watcher-azimuth:local
+    restart: unless-stopped
+    depends_on:
+      watcher-db:
+        condition: service_healthy
+    environment:
+      CERC_SCRIPT_DEBUG: ${CERC_SCRIPT_DEBUG}
+      CERC_IPLD_ETH_RPC: ${CERC_IPLD_ETH_RPC}
+      CERC_IPLD_ETH_GQL: ${CERC_IPLD_ETH_GQL}
+      CONTRACT_ADDRESS: 0x8C241098C3D3498Fe1261421633FD57986D74AeA
+      CONTRACT_NAME: ConditionalStarRelease
+      STARTING_BLOCK: 6828004
+    working_dir: /app/packages/conditional-star-release-watcher
+    command: "./start-job-runner.sh"
+    volumes:
+      - ../config/watcher-azimuth/watcher-config-template.toml:/app/packages/conditional-star-release-watcher/environments/watcher-config-template.toml
+      - ../config/watcher-azimuth/merge-toml.js:/app/packages/conditional-star-release-watcher/merge-toml.js
+      - ../config/watcher-azimuth/start-job-runner.sh:/app/packages/conditional-star-release-watcher/start-job-runner.sh
+    ports:
+      - "9006"
+    healthcheck:
+      test: ["CMD", "nc", "-vz", "localhost", "9006"]
+      interval: 20s
+      timeout: 5s
+      retries: 15
+      start_period: 5s
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
+
   # Starts the conditional-star-release-watcher server
   conditional-star-release-watcher-server:
     image: cerc/watcher-azimuth:local
@@ -119,8 +245,8 @@ services:
     depends_on:
       watcher-db:
         condition: service_healthy
-    env_file:
+      conditional-star-release-watcher-job-runner:
-      - ../config/watcher-azimuth/watcher-params.env
+        condition: service_healthy
     environment:
       CERC_SCRIPT_DEBUG: ${CERC_SCRIPT_DEBUG}
       CERC_IPLD_ETH_RPC: ${CERC_IPLD_ETH_RPC}
@@ -142,6 +268,37 @@ services:
     extra_hosts:
       - "host.docker.internal:host-gateway"
 
+  # Starts the delegated-sending-watcher job runner
+  delegated-sending-watcher-job-runner:
+    image: cerc/watcher-azimuth:local
+    restart: unless-stopped
+    depends_on:
+      watcher-db:
+        condition: service_healthy
+    environment:
+      CERC_SCRIPT_DEBUG: ${CERC_SCRIPT_DEBUG}
+      CERC_IPLD_ETH_RPC: ${CERC_IPLD_ETH_RPC}
+      CERC_IPLD_ETH_GQL: ${CERC_IPLD_ETH_GQL}
+      CONTRACT_ADDRESS: 0xf6b461fE1aD4bd2ce25B23Fe0aff2ac19B3dFA76
+      CONTRACT_NAME: DelegatedSending
+      STARTING_BLOCK: 6784956
+    working_dir: /app/packages/delegated-sending-watcher
+    command: "./start-job-runner.sh"
+    volumes:
+      - ../config/watcher-azimuth/watcher-config-template.toml:/app/packages/delegated-sending-watcher/environments/watcher-config-template.toml
+      - ../config/watcher-azimuth/merge-toml.js:/app/packages/delegated-sending-watcher/merge-toml.js
+      - ../config/watcher-azimuth/start-job-runner.sh:/app/packages/delegated-sending-watcher/start-job-runner.sh
+    ports:
+      - "9008"
+    healthcheck:
+      test: ["CMD", "nc", "-vz", "localhost", "9008"]
+      interval: 20s
+      timeout: 5s
+      retries: 15
+      start_period: 5s
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
+
   # Starts the delegated-sending-watcher server
   delegated-sending-watcher-server:
     image: cerc/watcher-azimuth:local
@@ -149,8 +306,8 @@ services:
     depends_on:
       watcher-db:
         condition: service_healthy
-    env_file:
+      delegated-sending-watcher-job-runner:
-      - ../config/watcher-azimuth/watcher-params.env
+        condition: service_healthy
     environment:
       CERC_SCRIPT_DEBUG: ${CERC_SCRIPT_DEBUG}
       CERC_IPLD_ETH_RPC: ${CERC_IPLD_ETH_RPC}
@@ -172,6 +329,37 @@ services:
     extra_hosts:
       - "host.docker.internal:host-gateway"
 
+  # Starts the ecliptic-watcher job runner
+  ecliptic-watcher-job-runner:
+    image: cerc/watcher-azimuth:local
+    restart: unless-stopped
+    depends_on:
+      watcher-db:
+        condition: service_healthy
+    environment:
+      CERC_SCRIPT_DEBUG: ${CERC_SCRIPT_DEBUG}
+      CERC_IPLD_ETH_RPC: ${CERC_IPLD_ETH_RPC}
+      CERC_IPLD_ETH_GQL: ${CERC_IPLD_ETH_GQL}
+      CONTRACT_ADDRESS: 0x33EeCbf908478C10614626A9D304bfe18B78DD73
+      CONTRACT_NAME: Ecliptic
+      STARTING_BLOCK: 13692129
+    working_dir: /app/packages/ecliptic-watcher
+    command: "./start-job-runner.sh"
+    volumes:
+      - ../config/watcher-azimuth/watcher-config-template.toml:/app/packages/ecliptic-watcher/environments/watcher-config-template.toml
+      - ../config/watcher-azimuth/merge-toml.js:/app/packages/ecliptic-watcher/merge-toml.js
+      - ../config/watcher-azimuth/start-job-runner.sh:/app/packages/ecliptic-watcher/start-job-runner.sh
+    ports:
+      - "9010"
+    healthcheck:
+      test: ["CMD", "nc", "-vz", "localhost", "9010"]
+      interval: 20s
+      timeout: 5s
+      retries: 15
+      start_period: 5s
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
+
   # Starts the ecliptic-watcher server
   ecliptic-watcher-server:
     image: cerc/watcher-azimuth:local
@@ -179,8 +367,8 @@ services:
     depends_on:
       watcher-db:
         condition: service_healthy
-    env_file:
+      ecliptic-watcher-job-runner:
-      - ../config/watcher-azimuth/watcher-params.env
+        condition: service_healthy
     environment:
       CERC_SCRIPT_DEBUG: ${CERC_SCRIPT_DEBUG}
       CERC_IPLD_ETH_RPC: ${CERC_IPLD_ETH_RPC}
@@ -202,6 +390,37 @@ services:
     extra_hosts:
       - "host.docker.internal:host-gateway"
 
+  # Starts the linear-star-release-watcher job runner
+  linear-star-release-watcher-job-runner:
+    image: cerc/watcher-azimuth:local
+    restart: unless-stopped
+    depends_on:
+      watcher-db:
+        condition: service_healthy
+    environment:
+      CERC_SCRIPT_DEBUG: ${CERC_SCRIPT_DEBUG}
+      CERC_IPLD_ETH_RPC: ${CERC_IPLD_ETH_RPC}
+      CERC_IPLD_ETH_GQL: ${CERC_IPLD_ETH_GQL}
+      CONTRACT_ADDRESS: 0x86cd9cd0992F04231751E3761De45cEceA5d1801
+      CONTRACT_NAME: LinearStarRelease
+      STARTING_BLOCK: 6784943
+    working_dir: /app/packages/linear-star-release-watcher
+    command: "./start-job-runner.sh"
+    volumes:
+      - ../config/watcher-azimuth/watcher-config-template.toml:/app/packages/linear-star-release-watcher/environments/watcher-config-template.toml
+      - ../config/watcher-azimuth/merge-toml.js:/app/packages/linear-star-release-watcher/merge-toml.js
+      - ../config/watcher-azimuth/start-job-runner.sh:/app/packages/linear-star-release-watcher/start-job-runner.sh
+    ports:
+      - "9012"
+    healthcheck:
+      test: ["CMD", "nc", "-vz", "localhost", "9012"]
+      interval: 20s
+      timeout: 5s
+      retries: 15
+      start_period: 5s
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
+
   # Starts the linear-star-release-watcher server
   linear-star-release-watcher-server:
     image: cerc/watcher-azimuth:local
@@ -209,8 +428,8 @@ services:
     depends_on:
       watcher-db:
         condition: service_healthy
-    env_file:
+      linear-star-release-watcher-job-runner:
-      - ../config/watcher-azimuth/watcher-params.env
+        condition: service_healthy
     environment:
       CERC_SCRIPT_DEBUG: ${CERC_SCRIPT_DEBUG}
       CERC_IPLD_ETH_RPC: ${CERC_IPLD_ETH_RPC}
@@ -232,6 +451,37 @@ services:
     extra_hosts:
       - "host.docker.internal:host-gateway"
 
+  # Starts the polls-watcher job runner
+  polls-watcher-job-runner:
+    image: cerc/watcher-azimuth:local
+    restart: unless-stopped
+    depends_on:
+      watcher-db:
+        condition: service_healthy
+    environment:
+      CERC_SCRIPT_DEBUG: ${CERC_SCRIPT_DEBUG}
+      CERC_IPLD_ETH_RPC: ${CERC_IPLD_ETH_RPC}
+      CERC_IPLD_ETH_GQL: ${CERC_IPLD_ETH_GQL}
+      CONTRACT_ADDRESS: 0x7fEcaB617c868Bb5996d99D95200D2Fa708218e4
+      CONTRACT_NAME: Polls
+      STARTING_BLOCK: 6784912
+    working_dir: /app/packages/polls-watcher
+    command: "./start-job-runner.sh"
+    volumes:
+      - ../config/watcher-azimuth/watcher-config-template.toml:/app/packages/polls-watcher/environments/watcher-config-template.toml
+      - ../config/watcher-azimuth/merge-toml.js:/app/packages/polls-watcher/merge-toml.js
+      - ../config/watcher-azimuth/start-job-runner.sh:/app/packages/polls-watcher/start-job-runner.sh
+    ports:
+      - "9014"
+    healthcheck:
+      test: ["CMD", "nc", "-vz", "localhost", "9014"]
+      interval: 20s
+      timeout: 5s
+      retries: 15
+      start_period: 5s
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
+
   # Starts the polls-watcher server
   polls-watcher-server:
     image: cerc/watcher-azimuth:local
@@ -239,8 +489,8 @@ services:
     depends_on:
       watcher-db:
         condition: service_healthy
-    env_file:
+      polls-watcher-job-runner:
-      - ../config/watcher-azimuth/watcher-params.env
+        condition: service_healthy
     environment:
       CERC_SCRIPT_DEBUG: ${CERC_SCRIPT_DEBUG}
       CERC_IPLD_ETH_RPC: ${CERC_IPLD_ETH_RPC}

stack_orchestrator/data/compose/docker-compose-watcher-merkl-sushiswap-v3.yml

@@ -0,0 +1,76 @@
+version: '3.2'
+
+services:
+  merkl-sushiswap-v3-watcher-db:
+    restart: unless-stopped
+    image: postgres:14-alpine
+    environment:
+      - POSTGRES_USER=vdbm
+      - POSTGRES_MULTIPLE_DATABASES=merkl-sushiswap-v3-watcher,merkl-sushiswap-v3-watcher-job-queue
+      - POSTGRES_EXTENSION=merkl-sushiswap-v3-watcher-job-queue:pgcrypto
+      - POSTGRES_PASSWORD=password
+    volumes:
+      - ../config/postgresql/multiple-postgressql-databases.sh:/docker-entrypoint-initdb.d/multiple-postgressql-databases.sh
+      - merkl_sushiswap_v3_watcher_db_data:/var/lib/postgresql/data
+    ports:
+      - "5432"
+    healthcheck:
+      test: ["CMD", "nc", "-v", "localhost", "5432"]
+      interval: 20s
+      timeout: 5s
+      retries: 15
+      start_period: 10s
+
+  merkl-sushiswap-v3-watcher-job-runner:
+    restart: unless-stopped
+    depends_on:
+      merkl-sushiswap-v3-watcher-db:
+        condition: service_healthy
+    image: cerc/watcher-merkl-sushiswap-v3:local
+    environment:
+      CERC_SCRIPT_DEBUG: ${CERC_SCRIPT_DEBUG}
+      CERC_ETH_RPC_ENDPOINT: ${CERC_ETH_RPC_ENDPOINT}
+    command: ["bash", "./start-job-runner.sh"]
+    volumes:
+      - ../config/watcher-merkl-sushiswap-v3/watcher-config-template.toml:/app/environments/watcher-config-template.toml
+      - ../config/watcher-merkl-sushiswap-v3/start-job-runner.sh:/app/start-job-runner.sh
+    ports:
+      - "9000"
+    healthcheck:
+      test: ["CMD", "nc", "-v", "localhost", "9000"]
+      interval: 20s
+      timeout: 5s
+      retries: 15
+      start_period: 5s
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
+
+  merkl-sushiswap-v3-watcher-server:
+    restart: unless-stopped
+    depends_on:
+      merkl-sushiswap-v3-watcher-db:
+        condition: service_healthy
+      merkl-sushiswap-v3-watcher-job-runner:
+        condition: service_healthy
+    image: cerc/watcher-merkl-sushiswap-v3:local
+    environment:
+      CERC_SCRIPT_DEBUG: ${CERC_SCRIPT_DEBUG}
+      CERC_ETH_RPC_ENDPOINT: ${CERC_ETH_RPC_ENDPOINT}
+    command: ["bash", "./start-server.sh"]
+    volumes:
+      - ../config/watcher-merkl-sushiswap-v3/watcher-config-template.toml:/app/environments/watcher-config-template.toml
+      - ../config/watcher-merkl-sushiswap-v3/start-server.sh:/app/start-server.sh
+    ports:
+      - "127.0.0.1:3007:3008"
+      - "9001"
+    healthcheck:
+      test: ["CMD", "nc", "-v", "localhost", "3008"]
+      interval: 20s
+      timeout: 5s
+      retries: 15
+      start_period: 5s
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
+
+volumes:
+  merkl_sushiswap_v3_watcher_db_data:

stack_orchestrator/data/compose/docker-compose-watcher-sushiswap-v3.yml

@@ -0,0 +1,76 @@
+version: '3.2'
+
+services:
+  sushiswap-v3-watcher-db:
+    restart: unless-stopped
+    image: postgres:14-alpine
+    environment:
+      - POSTGRES_USER=vdbm
+      - POSTGRES_MULTIPLE_DATABASES=sushiswap-v3-watcher,sushiswap-v3-watcher-job-queue
+      - POSTGRES_EXTENSION=sushiswap-v3-watcher-job-queue:pgcrypto
+      - POSTGRES_PASSWORD=password
+    volumes:
+      - ../config/postgresql/multiple-postgressql-databases.sh:/docker-entrypoint-initdb.d/multiple-postgressql-databases.sh
+      - sushiswap_v3_watcher_db_data:/var/lib/postgresql/data
+    ports:
+      - "5432"
+    healthcheck:
+      test: ["CMD", "nc", "-v", "localhost", "5432"]
+      interval: 20s
+      timeout: 5s
+      retries: 15
+      start_period: 10s
+
+  sushiswap-v3-watcher-job-runner:
+    restart: unless-stopped
+    depends_on:
+      sushiswap-v3-watcher-db:
+        condition: service_healthy
+    image: cerc/watcher-sushiswap-v3:local
+    environment:
+      CERC_SCRIPT_DEBUG: ${CERC_SCRIPT_DEBUG}
+      CERC_ETH_RPC_ENDPOINT: ${CERC_ETH_RPC_ENDPOINT}
+    command: ["bash", "./start-job-runner.sh"]
+    volumes:
+      - ../config/watcher-sushiswap-v3/watcher-config-template.toml:/app/environments/watcher-config-template.toml
+      - ../config/watcher-sushiswap-v3/start-job-runner.sh:/app/start-job-runner.sh
+    ports:
+      - "9000"
+    healthcheck:
+      test: ["CMD", "nc", "-v", "localhost", "9000"]
+      interval: 20s
+      timeout: 5s
+      retries: 15
+      start_period: 5s
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
+
+  sushiswap-v3-watcher-server:
+    restart: unless-stopped
+    depends_on:
+      sushiswap-v3-watcher-db:
+        condition: service_healthy
+      sushiswap-v3-watcher-job-runner:
+        condition: service_healthy
+    image: cerc/watcher-sushiswap-v3:local
+    environment:
+      CERC_SCRIPT_DEBUG: ${CERC_SCRIPT_DEBUG}
+      CERC_ETH_RPC_ENDPOINT: ${CERC_ETH_RPC_ENDPOINT}
+    command: ["bash", "./start-server.sh"]
+    volumes:
+      - ../config/watcher-sushiswap-v3/watcher-config-template.toml:/app/environments/watcher-config-template.toml
+      - ../config/watcher-sushiswap-v3/start-server.sh:/app/start-server.sh
+    ports:
+      - "127.0.0.1:3008:3008"
+      - "9001"
+    healthcheck:
+      test: ["CMD", "nc", "-v", "localhost", "3008"]
+      interval: 20s
+      timeout: 5s
+      retries: 15
+      start_period: 5s
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
+
+volumes:
+  sushiswap_v3_watcher_db_data:

stack_orchestrator/data/compose/docker-compose-webapp-template.yml

@@ -0,0 +1,8 @@
+services:
+  webapp:
+    image: cerc/webapp-container:local
+    restart: always
+    environment:
+      CERC_SCRIPT_DEBUG: ${CERC_SCRIPT_DEBUG}
+    ports:
+      - "3000"

stack_orchestrator/data/config/fixturenet-laconicd/create-fixturenet.sh

@@ -14,44 +14,48 @@ LOGLEVEL="info"
 TRACE="--trace"
 # TRACE=""
 
-# validate dependencies are installed
+if [ "$1" == "clean" ] || [ ! -d "$HOME/.laconicd/data/blockstore.db" ]; then
-command -v jq > /dev/null 2>&1 || { echo >&2 "jq not installed. More info: https://stedolan.github.io/jq/download/"; exit 1; }
+  # validate dependencies are installed
+  command -v jq > /dev/null 2>&1 || { echo >&2 "jq not installed. More info: https://stedolan.github.io/jq/download/"; exit 1; }
 
-# remove existing daemon and client
+  # remove existing daemon and client
-rm -rf ~/.laconic*
+  rm -rf $HOME/.laconicd/*
+  rm -rf $HOME/.laconic/*
 
-make install
+  if [ -n "`which make`" ]; then
+    make install
+  fi
 
-laconicd config keyring-backend $KEYRING
+  laconicd config keyring-backend $KEYRING
-laconicd config chain-id $CHAINID
+  laconicd config chain-id $CHAINID
 
-# if $KEY exists it should be deleted
+  # if $KEY exists it should be deleted
-laconicd keys add $KEY --keyring-backend $KEYRING --algo $KEYALGO
+  laconicd keys add $KEY --keyring-backend $KEYRING --algo $KEYALGO
 
-# Set moniker and chain-id for Ethermint (Moniker can be anything, chain-id must be an integer)
+  # Set moniker and chain-id for Ethermint (Moniker can be anything, chain-id must be an integer)
-laconicd init $MONIKER --chain-id $CHAINID
+  laconicd init $MONIKER --chain-id $CHAINID
 
-# Change parameter token denominations to aphoton
+  # Change parameter token denominations to aphoton
-cat $HOME/.laconicd/config/genesis.json | jq '.app_state["staking"]["params"]["bond_denom"]="aphoton"' > $HOME/.laconicd/config/tmp_genesis.json && mv $HOME/.laconicd/config/tmp_genesis.json $HOME/.laconicd/config/genesis.json
+  cat $HOME/.laconicd/config/genesis.json | jq '.app_state["staking"]["params"]["bond_denom"]="aphoton"' > $HOME/.laconicd/config/tmp_genesis.json && mv $HOME/.laconicd/config/tmp_genesis.json $HOME/.laconicd/config/genesis.json
-cat $HOME/.laconicd/config/genesis.json | jq '.app_state["crisis"]["constant_fee"]["denom"]="aphoton"' > $HOME/.laconicd/config/tmp_genesis.json && mv $HOME/.laconicd/config/tmp_genesis.json $HOME/.laconicd/config/genesis.json
+  cat $HOME/.laconicd/config/genesis.json | jq '.app_state["crisis"]["constant_fee"]["denom"]="aphoton"' > $HOME/.laconicd/config/tmp_genesis.json && mv $HOME/.laconicd/config/tmp_genesis.json $HOME/.laconicd/config/genesis.json
-cat $HOME/.laconicd/config/genesis.json | jq '.app_state["gov"]["deposit_params"]["min_deposit"][0]["denom"]="aphoton"' > $HOME/.laconicd/config/tmp_genesis.json && mv $HOME/.laconicd/config/tmp_genesis.json $HOME/.laconicd/config/genesis.json
+  cat $HOME/.laconicd/config/genesis.json | jq '.app_state["gov"]["deposit_params"]["min_deposit"][0]["denom"]="aphoton"' > $HOME/.laconicd/config/tmp_genesis.json && mv $HOME/.laconicd/config/tmp_genesis.json $HOME/.laconicd/config/genesis.json
-cat $HOME/.laconicd/config/genesis.json | jq '.app_state["mint"]["params"]["mint_denom"]="aphoton"' > $HOME/.laconicd/config/tmp_genesis.json && mv $HOME/.laconicd/config/tmp_genesis.json $HOME/.laconicd/config/genesis.json
+  cat $HOME/.laconicd/config/genesis.json | jq '.app_state["mint"]["params"]["mint_denom"]="aphoton"' > $HOME/.laconicd/config/tmp_genesis.json && mv $HOME/.laconicd/config/tmp_genesis.json $HOME/.laconicd/config/genesis.json
-# Custom modules
+  # Custom modules
-cat $HOME/.laconicd/config/genesis.json | jq '.app_state["registry"]["params"]["record_rent"]["denom"]="aphoton"' > $HOME/.laconicd/config/tmp_genesis.json && mv $HOME/.laconicd/config/tmp_genesis.json $HOME/.laconicd/config/genesis.json
+  cat $HOME/.laconicd/config/genesis.json | jq '.app_state["registry"]["params"]["record_rent"]["denom"]="aphoton"' > $HOME/.laconicd/config/tmp_genesis.json && mv $HOME/.laconicd/config/tmp_genesis.json $HOME/.laconicd/config/genesis.json
-cat $HOME/.laconicd/config/genesis.json | jq '.app_state["registry"]["params"]["authority_rent"]["denom"]="aphoton"' > $HOME/.laconicd/config/tmp_genesis.json && mv $HOME/.laconicd/config/tmp_genesis.json $HOME/.laconicd/config/genesis.json
+  cat $HOME/.laconicd/config/genesis.json | jq '.app_state["registry"]["params"]["authority_rent"]["denom"]="aphoton"' > $HOME/.laconicd/config/tmp_genesis.json && mv $HOME/.laconicd/config/tmp_genesis.json $HOME/.laconicd/config/genesis.json
-cat $HOME/.laconicd/config/genesis.json | jq '.app_state["registry"]["params"]["authority_auction_commit_fee"]["denom"]="aphoton"' > $HOME/.laconicd/config/tmp_genesis.json && mv $HOME/.laconicd/config/tmp_genesis.json $HOME/.laconicd/config/genesis.json
+  cat $HOME/.laconicd/config/genesis.json | jq '.app_state["registry"]["params"]["authority_auction_commit_fee"]["denom"]="aphoton"' > $HOME/.laconicd/config/tmp_genesis.json && mv $HOME/.laconicd/config/tmp_genesis.json $HOME/.laconicd/config/genesis.json
-cat $HOME/.laconicd/config/genesis.json | jq '.app_state["registry"]["params"]["authority_auction_reveal_fee"]["denom"]="aphoton"' > $HOME/.laconicd/config/tmp_genesis.json && mv $HOME/.laconicd/config/tmp_genesis.json $HOME/.laconicd/config/genesis.json
+  cat $HOME/.laconicd/config/genesis.json | jq '.app_state["registry"]["params"]["authority_auction_reveal_fee"]["denom"]="aphoton"' > $HOME/.laconicd/config/tmp_genesis.json && mv $HOME/.laconicd/config/tmp_genesis.json $HOME/.laconicd/config/genesis.json
-cat $HOME/.laconicd/config/genesis.json | jq '.app_state["registry"]["params"]["authority_auction_minimum_bid"]["denom"]="aphoton"' > $HOME/.laconicd/config/tmp_genesis.json && mv $HOME/.laconicd/config/tmp_genesis.json $HOME/.laconicd/config/genesis.json
+  cat $HOME/.laconicd/config/genesis.json | jq '.app_state["registry"]["params"]["authority_auction_minimum_bid"]["denom"]="aphoton"' > $HOME/.laconicd/config/tmp_genesis.json && mv $HOME/.laconicd/config/tmp_genesis.json $HOME/.laconicd/config/genesis.json
 
-if [[ "$TEST_REGISTRY_EXPIRY" == "true" ]]; then
+  if [[ "$TEST_REGISTRY_EXPIRY" == "true" ]]; then
     echo "Setting timers for expiry tests."
 
     cat $HOME/.laconicd/config/genesis.json | jq '.app_state["registry"]["params"]["record_rent_duration"]="60s"' > $HOME/.laconicd/config/tmp_genesis.json && mv $HOME/.laconicd/config/tmp_genesis.json $HOME/.laconicd/config/genesis.json
     cat $HOME/.laconicd/config/genesis.json | jq '.app_state["registry"]["params"]["authority_grace_period"]="60s"' > $HOME/.laconicd/config/tmp_genesis.json && mv $HOME/.laconicd/config/tmp_genesis.json $HOME/.laconicd/config/genesis.json
     cat $HOME/.laconicd/config/genesis.json | jq '.app_state["registry"]["params"]["authority_rent_duration"]="60s"' > $HOME/.laconicd/config/tmp_genesis.json && mv $HOME/.laconicd/config/tmp_genesis.json $HOME/.laconicd/config/genesis.json
-fi
+  fi
 
-if [[ "$TEST_AUCTION_ENABLED" == "true" ]]; then
+  if [[ "$TEST_AUCTION_ENABLED" == "true" ]]; then
     echo "Enabling auction and setting timers."
 
     cat $HOME/.laconicd/config/genesis.json | jq '.app_state["registry"]["params"]["authority_auction_enabled"]=true' > $HOME/.laconicd/config/tmp_genesis.json && mv $HOME/.laconicd/config/tmp_genesis.json $HOME/.laconicd/config/genesis.json
@@ -59,22 +63,22 @@ if [[ "$TEST_AUCTION_ENABLED" == "true" ]]; then
     cat $HOME/.laconicd/config/genesis.json | jq '.app_state["registry"]["params"]["authority_grace_period"]="300s"' > $HOME/.laconicd/config/tmp_genesis.json && mv $HOME/.laconicd/config/tmp_genesis.json $HOME/.laconicd/config/genesis.json
     cat $HOME/.laconicd/config/genesis.json | jq '.app_state["registry"]["params"]["authority_auction_commits_duration"]="60s"' > $HOME/.laconicd/config/tmp_genesis.json && mv $HOME/.laconicd/config/tmp_genesis.json $HOME/.laconicd/config/genesis.json
     cat $HOME/.laconicd/config/genesis.json | jq '.app_state["registry"]["params"]["authority_auction_reveals_duration"]="60s"' > $HOME/.laconicd/config/tmp_genesis.json && mv $HOME/.laconicd/config/tmp_genesis.json $HOME/.laconicd/config/genesis.json
-fi
+  fi
 
-# increase block time (?)
+  # increase block time (?)
-cat $HOME/.laconicd/config/genesis.json | jq '.consensus_params["block"]["time_iota_ms"]="1000"' > $HOME/.laconicd/config/tmp_genesis.json && mv $HOME/.laconicd/config/tmp_genesis.json $HOME/.laconicd/config/genesis.json
+  cat $HOME/.laconicd/config/genesis.json | jq '.consensus_params["block"]["time_iota_ms"]="1000"' > $HOME/.laconicd/config/tmp_genesis.json && mv $HOME/.laconicd/config/tmp_genesis.json $HOME/.laconicd/config/genesis.json
 
-# Set gas limit in genesis
+  # Set gas limit in genesis
-cat $HOME/.laconicd/config/genesis.json | jq '.consensus_params["block"]["max_gas"]="10000000"' > $HOME/.laconicd/config/tmp_genesis.json && mv $HOME/.laconicd/config/tmp_genesis.json $HOME/.laconicd/config/genesis.json
+  cat $HOME/.laconicd/config/genesis.json | jq '.consensus_params["block"]["max_gas"]="10000000"' > $HOME/.laconicd/config/tmp_genesis.json && mv $HOME/.laconicd/config/tmp_genesis.json $HOME/.laconicd/config/genesis.json
 
-# disable produce empty block
+  # disable produce empty block
-if [[ "$OSTYPE" == "darwin"* ]]; then
+  if [[ "$OSTYPE" == "darwin"* ]]; then
       sed -i '' 's/create_empty_blocks = true/create_empty_blocks = false/g' $HOME/.laconicd/config/config.toml
     else
       sed -i 's/create_empty_blocks = true/create_empty_blocks = false/g' $HOME/.laconicd/config/config.toml
-fi
+  fi
 
-if [[ $1 == "pending" ]]; then
+  if [[ $1 == "pending" ]]; then
     if [[ "$OSTYPE" == "darwin"* ]]; then
         sed -i '' 's/create_empty_blocks_interval = "0s"/create_empty_blocks_interval = "30s"/g' $HOME/.laconicd/config/config.toml
         sed -i '' 's/timeout_propose = "3s"/timeout_propose = "30s"/g' $HOME/.laconicd/config/config.toml
@@ -96,22 +100,25 @@ if [[ $1 == "pending" ]]; then
         sed -i 's/timeout_commit = "5s"/timeout_commit = "150s"/g' $HOME/.laconicd/config/config.toml
         sed -i 's/timeout_broadcast_tx_commit = "10s"/timeout_broadcast_tx_commit = "150s"/g' $HOME/.laconicd/config/config.toml
     fi
-fi
+  fi
 
-# Allocate genesis accounts (cosmos formatted addresses)
+  # Allocate genesis accounts (cosmos formatted addresses)
-laconicd add-genesis-account $KEY 100000000000000000000000000aphoton --keyring-backend $KEYRING
+  laconicd add-genesis-account $KEY 100000000000000000000000000aphoton --keyring-backend $KEYRING
 
-# Sign genesis transaction
+  # Sign genesis transaction
-laconicd gentx $KEY 1000000000000000000000aphoton --keyring-backend $KEYRING --chain-id $CHAINID
+  laconicd gentx $KEY 1000000000000000000000aphoton --keyring-backend $KEYRING --chain-id $CHAINID
 
-# Collect genesis tx
+  # Collect genesis tx
-laconicd collect-gentxs
+  laconicd collect-gentxs
 
-# Run this to ensure everything worked and that the genesis file is setup correctly
+  # Run this to ensure everything worked and that the genesis file is setup correctly
-laconicd validate-genesis
+  laconicd validate-genesis
 
-if [[ $1 == "pending" ]]; then
+  if [[ $1 == "pending" ]]; then
     echo "pending mode is on, please wait for the first block committed."
+  fi
+else
+  echo "Using existing database at $HOME/.laconicd.  To replace, run '`basename $0` clean'"
 fi
 
 # Start the node (remove the --pruning=nothing flag if historical queries are not needed)

stack_orchestrator/data/config/proxy-server/run.sh

@@ -0,0 +1,9 @@
+#!/bin/sh
+
+if [ "$ENABLE_PROXY" = "true" ]; then
+  echo "Proxy server enabled"
+  yarn proxy
+else
+  echo "Proxy server disabled, exiting"
+  exit 0
+fi

stack_orchestrator/data/config/uniswap-interface/build-app.sh

@@ -0,0 +1,18 @@
+#!/bin/bash
+
+set -e
+if [ -n "$CERC_SCRIPT_DEBUG" ]; then
+  set -x
+fi
+
+# Check and exit if a deployment already exists (on restarts)
+if [ -d /app-builds/uniswap/build ]; then
+  echo "Build already exists, remove volume to rebuild"
+  exit 0
+fi
+
+yarn build
+
+# Move build to app-builds so urbit can deploy it
+mkdir /app-builds/uniswap
+cp -r ./build /app-builds/uniswap/

stack_orchestrator/data/config/uniswap-interface/deploy-uniswap-app.sh

@@ -0,0 +1,149 @@
+#!/bin/bash
+
+if [ -n "$CERC_SCRIPT_DEBUG" ]; then
+  set -x
+fi
+
+echo "Using IPFS endpoint ${CERC_IPFS_GLOB_HOST_ENDPOINT} for hosting globs"
+echo "Using IPFS server endpoint ${CERC_IPFS_SERVER_ENDPOINT} for reading glob files"
+ipfs_host_endpoint=${CERC_IPFS_GLOB_HOST_ENDPOINT}
+ipfs_server_endpoint=${CERC_IPFS_SERVER_ENDPOINT}
+
+uniswap_app_build='/app-builds/uniswap/build'
+uniswap_desk_dir='/urbit/zod/uniswap'
+
+if [ -d ${uniswap_desk_dir} ]; then
+  echo "Uniswap desk dir already exists, skipping deployment..."
+  exit 0
+fi
+
+# Fire curl requests to perform operations on the ship
+dojo () {
+  curl -s --data '{"source":{"dojo":"'"$1"'"},"sink":{"stdout":null}}' http://localhost:12321
+}
+
+hood () {
+  curl -s --data '{"source":{"dojo":"+hood/'"$1"'"},"sink":{"app":"hood"}}' http://localhost:12321
+}
+
+# Create/mount a uniswap desk
+hood "merge %uniswap our %landscape"
+hood "mount %uniswap"
+
+# Loop until the uniswap build appears
+while [ ! -d ${uniswap_app_build} ]; do
+  echo "Uniswap app build not found, retrying in 5s..."
+  sleep 5
+done
+echo "Build found..."
+
+# Copy over build to desk data dir
+cp -r ${uniswap_app_build} ${uniswap_desk_dir}
+
+# Create a mark file for .map file type
+cat << EOF > "${uniswap_desk_dir}/mar/map.hoon"
+::
+::::  /hoon/map/mar
+  ::  Mark for js source maps
+/?    310
+::
+=,  eyre
+|_  mud=@
+++  grow
+  |%
+  ++  mime  [/application/octet-stream (as-octs:mimes:html (@t mud))]
+  --
+++  grab
+  |%                                                    ::  convert from
+  ++  mime  |=([p=mite q=octs] (@t q.q))
+  ++  noun  cord                                        ::  clam from %noun
+  --
+++  grad  %mime
+--
+EOF
+
+# Create a mark file for .woff file type
+cat << EOF > "${uniswap_desk_dir}/mar/woff.hoon"
+|_  dat=octs
+++  grow
+  |%
+  ++  mime  [/font/woff dat]
+  --
+++  grab
+  |%
+  ++  mime  |=([=mite =octs] octs)
+  ++  noun  octs
+  --
+++  grad  %mime
+--
+EOF
+
+# Create a mark file for .ttf file type
+cat << EOF > "${uniswap_desk_dir}/mar/ttf.hoon"
+|_  dat=octs
+++  grow
+  |%
+  ++  mime  [/font/ttf dat]
+  --
+++  grab
+  |%
+  ++  mime  |=([=mite =octs] octs)
+  ++  noun  octs
+  --
+++  grad  %mime
+--
+EOF
+
+rm "${uniswap_desk_dir}/desk.bill"
+rm "${uniswap_desk_dir}/desk.ship"
+
+# Commit changes and create a glob
+hood "commit %uniswap"
+dojo "-landscape!make-glob %uniswap /build"
+
+glob_file=$(ls -1 -c zod/.urb/put | head -1)
+echo "Created glob file: ${glob_file}"
+
+upload_response=$(curl -X POST -F file=@./zod/.urb/put/${glob_file} ${ipfs_host_endpoint}/api/v0/add)
+glob_cid=$(echo "$upload_response" | grep -o '"Hash":"[^"]*' | sed 's/"Hash":"//')
+
+echo "Glob file uploaded to IFPS:"
+echo "{ cid: ${glob_cid}, filename: ${glob_file} }"
+
+# Curl and wait for the glob to be hosted
+glob_url="${ipfs_server_endpoint}/ipfs/${glob_cid}?filename=${glob_file}"
+
+echo "Checking if glob file hosted at ${glob_url}"
+while true; do
+  response=$(curl -sL -w "%{http_code}" -o /dev/null "$glob_url")
+
+  if [ $response -eq 200 ]; then
+    echo "File found at $glob_url"
+    break  # Exit the loop if the file is found
+  else
+    echo "File not found. Retrying in a few seconds..."
+    sleep 5
+  fi
+done
+
+glob_hash=$(echo "$glob_file" | sed "s/glob-\([a-z0-9\.]*\).glob/\1/")
+
+# Update the docket file
+cat << EOF > "${uniswap_desk_dir}/desk.docket-0"
+:~  title+'Uniswap'
+    info+'Self-hosted uniswap frontend.'
+    color+0xcd.75df
+    image+'https://logowik.com/content/uploads/images/uniswap-uni7403.jpg'
+    base+'uniswap'
+    glob-http+['${glob_url}' ${glob_hash}]
+    version+[0 0 1]
+    website+'https://uniswap.org/'
+    license+'MIT'
+==
+EOF
+
+# Commit changes and install the app
+hood "commit %uniswap"
+hood "install our %uniswap"
+
+echo "Uniswap app installed"

stack_orchestrator/data/config/uniswap-interface/install-uniswap-app.sh

@@ -0,0 +1,110 @@
+#!/bin/bash
+
+# $1: Glob file URL (eg. https://xyz.com/glob-0vabcd.glob)
+# $2: Glob file hash (eg. 0vabcd)
+# $3: Urbit ship's pier dir (default: ./zod)
+
+if [ "$#" -lt 2 ]; then
+  echo "Insufficient arguments"
+  exit 0
+fi
+
+glob_url=$1
+glob_hash=$2
+echo "Using glob file from ${glob_url} with hash ${glob_hash}"
+
+# Default pier dir: ./zod
+# Default desk dir: ./zod/uniswap
+pier_dir="${3:-./zod}"
+uniswap_desk_dir="${pier_dir}/uniswap"
+echo "Using ${uniswap_desk_dir} as the Uniswap desk dir path"
+
+# Fire curl requests to perform operations on the ship
+dojo () {
+  curl -s --data '{"source":{"dojo":"'"$1"'"},"sink":{"stdout":null}}' http://localhost:12321
+}
+
+hood () {
+  curl -s --data '{"source":{"dojo":"+hood/'"$1"'"},"sink":{"app":"hood"}}' http://localhost:12321
+}
+
+# Create/mount a uniswap desk
+hood "merge %uniswap our %landscape"
+hood "mount %uniswap"
+
+# Create a mark file for .map file type
+cat << EOF > "${uniswap_desk_dir}/mar/map.hoon"
+::
+::::  /hoon/map/mar
+  ::  Mark for js source maps
+/?    310
+::
+=,  eyre
+|_  mud=@
+++  grow
+  |%
+  ++  mime  [/application/octet-stream (as-octs:mimes:html (@t mud))]
+  --
+++  grab
+  |%                                                    ::  convert from
+  ++  mime  |=([p=mite q=octs] (@t q.q))
+  ++  noun  cord                                        ::  clam from %noun
+  --
+++  grad  %mime
+--
+EOF
+
+# Create a mark file for .woff file type
+cat << EOF > "${uniswap_desk_dir}/mar/woff.hoon"
+|_  dat=octs
+++  grow
+  |%
+  ++  mime  [/font/woff dat]
+  --
+++  grab
+  |%
+  ++  mime  |=([=mite =octs] octs)
+  ++  noun  octs
+  --
+++  grad  %mime
+--
+EOF
+
+# Create a mark file for .ttf file type
+cat << EOF > "${uniswap_desk_dir}/mar/ttf.hoon"
+|_  dat=octs
+++  grow
+  |%
+  ++  mime  [/font/ttf dat]
+  --
+++  grab
+  |%
+  ++  mime  |=([=mite =octs] octs)
+  ++  noun  octs
+  --
+++  grad  %mime
+--
+EOF
+
+rm "${uniswap_desk_dir}/desk.bill"
+rm "${uniswap_desk_dir}/desk.ship"
+
+# Update the docket file
+cat << EOF > "${uniswap_desk_dir}/desk.docket-0"
+:~  title+'Uniswap'
+    info+'Self-hosted uniswap frontend.'
+    color+0xcd.75df
+    image+'https://logowik.com/content/uploads/images/uniswap-uni7403.jpg'
+    base+'uniswap'
+    glob-http+['${glob_url}' ${glob_hash}]
+    version+[0 0 1]
+    website+'https://uniswap.org/'
+    license+'MIT'
+==
+EOF
+
+# Commit changes and install the app
+hood "commit %uniswap"
+hood "install our %uniswap"
+
+echo "Uniswap app installed"

stack_orchestrator/data/config/uniswap-interface/remote-deploy-uniswap.sh

@@ -0,0 +1,21 @@
+#!/bin/bash
+
+# $1: Remote user host
+# $2: Remote Urbit ship's pier dir path (eg. /home/user/zod)
+# $3: Glob file URL (eg. https://xyz.com/glob-0vabcd.glob)
+# $4: Glob file hash (eg. 0vabcd)
+
+if [ "$#" -ne 4 ]; then
+  echo "Incorrect number of arguments"
+  echo "Usage: $0 <username@remote_host> </path/to/remote/pier/folder> <glob_url> <glob_hash>"
+  exit 1
+fi
+
+remote_user_host="$1"
+remote_pier_folder="$2"
+glob_url="$3"
+glob_hash="$4"
+
+installation_script="./install-uniswap-app.sh"
+
+ssh "$remote_user_host" "bash -s $glob_url $glob_hash $remote_pier_folder" < "$installation_script"

stack_orchestrator/data/config/urbit/run-urbit-ship.sh

@@ -0,0 +1,18 @@
+#!/bin/bash
+
+set -e
+if [ -n "$CERC_SCRIPT_DEBUG" ]; then
+  set -x
+fi
+
+pier_dir="/urbit/zod"
+
+# Run urbit ship in daemon mode
+# Check if the directory exists
+if [ -d "$pier_dir" ]; then
+  echo "Pier directory already exists, rebooting..."
+  urbit -d zod
+else
+  echo "Creating a new fake ship..."
+  urbit -d -F zod
+fi

stack_orchestrator/data/config/watcher-azimuth/start-job-runner.sh

@@ -0,0 +1,28 @@
+#!/bin/sh
+set -e
+if [ -n "$CERC_SCRIPT_DEBUG" ]; then
+  set -x
+fi
+
+echo "Using IPLD ETH RPC endpoint ${CERC_IPLD_ETH_RPC}"
+echo "Using IPLD GQL endpoint ${CERC_IPLD_ETH_GQL}"
+echo "Using historicalLogsBlockRange ${CERC_HISTORICAL_BLOCK_RANGE:-2000}"
+
+# Replace env variables in template TOML file
+# Read in the config template TOML file and modify it
+WATCHER_CONFIG_TEMPLATE=$(cat environments/watcher-config-template.toml)
+WATCHER_CONFIG=$(echo "$WATCHER_CONFIG_TEMPLATE" | \
+  sed -E "s|REPLACE_WITH_CERC_IPLD_ETH_RPC|${CERC_IPLD_ETH_RPC}|g; \
+          s|REPLACE_WITH_CERC_IPLD_ETH_GQL|${CERC_IPLD_ETH_GQL}|g; \
+          s|REPLACE_WITH_CERC_HISTORICAL_BLOCK_RANGE|${CERC_HISTORICAL_BLOCK_RANGE:-2000}| ")
+
+# Write the modified content to a new file
+echo "$WATCHER_CONFIG" > environments/watcher-config.toml
+
+# Merge SO watcher config with existing config file
+node merge-toml.js
+
+yarn watch:contract --address $CONTRACT_ADDRESS --kind $CONTRACT_NAME --checkpoint true --starting-block $STARTING_BLOCK
+
+echo 'yarn job-runner'
+yarn job-runner

stack_orchestrator/data/config/watcher-azimuth/start-server.sh

@@ -4,18 +4,17 @@ if [ -n "$CERC_SCRIPT_DEBUG" ]; then
   set -x
 fi
 
-CERC_IPLD_ETH_RPC="${CERC_IPLD_ETH_RPC:-${DEFAULT_CERC_IPLD_ETH_RPC}}"
-CERC_IPLD_ETH_GQL="${CERC_IPLD_ETH_GQL:-${DEFAULT_CERC_IPLD_ETH_GQL}}"
-
 echo "Using IPLD ETH RPC endpoint ${CERC_IPLD_ETH_RPC}"
 echo "Using IPLD GQL endpoint ${CERC_IPLD_ETH_GQL}"
+echo "Using historicalLogsBlockRange ${CERC_HISTORICAL_BLOCK_RANGE:-2000}"
 
 # Replace env variables in template TOML file
 # Read in the config template TOML file and modify it
 WATCHER_CONFIG_TEMPLATE=$(cat environments/watcher-config-template.toml)
 WATCHER_CONFIG=$(echo "$WATCHER_CONFIG_TEMPLATE" | \
   sed -E "s|REPLACE_WITH_CERC_IPLD_ETH_RPC|${CERC_IPLD_ETH_RPC}|g; \
-          s|REPLACE_WITH_CERC_IPLD_ETH_GQL|${CERC_IPLD_ETH_GQL}| ")
+          s|REPLACE_WITH_CERC_IPLD_ETH_GQL|${CERC_IPLD_ETH_GQL}|g; \
+          s|REPLACE_WITH_CERC_HISTORICAL_BLOCK_RANGE|${CERC_HISTORICAL_BLOCK_RANGE:-2000}| ")
 
 # Write the modified content to a new file
 echo "$WATCHER_CONFIG" > environments/watcher-config.toml

stack_orchestrator/data/config/watcher-azimuth/watcher-config-template.toml

@@ -2,6 +2,9 @@
   host = "0.0.0.0"
   maxSimultaneousRequests = -1
 
+[metrics]
+  host = "0.0.0.0"
+
 [database]
   host = "watcher-db"
   port = 5432
@@ -12,3 +15,7 @@
   [upstream.ethServer]
     gqlApiEndpoint = "REPLACE_WITH_CERC_IPLD_ETH_GQL"
     rpcProviderEndpoint = "REPLACE_WITH_CERC_IPLD_ETH_RPC"
+
+[jobQueue]
+  historicalLogsBlockRange = REPLACE_WITH_CERC_HISTORICAL_BLOCK_RANGE
+  blockDelayInMilliSecs = 12000

stack_orchestrator/data/config/watcher-azimuth/watcher-params.env

@@ -1,5 +0,0 @@
-# Defaults
-
-# ipld-eth-server endpoints
-DEFAULT_CERC_IPLD_ETH_RPC=
-DEFAULT_CERC_IPLD_ETH_GQL=

stack_orchestrator/data/config/watcher-merkl-sushiswap-v3/start-job-runner.sh

@@ -0,0 +1,20 @@
+#!/bin/sh
+
+set -e
+if [ -n "$CERC_SCRIPT_DEBUG" ]; then
+  set -x
+fi
+set -u
+
+echo "Using ETH RPC endpoint ${CERC_ETH_RPC_ENDPOINT}"
+
+# Read in the config template TOML file and modify it
+WATCHER_CONFIG_TEMPLATE=$(cat environments/watcher-config-template.toml)
+WATCHER_CONFIG=$(echo "$WATCHER_CONFIG_TEMPLATE" | \
+  sed -E "s|REPLACE_WITH_CERC_ETH_RPC_ENDPOINT|${CERC_ETH_RPC_ENDPOINT}| ")
+
+# Write the modified content to a new file
+echo "$WATCHER_CONFIG" > environments/local.toml
+
+echo "Running job-runner..."
+DEBUG=vulcanize:* exec node --enable-source-maps dist/job-runner.js

stack_orchestrator/data/config/watcher-merkl-sushiswap-v3/start-server.sh

@@ -0,0 +1,20 @@
+#!/bin/sh
+
+set -e
+if [ -n "$CERC_SCRIPT_DEBUG" ]; then
+  set -x
+fi
+set -u
+
+echo "Using ETH RPC endpoint ${CERC_ETH_RPC_ENDPOINT}"
+
+# Read in the config template TOML file and modify it
+WATCHER_CONFIG_TEMPLATE=$(cat environments/watcher-config-template.toml)
+WATCHER_CONFIG=$(echo "$WATCHER_CONFIG_TEMPLATE" | \
+  sed -E "s|REPLACE_WITH_CERC_ETH_RPC_ENDPOINT|${CERC_ETH_RPC_ENDPOINT}| ")
+
+# Write the modified content to a new file
+echo "$WATCHER_CONFIG" > environments/local.toml
+
+echo "Running server..."
+DEBUG=vulcanize:* exec node --enable-source-maps dist/server.js

stack_orchestrator/data/config/watcher-merkl-sushiswap-v3/watcher-config-template.toml

@@ -0,0 +1,100 @@
+[server]
+  host = "0.0.0.0"
+  port = 3008
+  kind = "active"
+  gqlPath = '/'
+
+  # Checkpointing state.
+  checkpointing = true
+
+  # Checkpoint interval in number of blocks.
+  checkpointInterval = 2000
+
+  # Enable state creation
+  # CAUTION: Disable only if state creation is not desired or can be filled subsequently
+  enableState = false
+
+  subgraphPath = "./subgraph-build"
+
+  # Interval to restart wasm instance periodically
+  wasmRestartBlocksInterval = 20
+
+  # Interval in number of blocks at which to clear entities cache.
+  clearEntitiesCacheInterval = 1000
+
+  # Max block range for which to return events in eventsInRange GQL query.
+  # Use -1 for skipping check on block range.
+  maxEventsBlockRange = 1000
+
+  # Flag to specify whether RPC endpoint supports block hash as block tag parameter
+  rpcSupportsBlockHashParam = false
+
+  # GQL cache settings
+  [server.gqlCache]
+    enabled = true
+
+    # Max in-memory cache size (in bytes) (default 8 MB)
+    # maxCacheSize
+
+    # GQL cache-control max-age settings (in seconds)
+    maxAge = 15
+    timeTravelMaxAge = 86400 # 1 day
+
+[metrics]
+  host = "127.0.0.1"
+  port = 9000
+  [metrics.gql]
+    port = 9001
+
+[database]
+  type = "postgres"
+  host = "merkl-sushiswap-v3-watcher-db"
+  port = 5432
+  database = "merkl-sushiswap-v3-watcher"
+  username = "vdbm"
+  password = "password"
+  synchronize = true
+  logging = false
+
+[upstream]
+  [upstream.ethServer]
+    rpcProviderEndpoint = "REPLACE_WITH_CERC_ETH_RPC_ENDPOINT"
+
+    # Boolean flag to specify if rpc-eth-client should be used for RPC endpoint instead of ipld-eth-client (ipld-eth-server GQL client)
+    rpcClient = true
+
+    # Boolean flag to specify if rpcProviderEndpoint is an FEVM RPC endpoint
+    isFEVM = true
+
+    # Boolean flag to filter event logs by contracts
+    filterLogsByAddresses = true
+    # Boolean flag to filter event logs by topics
+    filterLogsByTopics = false
+
+  [upstream.cache]
+    name = "requests"
+    enabled = false
+    deleteOnStart = false
+
+[jobQueue]
+  dbConnectionString = "postgres://vdbm:password@merkl-sushiswap-v3-watcher-db/merkl-sushiswap-v3-watcher-job-queue"
+  maxCompletionLagInSecs = 300
+  jobDelayInMilliSecs = 100
+  eventsInBatch = 50
+  subgraphEventsOrder = true
+  # Filecoin block time: https://docs.filecoin.io/basics/the-blockchain/blocks-and-tipsets#blocktime
+  blockDelayInMilliSecs = 30000
+  prefetchBlocksInMem = false
+  prefetchBlockCount = 10
+
+  # Boolean to switch between modes of processing events when starting the server.
+  # Setting to true will fetch filtered events and required blocks in a range of blocks and then process them.
+  # Setting to false will fetch blocks consecutively with its events and then process them (Behaviour is followed in realtime processing near head).
+  useBlockRanges = true
+
+  # Block range in which logs are fetched during historical blocks processing
+  historicalLogsBlockRange = 2000
+
+  # Max block range of historical processing after which it waits for completion of events processing
+  # If set to -1 historical processing does not wait for events processing and completes till latest canonical block
+  historicalMaxFetchAhead = 10000

stack_orchestrator/data/config/watcher-sushiswap-v3/start-job-runner.sh

@@ -0,0 +1,20 @@
+#!/bin/sh
+
+set -e
+if [ -n "$CERC_SCRIPT_DEBUG" ]; then
+  set -x
+fi
+set -u
+
+echo "Using ETH RPC endpoint ${CERC_ETH_RPC_ENDPOINT}"
+
+# Read in the config template TOML file and modify it
+WATCHER_CONFIG_TEMPLATE=$(cat environments/watcher-config-template.toml)
+WATCHER_CONFIG=$(echo "$WATCHER_CONFIG_TEMPLATE" | \
+  sed -E "s|REPLACE_WITH_CERC_ETH_RPC_ENDPOINT|${CERC_ETH_RPC_ENDPOINT}| ")
+
+# Write the modified content to a new file
+echo "$WATCHER_CONFIG" > environments/local.toml
+
+echo "Running job-runner..."
+DEBUG=vulcanize:* exec node --enable-source-maps dist/job-runner.js

stack_orchestrator/data/config/watcher-sushiswap-v3/start-server.sh

@@ -0,0 +1,20 @@
+#!/bin/sh
+
+set -e
+if [ -n "$CERC_SCRIPT_DEBUG" ]; then
+  set -x
+fi
+set -u
+
+echo "Using ETH RPC endpoint ${CERC_ETH_RPC_ENDPOINT}"
+
+# Read in the config template TOML file and modify it
+WATCHER_CONFIG_TEMPLATE=$(cat environments/watcher-config-template.toml)
+WATCHER_CONFIG=$(echo "$WATCHER_CONFIG_TEMPLATE" | \
+  sed -E "s|REPLACE_WITH_CERC_ETH_RPC_ENDPOINT|${CERC_ETH_RPC_ENDPOINT}| ")
+
+# Write the modified content to a new file
+echo "$WATCHER_CONFIG" > environments/local.toml
+
+echo "Running server..."
+DEBUG=vulcanize:* exec node --enable-source-maps dist/server.js

stack_orchestrator/data/config/watcher-sushiswap-v3/watcher-config-template.toml

@@ -0,0 +1,100 @@
+[server]
+  host = "0.0.0.0"
+  port = 3008
+  kind = "active"
+  gqlPath = "/"
+
+  # Checkpointing state.
+  checkpointing = true
+
+  # Checkpoint interval in number of blocks.
+  checkpointInterval = 2000
+
+  # Enable state creation
+  # CAUTION: Disable only if state creation is not desired or can be filled subsequently
+  enableState = false
+
+  subgraphPath = "./subgraph-build"
+
+  # Interval to restart wasm instance periodically
+  wasmRestartBlocksInterval = 20
+
+  # Interval in number of blocks at which to clear entities cache.
+  clearEntitiesCacheInterval = 1000
+
+  # Max block range for which to return events in eventsInRange GQL query.
+  # Use -1 for skipping check on block range.
+  maxEventsBlockRange = 1000
+
+  # Flag to specify whether RPC endpoint supports block hash as block tag parameter
+  rpcSupportsBlockHashParam = false
+
+  # GQL cache settings
+  [server.gqlCache]
+    enabled = true
+
+    # Max in-memory cache size (in bytes) (default 8 MB)
+    # maxCacheSize
+
+    # GQL cache-control max-age settings (in seconds)
+    maxAge = 15
+    timeTravelMaxAge = 86400 # 1 day
+
+[metrics]
+  host = "127.0.0.1"
+  port = 9000
+  [metrics.gql]
+    port = 9001
+
+[database]
+  type = "postgres"
+  host = "sushiswap-v3-watcher-db"
+  port = 5432
+  database = "sushiswap-v3-watcher"
+  username = "vdbm"
+  password = "password"
+  synchronize = true
+  logging = false
+
+[upstream]
+  [upstream.ethServer]
+    rpcProviderEndpoint = "REPLACE_WITH_CERC_ETH_RPC_ENDPOINT"
+
+    # Boolean flag to specify if rpc-eth-client should be used for RPC endpoint instead of ipld-eth-client (ipld-eth-server GQL client)
+    rpcClient = true
+
+    # Boolean flag to specify if rpcProviderEndpoint is an FEVM RPC endpoint
+    isFEVM = true
+
+    # Boolean flag to filter event logs by contracts
+    filterLogsByAddresses = true
+    # Boolean flag to filter event logs by topics
+    filterLogsByTopics = true
+
+  [upstream.cache]
+    name = "requests"
+    enabled = false
+    deleteOnStart = false
+
+[jobQueue]
+  dbConnectionString = "postgres://vdbm:password@sushiswap-v3-watcher-db/sushiswap-v3-watcher-job-queue"
+  maxCompletionLagInSecs = 300
+  jobDelayInMilliSecs = 100
+  eventsInBatch = 50
+  subgraphEventsOrder = true
+  # Filecoin block time: https://docs.filecoin.io/basics/the-blockchain/blocks-and-tipsets#blocktime
+  blockDelayInMilliSecs = 30000
+  prefetchBlocksInMem = false
+  prefetchBlockCount = 10
+
+  # Boolean to switch between modes of processing events when starting the server.
+  # Setting to true will fetch filtered events and required blocks in a range of blocks and then process them.
+  # Setting to false will fetch blocks consecutively with its events and then process them (Behaviour is followed in realtime processing near head).
+  useBlockRanges = true
+
+  # Block range in which logs are fetched during historical blocks processing
+  historicalLogsBlockRange = 2000
+
+  # Max block range of historical processing after which it waits for completion of events processing
+  # If set to -1 historical processing does not wait for events processing and completes till latest canonical block
+  historicalMaxFetchAhead = 10000

stack_orchestrator/data/container-build/cerc-laconic-console-host/config.yml

@@ -2,5 +2,5 @@
 
 services:
   wns:
-    server: 'LACONIC_HOSTED_ENDPOINT:9473/api'
+    server: 'LACONIC_HOSTED_ENDPOINT/api'
-    webui: 'LACONIC_HOSTED_ENDPOINT:9473/console'
+    webui: 'LACONIC_HOSTED_ENDPOINT/console'

stack_orchestrator/data/container-build/cerc-nextjs-base/Dockerfile

@@ -24,32 +24,21 @@ RUN \
     && su ${USERNAME} -c "npm config -g set prefix ${NPM_GLOBAL}" \
     # Install eslint
     && su ${USERNAME} -c "umask 0002 && npm install -g eslint" \
+    # Install semver
+    && su ${USERNAME} -c "umask 0002 && npm install -g semver" \
     && npm cache clean --force > /dev/null 2>&1
 
 # [Optional] Uncomment this section to install additional OS packages.
 RUN apt-get update && export DEBIAN_FRONTEND=noninteractive \
-    && apt-get -y install --no-install-recommends jq gettext-base
+    && apt-get -y install --no-install-recommends jq gettext-base moreutils
-
-# [Optional] Uncomment if you want to install an additional version of node using nvm
-# ARG EXTRA_NODE_VERSION=10
-# RUN su node -c "source /usr/local/share/nvm/nvm.sh && nvm install ${EXTRA_NODE_VERSION}"
-
-# We do this to get a yq binary from the published container, for the correct architecture we're building here
-# COPY --from=docker.io/mikefarah/yq:latest /usr/bin/yq /usr/local/bin/yq
-
-COPY /scripts /scripts
 
 # [Optional] Uncomment if you want to install more global node modules
 # RUN su node -c "npm install -g <your-package-list-here>"
 
-# RUN mkdir -p /config
-# COPY ./config.yml /config
-
-# Install simple web server for now (use nginx perhaps later)
-# RUN yarn global add http-server
-
 # Expose port for http
 EXPOSE 3000
 
+COPY /scripts /scripts
+
 # Default command sleeps forever so docker doesn't kill it
-CMD ["/scripts/start-serving-app.sh"]
+ENTRYPOINT ["/scripts/start-serving-app.sh"]

stack_orchestrator/data/container-build/cerc-nextjs-base/Dockerfile.webapp

@@ -1,4 +1,8 @@
 FROM cerc/nextjs-base:local
+
+ARG CERC_NEXT_VERSION=keep
+ARG CERC_BUILD_TOOL
+
 WORKDIR /app
 COPY . .
 RUN rm -rf node_modules build .next*

stack_orchestrator/data/container-build/cerc-nextjs-base/build.sh

@@ -11,3 +11,19 @@ CERC_CONTAINER_BUILD_DOCKERFILE=${CERC_CONTAINER_BUILD_DOCKERFILE:-$SCRIPT_DIR/D
 CERC_CONTAINER_BUILD_TAG=${CERC_CONTAINER_BUILD_TAG:-cerc/nextjs-base:local}
 
 docker build -t $CERC_CONTAINER_BUILD_TAG ${build_command_args} -f $CERC_CONTAINER_BUILD_DOCKERFILE $CERC_CONTAINER_BUILD_WORK_DIR
+
+if [ $? -eq 0 ] && [ "$CERC_CONTAINER_BUILD_TAG" != "cerc/nextjs-base:local" ]; then
+  cat <<EOF
+
+#################################################################
+
+Built host container for $CERC_CONTAINER_BUILD_WORK_DIR with tag:
+
+    $CERC_CONTAINER_BUILD_TAG
+
+To test locally run:
+
+    laconic-so run-webapp --image $CERC_CONTAINER_BUILD_TAG --env-file /path/to/environment.env
+
+EOF
+fi

stack_orchestrator/data/container-build/cerc-nextjs-base/scripts/apply-runtime-env.sh

@@ -8,6 +8,15 @@ WORK_DIR="${1:-./}"
 SRC_DIR="${2:-.next}"
 TRG_DIR="${3:-.next-r}"
 
+CERC_BUILD_TOOL="${CERC_BUILD_TOOL}"
+if [ -z "$CERC_BUILD_TOOL" ]; then
+  if [ -f "yarn.lock" ]; then
+    CERC_BUILD_TOOL=npm
+  else
+    CERC_BUILD_TOOL=yarn
+  fi
+fi
+
 cd "${WORK_DIR}" || exit 1
 
 rm -rf "$TRG_DIR"
@@ -29,6 +38,9 @@ for f in $(find "$TRG_DIR" -regex ".*.[tj]sx?$" -type f | grep -v 'node_modules'
     orig_name=$(echo -n "${e}" | sed 's/"//g')
     cur_name=$(echo -n "${orig_name}" | sed 's/CERC_RUNTIME_ENV_//g')
     cur_val=$(echo -n "\$${cur_name}" | envsubst)
+    if [ "$CERC_RETAIN_ENV_QUOTES" != "true" ]; then
+      cur_val=$(sed "s/^[\"']//" <<< "$cur_val" | sed "s/[\"']//")
+    fi
     esc_val=$(sed 's/[&/\]/\\&/g' <<< "$cur_val")
     echo "$f: $cur_name=$cur_val"
     sed -i "s/$orig_name/$esc_val/g" $f

stack_orchestrator/data/container-build/cerc-nextjs-base/scripts/build-app.sh

@@ -4,8 +4,19 @@ if [ -n "$CERC_SCRIPT_DEBUG" ]; then
     set -x
 fi
 
-SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
+CERC_MIN_NEXTVER=13.4.2
 
+CERC_NEXT_VERSION="${CERC_NEXT_VERSION:-keep}"
+CERC_BUILD_TOOL="${CERC_BUILD_TOOL}"
+if [ -z "$CERC_BUILD_TOOL" ]; then
+  if [ -f "yarn.lock" ]; then
+    CERC_BUILD_TOOL=yarn
+  else
+    CERC_BUILD_TOOL=npm
+  fi
+fi
+
+SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
 WORK_DIR="${1:-/app}"
 
 cd "${WORK_DIR}" || exit 1
@@ -20,6 +31,7 @@ if [ $? -ne 0 ]; then
 fi
 
 js-beautify next.config.dist > next.config.js
+echo "" >> next.config.js
 
 WEBPACK_REQ_LINE=$(grep -n "require([\'\"]webpack[\'\"])" next.config.js | cut -d':' -f1)
 if [ -z "$WEBPACK_REQ_LINE" ]; then
@@ -58,7 +70,7 @@ if [ -n "$WEBPACK_CONF_LINE" ]; then
   cat > next.config.js.3 <<EOF
       $WEBPACK_CONF_VAR.plugins.push(new webpack.DefinePlugin(envMap));
 EOF
-  NEXT_SECTION_LINE=$((WEBPACK_CONF_LINE - 1))
+  NEXT_SECTION_LINE=$((WEBPACK_CONF_LINE))
 elif [ -n "$ENV_LINE" ]; then
   head -$(( ${ENV_LINE} - 1 )) next.config.js > next.config.js.2
   cat > next.config.js.3 <<EOF
@@ -67,7 +79,7 @@ elif [ -n "$ENV_LINE" ]; then
       return config;
     },
 EOF
-  NEXT_SECTION_ADJUSTMENT=2
+  NEXT_SECTION_ADJUSTMENT=1
   NEXT_SECTION_LINE=$ENV_LINE
 else
   echo "WARNING: Cannot find location to insert environment variable map in next.config.js" 1>&2
@@ -88,7 +100,38 @@ fi
 
 cat package.dist | jq '.scripts.cerc_compile = "next experimental-compile"' | jq '.scripts.cerc_generate = "next experimental-generate"' > package.json
 
-npm install || exit 1
+CUR_NEXT_VERSION="`jq -r '.dependencies.next' package.json`"
-npm run cerc_compile || exit 1
+
+if [ "$CERC_NEXT_VERSION" != "keep" ] && [ "$CUR_NEXT_VERSION" != "$CERC_NEXT_VERSION" ]; then
+  echo "Changing 'next' version specifier from '$CUR_NEXT_VERSION' to '$CERC_NEXT_VERSION' (set with '--extra-build-args \"--build-arg CERC_NEXT_VERSION=$CERC_NEXT_VERSION\"')"
+  cat package.json | jq ".dependencies.next = \"$CERC_NEXT_VERSION\"" | sponge package.json
+fi
+
+$CERC_BUILD_TOOL install || exit 1
+
+CUR_NEXT_VERSION=`jq -r '.version' node_modules/next/package.json`
+
+semver -p -r ">=$CERC_MIN_NEXTVER" $CUR_NEXT_VERSION
+if [ $? -ne 0 ]; then
+  cat <<EOF
+
+###############################################################################
+
+WARNING: 'next' $CUR_NEXT_VERSION < minimum version $CERC_MIN_NEXTVER.
+
+Attempting to build with '^$CERC_MIN_NEXTVER'.  If this fails, you should upgrade
+the dependency in your webapp, or specify an explicit 'next' version
+to use for the build with:
+
+     --extra-build-args "--build-arg CERC_NEXT_VERSION=<version>"
+
+###############################################################################
+
+EOF
+  cat package.json | jq ".dependencies.next = \"^$CERC_MIN_NEXTVER\"" | sponge package.json
+  $CERC_BUILD_TOOL install || exit 1
+fi
+
+$CERC_BUILD_TOOL run cerc_compile || exit 1
 
 exit 0

stack_orchestrator/data/container-build/cerc-nextjs-base/scripts/start-serving-app.sh

@@ -3,7 +3,25 @@ if [ -n "$CERC_SCRIPT_DEBUG" ]; then
     set -x
 fi
 
+
 SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
+CERC_MAX_GENERATE_TIME=${CERC_MAX_GENERATE_TIME:-60}
+tpid=""
+
+ctrl_c() {
+    kill $tpid $(ps -ef | grep node | grep next | awk '{print $2}') 2>/dev/null
+}
+
+trap ctrl_c INT
+
+CERC_BUILD_TOOL="${CERC_BUILD_TOOL}"
+if [ -z "$CERC_BUILD_TOOL" ]; then
+  if [ -f "yarn.lock" ] && [ ! -f "package-lock.json" ]; then
+    CERC_BUILD_TOOL=yarn
+  else
+    CERC_BUILD_TOOL=npm
+  fi
+fi
 
 CERC_WEBAPP_FILES_DIR="${CERC_WEBAPP_FILES_DIR:-/app}"
 cd "$CERC_WEBAPP_FILES_DIR"
@@ -16,19 +34,28 @@ if [ "$CERC_NEXTJS_SKIP_GENERATE" != "true" ]; then
   jq -e '.scripts.cerc_generate' package.json >/dev/null
   if [ $? -eq 0 ]; then
     npm run cerc_generate > gen.out 2>&1 &
-    tail -n0 -f gen.out | sed '/rendered as static HTML/ q'
+    tail -f gen.out &
+    tpid=$!
+
     count=0
-    while [ $count -lt 10 ]; do
+    generate_done="false"
+    while [ $count -lt $CERC_MAX_GENERATE_TIME ] && [ "$generate_done" == "false" ]; do
       sleep 1
-      ps -ef | grep 'node' | grep 'next' | grep 'generate' >/dev/null
-      if [ $? -ne 0 ]; then 
-        break
-      else
       count=$((count + 1))
+      grep 'rendered as static HTML' gen.out > /dev/null
+      if [ $? -eq 0 ]; then
+        generate_done="true"
       fi
     done
-    kill $(ps -ef |grep node | grep next | grep generate | awk '{print $2}') 2>/dev/null
+
+    if [ $generate_done != "true" ]; then
+      echo "ERROR: 'npm run cerc_generate' not successful within CERC_MAX_GENERATE_TIME" 1>&2
+      exit 1
+    fi
+
+    kill $tpid $(ps -ef | grep node | grep next | grep generate | awk '{print $2}') 2>/dev/null
+    tpid=""
   fi
 fi
 
-npm start . -p ${CERC_LISTEN_PORT:-3000}
+$CERC_BUILD_TOOL start . -p ${CERC_LISTEN_PORT:-3000}

stack_orchestrator/data/container-build/cerc-uniswap-interface/Dockerfile

@@ -0,0 +1,10 @@
+FROM node:18.17.1-alpine3.18
+
+RUN apk --update --no-cache add git make alpine-sdk bash
+
+WORKDIR /app
+
+COPY . .
+
+RUN echo "Building uniswap-interface" && \
+    yarn

stack_orchestrator/data/container-build/cerc-uniswap-interface/build.sh

@@ -0,0 +1,8 @@
+#!/usr/bin/env bash
+# Build the uniswap-interface image
+source ${CERC_CONTAINER_BASE_DIR}/build-base.sh
+
+# See: https://stackoverflow.com/a/246128/1701505
+SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
+
+docker build -t cerc/uniswap-interface:local -f ${SCRIPT_DIR}/Dockerfile ${build_command_args} ${CERC_REPO_BASE_DIR}/uniswap-interface

stack_orchestrator/data/container-build/cerc-urbit-globs-host/Dockerfile

@@ -0,0 +1,7 @@
+FROM python:3.13.0a2-alpine3.18
+
+RUN apk --update --no-cache add alpine-sdk jq bash curl wget
+
+WORKDIR /app
+
+ENTRYPOINT [ "bash" ]

stack_orchestrator/data/container-build/cerc-urbit-globs-host/build.sh

@@ -0,0 +1,9 @@
+#!/usr/bin/env bash
+# Build the urbit-globs-host image
+
+source ${CERC_CONTAINER_BASE_DIR}/build-base.sh
+
+# See: https://stackoverflow.com/a/246128/1701505
+SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
+
+docker build -t cerc/urbit-globs-host:local -f ${SCRIPT_DIR}/Dockerfile ${build_command_args} ${SCRIPT_DIR}

stack_orchestrator/data/container-build/cerc-watcher-merkl-sushiswap-v3/Dockerfile

@@ -0,0 +1,10 @@
+FROM node:18.17.1-alpine3.18
+
+RUN apk --update --no-cache add git python3 alpine-sdk bash curl jq
+
+WORKDIR /app
+
+COPY . .
+
+RUN echo "Installing dependencies and building merkl-sushiswap-v3-watcher-ts" && \
+  yarn && yarn build

stack_orchestrator/data/container-build/cerc-watcher-merkl-sushiswap-v3/build.sh

@@ -0,0 +1,11 @@
+#!/usr/bin/env bash
+# Build cerc/watcher-merkl-sushiswap-v3
+
+source ${CERC_CONTAINER_BASE_DIR}/build-base.sh
+
+# See: https://stackoverflow.com/a/246128/1701505
+SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
+
+docker build -t cerc/my-new-stack:local -f ${CERC_REPO_BASE_DIR}/my-new-stack/Dockerfile ${build_command_args} ${CERC_REPO_BASE_DIR}/my-new-stack
+
+docker build -t cerc/watcher-merkl-sushiswap-v3:local -f ${SCRIPT_DIR}/Dockerfile ${build_command_args} ${CERC_REPO_BASE_DIR}/merkl-sushiswap-v3-watcher-ts

stack_orchestrator/data/container-build/cerc-watcher-sushiswap-v3/Dockerfile

@@ -0,0 +1,10 @@
+FROM node:18.17.1-alpine3.18
+
+RUN apk --update --no-cache add git python3 alpine-sdk bash curl jq
+
+WORKDIR /app
+
+COPY . .
+
+RUN echo "Installing dependencies and building sushiswap-v3-watcher-ts" && \
+  yarn && yarn build

stack_orchestrator/data/container-build/cerc-watcher-sushiswap-v3/build.sh

@@ -0,0 +1,9 @@
+#!/usr/bin/env bash
+# Build cerc/watcher-sushiswap-v3
+
+source ${CERC_CONTAINER_BASE_DIR}/build-base.sh
+
+# See: https://stackoverflow.com/a/246128/1701505
+SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
+
+docker build -t cerc/watcher-sushiswap-v3:local -f ${SCRIPT_DIR}/Dockerfile ${build_command_args} ${CERC_REPO_BASE_DIR}/sushiswap-v3-watcher-ts

stack_orchestrator/data/container-image-list.txt

@@ -57,3 +57,6 @@ cerc/nitro-contracts
 cerc/mobymask-snap
 cerc/ponder
 cerc/nitro-rpc-client
+cerc/watcher-merkl-sushiswap-v3
+cerc/watcher-sushiswap-v3
+cerc/uniswap-interface

stack_orchestrator/data/pod-list.txt

@@ -43,3 +43,5 @@ nitro-contracts
 mobymask-snap
 ponder
 ipld-eth-server-payments
+merkl-sushiswap-v3
+sushiswap-v3

stack_orchestrator/data/repository-list.txt

@@ -47,3 +47,6 @@ github.com/cerc-io/go-nitro
 github.com/cerc-io/ts-nitro
 github.com/cerc-io/mobymask-snap
 github.com/cerc-io/ponder
+github.com/cerc-io/merkl-sushiswap-v3-watcher-ts
+github.com/cerc-io/sushiswap-v3-watcher-ts
+github.com/cerc-io/uniswap-interface

stack_orchestrator/data/stacks/azimuth/README.md

@@ -9,19 +9,11 @@ Prerequisite: `ipld-eth-server` RPC and GQL endpoints
 Clone required repositories:
 
 ```bash
-laconic-so --stack azimuth setup-repositories
+laconic-so --stack azimuth setup-repositories --pull
 ```
 
 NOTE: If the repository already exists and checked out to a different version, `setup-repositories` command will throw an error.
-For getting around this, the `azimuth-watcher-ts` repository can be removed and then run the command.
+For getting around this, the `azimuth-watcher-ts` repository can be removed and then run the command again.
-
-Checkout to the required versions and branches in repos
-
-```bash
-# azimuth-watcher-ts
-cd ~/cerc/azimuth-watcher-ts
-git checkout v0.1.0
-```
 
 Build the container images:
 
@@ -31,42 +23,85 @@ laconic-so --stack azimuth build-containers
 
 This should create the required docker images in the local image registry.
 
-### Configuration
+## Create a deployment
 
-* Create and update an env file to be used in the next step:
+First, create a spec file for the deployment, which will map the stack's ports and volumes to the host:
+```bash
+laconic-so --stack azimuth deploy init --output azimuth-spec.yml
+```
+
+### Ports
+
+Edit `network` in spec file to map container ports to same ports in host
+
+```yaml
+...
+network:
+  ports:
+    watcher-db:
+     - 0.0.0.0:15432:5432
+    azimuth-watcher-server:
+     - 0.0.0.0:3001:3001
+    censures-watcher-server:
+     - 0.0.0.0:3002:3002
+    claims-watcher-server:
+     - 0.0.0.0:3003:3003
+    conditional-star-release-watcher-server:
+     - 0.0.0.0:3004:3004
+    delegated-sending-watcher-server:
+     - 0.0.0.0:3005:3005
+    ecliptic-watcher-server:
+     - 0.0.0.0:3006:3006
+    linear-star-release-watcher-server:
+     - 0.0.0.0:3007:3007
+    polls-watcher-server:
+     - 0.0.0.0:3008:3008
+    gateway-server:
+     - 0.0.0.0:4000:4000
+...
+```
+
+### Data volumes
+Container data volumes are bind-mounted to specified paths in the host filesystem.
+The default setup (generated by `laconic-so deploy init`) places the volumes in the `./data` subdirectory of the deployment directory. The default mappings can be customized by editing the "spec" file generated by `laconic-so deploy init`.
+
+---
+
+Once you've made any needed changes to the spec file, create a deployment from it:
+```bash
+laconic-so --stack azimuth deploy create --spec-file azimuth-spec.yml --deployment-dir azimuth-deployment
+```
+
+## Set env variables
+
+Inside the deployment directory, open the file `config.env` and add variable to update RPC endpoint :
 
   ```bash
-  # External ipld-eth-server endpoints
+  # External RPC endpoints
   CERC_IPLD_ETH_RPC=
-  CERC_IPLD_ETH_GQL=
   ```
 
-* NOTE: If `ipld-eth-server` is running on the host machine, use `host.docker.internal` as the hostname to access host ports
+* NOTE: If RPC endpoint is on the host machine, use `host.docker.internal` as the hostname to access the host port, or use the `ip a` command to find the IP address of the `docker0` interface (this will usually be something like `172.17.0.1` or `172.18.0.1`)
 
-### Deploy the stack
+## Start the stack
 
-* Deploy the containers:
+Start the deployment:
-
+```bash
-  ```bash
+laconic-so deployment --dir azimuth-deployment start
-  laconic-so --stack azimuth deploy-system --env-file <PATH_TO_ENV_FILE> up
+```
-  ```
 
 * List and check the health status of all the containers using `docker ps` and wait for them to be `healthy`
 
 ## Clean up
 
-Stop all the services running in background:
+To stop all azimuth services running in the background, while preserving chain data:
 
 ```bash
-laconic-so --stack azimuth deploy-system down
+laconic-so deployment --dir azimuth-deployment stop
 ```
 
-Clear volumes created by this stack:
+To stop all azimuth services and also delete data:
 
 ```bash
-# List all relevant volumes
+laconic-so deployment --dir azimuth-deployment stop --delete-volumes
-docker volume ls -q --filter "name=.*watcher_db_data"
-
-# Remove all the listed volumes
-docker volume rm $(docker volume ls -q --filter "name=.*watcher_db_data")
 ```

stack_orchestrator/data/stacks/azimuth/stack.yml

@@ -1,7 +1,7 @@
 version: "1.0"
 name: azimuth
 repos:
-  - github.com/cerc-io/azimuth-watcher-ts@v0.1.1
+  - github.com/cerc-io/azimuth-watcher-ts@v0.1.2
 containers:
   - cerc/watcher-azimuth
 pods:

stack_orchestrator/data/stacks/graph-node/README.md

@@ -59,7 +59,7 @@ ports:
 Create deployment:
 
 ```bash
-laconic-so deploy create --spec-file graph-node-spec.yml --deployment-dir graph-node-deployment
+laconic-so --stack graph-node deploy create --spec-file graph-node-spec.yml --deployment-dir graph-node-deployment
 ```
 
 ## Start the stack

stack_orchestrator/data/stacks/merkl-sushiswap-v3/README.md

@@ -0,0 +1,81 @@
+# Merkl SushiSwap v3 Watcher
+
+## Setup
+
+Clone required repositories:
+
+```bash
+laconic-so --stack merkl-sushiswap-v3 setup-repositories --git-ssh --pull
+```
+
+Build the container images:
+
+```bash
+laconic-so --stack merkl-sushiswap-v3 build-containers
+```
+
+## Deploy
+
+### Configuration
+
+Create and update an env file to be used in the next step:
+
+  ```bash
+  # External Filecoin (ETH RPC) endpoint to point the watcher
+  CERC_ETH_RPC_ENDPOINT=
+  ```
+
+### Deploy the stack
+
+```bash
+laconic-so --stack merkl-sushiswap-v3 deploy --cluster merkl_sushiswap_v3 --env-file <PATH_TO_ENV_FILE> up
+```
+
+* To list down and monitor the running containers:
+
+  ```bash
+  laconic-so --stack merkl-sushiswap-v3 deploy --cluster merkl_sushiswap_v3 ps
+
+  # With status
+  docker ps -a
+
+  # Check logs for a container
+  docker logs -f <CONTAINER_ID>
+  ```
+
+* Open the GQL playground at http://localhost:3007/graphql
+
+  ```graphql
+  {
+    _meta {
+      block {
+        number
+        timestamp
+      }
+      hasIndexingErrors
+    }
+    
+    factories {
+      id
+      poolCount
+    }
+  }
+  ```
+
+## Clean up
+
+Stop all the services running in background:
+
+```bash
+laconic-so --stack merkl-sushiswap-v3 deploy --cluster merkl_sushiswap_v3 down
+```
+
+Clear volumes created by this stack:
+
+```bash
+# List all relevant volumes
+docker volume ls -q --filter "name=merkl_sushiswap_v3"
+
+# Remove all the listed volumes
+docker volume rm $(docker volume ls -q --filter "name=merkl_sushiswap_v3")
+```

stack_orchestrator/data/stacks/merkl-sushiswap-v3/stack.yml

@@ -0,0 +1,9 @@
+version: "1.0"
+name: merkl-sushiswap-v3
+description: "SushiSwap v3 watcher stack"
+repos:
+  - github.com/cerc-io/merkl-sushiswap-v3-watcher-ts@v0.1.4
+containers:
+  - cerc/watcher-merkl-sushiswap-v3
+pods:
+  - watcher-merkl-sushiswap-v3

stack_orchestrator/data/stacks/proxy-server/README.md

@@ -0,0 +1,79 @@
+# Proxy Server
+
+Instructions to setup and deploy a HTTP proxy server
+
+## Setup
+
+Clone required repository:
+
+```bash
+laconic-so --stack proxy-server setup-repositories --pull
+
+# If this throws an error as a result of being already checked out to a branch/tag in a repo, remove the repositories mentioned below and re-run the command
+```
+
+Build the container image:
+
+```bash
+laconic-so --stack proxy-server build-containers
+```
+
+## Create a deployment
+
+* First, create a spec file for the deployment, which will allow mapping the stack's ports and volumes to the host:
+
+  ```bash
+  laconic-so --stack proxy-server deploy init --output proxy-server-spec.yml
+  ```
+
+* Edit `network` in spec file to map container ports to same ports in host:
+
+  ```yml
+  ...
+  network:
+    ports:
+      proxy-server:
+        - '4000:4000'
+  ...
+  ```
+
+* Once you've made any needed changes to the spec file, create a deployment from it:
+
+  ```bash
+  laconic-so --stack proxy-server deploy create --spec-file proxy-server-spec.yml --deployment-dir proxy-server-deployment
+  ```
+
+* Inside the deployment directory, open the file `config.env` and set the following env variables:
+
+  ```bash
+  # Whether to run the proxy server (Optional) (Default: true)
+  ENABLE_PROXY=
+
+  # Upstream endpoint
+  # (Eg. https://api.example.org)
+  CERC_PROXY_UPSTREAM=
+
+  # Origin header to be used (Optional)
+  # (Eg. https://app.example.org)
+  CERC_PROXY_ORIGIN_HEADER=
+  ```
+
+## Start the stack
+
+Start the deployment:
+
+```bash
+laconic-so deployment --dir proxy-server-deployment start
+```
+
+* List and check the health status of the container using `docker ps`
+
+* The proxy server will now be listening at http://localhost:4000
+
+## Clean up
+
+To stop the service running in background:
+
+```bash
+laconic-so deployment --dir proxy-server-deployment stop
+```

stack_orchestrator/data/stacks/proxy-server/stack.yml

@@ -0,0 +1,8 @@
+version: "0.1"
+name: proxy-server
+repos:
+  - github.com/cerc-io/watcher-ts@v0.2.78
+containers:
+  - cerc/watcher-ts
+pods:
+  - proxy-server

stack_orchestrator/data/stacks/sushiswap-v3/README.md

@@ -0,0 +1,62 @@
+# SushiSwap v3 Watcher
+
+## Setup
+
+Clone required repositories:
+
+```bash
+laconic-so --stack sushiswap-v3 setup-repositories --git-ssh --pull
+```
+
+Build the container images:
+
+```bash
+laconic-so --stack sushiswap-v3 build-containers
+```
+
+## Deploy
+
+### Configuration
+
+Create and update an env file to be used in the next step:
+
+  ```bash
+  # External Filecoin (ETH RPC) endpoint to point the watcher
+  CERC_ETH_RPC_ENDPOINT=
+  ```
+
+### Deploy the stack
+
+```bash
+laconic-so --stack sushiswap-v3 deploy --cluster sushiswap_v3 --env-file <PATH_TO_ENV_FILE> up
+```
+
+* To list down and monitor the running containers:
+
+  ```bash
+  laconic-so --stack sushiswap-v3 deploy --cluster sushiswap_v3 ps
+
+  # With status
+  docker ps -a
+
+  # Check logs for a container
+  docker logs -f <CONTAINER_ID>
+  ```
+
+## Clean up
+
+Stop all the services running in background:
+
+```bash
+laconic-so --stack sushiswap-v3 deploy --cluster sushiswap_v3 down
+```
+
+Clear volumes created by this stack:
+
+```bash
+# List all relevant volumes
+docker volume ls -q --filter "name=sushiswap_v3"
+
+# Remove all the listed volumes
+docker volume rm $(docker volume ls -q --filter "name=sushiswap_v3")
+```

stack_orchestrator/data/stacks/sushiswap-v3/stack.yml

@@ -0,0 +1,9 @@
+version: "1.0"
+name: sushiswap-v3
+description: "SushiSwap v3 watcher stack"
+repos:
+  - github.com/cerc-io/sushiswap-v3-watcher-ts@v0.1.4
+containers:
+  - cerc/watcher-sushiswap-v3
+pods:
+  - watcher-sushiswap-v3

stack_orchestrator/data/stacks/uniswap-urbit-app/README.md

@@ -0,0 +1,151 @@
+# Self-hosted Uniswap Frontend
+
+Instructions to setup and deploy Uniswap app on Urbit
+
+Build and deploy:
+
+- Urbit
+- Uniswap app
+
+## Setup
+
+Clone required repositories:
+
+```bash
+laconic-so --stack uniswap-urbit-app setup-repositories --pull
+
+# If this throws an error as a result of being already checked out to a branch/tag in a repo, remove the repositories mentioned below and re-run the command
+```
+
+Build the container images:
+
+```bash
+laconic-so --stack uniswap-urbit-app build-containers
+```
+
+## Create a deployment
+
+First, create a spec file for the deployment, which will map the stack's ports and volumes to the host:
+
+```bash
+laconic-so --stack uniswap-urbit-app deploy init --output uniswap-urbit-app-spec.yml
+```
+
+### Ports
+
+Edit `network` in spec file to map container ports to same ports in host
+
+```
+...
+network:
+  ports:
+    urbit-fake-ship:
+      - '8080:80'
+    proxy-server:
+      - '4000:4000'
+    ipfs-glob-host:
+      - '8081:8080'
+      - '5001:5001'
+...
+```
+
+### Data volumes
+
+Container data volumes are bind-mounted to specified paths in the host filesystem.
+The default setup (generated by `laconic-so deploy init`) places the volumes in the `./data` subdirectory of the deployment directory. The default mappings can be customized by editing the "spec" file generated by `laconic-so deploy init`.
+
+---
+
+Once you've made any needed changes to the spec file, create a deployment from it:
+
+```bash
+laconic-so --stack uniswap-urbit-app deploy create --spec-file uniswap-urbit-app-spec.yml --deployment-dir uniswap-urbit-app-deployment
+```
+
+## Set env variables
+
+Inside the deployment directory, open the file `config.env` and set the following env variables:
+
+  ```bash
+  # External RPC endpoints
+  # https://docs.infura.io/getting-started#2-create-an-api-key
+  CERC_INFURA_KEY=
+
+  # Uniswap API GQL Endpoint
+  # Set this to GQL proxy server endpoint for uniswap app
+  # (Eg. http://localhost:4000/v1/graphql)
+  # (Eg. https://abc.xyz.com/v1/graphql)
+  CERC_UNISWAP_GQL=
+
+  # Optional
+
+  # Whether to run the proxy GQL server
+  # (Disable only if proxy not required to be run) (Default: true)
+  ENABLE_PROXY=
+
+  # Proxy server configuration
+  # Used only if proxy is enabled
+
+  # Upstream API URL
+  # (Eg. https://api.example.org)
+  CERC_PROXY_UPSTREAM=https://api.uniswap.org
+
+  # Origin header to be used in the proxy
+  # (Eg. https://app.example.org)
+  CERC_PROXY_ORIGIN_HEADER=https://app.uniswap.org
+
+  # IPFS configuration
+
+  # IFPS endpoint to host the glob file on
+  # (Default: http://ipfs-glob-host:5001 pointing to in-stack IPFS node)
+  CERC_IPFS_GLOB_HOST_ENDPOINT=
+
+  # IFPS endpoint to fetch the glob file from
+  # (Default: http://ipfs-glob-host:8080 pointing to in-stack IPFS node)
+  CERC_IPFS_SERVER_ENDPOINT=
+  ```
+
+## Start the stack
+
+Start the deployment:
+
+```bash
+laconic-so deployment --dir uniswap-urbit-app-deployment start
+```
+
+* List and check the health status of all the containers using `docker ps` and wait for them to be `healthy`
+
+* Run the following to get login password for Urbit web interface:
+
+  ```bash
+  laconic-so deployment --dir uniswap-urbit-app-deployment exec urbit-fake-ship "curl -s --data '{\"source\":{\"dojo\":\"+code\"},\"sink\":{\"stdout\":null}}' http://localhost:12321"
+
+  # Expected output: "<PASSWORD>\n"%
+  ```
+
+* Open the Urbit web UI at http://localhost:8080 and use the `PASSWORD` from previous step to login
+
+* The uniswap app is not available when starting stack for the first time. Check `urbit-fake-ship` logs to see that app has installed
+  ```
+  laconic-so deployment --dir uniswap-urbit-app-deployment logs -f
+
+  # Expected output:
+  # laconic-3ccf7ee79bdae874-urbit-fake-ship-1    | docket: fetching %http glob for %uniswap desk
+  # laconic-3ccf7ee79bdae874-urbit-fake-ship-1    | ">="">="Uniswap app installed
+  ```
+
+* The uniswap app will be now visible at http://localhost:8080
+
+## Clean up
+
+To stop all uniswap-urbit-app services running in the background, while preserving data:
+
+```bash
+laconic-so deployment --dir uniswap-urbit-app-deployment stop
+```
+
+To stop all uniswap-urbit-app services and also delete data:
+
+```bash
+laconic-so deployment --dir uniswap-urbit-app-deployment stop --delete-volumes
+```

stack_orchestrator/data/stacks/uniswap-urbit-app/stack.yml

@@ -0,0 +1,13 @@
+version: "0.1"
+name: uniswap-urbit-app
+repos:
+  - github.com/cerc-io/uniswap-interface@laconic # TODO: Use release
+  - github.com/cerc-io/watcher-ts@v0.2.78
+containers:
+  - cerc/uniswap-interface
+  - cerc/watcher-ts
+  - cerc/urbit-globs-host
+pods:
+  - uniswap-interface
+  - proxy-server
+  - uniswap-urbit

stack_orchestrator/data/stacks/webapp-template/README.md

@@ -0,0 +1 @@
+# Template stack for webapp deployments

stack_orchestrator/data/stacks/webapp-template/stack.yml

@@ -0,0 +1,7 @@
+version: "1.0"
+name: test
+description: "Webapp deployment stack"
+containers:
+  - cerc/webapp-template-container
+pods:
+  - webapp-template

stack_orchestrator/deploy/compose/deploy_docker.py

@@ -16,14 +16,17 @@
 from pathlib import Path
 from python_on_whales import DockerClient, DockerException
 from stack_orchestrator.deploy.deployer import Deployer, DeployerException, DeployerConfigGenerator
+from stack_orchestrator.deploy.deployment_context import DeploymentContext
 
 
 class DockerDeployer(Deployer):
     name: str = "compose"
+    type: str
 
-    def __init__(self, deployment_dir, compose_files, compose_project_name, compose_env_file) -> None:
+    def __init__(self, type, deployment_context: DeploymentContext, compose_files, compose_project_name, compose_env_file) -> None:
         self.docker = DockerClient(compose_files=compose_files, compose_project_name=compose_project_name,
                                    compose_env_file=compose_env_file)
+        self.type = type
 
     def up(self, detach, services):
         try:
@@ -37,6 +40,13 @@ class DockerDeployer(Deployer):
         except DockerException as e:
             raise DeployerException(e)
 
+    def status(self):
+        try:
+            for p in self.docker.compose.ps():
+                print(f"{p.name}\t{p.state.status}")
+        except DockerException as e:
+            raise DeployerException(e)
+
     def ps(self):
         try:
             return self.docker.compose.ps()
@@ -61,17 +71,17 @@ class DockerDeployer(Deployer):
         except DockerException as e:
             raise DeployerException(e)
 
-    def run(self, image, command, user, volumes, entrypoint=None):
+    def run(self, image: str, command=None, user=None, volumes=None, entrypoint=None, env={}, ports=[], detach=False):
         try:
-            return self.docker.run(image=image, command=command, user=user, volumes=volumes, entrypoint=entrypoint)
+            return self.docker.run(image=image, command=command, user=user, volumes=volumes,
+                                   entrypoint=entrypoint, envs=env, detach=detach, publish=ports, publish_all=len(ports) == 0)
         except DockerException as e:
             raise DeployerException(e)
 
 
 class DockerDeployerConfigGenerator(DeployerConfigGenerator):
-    config_file_name: str = "kind-config.yml"
 
-    def __init__(self) -> None:
+    def __init__(self, type: str) -> None:
         super().__init__()
 
     # Nothing needed at present for the docker deployer

stack_orchestrator/deploy/deploy.py

@@ -24,6 +24,8 @@ from importlib import resources
 import subprocess
 import click
 from pathlib import Path
+from stack_orchestrator import constants
+from stack_orchestrator.opts import opts
 from stack_orchestrator.util import include_exclude_check, get_parsed_stack_config, global_options2, get_dev_root_path
 from stack_orchestrator.deploy.deployer import Deployer, DeployerException
 from stack_orchestrator.deploy.deployer_factory import getDeployer
@@ -39,7 +41,7 @@ from stack_orchestrator.deploy.deployment_create import setup as deployment_setu
 @click.option("--exclude", help="don\'t start these components")
 @click.option("--env-file", help="env file to be used")
 @click.option("--cluster", help="specify a non-default cluster name")
-@click.option("--deploy-to", help="cluster system to deploy to (compose or k8s)")
+@click.option("--deploy-to", help="cluster system to deploy to (compose or k8s or k8s-kind)")
 @click.pass_context
 def command(ctx, include, exclude, env_file, cluster, deploy_to):
     '''deploy a stack'''
@@ -62,11 +64,19 @@ def command(ctx, include, exclude, env_file, cluster, deploy_to):
 
 
 def create_deploy_context(
-        global_context, deployment_context: DeploymentContext, stack, include, exclude, cluster, env_file, deployer):
+        global_context,
+        deployment_context: DeploymentContext,
+        stack,
+        include,
+        exclude,
+        cluster,
+        env_file,
+        deploy_to) -> DeployCommandContext:
+    # Extract the cluster name from the deployment, if we have one
+    if deployment_context and cluster is None:
+        cluster = deployment_context.get_cluster_id()
     cluster_context = _make_cluster_context(global_context, stack, include, exclude, cluster, env_file)
-    deployment_dir = deployment_context.deployment_dir if deployment_context else None
+    deployer = getDeployer(deploy_to, deployment_context, compose_files=cluster_context.compose_files,
-    # See: https://gabrieldemarmiesse.github.io/python-on-whales/sub-commands/compose/
-    deployer = getDeployer(deployer, deployment_dir, compose_files=cluster_context.compose_files,
                            compose_project_name=cluster_context.cluster,
                            compose_env_file=cluster_context.env_file)
     return DeployCommandContext(stack, cluster_context, deployer)
@@ -102,6 +112,14 @@ def down_operation(ctx, delete_volumes, extra_args_list):
         ctx.obj.deployer.down(timeout=timeout_arg, volumes=delete_volumes)
 
 
+def status_operation(ctx):
+    global_context = ctx.parent.parent.obj
+    if not global_context.dry_run:
+        if global_context.verbose:
+            print("Running compose status")
+        ctx.obj.deployer.status()
+
+
 def ps_operation(ctx):
     global_context = ctx.parent.parent.obj
     if not global_context.dry_run:
@@ -155,7 +173,7 @@ def exec_operation(ctx, extra_args):
         if global_context.verbose:
             print(f"Running compose exec {service_name} {command_to_exec}")
         try:
-            ctx.obj.deployer.execute(service_name, command_to_exec, envs=container_exec_env)
+            ctx.obj.deployer.execute(service_name, command_to_exec, envs=container_exec_env, tty=True)
         except DeployerException:
             print("container command returned error exit status")
 
@@ -248,6 +266,22 @@ def _make_runtime_env(ctx):
     return container_exec_env
 
 
+def _make_default_cluster_name(deployment, compose_dir, stack, include, exclude):
+    # Create default unique, stable cluster name from confile file path and stack name if provided
+    if deployment:
+        path = os.path.realpath(os.path.abspath(compose_dir))
+    else:
+        path = "internal"
+    unique_cluster_descriptor = f"{path},{stack},{include},{exclude}"
+    if opts.o.debug:
+        print(f"pre-hash descriptor: {unique_cluster_descriptor}")
+    hash = hashlib.md5(unique_cluster_descriptor.encode()).hexdigest()[:16]
+    cluster = f"{constants.cluster_name_prefix}{hash}"
+    if opts.o.debug:
+        print(f"Using cluster name: {cluster}")
+    return cluster
+
+
 # stack has to be either PathLike pointing to a stack yml file, or a string with the name of a known stack
 def _make_cluster_context(ctx, stack, include, exclude, cluster, env_file):
 
@@ -265,16 +299,9 @@ def _make_cluster_context(ctx, stack, include, exclude, cluster, env_file):
         compose_dir = Path(__file__).absolute().parent.parent.joinpath("data", "compose")
 
     if cluster is None:
-        # Create default unique, stable cluster name from confile file path and stack name if provided
+        cluster = _make_default_cluster_name(deployment, compose_dir, stack, include, exclude)
-        # TODO: change this to the config file path
+    else:
-        path = os.path.realpath(sys.argv[0])
+        _make_default_cluster_name(deployment, compose_dir, stack, include, exclude)
-        unique_cluster_descriptor = f"{path},{stack},{include},{exclude}"
-        if ctx.debug:
-            print(f"pre-hash descriptor: {unique_cluster_descriptor}")
-        hash = hashlib.md5(unique_cluster_descriptor.encode()).hexdigest()
-        cluster = f"laconic-{hash}"
-        if ctx.verbose:
-            print(f"Using cluster name: {cluster}")
 
     # See: https://stackoverflow.com/a/20885799/1701505
     from stack_orchestrator import data

stack_orchestrator/deploy/deploy_util.py

@@ -14,9 +14,10 @@
 # along with this program.  If not, see <http:#www.gnu.org/licenses/>.
 
 import os
-from typing import List
+from typing import List, Any
 from stack_orchestrator.deploy.deploy_types import DeployCommandContext, VolumeMapping
 from stack_orchestrator.util import get_parsed_stack_config, get_yaml, get_compose_file_dir, get_pod_list
+from stack_orchestrator.opts import opts
 
 
 def _container_image_from_service(stack: str, service: str):
@@ -37,6 +38,33 @@ def _container_image_from_service(stack: str, service: str):
     return image_name
 
 
+def parsed_pod_files_map_from_file_names(pod_files):
+    parsed_pod_yaml_map : Any = {}
+    for pod_file in pod_files:
+        with open(pod_file, "r") as pod_file_descriptor:
+            parsed_pod_file = get_yaml().load(pod_file_descriptor)
+            parsed_pod_yaml_map[pod_file] = parsed_pod_file
+    if opts.o.debug:
+        print(f"parsed_pod_yaml_map: {parsed_pod_yaml_map}")
+    return parsed_pod_yaml_map
+
+
+def images_for_deployment(pod_files: List[str]):
+    image_set = set()
+    parsed_pod_yaml_map = parsed_pod_files_map_from_file_names(pod_files)
+    # Find the set of images in the pods
+    for pod_name in parsed_pod_yaml_map:
+        pod = parsed_pod_yaml_map[pod_name]
+        services = pod["services"]
+        for service_name in services:
+            service_info = services[service_name]
+            image = service_info["image"]
+            image_set.add(image)
+    if opts.o.debug:
+        print(f"image_set: {image_set}")
+    return image_set
+
+
 def _volumes_to_docker(mounts: List[VolumeMapping]):
     # Example from doc: [("/", "/host"), ("/etc/hosts", "/etc/hosts", "rw")]
     result = []

stack_orchestrator/deploy/deployer.py

@@ -31,6 +31,10 @@ class Deployer(ABC):
     def ps(self):
         pass
 
+    @abstractmethod
+    def status(self):
+        pass
+
     @abstractmethod
     def port(self, service, private_port):
         pass
@@ -44,7 +48,7 @@ class Deployer(ABC):
         pass
 
     @abstractmethod
-    def run(self, image, command, user, volumes, entrypoint):
+    def run(self, image: str, command=None, user=None, volumes=None, entrypoint=None, env={}, ports=[], detach=False):
         pass
 
 

stack_orchestrator/deploy/deployer_factory.py

@@ -13,23 +13,24 @@
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <http:#www.gnu.org/licenses/>.
 
+from stack_orchestrator import constants
 from stack_orchestrator.deploy.k8s.deploy_k8s import K8sDeployer, K8sDeployerConfigGenerator
 from stack_orchestrator.deploy.compose.deploy_docker import DockerDeployer, DockerDeployerConfigGenerator
 
 
 def getDeployerConfigGenerator(type: str):
     if type == "compose" or type is None:
-        return DockerDeployerConfigGenerator()
+        return DockerDeployerConfigGenerator(type)
-    elif type == "k8s":
+    elif type == constants.k8s_deploy_type or type == constants.k8s_kind_deploy_type:
-        return K8sDeployerConfigGenerator()
+        return K8sDeployerConfigGenerator(type)
     else:
         print(f"ERROR: deploy-to {type} is not valid")
 
 
-def getDeployer(type: str, deployment_dir, compose_files, compose_project_name, compose_env_file):
+def getDeployer(type: str, deployment_context, compose_files, compose_project_name, compose_env_file):
     if type == "compose" or type is None:
-        return DockerDeployer(deployment_dir, compose_files, compose_project_name, compose_env_file)
+        return DockerDeployer(type, deployment_context, compose_files, compose_project_name, compose_env_file)
-    elif type == "k8s":
+    elif type == type == constants.k8s_deploy_type or type == constants.k8s_kind_deploy_type:
-        return K8sDeployer(deployment_dir, compose_files, compose_project_name, compose_env_file)
+        return K8sDeployer(type, deployment_context, compose_files, compose_project_name, compose_env_file)
     else:
         print(f"ERROR: deploy-to {type} is not valid")

stack_orchestrator/deploy/deployment.py

@@ -16,8 +16,11 @@
 import click
 from pathlib import Path
 import sys
-from stack_orchestrator.deploy.deploy import up_operation, down_operation, ps_operation, port_operation
+from stack_orchestrator import constants
+from stack_orchestrator.deploy.images import push_images_operation
+from stack_orchestrator.deploy.deploy import up_operation, down_operation, ps_operation, port_operation, status_operation
 from stack_orchestrator.deploy.deploy import exec_operation, logs_operation, create_deploy_context
+from stack_orchestrator.deploy.deploy_types import DeployCommandContext
 from stack_orchestrator.deploy.deployment_context import DeploymentContext
 
 
@@ -45,13 +48,17 @@ def command(ctx, dir):
     ctx.obj = deployment_context
 
 
-def make_deploy_context(ctx):
+def make_deploy_context(ctx) -> DeployCommandContext:
     context: DeploymentContext = ctx.obj
     stack_file_path = context.get_stack_file()
     env_file = context.get_env_file()
-    cluster_name = context.get_cluster_name()
+    cluster_name = context.get_cluster_id()
+    if constants.deploy_to_key in context.spec.obj:
+        deployment_type = context.spec.obj[constants.deploy_to_key]
+    else:
+        deployment_type = constants.compose_deploy_type
     return create_deploy_context(ctx.parent.parent.obj, context, stack_file_path, None, None, cluster_name, env_file,
-                                 context.spec.obj["deploy-to"])
+                                 deployment_type)
 
 
 @command.command()
@@ -104,6 +111,14 @@ def ps(ctx):
     ps_operation(ctx)
 
 
+@command.command()
+@click.pass_context
+def push_images(ctx):
+    deploy_command_context: DeployCommandContext = make_deploy_context(ctx)
+    deployment_context: DeploymentContext = ctx.obj
+    push_images_operation(deploy_command_context, deployment_context)
+
+
 @command.command()
 @click.argument('extra_args', nargs=-1)  # help: command: port <service1> <service2>
 @click.pass_context
@@ -132,4 +147,5 @@ def logs(ctx, tail, follow, extra_args):
 @command.command()
 @click.pass_context
 def status(ctx):
-    print(f"Context: {ctx.parent.obj}")
+    ctx.obj = make_deploy_context(ctx)
+    status_operation(ctx)

stack_orchestrator/deploy/deployment_context.py

@@ -14,29 +14,39 @@
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <http:#www.gnu.org/licenses/>.
 
+import hashlib
+import os
 from pathlib import Path
 
+from stack_orchestrator import constants
+from stack_orchestrator.util import get_yaml
 from stack_orchestrator.deploy.stack import Stack
 from stack_orchestrator.deploy.spec import Spec
 
 
 class DeploymentContext:
     deployment_dir: Path
+    id: str
     spec: Spec
     stack: Stack
 
     def get_stack_file(self):
-        return self.deployment_dir.joinpath("stack.yml")
+        return self.deployment_dir.joinpath(constants.stack_file_name)
 
     def get_spec_file(self):
-        return self.deployment_dir.joinpath("spec.yml")
+        return self.deployment_dir.joinpath(constants.spec_file_name)
 
     def get_env_file(self):
-        return self.deployment_dir.joinpath("config.env")
+        return self.deployment_dir.joinpath(constants.config_file_name)
 
-    # TODO: implement me
+    def get_deployment_file(self):
-    def get_cluster_name(self):
+        return self.deployment_dir.joinpath(constants.deployment_file_name)
-        return None
+
+    def get_compose_dir(self):
+        return self.deployment_dir.joinpath(constants.compose_dir_name)
+
+    def get_cluster_id(self):
+        return self.id
 
     def init(self, dir):
         self.deployment_dir = dir
@@ -44,3 +54,16 @@ class DeploymentContext:
         self.spec.init_from_file(self.get_spec_file())
         self.stack = Stack(self.spec.obj["stack"])
         self.stack.init_from_file(self.get_stack_file())
+        deployment_file_path = self.get_deployment_file()
+        if deployment_file_path.exists():
+            with deployment_file_path:
+                obj = get_yaml().load(open(deployment_file_path, "r"))
+                self.id = obj[constants.cluster_id_key]
+        # Handle the case of a legacy deployment with no file
+        # Code below is intended to match the output from _make_default_cluster_name()
+        # TODO: remove when we no longer need to support legacy deployments
+        else:
+            path = os.path.realpath(os.path.abspath(self.get_compose_dir()))
+            unique_cluster_descriptor = f"{path},{self.get_stack_file()},None,None"
+            hash = hashlib.md5(unique_cluster_descriptor.encode()).hexdigest()[:16]
+            self.id = f"{constants.cluster_name_prefix}{hash}"

stack_orchestrator/deploy/deployment_create.py

@@ -20,17 +20,20 @@ from pathlib import Path
 from typing import List
 import random
 from shutil import copy, copyfile, copytree
+from secrets import token_hex
 import sys
+from stack_orchestrator import constants
+from stack_orchestrator.opts import opts
 from stack_orchestrator.util import (get_stack_file_path, get_parsed_deployment_spec, get_parsed_stack_config,
                                      global_options, get_yaml, get_pod_list, get_pod_file_path, pod_has_scripts,
-                                     get_pod_script_paths, get_plugin_code_paths)
+                                     get_pod_script_paths, get_plugin_code_paths, error_exit, env_var_map_from_file)
 from stack_orchestrator.deploy.deploy_types import LaconicStackSetupCommand
 from stack_orchestrator.deploy.deployer_factory import getDeployerConfigGenerator
 from stack_orchestrator.deploy.deployment_context import DeploymentContext
 
 
 def _make_default_deployment_dir():
-    return "deployment-001"
+    return Path("deployment-001")
 
 
 def _get_ports(stack):
@@ -102,8 +105,8 @@ def _fixup_pod_file(pod, spec, compose_dir):
                                        }
                     pod["volumes"][volume] = new_volume_spec
     # Fix up ports
-    if "ports" in spec:
+    if "network" in spec and "ports" in spec["network"]:
-        spec_ports = spec["ports"]
+        spec_ports = spec["network"]["ports"]
         for container_name, container_ports in spec_ports.items():
             if container_name in pod["services"]:
                 pod["services"][container_name]["ports"] = container_ports
@@ -242,37 +245,76 @@ def _parse_config_variables(variable_values: str):
                 variable_name = variable_value_pair[0]
                 variable_value = variable_value_pair[1]
                 result_values[variable_name] = variable_value
-            result = {"config": result_values}
+            result = result_values
     return result
 
 
 @click.command()
 @click.option("--config", help="Provide config variables for the deployment")
+@click.option("--config-file", help="Provide config variables in a file for the deployment")
+@click.option("--kube-config", help="Provide a config file for a k8s deployment")
+@click.option("--image-registry", help="Provide a container image registry url for this k8s cluster")
 @click.option("--output", required=True, help="Write yaml spec file here")
 @click.option("--map-ports-to-host", required=False,
               help="Map ports to the host as one of: any-variable-random (default), "
               "localhost-same, any-same, localhost-fixed-random, any-fixed-random")
 @click.pass_context
-def init(ctx, config, output, map_ports_to_host):
+def init(ctx, config, config_file, kube_config, image_registry, output, map_ports_to_host):
-    yaml = get_yaml()
     stack = global_options(ctx).stack
-    debug = global_options(ctx).debug
+    deployer_type = ctx.obj.deployer.type
-    default_spec_file_content = call_stack_deploy_init(ctx.obj)
+    deploy_command_context = ctx.obj
-    spec_file_content = {"stack": stack, "deploy-to": ctx.obj.deployer.name}
+    return init_operation(
+        deploy_command_context,
+        stack, deployer_type,
+        config, config_file,
+        kube_config,
+        image_registry,
+        output,
+        map_ports_to_host)
+
+
+# The init command's implementation is in a separate function so that we can
+# call it from other commands, bypassing the click decoration stuff
+def init_operation(deploy_command_context, stack, deployer_type, config,
+                   config_file, kube_config, image_registry, output, map_ports_to_host):
+
+    default_spec_file_content = call_stack_deploy_init(deploy_command_context)
+    spec_file_content = {"stack": stack, constants.deploy_to_key: deployer_type}
+    if deployer_type == "k8s":
+        if kube_config is None:
+            error_exit("--kube-config must be supplied with --deploy-to k8s")
+        if image_registry is None:
+            error_exit("--image-registry must be supplied with --deploy-to k8s")
+        spec_file_content.update({constants.kube_config_key: kube_config})
+        spec_file_content.update({constants.image_resigtry_key: image_registry})
+    else:
+        # Check for --kube-config supplied for non-relevant deployer types
+        if kube_config is not None:
+            error_exit(f"--kube-config is not allowed with a {deployer_type} deployment")
+        if image_registry is not None:
+            error_exit(f"--image-registry is not allowed with a {deployer_type} deployment")
     if default_spec_file_content:
         spec_file_content.update(default_spec_file_content)
     config_variables = _parse_config_variables(config)
-    if config_variables:
     # Implement merge, since update() overwrites
+    if config_variables:
         orig_config = spec_file_content.get("config", {})
-        new_config = config_variables["config"]
+        new_config = config_variables
+        merged_config = {**new_config, **orig_config}
+        spec_file_content.update({"config": merged_config})
+    if config_file:
+        config_file_path = Path(config_file)
+        if not config_file_path.exists():
+            error_exit(f"config file: {config_file} does not exist")
+        config_file_variables = env_var_map_from_file(config_file_path)
+        if config_file_variables:
+            orig_config = spec_file_content.get("config", {})
+            new_config = config_file_variables
             merged_config = {**new_config, **orig_config}
             spec_file_content.update({"config": merged_config})
-    if debug:
-        print(f"Creating spec file for stack: {stack} with content: {spec_file_content}")
 
     ports = _get_mapped_ports(stack, map_ports_to_host)
-    spec_file_content["ports"] = ports
+    spec_file_content.update({"network": {"ports": ports}})
 
     named_volumes = _get_named_volumes(stack)
     if named_volumes:
@@ -281,8 +323,11 @@ def init(ctx, config, output, map_ports_to_host):
             volume_descriptors[named_volume] = f"./data/{named_volume}"
         spec_file_content["volumes"] = volume_descriptors
 
+    if opts.o.debug:
+        print(f"Creating spec file for stack: {stack} with content: {spec_file_content}")
+
     with open(output, "w") as output_file:
-        yaml.dump(spec_file_content, output_file)
+        get_yaml().dump(spec_file_content, output_file)
 
 
 def _write_config_file(spec_file: Path, config_env_file: Path):
@@ -296,12 +341,25 @@ def _write_config_file(spec_file: Path, config_env_file: Path):
                     output_file.write(f"{variable_name}={variable_value}\n")
 
 
+def _write_kube_config_file(external_path: Path, internal_path: Path):
+    if not external_path.exists():
+        error_exit(f"Kube config file {external_path} does not exist")
+    copyfile(external_path, internal_path)
+
+
 def _copy_files_to_directory(file_paths: List[Path], directory: Path):
     for path in file_paths:
         # Using copy to preserve the execute bit
         copy(path, os.path.join(directory, os.path.basename(path)))
 
 
+def _create_deployment_file(deployment_dir: Path):
+    deployment_file_path = deployment_dir.joinpath(constants.deployment_file_name)
+    cluster = f"{constants.cluster_name_prefix}{token_hex(8)}"
+    with open(deployment_file_path, "w") as output_file:
+        output_file.write(f"{constants.cluster_id_key}: {cluster}\n")
+
+
 @click.command()
 @click.option("--spec-file", required=True, help="Spec file to use to create this deployment")
 @click.option("--deployment-dir", help="Create deployment files in this directory")
@@ -310,29 +368,42 @@ def _copy_files_to_directory(file_paths: List[Path], directory: Path):
 @click.option("--initial-peers", help="Initial set of persistent peers")
 @click.pass_context
 def create(ctx, spec_file, deployment_dir, network_dir, initial_peers):
-    # This function fails with a useful error message if the file doens't exist
+    deployment_command_context = ctx.obj
+    return create_operation(deployment_command_context, spec_file, deployment_dir, network_dir, initial_peers)
+
+
+# The init command's implementation is in a separate function so that we can
+# call it from other commands, bypassing the click decoration stuff
+def create_operation(deployment_command_context, spec_file, deployment_dir, network_dir, initial_peers):
     parsed_spec = get_parsed_deployment_spec(spec_file)
     stack_name = parsed_spec["stack"]
+    deployment_type = parsed_spec[constants.deploy_to_key]
     stack_file = get_stack_file_path(stack_name)
     parsed_stack = get_parsed_stack_config(stack_name)
-    if global_options(ctx).debug:
+    if opts.o.debug:
         print(f"parsed spec: {parsed_spec}")
     if deployment_dir is None:
-        deployment_dir = _make_default_deployment_dir()
+        deployment_dir_path = _make_default_deployment_dir()
-    if os.path.exists(deployment_dir):
+    else:
-        print(f"Error: {deployment_dir} already exists")
+        deployment_dir_path = Path(deployment_dir)
-        sys.exit(1)
+    if deployment_dir_path.exists():
-    os.mkdir(deployment_dir)
+        error_exit(f"{deployment_dir_path} already exists")
+    os.mkdir(deployment_dir_path)
     # Copy spec file and the stack file into the deployment dir
-    copyfile(spec_file, os.path.join(deployment_dir, "spec.yml"))
+    copyfile(spec_file, deployment_dir_path.joinpath(constants.spec_file_name))
-    copyfile(stack_file, os.path.join(deployment_dir, os.path.basename(stack_file)))
+    copyfile(stack_file, deployment_dir_path.joinpath(os.path.basename(stack_file)))
+    _create_deployment_file(deployment_dir_path)
     # Copy any config varibles from the spec file into an env file suitable for compose
-    _write_config_file(spec_file, os.path.join(deployment_dir, "config.env"))
+    _write_config_file(spec_file, deployment_dir_path.joinpath(constants.config_file_name))
+    # Copy any k8s config file into the deployment dir
+    if deployment_type == "k8s":
+        _write_kube_config_file(Path(parsed_spec[constants.kube_config_key]),
+                                deployment_dir_path.joinpath(constants.kube_config_filename))
     # Copy the pod files into the deployment dir, fixing up content
     pods = get_pod_list(parsed_stack)
-    destination_compose_dir = os.path.join(deployment_dir, "compose")
+    destination_compose_dir = deployment_dir_path.joinpath("compose")
     os.mkdir(destination_compose_dir)
-    destination_pods_dir = os.path.join(deployment_dir, "pods")
+    destination_pods_dir = deployment_dir_path.joinpath("pods")
     os.mkdir(destination_pods_dir)
     data_dir = Path(__file__).absolute().parent.parent.joinpath("data")
     yaml = get_yaml()
@@ -340,12 +411,12 @@ def create(ctx, spec_file, deployment_dir, network_dir, initial_peers):
         pod_file_path = get_pod_file_path(parsed_stack, pod)
         parsed_pod_file = yaml.load(open(pod_file_path, "r"))
         extra_config_dirs = _find_extra_config_dirs(parsed_pod_file, pod)
-        destination_pod_dir = os.path.join(destination_pods_dir, pod)
+        destination_pod_dir = destination_pods_dir.joinpath(pod)
         os.mkdir(destination_pod_dir)
-        if global_options(ctx).debug:
+        if opts.o.debug:
             print(f"extra config dirs: {extra_config_dirs}")
         _fixup_pod_file(parsed_pod_file, parsed_spec, destination_compose_dir)
-        with open(os.path.join(destination_compose_dir, "docker-compose-%s.yml" % pod), "w") as output_file:
+        with open(destination_compose_dir.joinpath("docker-compose-%s.yml" % pod), "w") as output_file:
             yaml.dump(parsed_pod_file, output_file)
         # Copy the config files for the pod, if any
         config_dirs = {pod}
@@ -353,27 +424,26 @@ def create(ctx, spec_file, deployment_dir, network_dir, initial_peers):
         for config_dir in config_dirs:
             source_config_dir = data_dir.joinpath("config", config_dir)
             if os.path.exists(source_config_dir):
-                destination_config_dir = os.path.join(deployment_dir, "config", config_dir)
+                destination_config_dir = deployment_dir_path.joinpath("config", config_dir)
                 # If the same config dir appears in multiple pods, it may already have been copied
                 if not os.path.exists(destination_config_dir):
                     copytree(source_config_dir, destination_config_dir)
         # Copy the script files for the pod, if any
         if pod_has_scripts(parsed_stack, pod):
-            destination_script_dir = os.path.join(destination_pod_dir, "scripts")
+            destination_script_dir = destination_pod_dir.joinpath("scripts")
             os.mkdir(destination_script_dir)
             script_paths = get_pod_script_paths(parsed_stack, pod)
             _copy_files_to_directory(script_paths, destination_script_dir)
     # Delegate to the stack's Python code
     # The deploy create command doesn't require a --stack argument so we need to insert the
     # stack member here.
-    deployment_command_context = ctx.obj
     deployment_command_context.stack = stack_name
     deployment_context = DeploymentContext()
-    deployment_context.init(Path(deployment_dir))
+    deployment_context.init(deployment_dir_path)
     # Call the deployer to generate any deployer-specific files (e.g. for kind)
-    deployer_config_generator = getDeployerConfigGenerator(parsed_spec["deploy-to"])
+    deployer_config_generator = getDeployerConfigGenerator(deployment_type)
-    # TODO: make deployment_dir a Path above
+    # TODO: make deployment_dir_path a Path above
-    deployer_config_generator.generate(Path(deployment_dir))
+    deployer_config_generator.generate(deployment_dir_path)
     call_stack_deploy_create(deployment_context, [network_dir, initial_peers])
 
 

stack_orchestrator/deploy/images.py

@@ -0,0 +1,62 @@
+# Copyright © 2023 Vulcanize
+
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http:#www.gnu.org/licenses/>.
+
+from typing import Set
+
+from python_on_whales import DockerClient
+
+from stack_orchestrator import constants
+from stack_orchestrator.opts import opts
+from stack_orchestrator.deploy.deployment_context import DeploymentContext
+from stack_orchestrator.deploy.deploy_types import DeployCommandContext
+from stack_orchestrator.deploy.deploy_util import images_for_deployment
+
+
+def _image_needs_pushed(image: str):
+    # TODO: this needs to be more intelligent
+    return image.endswith(":local")
+
+
+def remote_tag_for_image(image: str, remote_repo_url: str):
+    # Turns image tags of the form: foo/bar:local into remote.repo/org/bar:deploy
+    (org, image_name_with_version) = image.split("/")
+    (image_name, image_version) = image_name_with_version.split(":")
+    if image_version == "local":
+        return f"{remote_repo_url}/{image_name}:deploy"
+    else:
+        return image
+
+
+# TODO: needs lots of error handling
+def push_images_operation(command_context: DeployCommandContext, deployment_context: DeploymentContext):
+    # Get the list of images for the stack
+    cluster_context = command_context.cluster_context
+    images: Set[str] = images_for_deployment(cluster_context.compose_files)
+    # Tag the images for the remote repo
+    remote_repo_url = deployment_context.spec.obj[constants.image_resigtry_key]
+    docker = DockerClient()
+    for image in images:
+        if _image_needs_pushed(image):
+            remote_tag = remote_tag_for_image(image, remote_repo_url)
+            if opts.o.verbose:
+                print(f"Tagging {image} to {remote_tag}")
+            docker.image.tag(image, remote_tag)
+    # Run docker push commands to upload
+    for image in images:
+        if _image_needs_pushed(image):
+            remote_tag = remote_tag_for_image(image, remote_repo_url)
+            if opts.o.verbose:
+                print(f"Pushing image {remote_tag}")
+            docker.image.push(remote_tag)

stack_orchestrator/deploy/k8s/cluster_info.py

@@ -17,37 +17,116 @@ from kubernetes import client
 from typing import Any, List, Set
 
 from stack_orchestrator.opts import opts
+from stack_orchestrator.util import env_var_map_from_file
 from stack_orchestrator.deploy.k8s.helpers import named_volumes_from_pod_files, volume_mounts_for_service, volumes_for_pod_files
-from stack_orchestrator.deploy.k8s.helpers import parsed_pod_files_map_from_file_names, get_node_pv_mount_path
+from stack_orchestrator.deploy.k8s.helpers import get_node_pv_mount_path
-from stack_orchestrator.deploy.k8s.helpers import env_var_map_from_file, envs_from_environment_variables_map
+from stack_orchestrator.deploy.k8s.helpers import envs_from_environment_variables_map
+from stack_orchestrator.deploy.deploy_util import parsed_pod_files_map_from_file_names, images_for_deployment
 from stack_orchestrator.deploy.deploy_types import DeployEnvVars
+from stack_orchestrator.deploy.spec import Spec
+from stack_orchestrator.deploy.images import remote_tag_for_image
 
 
 class ClusterInfo:
-    parsed_pod_yaml_map: Any = {}
+    parsed_pod_yaml_map: Any
     image_set: Set[str] = set()
-    app_name: str = "test-app"
+    app_name: str
-    deployment_name: str = "test-deployment"
     environment_variables: DeployEnvVars
+    spec: Spec
 
     def __init__(self) -> None:
         pass
 
-    def int(self, pod_files: List[str], compose_env_file):
+    def int(self, pod_files: List[str], compose_env_file, deployment_name, spec: Spec):
         self.parsed_pod_yaml_map = parsed_pod_files_map_from_file_names(pod_files)
         # Find the set of images in the pods
+        self.image_set = images_for_deployment(pod_files)
+        self.environment_variables = DeployEnvVars(env_var_map_from_file(compose_env_file))
+        self.app_name = deployment_name
+        self.spec = spec
+        if (opts.o.debug):
+            print(f"Env vars: {self.environment_variables.map}")
+
+    def get_ingress(self):
+        # No ingress for a deployment that has no http-proxy defined, for now
+        http_proxy_info_list = self.spec.get_http_proxy()
+        ingress = None
+        if http_proxy_info_list:
+            # TODO: handle multiple definitions
+            http_proxy_info = http_proxy_info_list[0]
+            if opts.o.debug:
+                print(f"http-proxy: {http_proxy_info}")
+            # TODO: good enough parsing for webapp deployment for now
+            host_name = http_proxy_info["host-name"]
+            rules = []
+            tls = [client.V1IngressTLS(
+                hosts=[host_name],
+                secret_name=f"{self.app_name}-tls"
+            )]
+            paths = []
+            for route in http_proxy_info["routes"]:
+                path = route["path"]
+                proxy_to = route["proxy-to"]
+                if opts.o.debug:
+                    print(f"proxy config: {path} -> {proxy_to}")
+                # proxy_to has the form <service>:<port>
+                proxy_to_port = int(proxy_to.split(":")[1])
+                paths.append(client.V1HTTPIngressPath(
+                    path_type="Prefix",
+                    path=path,
+                    backend=client.V1IngressBackend(
+                        service=client.V1IngressServiceBackend(
+                            # TODO: this looks wrong
+                            name=f"{self.app_name}-service",
+                            # TODO: pull port number from the service
+                            port=client.V1ServiceBackendPort(number=proxy_to_port)
+                        )
+                    )
+                ))
+            rules.append(client.V1IngressRule(
+                host=host_name,
+                http=client.V1HTTPIngressRuleValue(
+                    paths=paths
+                )
+            ))
+            spec = client.V1IngressSpec(
+                tls=tls,
+                rules=rules
+            )
+            ingress = client.V1Ingress(
+                metadata=client.V1ObjectMeta(
+                    name=f"{self.app_name}-ingress",
+                    annotations={
+                        "kubernetes.io/ingress.class": "nginx",
+                        "cert-manager.io/cluster-issuer": "letsencrypt-prod"
+                    }
+                ),
+                spec=spec
+            )
+        return ingress
+
+    # TODO: suppoprt multiple services
+    def get_service(self):
         for pod_name in self.parsed_pod_yaml_map:
             pod = self.parsed_pod_yaml_map[pod_name]
             services = pod["services"]
             for service_name in services:
                 service_info = services[service_name]
-                image = service_info["image"]
+                port = int(service_info["ports"][0])
-                self.image_set.add(image)
                 if opts.o.debug:
-            print(f"image_set: {self.image_set}")
+                    print(f"service port: {port}")
-        self.environment_variables = DeployEnvVars(env_var_map_from_file(compose_env_file))
+        service = client.V1Service(
-        if (opts.o.debug):
+            metadata=client.V1ObjectMeta(name=f"{self.app_name}-service"),
-            print(f"Env vars: {self.environment_variables.map}")
+            spec=client.V1ServiceSpec(
+                type="ClusterIP",
+                ports=[client.V1ServicePort(
+                    port=port,
+                    target_port=port
+                )],
+                selector={"app": self.app_name}
+            )
+        )
+        return service
 
     def get_pvcs(self):
         result = []
@@ -99,12 +178,19 @@ class ClusterInfo:
                 container_name = service_name
                 service_info = services[service_name]
                 image = service_info["image"]
+                port = int(service_info["ports"][0])
+                if opts.o.debug:
+                    print(f"image: {image}")
+                    print(f"service port: {port}")
+                # Re-write the image tag for remote deployment
+                image_to_use = remote_tag_for_image(
+                    image, self.spec.get_image_registry()) if self.spec.get_image_registry() is not None else image
                 volume_mounts = volume_mounts_for_service(self.parsed_pod_yaml_map, service_name)
                 container = client.V1Container(
                     name=container_name,
-                    image=image,
+                    image=image_to_use,
                     env=envs_from_environment_variables_map(self.environment_variables.map),
-                    ports=[client.V1ContainerPort(container_port=80)],
+                    ports=[client.V1ContainerPort(container_port=port)],
                     volume_mounts=volume_mounts,
                     resources=client.V1ResourceRequirements(
                         requests={"cpu": "100m", "memory": "200Mi"},
@@ -125,7 +211,7 @@ class ClusterInfo:
         deployment = client.V1Deployment(
             api_version="apps/v1",
             kind="Deployment",
-            metadata=client.V1ObjectMeta(name=self.deployment_name),
+            metadata=client.V1ObjectMeta(name=f"{self.app_name}-deployment"),
             spec=spec,
         )
         return deployment

stack_orchestrator/deploy/k8s/deploy_k8s.py

@@ -16,44 +16,78 @@
 from pathlib import Path
 from kubernetes import client, config
 
+from stack_orchestrator import constants
 from stack_orchestrator.deploy.deployer import Deployer, DeployerConfigGenerator
 from stack_orchestrator.deploy.k8s.helpers import create_cluster, destroy_cluster, load_images_into_kind
 from stack_orchestrator.deploy.k8s.helpers import pods_in_deployment, log_stream_from_string, generate_kind_config
 from stack_orchestrator.deploy.k8s.cluster_info import ClusterInfo
 from stack_orchestrator.opts import opts
+from stack_orchestrator.deploy.deployment_context import DeploymentContext
+from stack_orchestrator.util import error_exit
+
+
+class AttrDict(dict):
+    def __init__(self, *args, **kwargs):
+        super(AttrDict, self).__init__(*args, **kwargs)
+        self.__dict__ = self
+
+
+def _check_delete_exception(e: client.exceptions.ApiException):
+    if e.status == 404:
+        if opts.o.debug:
+            print("Failed to delete object, continuing")
+    else:
+        error_exit(f"k8s api error: {e}")
 
 
 class K8sDeployer(Deployer):
     name: str = "k8s"
+    type: str
     core_api: client.CoreV1Api
     apps_api: client.AppsV1Api
+    networking_api: client.NetworkingV1Api
     k8s_namespace: str = "default"
     kind_cluster_name: str
-    cluster_info : ClusterInfo
+    cluster_info: ClusterInfo
     deployment_dir: Path
+    deployment_context: DeploymentContext
 
-    def __init__(self, deployment_dir, compose_files, compose_project_name, compose_env_file) -> None:
+    def __init__(self, type, deployment_context: DeploymentContext, compose_files, compose_project_name, compose_env_file) -> None:
+        self.type = type
+        # TODO: workaround pending refactoring above to cope with being created with a null deployment_context
+        if deployment_context is None:
+            return
+        self.deployment_dir = deployment_context.deployment_dir
+        self.deployment_context = deployment_context
+        self.kind_cluster_name = compose_project_name
+        self.cluster_info = ClusterInfo()
+        self.cluster_info.int(compose_files, compose_env_file, compose_project_name, deployment_context.spec)
         if (opts.o.debug):
-            print(f"Deployment dir: {deployment_dir}")
+            print(f"Deployment dir: {deployment_context.deployment_dir}")
             print(f"Compose files: {compose_files}")
             print(f"Project name: {compose_project_name}")
             print(f"Env file: {compose_env_file}")
-        self.deployment_dir = deployment_dir
+            print(f"Type: {type}")
-        self.kind_cluster_name = compose_project_name
-        self.cluster_info = ClusterInfo()
-        self.cluster_info.int(compose_files, compose_env_file)
 
     def connect_api(self):
+        if self.is_kind():
             config.load_kube_config(context=f"kind-{self.kind_cluster_name}")
+        else:
+            # Get the config file and pass to load_kube_config()
+            config.load_kube_config(config_file=self.deployment_dir.joinpath(constants.kube_config_filename).as_posix())
         self.core_api = client.CoreV1Api()
+        self.networking_api = client.NetworkingV1Api()
         self.apps_api = client.AppsV1Api()
+        self.custom_obj_api = client.CustomObjectsApi()
 
     def up(self, detach, services):
+
+        if self.is_kind():
             # Create the kind cluster
-        create_cluster(self.kind_cluster_name, self.deployment_dir.joinpath("kind-config.yml"))
+            create_cluster(self.kind_cluster_name, self.deployment_dir.joinpath(constants.kind_config_filename))
-        self.connect_api()
             # Ensure the referenced containers are copied into kind
             load_images_into_kind(self.kind_cluster_name, self.cluster_info.image_set)
+        self.connect_api()
 
         # Create the host-path-mounted PVs for this deployment
         pvs = self.cluster_info.get_pvs()
@@ -87,20 +121,170 @@ class K8sDeployer(Deployer):
             print(f"{deployment_resp.metadata.namespace} {deployment_resp.metadata.name} \
                   {deployment_resp.metadata.generation} {deployment_resp.spec.template.spec.containers[0].image}")
 
+        service: client.V1Service = self.cluster_info.get_service()
+        service_resp = self.core_api.create_namespaced_service(
+            namespace=self.k8s_namespace,
+            body=service
+        )
+        if opts.o.debug:
+            print("Service created:")
+            print(f"{service_resp}")
+
+        # TODO: disable ingress for kind
+        ingress: client.V1Ingress = self.cluster_info.get_ingress()
+
+        if opts.o.debug:
+            print(f"Sending this ingress: {ingress}")
+        ingress_resp = self.networking_api.create_namespaced_ingress(
+            namespace=self.k8s_namespace,
+            body=ingress
+        )
+        if opts.o.debug:
+            print("Ingress created:")
+            print(f"{ingress_resp}")
+
     def down(self, timeout, volumes):
+        self.connect_api()
         # Delete the k8s objects
+        # Create the host-path-mounted PVs for this deployment
+        pvs = self.cluster_info.get_pvs()
+        for pv in pvs:
+            if opts.o.debug:
+                print(f"Deleting this pv: {pv}")
+            try:
+                pv_resp = self.core_api.delete_persistent_volume(name=pv.metadata.name)
+                if opts.o.debug:
+                    print("PV deleted:")
+                    print(f"{pv_resp}")
+            except client.exceptions.ApiException as e:
+                _check_delete_exception(e)
+
+        # Figure out the PVCs for this deployment
+        pvcs = self.cluster_info.get_pvcs()
+        for pvc in pvcs:
+            if opts.o.debug:
+                print(f"Deleting this pvc: {pvc}")
+            try:
+                pvc_resp = self.core_api.delete_namespaced_persistent_volume_claim(
+                    name=pvc.metadata.name, namespace=self.k8s_namespace
+                )
+                if opts.o.debug:
+                    print("PVCs deleted:")
+                    print(f"{pvc_resp}")
+            except client.exceptions.ApiException as e:
+                _check_delete_exception(e)
+        deployment = self.cluster_info.get_deployment()
+        if opts.o.debug:
+            print(f"Deleting this deployment: {deployment}")
+        try:
+            self.apps_api.delete_namespaced_deployment(
+                name=deployment.metadata.name, namespace=self.k8s_namespace
+            )
+        except client.exceptions.ApiException as e:
+            _check_delete_exception(e)
+
+        service: client.V1Service = self.cluster_info.get_service()
+        if opts.o.debug:
+            print(f"Deleting service: {service}")
+        try:
+            self.core_api.delete_namespaced_service(
+                namespace=self.k8s_namespace,
+                name=service.metadata.name
+            )
+        except client.exceptions.ApiException as e:
+            _check_delete_exception(e)
+
+        # TODO: disable ingress for kind
+        ingress: client.V1Ingress = self.cluster_info.get_ingress()
+        if opts.o.debug:
+            print(f"Deleting this ingress: {ingress}")
+        try:
+            self.networking_api.delete_namespaced_ingress(
+                name=ingress.metadata.name, namespace=self.k8s_namespace
+            )
+        except client.exceptions.ApiException as e:
+            _check_delete_exception(e)
+
+        if self.is_kind():
             # Destroy the kind cluster
             destroy_cluster(self.kind_cluster_name)
 
-    def ps(self):
+    def status(self):
         self.connect_api()
         # Call whatever API we need to get the running container list
-        ret = self.core_api.list_pod_for_all_namespaces(watch=False)
+        all_pods = self.core_api.list_pod_for_all_namespaces(watch=False)
-        if ret.items:
+        pods = []
-            for i in ret.items:
+
-                print("%s\t%s\t%s" % (i.status.pod_ip, i.metadata.namespace, i.metadata.name))
+        if all_pods.items:
-        ret = self.core_api.list_node(pretty=True, watch=False)
+            for p in all_pods.items:
-        return []
+                if self.cluster_info.app_name in p.metadata.name:
+                    pods.append(p)
+
+        if not pods:
+            return
+
+        hostname = "?"
+        ip = "?"
+        tls = "?"
+        try:
+            ingress = self.networking_api.read_namespaced_ingress(namespace=self.k8s_namespace,
+                                                                  name=self.cluster_info.get_ingress().metadata.name)
+
+            cert = self.custom_obj_api.get_namespaced_custom_object(
+                group="cert-manager.io",
+                version="v1",
+                namespace=self.k8s_namespace,
+                plural="certificates",
+                name=ingress.spec.tls[0].secret_name
+            )
+
+            hostname = ingress.spec.tls[0].hosts[0]
+            ip = ingress.status.load_balancer.ingress[0].ip
+            tls = "notBefore: %s, notAfter: %s" % (cert["status"]["notBefore"], cert["status"]["notAfter"])
+        except:  # noqa: E722
+            pass
+
+        print("Ingress:")
+        print("\tHostname:", hostname)
+        print("\tIP:", ip)
+        print("\tTLS:", tls)
+        print("")
+        print("Pods:")
+
+        for p in pods:
+            if p.metadata.deletion_timestamp:
+                print(f"\t{p.metadata.namespace}/{p.metadata.name}: Terminating ({p.metadata.deletion_timestamp})")
+            else:
+                print(f"\t{p.metadata.namespace}/{p.metadata.name}: Running ({p.metadata.creation_timestamp})")
+
+    def ps(self):
+        self.connect_api()
+        pods = self.core_api.list_pod_for_all_namespaces(watch=False)
+
+        ret = []
+
+        for p in pods.items:
+            if self.cluster_info.app_name in p.metadata.name:
+                pod_ip = p.status.pod_ip
+                ports = AttrDict()
+                for c in p.spec.containers:
+                    if c.ports:
+                        for prt in c.ports:
+                            ports[str(prt.container_port)] = [AttrDict({
+                                "HostIp": pod_ip,
+                                "HostPort": prt.container_port
+                            })]
+
+                ret.append(AttrDict({
+                    "id": f"{p.metadata.namespace}/{p.metadata.name}",
+                    "name": p.metadata.name,
+                    "namespace": p.metadata.namespace,
+                    "network_settings": AttrDict({
+                        "ports": ports
+                    })
+                }))
+
+        return ret
 
     def port(self, service, private_port):
         # Since we handle the port mapping, need to figure out where this comes from
@@ -120,24 +304,30 @@ class K8sDeployer(Deployer):
         log_data = self.core_api.read_namespaced_pod_log(k8s_pod_name, namespace="default", container="test")
         return log_stream_from_string(log_data)
 
-    def run(self, image, command, user, volumes, entrypoint=None):
+    def run(self, image: str, command=None, user=None, volumes=None, entrypoint=None, env={}, ports=[], detach=False):
         # We need to figure out how to do this -- check why we're being called first
         pass
 
+    def is_kind(self):
+        return self.type == "k8s-kind"
+
 
 class K8sDeployerConfigGenerator(DeployerConfigGenerator):
-    config_file_name: str = "kind-config.yml"
+    type: str
 
-    def __init__(self) -> None:
+    def __init__(self, type: str) -> None:
+        self.type = type
         super().__init__()
 
     def generate(self, deployment_dir: Path):
+        # No need to do this for the remote k8s case
+        if self.type == "k8s-kind":
             # Check the file isn't already there
             # Get the config file contents
             content = generate_kind_config(deployment_dir)
             if opts.o.debug:
                 print(f"kind config is: {content}")
-        config_file = deployment_dir.joinpath(self.config_file_name)
+            config_file = deployment_dir.joinpath(constants.kind_config_filename)
             # Write the file
             with open(config_file, "w") as output_file:
                 output_file.write(content)

stack_orchestrator/deploy/k8s/helpers.py

@@ -14,14 +14,13 @@
 # along with this program.  If not, see <http:#www.gnu.org/licenses/>.
 
 from kubernetes import client
-from dotenv import dotenv_values
 import os
 from pathlib import Path
 import subprocess
-from typing import Any, Set, Mapping, List
+from typing import Set, Mapping, List
 
 from stack_orchestrator.opts import opts
-from stack_orchestrator.util import get_yaml
+from stack_orchestrator.deploy.deploy_util import parsed_pod_files_map_from_file_names
 
 
 def _run_command(command: str):
@@ -133,17 +132,6 @@ def _make_absolute_host_path(data_mount_path: Path, deployment_dir: Path) -> Pat
         return Path.cwd().joinpath(deployment_dir.joinpath("compose").joinpath(data_mount_path)).resolve()
 
 
-def parsed_pod_files_map_from_file_names(pod_files):
-    parsed_pod_yaml_map : Any = {}
-    for pod_file in pod_files:
-        with open(pod_file, "r") as pod_file_descriptor:
-            parsed_pod_file = get_yaml().load(pod_file_descriptor)
-            parsed_pod_yaml_map[pod_file] = parsed_pod_file
-    if opts.o.debug:
-        print(f"parsed_pod_yaml_map: {parsed_pod_yaml_map}")
-    return parsed_pod_yaml_map
-
-
 def _generate_kind_mounts(parsed_pod_files, deployment_dir):
     volume_definitions = []
     volume_host_path_map = _get_host_paths_for_volumes(parsed_pod_files)
@@ -235,7 +223,3 @@ def generate_kind_config(deployment_dir: Path):
         f"{port_mappings_yml}\n"
         f"{mounts_yml}\n"
     )
-
-
-def env_var_map_from_file(file: Path) -> Mapping[str, str]:
-    return dotenv_values(file)

stack_orchestrator/deploy/spec.py

@@ -16,6 +16,7 @@
 from pathlib import Path
 import typing
 from stack_orchestrator.util import get_yaml
+from stack_orchestrator import constants
 
 
 class Spec:
@@ -28,3 +29,14 @@ class Spec:
     def init_from_file(self, file_path: Path):
         with file_path:
             self.obj = get_yaml().load(open(file_path, "r"))
+
+    def get_image_registry(self):
+        return (self.obj[constants.image_resigtry_key]
+                if self.obj and constants.image_resigtry_key in self.obj
+                else None)
+
+    def get_http_proxy(self):
+        return (self.obj[constants.network_key][constants.http_proxy_key]
+                if self.obj and constants.network_key in self.obj
+                and constants.http_proxy_key in self.obj[constants.network_key]
+                else None)

stack_orchestrator/deploy/webapp/deploy_webapp.py

@@ -0,0 +1,118 @@
+# Copyright ©2023 Vulcanize
+
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http:#www.gnu.org/licenses/>.
+
+import click
+import os
+from pathlib import Path
+from urllib.parse import urlparse
+from tempfile import NamedTemporaryFile
+
+from stack_orchestrator.util import error_exit, global_options2
+from stack_orchestrator.deploy.deployment_create import init_operation, create_operation
+from stack_orchestrator.deploy.deploy import create_deploy_context
+from stack_orchestrator.deploy.deploy_types import DeployCommandContext
+
+
+def _fixup_container_tag(deployment_dir: str, image: str):
+    deployment_dir_path = Path(deployment_dir)
+    compose_file = deployment_dir_path.joinpath("compose", "docker-compose-webapp-template.yml")
+    # replace "cerc/webapp-container:local" in the file with our image tag
+    with open(compose_file) as rfile:
+        contents = rfile.read()
+        contents = contents.replace("cerc/webapp-container:local", image)
+    with open(compose_file, "w") as wfile:
+        wfile.write(contents)
+
+
+def _fixup_url_spec(spec_file_name: str, url: str):
+    # url is like: https://example.com/path
+    parsed_url = urlparse(url)
+    http_proxy_spec = f'''
+  http-proxy:
+    - host-name: {parsed_url.hostname}
+      routes:
+        - path: '{parsed_url.path if parsed_url.path else "/"}'
+          proxy-to: webapp:3000
+    '''
+    spec_file_path = Path(spec_file_name)
+    with open(spec_file_path) as rfile:
+        contents = rfile.read()
+        contents = contents + http_proxy_spec
+    with open(spec_file_path, "w") as wfile:
+        wfile.write(contents)
+
+
+@click.group()
+@click.pass_context
+def command(ctx):
+    '''manage a webapp deployment'''
+
+    # Check that --stack wasn't supplied
+    if ctx.parent.obj.stack:
+        error_exit("--stack can't be supplied with the deploy-webapp command")
+
+
+@command.command()
+@click.option("--kube-config", help="Provide a config file for a k8s deployment")
+@click.option("--image-registry", help="Provide a container image registry url for this k8s cluster")
+@click.option("--deployment-dir", help="Create deployment files in this directory", required=True)
+@click.option("--image", help="image to deploy", required=True)
+@click.option("--url", help="url to serve", required=True)
+@click.option("--env-file", help="environment file for webapp")
+@click.pass_context
+def create(ctx, deployment_dir, image, url, kube_config, image_registry, env_file):
+    '''create a deployment for the specified webapp container'''
+    # Do the equivalent of:
+    # 1. laconic-so --stack webapp-template deploy --deploy-to k8s init --output webapp-spec.yml
+    #   --config (eqivalent of the contents of my-config.env)
+    # 2. laconic-so  --stack webapp-template deploy --deploy-to k8s create --deployment-dir test-deployment
+    #   --spec-file webapp-spec.yml
+    # 3. Replace the container image tag with the specified image
+    deployment_dir_path = Path(deployment_dir)
+    # Check the deployment dir does not exist
+    if deployment_dir_path.exists():
+        error_exit(f"Deployment dir {deployment_dir} already exists")
+    # Generate a temporary file name for the spec file
+    tf = NamedTemporaryFile(prefix="webapp-", suffix=".yml", delete=False)
+    spec_file_name = tf.name
+    # Specify the webapp template stack
+    stack = "webapp-template"
+    # TODO: support env file
+    deploy_command_context: DeployCommandContext = create_deploy_context(
+        global_options2(ctx), None, stack, None, None, None, env_file, "k8s"
+        )
+    init_operation(
+        deploy_command_context,
+        stack,
+        "k8s",
+        None,
+        env_file,
+        kube_config,
+        image_registry,
+        spec_file_name,
+        None
+    )
+    # Add the TLS and DNS spec
+    _fixup_url_spec(spec_file_name, url)
+    create_operation(
+        deploy_command_context,
+        spec_file_name,
+        deployment_dir,
+        None,
+        None
+    )
+    # Fix up the container tag inside the deployment compose file
+    _fixup_container_tag(deployment_dir, image)
+    os.remove(spec_file_name)

stack_orchestrator/deploy/webapp/run_webapp.py

@@ -0,0 +1,65 @@
+# Copyright © 2022, 2023 Vulcanize
+
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http:#www.gnu.org/licenses/>.
+
+# Builds webapp containers
+
+# env vars:
+# CERC_REPO_BASE_DIR defaults to ~/cerc
+
+# TODO: display the available list of containers; allow re-build of either all or specific containers
+
+import hashlib
+import click
+from dotenv import dotenv_values
+
+from stack_orchestrator import constants
+from stack_orchestrator.deploy.deployer_factory import getDeployer
+
+WEBAPP_PORT = 3000
+
+
+@click.command()
+@click.option("--image", help="image to deploy", required=True)
+@click.option("--env-file", help="environment file for webapp")
+@click.option("--port", help="port to use (default random)")
+@click.pass_context
+def command(ctx, image, env_file, port):
+    '''run the specified webapp container'''
+
+    env = {}
+    if env_file:
+        env = dotenv_values(env_file)
+
+    unique_cluster_descriptor = f"{image},{env}"
+    hash = hashlib.md5(unique_cluster_descriptor.encode()).hexdigest()
+    cluster = f"laconic-webapp-{hash}"
+
+    deployer = getDeployer(type=constants.compose_deploy_type,
+                           deployment_context=None,
+                           compose_files=None,
+                           compose_project_name=cluster,
+                           compose_env_file=None)
+
+    ports = []
+    if port:
+        ports = [(port, WEBAPP_PORT)]
+    container = deployer.run(image, command=[], user=None, volumes=[], entrypoint=None, env=env, ports=ports, detach=True)
+
+    # Make configurable?
+    webappPort = f"{WEBAPP_PORT}/tcp"
+    # TODO: This assumes a Docker container object...
+    if webappPort in container.network_settings.ports:
+        mapping = container.network_settings.ports[webappPort][0]
+        print(f"""Image: {image}\nID: {container.id}\nURL: http://localhost:{mapping['HostPort']}""")

stack_orchestrator/main.py

@@ -20,6 +20,7 @@ from stack_orchestrator.repos import setup_repositories
 from stack_orchestrator.build import build_containers
 from stack_orchestrator.build import build_npms
 from stack_orchestrator.build import build_webapp
+from stack_orchestrator.deploy.webapp import run_webapp, deploy_webapp
 from stack_orchestrator.deploy import deploy
 from stack_orchestrator import version
 from stack_orchestrator.deploy import deployment
@@ -50,6 +51,8 @@ cli.add_command(setup_repositories.command, "setup-repositories")
 cli.add_command(build_containers.command, "build-containers")
 cli.add_command(build_npms.command, "build-npms")
 cli.add_command(build_webapp.command, "build-webapp")
+cli.add_command(run_webapp.command, "run-webapp")
+cli.add_command(deploy_webapp.command, "deploy-webapp")
 cli.add_command(deploy.command, "deploy")  # deploy is an alias for deploy-system
 cli.add_command(deploy.command, "deploy-system")
 cli.add_command(deployment.command, "deployment")

stack_orchestrator/repos/setup_repositories.py

@@ -25,7 +25,8 @@ import click
 import importlib.resources
 from pathlib import Path
 import yaml
-from stack_orchestrator.util import include_exclude_check
+from stack_orchestrator.constants import stack_file_name
+from stack_orchestrator.util import include_exclude_check, stack_is_external, error_exit
 
 
 class GitProgress(git.RemoteProgress):
@@ -238,13 +239,20 @@ def command(ctx, include, exclude, git_ssh, check_only, pull, branches, branches
 
     repos_in_scope = []
     if stack:
+        if stack_is_external(stack):
+            stack_file_path = Path(stack).joinpath(stack_file_name)
+        else:
             # In order to be compatible with Python 3.8 we need to use this hack to get the path:
             # See: https://stackoverflow.com/questions/25389095/python-get-path-of-root-project-structure
-        stack_file_path = Path(__file__).absolute().parent.parent.joinpath("data", "stacks", stack, "stack.yml")
+            stack_file_path = Path(__file__).absolute().parent.parent.joinpath("data", "stacks", stack, stack_file_name)
+        if not stack_file_path.exists():
+            error_exit(f"stack {stack} does not exist")
         with stack_file_path:
             stack_config = yaml.safe_load(open(stack_file_path, "r"))
-            # TODO: syntax check the input here
+            if "repos" not in stack_config:
-            repos_in_scope = stack_config['repos']
+                error_exit(f"stack {stack} does not define any repositories")
+            else:
+                repos_in_scope = stack_config["repos"]
     else:
         repos_in_scope = all_repos
 

stack_orchestrator/util.py

@@ -18,6 +18,8 @@ import os.path
 import sys
 import ruamel.yaml
 from pathlib import Path
+from dotenv import dotenv_values
+from typing import Mapping
 
 
 def include_exclude_check(s, include, exclude):
@@ -150,6 +152,12 @@ def get_parsed_deployment_spec(spec_file):
         sys.exit(1)
 
 
+def stack_is_external(stack: str):
+    # Bit of a hack: if the supplied stack string represents
+    # a path that exists then we assume it must be external
+    return Path(stack).exists() if stack is not None else False
+
+
 def get_yaml():
     # See: https://stackoverflow.com/a/45701840/1701505
     yaml = ruamel.yaml.YAML()
@@ -167,3 +175,12 @@ def global_options(ctx):
 # TODO: hack
 def global_options2(ctx):
     return ctx.parent.obj
+
+
+def error_exit(s):
+    print(f"ERROR: {s}")
+    sys.exit(1)
+
+
+def env_var_map_from_file(file: Path) -> Mapping[str, str]:
+    return dotenv_values(file)

tests/k8s-deploy/run-deploy-test.sh

@@ -21,7 +21,7 @@ mkdir -p $CERC_REPO_BASE_DIR
 # Test basic stack-orchestrator deploy
 test_deployment_dir=$CERC_REPO_BASE_DIR/test-deployment-dir
 test_deployment_spec=$CERC_REPO_BASE_DIR/test-deployment-spec.yml
-$TEST_TARGET_SO --stack test deploy --deploy-to k8s init --output $test_deployment_spec --config CERC_TEST_PARAM_1=PASSED
+$TEST_TARGET_SO --stack test deploy --deploy-to k8s-kind init --output $test_deployment_spec --config CERC_TEST_PARAM_1=PASSED
 # Check the file now exists
 if [ ! -f "$test_deployment_spec" ]; then
     echo "deploy init test: spec file not present"

tests/webapp-test/run-webapp-test.sh

@@ -1,8 +1,10 @@
 #!/usr/bin/env bash
 set -e
+
 if [ -n "$CERC_SCRIPT_DEBUG" ]; then
   set -x
 fi
+
 # Dump environment variables for debugging
 echo "Environment variables:"
 env
@@ -24,26 +26,28 @@ git clone https://git.vdb.to/cerc-io/test-progressive-web-app.git $CERC_REPO_BAS
 # Test webapp command execution
 $TEST_TARGET_SO build-webapp --source-repo $CERC_REPO_BASE_DIR/test-progressive-web-app
 
-UUID=`uuidgen`
+CHECK="SPECIAL_01234567890_TEST_STRING"
 
 set +e
 
-CONTAINER_ID=$(docker run -p 3000:3000 -d cerc/test-progressive-web-app:local)
+CONTAINER_ID=$(docker run -p 3000:3000 -d -e CERC_SCRIPT_DEBUG=$CERC_SCRIPT_DEBUG cerc/test-progressive-web-app:local)
 sleep 3
-wget -O test.before -m http://localhost:3000
+wget -t 7 -O test.before -m http://localhost:3000
 
+docker logs $CONTAINER_ID
 docker remove -f $CONTAINER_ID
 
-CONTAINER_ID=$(docker run -p 3000:3000 -e CERC_WEBAPP_DEBUG=$UUID -d cerc/test-progressive-web-app:local)
+CONTAINER_ID=$(docker run -p 3000:3000 -e CERC_WEBAPP_DEBUG=$CHECK -e CERC_SCRIPT_DEBUG=$CERC_SCRIPT_DEBUG -d cerc/test-progressive-web-app:local)
 sleep 3
-wget -O test.after -m http://localhost:3000
+wget -t 7 -O test.after -m http://localhost:3000
 
+docker logs $CONTAINER_ID
 docker remove -f $CONTAINER_ID
 
 echo "###########################################################################"
 echo ""
 
-grep "$UUID" test.before > /dev/null
+grep "$CHECK" test.before > /dev/null
 if [ $? -ne 1 ]; then
   echo "BEFORE: FAILED"
   exit 1
@@ -51,7 +55,7 @@ else
   echo "BEFORE: PASSED"
 fi
 
-grep "$UUID" test.after > /dev/null
+grep "$CHECK" test.after > /dev/null
 if [ $? -ne 0 ]; then
   echo "AFTER: FAILED"
   exit 1