WIP

`laconic cns` got renamed to `laconic registry` which breaks all the scripts and commands that use it.

Reviewed-on: cerc-io/stack-orchestrator#784
Co-authored-by: Thomas E Lackey <telackey@bozemanpass.com>
Co-committed-by: Thomas E Lackey <telackey@bozemanpass.com>

docs/deploy-webapps-via-registry.md

@@ -0,0 +1,271 @@
+# Deploying WebApps from the Laconic Registry to Kubernetes
+
+First, ensure you have `laconicd` and the console running via [this tutorial](./laconicd-with-console.md).
+
+## Setup Kubernetes
+
+If merely requesting deployment, access to the Laconic registry is sufficient, but to *process* deployment requests, Kubernetes configuration is also necessary. The configuration format is the same as used by `kubectl`, and should be places in `/etc/config/kube.yml`
+
+## Setup the Deployer
+
+### Set envs
+
+
+```
+$DEPLOYMENT_DNS_SUFFIX
+$DEPLOYMENT_RECORD_NAMESPACE
+$IMAGE_REGISTRY
+$IMAGE_REGISTRY_CREDS
+$IMAGE_REGISTRY_USER
+
+```
+
+
+```
+laconic-so --stack webapp-deployer-backend setup-repositories
+laconic-so --stack webapp-deployer-backend build-containers
+laconic-so --stack webapp-deployer-backend deploy up
+```
+
+## Setup Your Account
+
+Writing to the Laconic Registry, in order to create records for applications and deployment requests, requires a user key, and a bond with adequate funds available.
+
+### Install the registry CLI
+
+```
+npm config set @cerc-io:registry https://git.vdb.to/api/packages/cerc-io/npm/
+npm install -g @cerc-io/laconic-registry-cli
+```
+
+Alternatively, use the pre-configured container that was setup alongside `laconicd`
+
+### Get your private key
+
+```
+exec into laconicd
+```
+
+### config.yml
+
+```
+services:
+  cns:
+    restEndpoint: http://console.laconic.com:1317
+    gqlEndpoint: http://console.laconic.com:9473/api
+    userKey: 3d4789e88508c6230d973ccea283032d3e3948775dbe02f4f0a80dc6c1f7c8d5
+   # bondId: 3c61577f3c6197ced599e2fc21ccf76f43373004fd29c29f2e560c77f7c4bb6d
+    chainId: laconic_9000-1
+    gas: 550000
+    fees: 200000aphoton
+```
+
+### Create a bond
+
+```
+laconic cns -c config.yml bond create --type aphoton --quantity 1000000000 --gas 200000 --fees 200000aphoton
+```
+
+
+
+
+
+# Creating Deployments
+
+Deploying an application from the registry requires (and generates)
+several records:
+
+1. An `ApplicationRecord` which describes the application itself, including its name, version, and repository.
+2. An `ApplicationDeploymentRequest` which references an`ApplicationRecord`, and provides additional information such as the DNS name and configuration.
+3. A `DnsRecord`, which contains the FQDN of the deployment. The ownership of this record will be checked against any future deployment or removal requests.
+4. An `ApplicationDeploymentRecord` which records the successful processing of the deployment request.
+
+Additionally, since names need to be registered, namespace authorities need to be reserved, for example:
+
+```
+$ laconic -c $LACONIC_CONFIG cns authority reserve my-org-name
+
+$ laconic -c $LACONIC_CONFIG cns authority bond set my-org-name 0e9176d854bc3c20528b6361aab632f0b252a0f69717bf035fa68d1ef7647ba7
+```
+
+## Application Records
+
+The `ApplicationRecord` should, at a minimum, specify the name of the application (`name`), its version (`app_version`), type (`app_type`), one or more repository URLs (`repository`), and the repository reference (eg, branch, tag, hash, etc.) to use (`repository_ref`).
+
+```
+$ cat app.yml
+record:
+  type: ApplicationRecord
+  version: 0.0.4
+  name: "@my-org-name/my-test-webapp"
+  repository:
+    - "https://github.com/my-org-name/my-test-webapp"
+  repository_ref: "v0.1.5"
+  app_version: "0.1.5"
+  app_type: "webapp"
+
+$ laconic -c $LACONIC_CONFIG cns record publish -f app.yml
+bafyreihwvu6ynmk4nfrxg2vdcx2ep3tqry775ksqyehjitj2i4kphhyuky
+```
+
+One or more names should be registered for the application, which deployment requests can reference.
+
+```
+$ laconic -c $LACONIC_CONFIG cns name set "crn://my-org-name/applications/my-test-webapp" bafyreihwvu6ynmk4nfrxg2vdcx2ep3tqry775ksqyehjitj2i4kphhyuky
+
+$ laconic -c $LACONIC_CONFIG cns name set "crn://my-org-name/applications/my-test-webapp@0.1.5" bafyreihwvu6ynmk4nfrxg2vdcx2ep3tqry775ksqyehjitj2i4kphhyuky
+
+$ laconic -c $LACONIC_CONFIG cns record get --id bafyreihwvu6ynmk4nfrxg2vdcx2ep3tqry775ksqyehjitj2i4kphhyuky
+[
+  {
+    "id": "bafyreihwvu6ynmk4nfrxg2vdcx2ep3tqry775ksqyehjitj2i4kphhyuky",
+    "names": [
+      "crn://my-org-name/applications/my-test-webapp",
+      "crn://my-org-name/applications/my-test-webapp@0.1.5",
+    ],
+    "owners": [
+      "2671D38525BDC91A5DF4794EF2059D5771133702"
+    ],
+    "bondId": "0e9176d854bc3c20528b6361aab632f0b252a0f69717bf035fa68d1ef7647ba7",
+    "createTime": "2024-01-12T20:30:33Z",
+    "expiryTime": "2025-01-11T20:30:33Z",
+    "attributes": {
+      "app_type": "webapp",
+      "app_version": "0.1.5",
+      "name": "@my-org-name/my-test-webapp",
+      "repository": [
+        "https://github.com/my-org-name/my-test-webapp"
+      ],
+      "repository_ref": "v0.1.5",
+      "type": "ApplicationRecord",
+      "version": "0.0.4"
+    }
+  }
+]
+```
+
+## Application Deployment Requests
+
+To create the deployment of specific application, and `ApplicationDeploymentRequest` must be published. This request must reference the application to be deployed (`application`) and may optionally provide configuration (`config`) and a DNS name (`dns`). If no DNS name is supplied, one will be generated. A supplied DNS is usually just the short hostname, not the FQDN, since the suffix is supplied by the deployer.
+
+```
+$ cat req.yml
+record:
+  type: ApplicationDeploymentRequest
+  version: 1.0.0
+  name: "my-org-name/my-test-webapp@0.1.5"
+  application: "crn://my-org-name/applications/my-test-webapp@0.1.5"
+  dns: "my-test-app"
+  config:
+    env:
+      CERC_WEBAPP_DEBUG: my_debug_value_here
+
+$ laconic -c $LACONIC_CONFIG cns record publish -f req.yml
+bafyreihtpvwjka5ecjmca46y4dip5gb2h25vvmc7t27d7g4zecngjssvky
+```
+
+It is not necessary to assign a name to the request.
+
+## Building and Deploying
+
+Building and deploying will happen automatically for records published the production registry, but in other environments it can be triggered manually.
+
+Laconic `stack-orchestrator` is used build and launch the application. It will clone the repository, build the application, upload the container to a registry, and launch the instance in Kubernetes, with
+automatic DNS and TLS provisioning.
+
+```
+$ laconic-so deploy-webapp-from-registry \
+    --kube-config $KUBE_CONFIG \
+    --image-registry registry.mydeployer.org/app-registry \
+    --deployment-parent-dir /opt/deployments \
+    --laconic-config $LACONIC_CONFIG \
+    --dns-suffix mydeployer.servesthe.world \
+    --record-namespace-dns crn://my-deployer-org/dns \
+    --record-namespace-deployments crn://my-deployer-org/deployments \
+    --request-id bafyreihtpvwjka5ecjmca46y4dip5gb2h25vvmc7t27d7g4zecngjssvky
+```
+
+When deployment is complete, an `ApplicationDeploymentRecord` will be created:
+
+```
+$ laconic -c $LACONIC_CONFIG cns name resolve crn://my-deployer-org/deployments/my-test-app.mydeployer.servesthe.world
+[
+  {
+    "id": "bafyreiemfmxsue4svzys6tcwsqmhfjeyoo3gp63n7kydi2izbrwuzd4rga",
+    "names": [
+      "crn://my-deployer-org/deployments/my-test-app.laconic.servesthe.world"
+    ],
+    "owners": [
+      "2671D38525BDC91A5DF4794EF2059D5771133702"
+    ],
+    "bondId": "0e9176d854bc3c20528b6361aab632f0b252a0f69717bf035fa68d1ef7647ba7",
+    "createTime": "2024-01-11T20:45:29Z",
+    "expiryTime": "2025-01-10T20:45:29Z",
+    "attributes": {
+      "application": "bafyreibs6y7jhgjlsoxyqlugvkweanp3bi7ippfmhehk2rslcpoaqic2xi",
+      "dns": "bafyreicpac7tar5ua5e42zo7d5zwyp5yv7iord23p3gdox7zdfslrarjle",
+      "meta": {
+        "config": "da39a3ee5e6b4b0d3255bfef95601890afd80709",
+        "so": "82a5f9a3e4924cbb92e28be68759a487"
+      },
+      "name": "@cerc-io/my-test-webapp",
+      "request": "bafyreiex7vmtruiiasra4wvqe7wirl6iuwiso6p3cvb3sjhq55e74zh4ke",
+      "type": "ApplicationDeploymentRecord",
+      "url": "https://my-test-app.mydeployer.servesthe.world",
+      "version": "0.0.1"
+    }
+  }
+]
+```
+
+## Checking Status
+
+The status of the deployment may be checked with `stack-orchestrator`.
+
+```
+$ laconic-so deployment --dir /opt/deployments/my-test-app.snowball.servesthe.world status
+Ingress:
+        Hostname: my-test-app.mydeployer.servesthe.world
+        IP: 204.130.133.199
+        TLS: notBefore: 2024-01-12T23:11:09Z, notAfter: 2024-04-11T23:11:08Z
+
+Pods:
+        default/laconic-9560ffc64512e453-deployment-fb58d756f-l77pm: Running (2024-01-13 00:10:42+00:00)
+```
+
+# Removing Deployments
+
+As with deployment, removal involves publishing a request, which is then fulfilled by deployment processor.
+
+## Application Deployment Removal Requests
+
+```
+$ cat apprm.yml
+record:
+  type: ApplicationDeploymentRemovalRequest
+  version: 1.0.0
+  deployment: bafyreiemfmxsue4svzys6tcwsqmhfjeyoo3gp63n7kydi2izbrwuzd4rga
+
+$ laconic -c $LACONIC_CONFIG cns record publish -f apprm.yml
+bafyreiejnqhsn5ibc3c6pzlsc26co3mt63djdeuntt3gmfuckcntdmisge
+```
+
+## Removal
+
+As with deployment, for records publish to the production service removal should be processed automatically. In other environments, removal can be processed with `stack-orchestrator`.
+
+```
+$ laconic-so undeploy-webapp-from-registry   \
+    --deployment-parent-dir /opt/deployments   \
+    --laconic-config ~/.laconic/local.yml   \
+    --request-id bafyreiejnqhsn5ibc3c6pzlsc26co3mt63djdeuntt3gmfuckcntdmisge
+
+Request bafyreiejnqhsn5ibc3c6pzlsc26co3mt63djdeuntt3gmfuckcntdmisge needs to processed.
+Found 1 unsatisfied request(s) to process.
+Matched deployment ownership: 2671D38525BDC91A5DF4794EF2059D5771133702
+record:
+  type: ApplicationDeploymentRemovalRecord
+  deployment: bafyreiemfmxsue4svzys6tcwsqmhfjeyoo3gp63n7kydi2izbrwuzd4rga
+  request: bafyreiejnqhsn5ibc3c6pzlsc26co3mt63djdeuntt3gmfuckcntdmisge
+  version: 1.0.0
+```

docs/fetching-containers.md

@@ -0,0 +1,9 @@
+# Fetching pre-built container images
+When Stack Orchestrator deploys a stack containing a suite of one or more containers it expects images for those containers to be on the local machine with a tag of the form `<image-name>:local` Images for these containers can be built from source (and optionally base container images from public registries) with the `build-containers` subcommand. 
+
+However, the task of building a large number of containers from source may consume considerable time and machine resources. This is where the `fetch-containers` subcommand steps in. It is designed to work exactly like `build-containers` but instead the images, pre-built, are fetched from an image registry then re-tagged for deployment. It can be used in place of `build-containers` for any stack provided the necessary containers, built for the local machine architecture (e.g. arm64 or x86-64) have already been published in an image registry.
+## Usage
+To use `fetch-containers`, provide an image registry path, a username and token/password with read access to the registry, and optionally specify `--force-local-overwrite`. If this argument is not specified, if there is already a locally built or previously fetched image for a stack container on the machine, it will not be overwritten and a warning issued.
+```
+$ laconic-so --stack mobymask-v3-demo fetch-containers --image-registry git.vdb.to/cerc-io --registry-username <registry-user> --registry-token <registry-token> --force-local-overwrite
+```

stack_orchestrator/build/build_webapp.py

@@ -27,7 +27,7 @@ from decouple import config
 import click
 from pathlib import Path
 from stack_orchestrator.build import build_containers
-from stack_orchestrator.deploy.webapp.util import determine_base_container
+from stack_orchestrator.deploy.webapp.util import determine_base_container, TimedLogger
 from stack_orchestrator.build.build_types import BuildContext
 
 
@@ -40,9 +40,11 @@ from stack_orchestrator.build.build_types import BuildContext
 @click.pass_context
 def command(ctx, base_container, source_repo, force_rebuild, extra_build_args, tag):
     '''build the specified webapp container'''
+    logger = TimedLogger()
 
     quiet = ctx.obj.quiet
     debug = ctx.obj.debug
+    verbose = ctx.obj.verbose
     local_stack = ctx.obj.local_stack
     stack = ctx.obj.stack
 
@@ -51,12 +53,12 @@ def command(ctx, base_container, source_repo, force_rebuild, extra_build_args, t
 
     if local_stack:
         dev_root_path = os.getcwd()[0:os.getcwd().rindex("stack-orchestrator")]
-        print(f'Local stack dev_root_path (CERC_REPO_BASE_DIR) overridden to: {dev_root_path}')
+        logger.log(f'Local stack dev_root_path (CERC_REPO_BASE_DIR) overridden to: {dev_root_path}')
     else:
         dev_root_path = os.path.expanduser(config("CERC_REPO_BASE_DIR", default="~/cerc"))
 
-    if not quiet:
-        print(f'Dev Root is: {dev_root_path}')
+    if verbose:
+        logger.log(f'Dev Root is: {dev_root_path}')
 
     if not base_container:
         base_container = determine_base_container(source_repo)
@@ -65,6 +67,9 @@ def command(ctx, base_container, source_repo, force_rebuild, extra_build_args, t
     container_build_env = build_containers.make_container_build_env(dev_root_path, container_build_dir, debug,
                                                                     force_rebuild, extra_build_args)
 
+    if verbose:
+        logger.log(f"Building base container: {base_container}")
+
     build_context_1 = BuildContext(
         stack,
         base_container,
@@ -74,9 +79,12 @@ def command(ctx, base_container, source_repo, force_rebuild, extra_build_args, t
     )
     ok = build_containers.process_container(build_context_1)
     if not ok:
-        print("ERROR: Build failed.", file=sys.stderr)
+        logger.log("ERROR: Build failed.")
         sys.exit(1)
 
+    if verbose:
+        logger.log(f"Base container {base_container} build finished.")
+
     # Now build the target webapp.  We use the same build script, but with a different Dockerfile and work dir.
     container_build_env["CERC_WEBAPP_BUILD_RUNNING"] = "true"
     container_build_env["CERC_CONTAINER_BUILD_WORK_DIR"] = os.path.abspath(source_repo)
@@ -85,9 +93,12 @@ def command(ctx, base_container, source_repo, force_rebuild, extra_build_args, t
                                                                           "Dockerfile.webapp")
     if not tag:
         webapp_name = os.path.abspath(source_repo).split(os.path.sep)[-1]
-        container_build_env["CERC_CONTAINER_BUILD_TAG"] = f"cerc/{webapp_name}:local"
-    else:
-        container_build_env["CERC_CONTAINER_BUILD_TAG"] = tag
+        tag = f"cerc/{webapp_name}:local"
+
+    container_build_env["CERC_CONTAINER_BUILD_TAG"] = tag
+
+    if verbose:
+        logger.log(f"Building app container: {tag}")
 
     build_context_2 = BuildContext(
         stack,
@@ -98,5 +109,9 @@ def command(ctx, base_container, source_repo, force_rebuild, extra_build_args, t
     )
     ok = build_containers.process_container(build_context_2)
     if not ok:
-        print("ERROR: Build failed.", file=sys.stderr)
-        sys.exit(1)
+        logger.log("ERROR: Build failed.")
+        sys.exit(1)
+
+    if verbose:
+        logger.log(f"App container {base_container} build finished.")
+        logger.log("build-webapp complete", show_step_time=False, show_total_time=True)

stack_orchestrator/data/compose/docker-compose-watcher-ajna.yml

@@ -0,0 +1,76 @@
+version: '3.2'
+
+services:
+  ajna-watcher-db:
+    restart: unless-stopped
+    image: postgres:14-alpine
+    environment:
+      - POSTGRES_USER=vdbm
+      - POSTGRES_MULTIPLE_DATABASES=ajna-watcher,ajna-watcher-job-queue
+      - POSTGRES_EXTENSION=ajna-watcher-job-queue:pgcrypto
+      - POSTGRES_PASSWORD=password
+    volumes:
+      - ../config/postgresql/multiple-postgressql-databases.sh:/docker-entrypoint-initdb.d/multiple-postgressql-databases.sh
+      - ajna_watcher_db_data:/var/lib/postgresql/data
+    ports:
+      - "5432"
+    healthcheck:
+      test: ["CMD", "nc", "-v", "localhost", "5432"]
+      interval: 20s
+      timeout: 5s
+      retries: 15
+      start_period: 10s
+
+  ajna-watcher-job-runner:
+    restart: unless-stopped
+    depends_on:
+      ajna-watcher-db:
+        condition: service_healthy
+    image: cerc/watcher-ajna:local
+    environment:
+      CERC_SCRIPT_DEBUG: ${CERC_SCRIPT_DEBUG}
+      CERC_ETH_RPC_ENDPOINT: ${CERC_ETH_RPC_ENDPOINT}
+    command: ["bash", "./start-job-runner.sh"]
+    volumes:
+      - ../config/watcher-ajna/watcher-config-template.toml:/app/environments/watcher-config-template.toml
+      - ../config/watcher-ajna/start-job-runner.sh:/app/start-job-runner.sh
+    ports:
+      - "9000"
+    healthcheck:
+      test: ["CMD", "nc", "-v", "localhost", "9000"]
+      interval: 20s
+      timeout: 5s
+      retries: 15
+      start_period: 5s
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
+
+  ajna-watcher-server:
+    restart: unless-stopped
+    depends_on:
+      ajna-watcher-db:
+        condition: service_healthy
+      ajna-watcher-job-runner:
+        condition: service_healthy
+    image: cerc/watcher-ajna:local
+    environment:
+      CERC_SCRIPT_DEBUG: ${CERC_SCRIPT_DEBUG}
+      CERC_ETH_RPC_ENDPOINT: ${CERC_ETH_RPC_ENDPOINT}
+    command: ["bash", "./start-server.sh"]
+    volumes:
+      - ../config/watcher-ajna/watcher-config-template.toml:/app/environments/watcher-config-template.toml
+      - ../config/watcher-ajna/start-server.sh:/app/start-server.sh
+    ports:
+      - "3008"
+      - "9001"
+    healthcheck:
+      test: ["CMD", "nc", "-v", "localhost", "3008"]
+      interval: 20s
+      timeout: 5s
+      retries: 15
+      start_period: 5s
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
+
+volumes:
+  ajna_watcher_db_data:

stack_orchestrator/data/config/monitoring/watcher-alert-rules.yml

@@ -50,22 +50,6 @@ groups:
               legendFormat: __auto
               range: false
               refId: latest_external
-          - refId: latest_indexed
-            relativeTimeRange:
-              from: 600
-              to: 0
-            datasourceUid: PBFA97CFB590B2093
-            model:
-              datasource:
-                type: prometheus
-                uid: PBFA97CFB590B2093
-              editorMode: code
-              expr: sync_status_block_number{job="azimuth", instance="azimuth", kind="latest_indexed"}
-              hide: false
-              instant: true
-              legendFormat: __auto
-              range: false
-              refId: latest_indexed
           - refId: condition
             relativeTimeRange:
               from: 600
@@ -142,22 +126,6 @@ groups:
               legendFormat: __auto
               range: false
               refId: latest_external
-          - refId: latest_indexed
-            relativeTimeRange:
-              from: 600
-              to: 0
-            datasourceUid: PBFA97CFB590B2093
-            model:
-              datasource:
-                type: prometheus
-                uid: PBFA97CFB590B2093
-              editorMode: code
-              expr: sync_status_block_number{job="azimuth", instance="censures", kind="latest_indexed"}
-              hide: false
-              instant: true
-              legendFormat: __auto
-              range: false
-              refId: latest_indexed
           - refId: condition
             relativeTimeRange:
               from: 600
@@ -234,22 +202,6 @@ groups:
               legendFormat: __auto
               range: false
               refId: latest_external
-          - refId: latest_indexed
-            relativeTimeRange:
-              from: 600
-              to: 0
-            datasourceUid: PBFA97CFB590B2093
-            model:
-              datasource:
-                type: prometheus
-                uid: PBFA97CFB590B2093
-              editorMode: code
-              expr: sync_status_block_number{job="azimuth", instance="claims", kind="latest_indexed"}
-              hide: false
-              instant: true
-              legendFormat: __auto
-              range: false
-              refId: latest_indexed
           - refId: condition
             relativeTimeRange:
               from: 600
@@ -326,22 +278,6 @@ groups:
               legendFormat: __auto
               range: false
               refId: latest_external
-          - refId: latest_indexed
-            relativeTimeRange:
-              from: 600
-              to: 0
-            datasourceUid: PBFA97CFB590B2093
-            model:
-              datasource:
-                type: prometheus
-                uid: PBFA97CFB590B2093
-              editorMode: code
-              expr: sync_status_block_number{job="azimuth", instance="conditional_star_release", kind="latest_indexed"}
-              hide: false
-              instant: true
-              legendFormat: __auto
-              range: false
-              refId: latest_indexed
           - refId: condition
             relativeTimeRange:
               from: 600
@@ -418,22 +354,6 @@ groups:
               legendFormat: __auto
               range: false
               refId: latest_external
-          - refId: latest_indexed
-            relativeTimeRange:
-              from: 600
-              to: 0
-            datasourceUid: PBFA97CFB590B2093
-            model:
-              datasource:
-                type: prometheus
-                uid: PBFA97CFB590B2093
-              editorMode: code
-              expr: sync_status_block_number{job="azimuth", instance="delegated_sending", kind="latest_indexed"}
-              hide: false
-              instant: true
-              legendFormat: __auto
-              range: false
-              refId: latest_indexed
           - refId: condition
             relativeTimeRange:
               from: 600
@@ -510,22 +430,6 @@ groups:
               legendFormat: __auto
               range: false
               refId: latest_external
-          - refId: latest_indexed
-            relativeTimeRange:
-              from: 600
-              to: 0
-            datasourceUid: PBFA97CFB590B2093
-            model:
-              datasource:
-                type: prometheus
-                uid: PBFA97CFB590B2093
-              editorMode: code
-              expr: sync_status_block_number{job="azimuth", instance="ecliptic", kind="latest_indexed"}
-              hide: false
-              instant: true
-              legendFormat: __auto
-              range: false
-              refId: latest_indexed
           - refId: condition
             relativeTimeRange:
               from: 600
@@ -602,22 +506,6 @@ groups:
               legendFormat: __auto
               range: false
               refId: latest_external
-          - refId: latest_indexed
-            relativeTimeRange:
-              from: 600
-              to: 0
-            datasourceUid: PBFA97CFB590B2093
-            model:
-              datasource:
-                type: prometheus
-                uid: PBFA97CFB590B2093
-              editorMode: code
-              expr: sync_status_block_number{job="azimuth", instance="azimuth", kind="latest_indexed"}
-              hide: false
-              instant: true
-              legendFormat: __auto
-              range: false
-              refId: latest_indexed
           - refId: condition
             relativeTimeRange:
               from: 600
@@ -694,22 +582,6 @@ groups:
               legendFormat: __auto
               range: false
               refId: latest_external
-          - refId: latest_indexed
-            relativeTimeRange:
-              from: 600
-              to: 0
-            datasourceUid: PBFA97CFB590B2093
-            model:
-              datasource:
-                type: prometheus
-                uid: PBFA97CFB590B2093
-              editorMode: code
-              expr: sync_status_block_number{job="azimuth", instance="polls", kind="latest_indexed"}
-              hide: false
-              instant: true
-              legendFormat: __auto
-              range: false
-              refId: latest_indexed
           - refId: condition
             relativeTimeRange:
               from: 600
@@ -788,22 +660,6 @@ groups:
               legendFormat: __auto
               range: false
               refId: latest_external
-          - refId: latest_indexed
-            relativeTimeRange:
-              from: 600
-              to: 0
-            datasourceUid: PBFA97CFB590B2093
-            model:
-              datasource:
-                type: prometheus
-                uid: PBFA97CFB590B2093
-              editorMode: code
-              expr: sync_status_block_number{job="sushi", instance="sushiswap", kind="latest_indexed"}
-              hide: false
-              instant: true
-              legendFormat: __auto
-              range: false
-              refId: latest_indexed
           - refId: condition
             relativeTimeRange:
               from: 600
@@ -880,22 +736,6 @@ groups:
               legendFormat: __auto
               range: false
               refId: latest_external
-          - refId: latest_indexed
-            relativeTimeRange:
-              from: 600
-              to: 0
-            datasourceUid: PBFA97CFB590B2093
-            model:
-              datasource:
-                type: prometheus
-                uid: PBFA97CFB590B2093
-              editorMode: code
-              expr: sync_status_block_number{job="sushi", instance="merkl_sushiswap", kind="latest_indexed"}
-              hide: false
-              instant: true
-              legendFormat: __auto
-              range: false
-              refId: latest_indexed
           - refId: condition
             relativeTimeRange:
               from: 600

stack_orchestrator/data/config/watcher-ajna/start-job-runner.sh

@@ -0,0 +1,20 @@
+#!/bin/sh
+
+set -e
+if [ -n "$CERC_SCRIPT_DEBUG" ]; then
+  set -x
+fi
+set -u
+
+echo "Using ETH RPC endpoint ${CERC_ETH_RPC_ENDPOINT}"
+
+# Read in the config template TOML file and modify it
+WATCHER_CONFIG_TEMPLATE=$(cat environments/watcher-config-template.toml)
+WATCHER_CONFIG=$(echo "$WATCHER_CONFIG_TEMPLATE" | \
+  sed -E "s|REPLACE_WITH_CERC_ETH_RPC_ENDPOINT|${CERC_ETH_RPC_ENDPOINT}| ")
+
+# Write the modified content to a new file
+echo "$WATCHER_CONFIG" > environments/local.toml
+
+echo "Running job-runner..."
+DEBUG=vulcanize:* exec node --enable-source-maps dist/job-runner.js

stack_orchestrator/data/config/watcher-ajna/start-server.sh

@@ -0,0 +1,20 @@
+#!/bin/sh
+
+set -e
+if [ -n "$CERC_SCRIPT_DEBUG" ]; then
+  set -x
+fi
+set -u
+
+echo "Using ETH RPC endpoint ${CERC_ETH_RPC_ENDPOINT}"
+
+# Read in the config template TOML file and modify it
+WATCHER_CONFIG_TEMPLATE=$(cat environments/watcher-config-template.toml)
+WATCHER_CONFIG=$(echo "$WATCHER_CONFIG_TEMPLATE" | \
+  sed -E "s|REPLACE_WITH_CERC_ETH_RPC_ENDPOINT|${CERC_ETH_RPC_ENDPOINT}| ")
+
+# Write the modified content to a new file
+echo "$WATCHER_CONFIG" > environments/local.toml
+
+echo "Running server..."
+DEBUG=vulcanize:* exec node --enable-source-maps dist/server.js

stack_orchestrator/data/config/watcher-ajna/watcher-config-template.toml

@@ -0,0 +1,98 @@
+[server]
+  host = "0.0.0.0"
+  port = 3008
+  kind = "active"
+  gqlPath = "/graphql"
+
+  # Checkpointing state.
+  checkpointing = true
+
+  # Checkpoint interval in number of blocks.
+  checkpointInterval = 2000
+
+  # Enable state creation
+  # CAUTION: Disable only if state creation is not desired or can be filled subsequently
+  enableState = false
+
+  subgraphPath = "./subgraph-build"
+
+  # Interval to restart wasm instance periodically
+  wasmRestartBlocksInterval = 20
+
+  # Interval in number of blocks at which to clear entities cache.
+  clearEntitiesCacheInterval = 1000
+
+  # Max block range for which to return events in eventsInRange GQL query.
+  # Use -1 for skipping check on block range.
+  maxEventsBlockRange = 1000
+
+  # Flag to specify whether RPC endpoint supports block hash as block tag parameter
+  rpcSupportsBlockHashParam = false
+
+  # GQL cache settings
+  [server.gqlCache]
+    enabled = true
+
+    # Max in-memory cache size (in bytes) (default 8 MB)
+    # maxCacheSize
+
+    # GQL cache-control max-age settings (in seconds)
+    maxAge = 15
+    timeTravelMaxAge = 86400 # 1 day
+
+[metrics]
+  host = "0.0.0.0"
+  port = 9000
+  [metrics.gql]
+    port = 9001
+
+[database]
+  type = "postgres"
+  host = "ajna-watcher-db"
+  port = 5432
+  database = "ajna-watcher"
+  username = "vdbm"
+  password = "password"
+  synchronize = true
+  logging = false
+
+[upstream]
+  [upstream.ethServer]
+    rpcProviderEndpoint = "REPLACE_WITH_CERC_ETH_RPC_ENDPOINT"
+
+    # Boolean flag to specify if rpc-eth-client should be used for RPC endpoint instead of ipld-eth-client (ipld-eth-server GQL client)
+    rpcClient = true
+
+    # Boolean flag to specify if rpcProviderEndpoint is an FEVM RPC endpoint
+    isFEVM = true
+
+    # Boolean flag to filter event logs by contracts
+    filterLogsByAddresses = true
+    # Boolean flag to filter event logs by topics
+    filterLogsByTopics = true
+
+  [upstream.cache]
+    name = "requests"
+    enabled = false
+    deleteOnStart = false
+
+[jobQueue]
+  dbConnectionString = "postgres://vdbm:password@ajna-watcher-db/ajna-watcher-job-queue"
+  maxCompletionLagInSecs = 300
+  jobDelayInMilliSecs = 100
+  eventsInBatch = 50
+  subgraphEventsOrder = true
+  # Filecoin block time: https://docs.filecoin.io/basics/the-blockchain/blocks-and-tipsets#blocktime
+  blockDelayInMilliSecs = 30000
+
+  # Boolean to switch between modes of processing events when starting the server.
+  # Setting to true will fetch filtered events and required blocks in a range of blocks and then process them.
+  # Setting to false will fetch blocks consecutively with its events and then process them (Behaviour is followed in realtime processing near head).
+  useBlockRanges = true
+
+  # Block range in which logs are fetched during historical blocks processing
+  historicalLogsBlockRange = 2000
+
+  # Max block range of historical processing after which it waits for completion of events processing
+  # If set to -1 historical processing does not wait for events processing and completes till latest canonical block
+  historicalMaxFetchAhead = 10000

stack_orchestrator/data/container-build/cerc-laconic-registry-cli/create-demo-records.sh

@@ -5,7 +5,7 @@ if [ -n "$CERC_SCRIPT_DEBUG" ]; then
   set -x
 fi
 
-registry_command="laconic cns"
+registry_command="laconic registry"
 demo_records_dir="scripts/demo-records"
 
 # Check we have funds

stack_orchestrator/data/container-build/cerc-nextjs-base/build.sh

@@ -11,8 +11,14 @@ CERC_CONTAINER_BUILD_DOCKERFILE=${CERC_CONTAINER_BUILD_DOCKERFILE:-$SCRIPT_DIR/D
 CERC_CONTAINER_BUILD_TAG=${CERC_CONTAINER_BUILD_TAG:-cerc/nextjs-base:local}
 
 docker build -t $CERC_CONTAINER_BUILD_TAG ${build_command_args} -f $CERC_CONTAINER_BUILD_DOCKERFILE $CERC_CONTAINER_BUILD_WORK_DIR
+rc=$?
 
-if [ $? -eq 0 ] && [ "$CERC_CONTAINER_BUILD_TAG" != "cerc/nextjs-base:local" ]; then
+if [ $rc -ne 0 ]; then
+  echo "BUILD FAILED" 1>&2
+  exit $rc
+fi
+
+if [ "$CERC_CONTAINER_BUILD_TAG" != "cerc/nextjs-base:local" ]; then
   cat <<EOF
 
 #################################################################

stack_orchestrator/data/container-build/cerc-nextjs-base/scripts/build-app.sh

@@ -115,7 +115,7 @@ if [ "$CERC_NEXT_VERSION" != "keep" ] && [ "$CUR_NEXT_VERSION" != "$CERC_NEXT_VE
   mv package.json.$$ package.json
 fi
 
-$CERC_BUILD_TOOL install || exit 1
+time $CERC_BUILD_TOOL install || exit 1
 
 CUR_NEXT_VERSION=`jq -r '.version' node_modules/next/package.json`
 
@@ -138,9 +138,9 @@ to use for the build with:
 EOF
   cat package.json | jq ".dependencies.next = \"^$CERC_MIN_NEXTVER\"" > package.json.$$
   mv package.json.$$ package.json
-  $CERC_BUILD_TOOL install || exit 1
+  time $CERC_BUILD_TOOL install || exit 1
 fi
 
-$CERC_BUILD_TOOL run cerc_compile || exit 1
+time $CERC_BUILD_TOOL run cerc_compile || exit 1
 
 exit 0

stack_orchestrator/data/container-build/cerc-watcher-ajna/Dockerfile

@@ -0,0 +1,10 @@
+FROM node:18.17.1-alpine3.18
+
+RUN apk --update --no-cache add git python3 alpine-sdk bash curl jq
+
+WORKDIR /app
+
+COPY . .
+
+RUN echo "Installing dependencies and building ajna-watcher-ts" && \
+  yarn && yarn build

stack_orchestrator/data/container-build/cerc-watcher-ajna/build.sh

@@ -0,0 +1,9 @@
+#!/usr/bin/env bash
+# Build cerc/watcher-ajna
+
+source ${CERC_CONTAINER_BASE_DIR}/build-base.sh
+
+# See: https://stackoverflow.com/a/246128/1701505
+SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
+
+docker build -t cerc/watcher-ajna:local -f ${SCRIPT_DIR}/Dockerfile ${build_command_args} ${CERC_REPO_BASE_DIR}/ajna-watcher-ts

stack_orchestrator/data/container-build/cerc-webapp-base/Dockerfile

@@ -1,6 +1,6 @@
 # Originally from: https://github.com/devcontainers/images/blob/main/src/javascript-node/.devcontainer/Dockerfile
 # [Choice] Node.js version (use -bullseye variants on local arm64/Apple Silicon): 18, 16, 14, 18-bullseye, 16-bullseye, 14-bullseye, 18-buster, 16-buster, 14-buster
-ARG VARIANT=20-bullseye
+ARG VARIANT=20-bullseye-slim
 FROM node:${VARIANT}
 
 ARG USERNAME=node

stack_orchestrator/data/container-build/cerc-webapp-base/build.sh

@@ -11,8 +11,14 @@ CERC_CONTAINER_BUILD_DOCKERFILE=${CERC_CONTAINER_BUILD_DOCKERFILE:-$SCRIPT_DIR/D
 CERC_CONTAINER_BUILD_TAG=${CERC_CONTAINER_BUILD_TAG:-cerc/webapp-base:local}
 
 docker build -t $CERC_CONTAINER_BUILD_TAG ${build_command_args} -f $CERC_CONTAINER_BUILD_DOCKERFILE $CERC_CONTAINER_BUILD_WORK_DIR
+rc=$?
 
-if [ $? -eq 0 ] && [ "$CERC_CONTAINER_BUILD_TAG" != "cerc/webapp-base:local" ]; then
+if [ $rc -ne 0 ]; then
+  echo "BUILD FAILED" 1>&2
+  exit $rc
+fi
+
+if [ "$CERC_CONTAINER_BUILD_TAG" != "cerc/webapp-base:local" ]; then
   cat <<EOF
 
 #################################################################

stack_orchestrator/data/container-build/cerc-webapp-base/scripts/build-app.sh

@@ -32,8 +32,8 @@ elif [ -f "${WORK_DIR}/package.json" ]; then
     fi
   fi
 
-  $CERC_BUILD_TOOL install || exit 1
-  $CERC_BUILD_TOOL build || exit 1
+  time $CERC_BUILD_TOOL install || exit 1
+  time $CERC_BUILD_TOOL build || exit 1
 
   rm -rf "${DEST_DIR}"
   if [ -z "${CERC_BUILD_OUTPUT_DIR}" ]; then

stack_orchestrator/data/stacks/ajna/README.md

@@ -0,0 +1,118 @@
+# Ajna Watcher
+
+## Setup
+
+Clone required repositories:
+
+```bash
+laconic-so --stack ajna setup-repositories --git-ssh --pull
+```
+
+Build the container images:
+
+```bash
+laconic-so --stack ajna build-containers
+```
+
+## Deploy
+
+Create a spec file for the deployment:
+
+```bash
+laconic-so --stack ajna deploy init --output ajna-spec.yml
+```
+
+### Ports
+
+Edit `network` in the spec file to map container ports to host ports as required:
+
+```yml
+...
+network:
+  ports:
+    ajna-watcher-db:
+     - 15432:5432
+    ajna-watcher-job-runner:
+     - 9000:9000
+    ajna-watcher-server:
+     - 3008:3008
+     - 9001:9001
+```
+
+### Create a deployment
+
+Create a deployment from the spec file:
+
+```bash
+laconic-so --stack ajna deploy create --spec-file ajna-spec.yml --deployment-dir ajna-deployment
+```
+
+### Configuration
+
+Inside deployment directory, open the `config.env` file  and set following env variables:
+
+```bash
+# External Filecoin (ETH RPC) endpoint to point the watcher to
+CERC_ETH_RPC_ENDPOINT=https://example-lotus-endpoint/rpc/v1
+```
+
+### Start the deployment
+
+```bash
+laconic-so deployment --dir ajna-deployment start
+```
+
+* To list down and monitor the running containers:
+
+  ```bash
+  # With status
+  docker ps -a
+
+  # Check logs for a container
+  docker logs -f <CONTAINER_ID>
+  ```
+
+* Open the GQL playground at <http://localhost:3008/graphql>
+
+  ```graphql
+  # Example query
+  query {
+    _meta {
+      block {
+        hash
+        number
+        timestamp
+      }
+      deployment
+      hasIndexingErrors
+    }
+
+    accounts {
+      id
+      txCount
+      tokensDelegated
+      rewardsClaimed
+    }
+  }
+  ```
+
+## Clean up
+
+Stop all the ajna services running in background:
+
+```bash
+# Only stop the docker containers
+laconic-so deployment --dir ajna-deployment stop
+
+# Run 'start' to restart the deployment
+```
+
+To stop all the ajna services and also delete data:
+
+```bash
+# Stop the docker containers
+laconic-so deployment --dir ajna-deployment stop --delete-volumes
+
+# Remove deployment directory (deployment will have to be recreated for a re-run)
+rm -r ajna-deployment
+```

stack_orchestrator/data/stacks/ajna/stack.yml

@@ -0,0 +1,9 @@
+version: "1.0"
+name: ajna
+description: "Ajna watcher stack"
+repos:
+  - git.vdb.to/cerc-io/ajna-watcher-ts
+containers:
+  - cerc/watcher-ajna
+pods:
+  - watcher-ajna

stack_orchestrator/data/stacks/fixturenet-laconic-loaded/README.md

@@ -49,7 +49,7 @@ $ laconic-so --stack fixturenet-laconic-loaded deploy logs
 ```
 ## 6. Test with the Registry CLI
 ```
-$ laconic-so --stack fixturenet-laconic-loaded deploy exec cli "laconic cns status"
+$ laconic-so --stack fixturenet-laconic-loaded deploy exec cli "laconic registry status"
 ```
 ## 7. View the laconic console
 Get the URL for the console web app with this command (the port number will be different for each deployment):

stack_orchestrator/data/stacks/fixturenet-laconicd/README.md

@@ -44,5 +44,5 @@ $ laconic-so --stack fixturenet-laconicd deploy logs
 ```
 ## 6. Test with the Registry CLI
 ```
-$ laconic-so --stack fixturenet-laconicd deploy exec cli "laconic cns status"
+$ laconic-so --stack fixturenet-laconicd deploy exec cli "laconic registry status"
 ```

stack_orchestrator/data/stacks/merkl-sushiswap-v3/README.md

@@ -16,26 +16,55 @@ laconic-so --stack merkl-sushiswap-v3 build-containers
 
 ## Deploy
 
-### Configuration
-
-Create and update an env file to be used in the next step:
-
-  ```bash
-  # External Filecoin (ETH RPC) endpoint to point the watcher
-  CERC_ETH_RPC_ENDPOINT=
-  ```
-
-### Deploy the stack
+Create a spec file for the deployment:
 
 ```bash
-laconic-so --stack merkl-sushiswap-v3 deploy --cluster merkl_sushiswap_v3 --env-file <PATH_TO_ENV_FILE> up
+laconic-so --stack merkl-sushiswap-v3 deploy init --output merkl-sushiswap-v3-spec.yml
+```
+
+### Ports
+
+Edit `network` in the spec file to map container ports to host ports as required:
+
+```
+...
+network:
+  ports:
+    merkl-sushiswap-v3-watcher-db:
+     - '5432'
+    merkl-sushiswap-v3-watcher-job-runner:
+     - 9002:9000
+    merkl-sushiswap-v3-watcher-server:
+     - 127.0.0.1:3007:3008
+     - 9003:9001
+```
+
+### Create a deployment
+
+Create a deployment from the spec file:
+
+```bash
+laconic-so --stack merkl-sushiswap-v3 deploy create --spec-file merkl-sushiswap-v3-spec.yml --deployment-dir merkl-sushiswap-v3-deployment
+```
+
+### Configuration
+
+Inside deployment directory, open the `config.env` file  and set following env variables:
+
+```bash
+# External Filecoin (ETH RPC) endpoint to point the watcher to
+CERC_ETH_RPC_ENDPOINT=https://example-lotus-endpoint/rpc/v1
+```
+
+### Start the deployment
+
+```bash
+laconic-so deployment --dir merkl-sushiswap-v3-deployment start
 ```
 
 * To list down and monitor the running containers:
 
   ```bash
-  laconic-so --stack merkl-sushiswap-v3 deploy --cluster merkl_sushiswap_v3 ps
-
   # With status
   docker ps -a
 
@@ -46,6 +75,7 @@ laconic-so --stack merkl-sushiswap-v3 deploy --cluster merkl_sushiswap_v3 --env-
 * Open the GQL playground at http://localhost:3007/graphql
 
   ```graphql
+  # Example query
   {
     _meta {
       block {
@@ -54,7 +84,7 @@ laconic-so --stack merkl-sushiswap-v3 deploy --cluster merkl_sushiswap_v3 --env-
       }
       hasIndexingErrors
     }
-    
+
     factories {
       id
       poolCount
@@ -64,18 +94,21 @@ laconic-so --stack merkl-sushiswap-v3 deploy --cluster merkl_sushiswap_v3 --env-
 
 ## Clean up
 
-Stop all the services running in background:
+Stop all the merkl-sushiswap-v3 services running in background:
 
 ```bash
-laconic-so --stack merkl-sushiswap-v3 deploy --cluster merkl_sushiswap_v3 down
+# Only stop the docker containers
+laconic-so deployment --dir merkl-sushiswap-v3-deployment stop
+
+# Run 'start' to restart the deployment
 ```
 
-Clear volumes created by this stack:
+To stop all the merkl-sushiswap-v3 services and also delete data:
 
 ```bash
-# List all relevant volumes
-docker volume ls -q --filter "name=merkl_sushiswap_v3"
+# Stop the docker containers
+laconic-so deployment --dir merkl-sushiswap-v3-deployment stop --delete-volumes
 
-# Remove all the listed volumes
-docker volume rm $(docker volume ls -q --filter "name=merkl_sushiswap_v3")
+# Remove deployment directory (deployment will have to be recreated for a re-run)
+rm -r merkl-sushiswap-v3-deployment
 ```

stack_orchestrator/data/stacks/merkl-sushiswap-v3/stack.yml

@@ -2,7 +2,7 @@ version: "1.0"
 name: merkl-sushiswap-v3
 description: "SushiSwap v3 watcher stack"
 repos:
-  - github.com/cerc-io/merkl-sushiswap-v3-watcher-ts@v0.1.6
+  - github.com/cerc-io/merkl-sushiswap-v3-watcher-ts@v0.1.7
 containers:
   - cerc/watcher-merkl-sushiswap-v3
 pods:

stack_orchestrator/data/stacks/monitoring/stack.yml

@@ -1,7 +1,7 @@
 version: "0.1"
 name: monitoring
 repos:
-  - github.com/cerc-io/watcher-ts@v0.2.79
+  - github.com/cerc-io/watcher-ts@v0.2.81
 containers:
   - cerc/watcher-ts
 pods:

stack_orchestrator/data/stacks/sushiswap-subgraph/README.md

@@ -55,7 +55,7 @@ ports:
 Create deployment:
 
 ```bash
-laconic-so deploy create --spec-file sushiswap-subgraph-spec.yml --deployment-dir sushiswap-subgraph-deployment
+laconic-so --stack sushiswap-subgraph deploy create --spec-file sushiswap-subgraph-spec.yml --deployment-dir sushiswap-subgraph-deployment
 ```
 
 ## Start the stack

stack_orchestrator/data/stacks/sushiswap-v3/README.md

@@ -16,26 +16,55 @@ laconic-so --stack sushiswap-v3 build-containers
 
 ## Deploy
 
-### Configuration
-
-Create and update an env file to be used in the next step:
-
-  ```bash
-  # External Filecoin (ETH RPC) endpoint to point the watcher
-  CERC_ETH_RPC_ENDPOINT=
-  ```
-
-### Deploy the stack
+Create a spec file for the deployment:
 
 ```bash
-laconic-so --stack sushiswap-v3 deploy --cluster sushiswap_v3 --env-file <PATH_TO_ENV_FILE> up
+laconic-so --stack sushiswap-v3 deploy init --output sushiswap-v3-spec.yml
+```
+
+### Ports
+
+Edit `network` in the spec file to map container ports to host ports as required:
+
+```
+...
+network:
+  ports:
+    sushiswap-v3-watcher-db:
+     - '5432'
+    sushiswap-v3-watcher-job-runner:
+     - 9000:9000
+    sushiswap-v3-watcher-server:
+     - 127.0.0.1:3008:3008
+     - 9001:9001
+```
+
+### Create a deployment
+
+Create a deployment from the spec file:
+
+```bash
+laconic-so --stack sushiswap-v3 deploy create --spec-file sushiswap-v3-spec.yml --deployment-dir sushiswap-v3-deployment
+```
+
+### Configuration
+
+Inside deployment directory, open the `config.env` file  and set following env variables:
+
+```bash
+# External Filecoin (ETH RPC) endpoint to point the watcher to
+CERC_ETH_RPC_ENDPOINT=https://example-lotus-endpoint/rpc/v1
+```
+
+### Start the deployment
+
+```bash
+laconic-so deployment --dir sushiswap-v3-deployment start
 ```
 
 * To list down and monitor the running containers:
 
   ```bash
-  laconic-so --stack sushiswap-v3 deploy --cluster sushiswap_v3 ps
-
   # With status
   docker ps -a
 
@@ -43,20 +72,43 @@ laconic-so --stack sushiswap-v3 deploy --cluster sushiswap_v3 --env-file <PATH_T
   docker logs -f <CONTAINER_ID>
   ```
 
+* Open the GQL playground at http://localhost:3008/graphql
+
+  ```graphql
+  # Example query
+  {
+    _meta {
+      block {
+        number
+        timestamp
+      }
+      hasIndexingErrors
+    }
+
+    factories {
+      id
+      poolCount
+    }
+  }
+  ```
+
 ## Clean up
 
-Stop all the services running in background:
+Stop all the sushiswap-v3 services running in background:
 
 ```bash
-laconic-so --stack sushiswap-v3 deploy --cluster sushiswap_v3 down
+# Only stop the docker containers
+laconic-so deployment --dir sushiswap-v3-deployment stop
+
+# Run 'start' to restart the deployment
 ```
 
-Clear volumes created by this stack:
+To stop all the sushiswap-v3 services and also delete data:
 
 ```bash
-# List all relevant volumes
-docker volume ls -q --filter "name=sushiswap_v3"
+# Stop the docker containers
+laconic-so deployment --dir sushiswap-v3-deployment stop --delete-volumes
 
-# Remove all the listed volumes
-docker volume rm $(docker volume ls -q --filter "name=sushiswap_v3")
+# Remove deployment directory (deployment will have to be recreated for a re-run)
+rm -r sushiswap-v3-deployment
 ```

stack_orchestrator/data/stacks/sushiswap-v3/stack.yml

@@ -2,7 +2,7 @@ version: "1.0"
 name: sushiswap-v3
 description: "SushiSwap v3 watcher stack"
 repos:
-  - github.com/cerc-io/sushiswap-v3-watcher-ts@v0.1.6
+  - github.com/cerc-io/sushiswap-v3-watcher-ts@v0.1.7
 containers:
   - cerc/watcher-sushiswap-v3
 pods:

stack_orchestrator/deploy/k8s/cluster_info.py

@@ -101,7 +101,7 @@ class ClusterInfo:
         )
         return service
 
-    def get_ingress(self, use_tls=False):
+    def get_ingress(self, use_tls=False, certificate=None, cluster_issuer="letsencrypt-prod"):
         # No ingress for a deployment that has no http-proxy defined, for now
         http_proxy_info_list = self.spec.get_http_proxy()
         ingress = None
@@ -114,8 +114,8 @@ class ClusterInfo:
             host_name = http_proxy_info["host-name"]
             rules = []
             tls = [client.V1IngressTLS(
-                hosts=[host_name],
-                secret_name=f"{self.app_name}-tls"
+                hosts=certificate["spec"]["dnsNames"] if certificate else [host_name],
+                secret_name=certificate["spec"]["secretName"] if certificate else f"{self.app_name}-tls"
             )] if use_tls else None
             paths = []
             for route in http_proxy_info["routes"]:
@@ -147,13 +147,17 @@ class ClusterInfo:
                 tls=tls,
                 rules=rules
             )
+
+            ingress_annotations = {
+                "kubernetes.io/ingress.class": "nginx",
+            }
+            if not certificate:
+                ingress_annotations["cert-manager.io/cluster-issuer"] = cluster_issuer
+
             ingress = client.V1Ingress(
                 metadata=client.V1ObjectMeta(
                     name=f"{self.app_name}-ingress",
-                    annotations={
-                        "kubernetes.io/ingress.class": "nginx",
-                        "cert-manager.io/cluster-issuer": "letsencrypt-prod"
-                    }
+                    annotations=ingress_annotations
                 ),
                 spec=spec
             )

stack_orchestrator/deploy/k8s/deploy_k8s.py

@@ -169,6 +169,39 @@ class K8sDeployer(Deployer):
                 print("Service created:")
                 print(f"{service_resp}")
 
+    def _find_certificate_for_host_name(self, host_name):
+        all_certificates = self.custom_obj_api.list_namespaced_custom_object(
+            group="cert-manager.io",
+            version="v1",
+            namespace=self.k8s_namespace,
+            plural="certificates"
+        )
+
+        host_parts = host_name.split(".", 1)
+        host_as_wild = None
+        if len(host_parts) == 2:
+            host_as_wild = f"*.{host_parts[1]}"
+
+        now = datetime.utcnow().replace(tzinfo=timezone.utc)
+        fmt = "%Y-%m-%dT%H:%M:%S%z"
+
+        # Walk over all the configured certificates.
+        for cert in all_certificates["items"]:
+            dns = cert["spec"]["dnsNames"]
+            # Check for an exact hostname match or a wildcard match.
+            if host_name in dns or host_as_wild in dns:
+                status = cert.get("status", {})
+                # Check the certificate date.
+                if "notAfter" in status and "notBefore" in status:
+                    before = datetime.strptime(status["notBefore"], fmt)
+                    after = datetime.strptime(status["notAfter"], fmt)
+                    if before < now < after:
+                        # Check the status is Ready
+                        for condition in status.get("conditions", []):
+                            if "True" == condition.get("status") and "Ready" == condition.get("type"):
+                                return cert
+        return None
+
     def up(self, detach, services):
         if not opts.o.dry_run:
             if self.is_kind():
@@ -189,8 +222,15 @@ class K8sDeployer(Deployer):
         self._create_volume_data()
         self._create_deployment()
 
+        http_proxy_info = self.cluster_info.spec.get_http_proxy()
         # Note: at present we don't support tls for kind (and enabling tls causes errors)
-        ingress: client.V1Ingress = self.cluster_info.get_ingress(use_tls=not self.is_kind())
+        use_tls = http_proxy_info and not self.is_kind()
+        certificate = self._find_certificate_for_host_name(http_proxy_info[0]["host-name"]) if use_tls else None
+        if opts.o.debug:
+            if certificate:
+                print(f"Using existing certificate: {certificate}")
+
+        ingress: client.V1Ingress = self.cluster_info.get_ingress(use_tls=use_tls, certificate=certificate)
         if ingress:
             if opts.o.debug:
                 print(f"Sending this ingress: {ingress}")
@@ -350,9 +390,11 @@ class K8sDeployer(Deployer):
                 name=ingress.spec.tls[0].secret_name
             )
 
-            hostname = ingress.spec.tls[0].hosts[0]
+            hostname = ingress.spec.rules[0].host
             ip = ingress.status.load_balancer.ingress[0].ip
-            tls = "notBefore: %s, notAfter: %s" % (cert["status"]["notBefore"], cert["status"]["notAfter"])
+            tls = "notBefore: %s; notAfter: %s; names: %s" % (
+                cert["status"]["notBefore"], cert["status"]["notAfter"], ingress.spec.tls[0].hosts
+            )
         except:  # noqa: E722
             pass
 

stack_orchestrator/deploy/spec.py

@@ -27,7 +27,9 @@ class ResourceLimits:
     memory: int = None
     storage: int = None
 
-    def __init__(self, obj={}):
+    def __init__(self, obj=None):
+        if obj is None:
+            obj = {}
         if "cpus" in obj:
             self.cpus = float(obj["cpus"])
         if "memory" in obj:
@@ -50,7 +52,9 @@ class Resources:
     limits: ResourceLimits = None
     reservations: ResourceLimits = None
 
-    def __init__(self, obj={}):
+    def __init__(self, obj=None):
+        if obj is None:
+            obj = {}
         if "reservations" in obj:
             self.reservations = ResourceLimits(obj["reservations"])
         if "limits" in obj:
@@ -72,7 +76,9 @@ class Spec:
     obj: typing.Any
     file_path: Path
 
-    def __init__(self, file_path: Path = None, obj={}) -> None:
+    def __init__(self, file_path: Path = None, obj=None) -> None:
+        if obj is None:
+            obj = {}
         self.file_path = file_path
         self.obj = obj
 
@@ -91,49 +97,41 @@ class Spec:
             self.file_path = file_path
 
     def get_image_registry(self):
-        return (self.obj[constants.image_registry_key]
-                if self.obj and constants.image_registry_key in self.obj
-                else None)
+        return self.obj.get(constants.image_registry_key)
 
     def get_volumes(self):
-        return (self.obj["volumes"]
-                if self.obj and "volumes" in self.obj
-                else {})
+        return self.obj.get(constants.volumes_key, {})
 
     def get_configmaps(self):
-        return (self.obj["configmaps"]
-                if self.obj and "configmaps" in self.obj
-                else {})
+        return self.obj.get(constants.configmaps_key, {})
 
     def get_container_resources(self):
-        return Resources(self.obj.get("resources", {}).get("containers", {}))
+        return Resources(self.obj.get(constants.resources_key, {}).get("containers", {}))
 
     def get_volume_resources(self):
-        return Resources(self.obj.get("resources", {}).get("volumes", {}))
+        return Resources(self.obj.get(constants.resources_key, {}).get(constants.volumes_key, {}))
 
     def get_http_proxy(self):
-        return (self.obj[constants.network_key][constants.http_proxy_key]
-                if self.obj and constants.network_key in self.obj
-                and constants.http_proxy_key in self.obj[constants.network_key]
-                else None)
+        return self.obj.get(constants.network_key, {}).get(constants.http_proxy_key, [])
 
     def get_annotations(self):
-        return self.obj.get("annotations", {})
+        return self.obj.get(constants.annotations_key, {})
 
     def get_labels(self):
-        return self.obj.get("labels", {})
+        return self.obj.get(constants.labels_key, {})
 
     def get_privileged(self):
-        return "true" == str(self.obj.get("security", {}).get("privileged", "false")).lower()
+        return "true" == str(self.obj.get(constants.security_key, {}).get("privileged", "false")).lower()
 
     def get_capabilities(self):
-        return self.obj.get("security", {}).get("capabilities", [])
+        return self.obj.get(constants.security_key, {}).get("capabilities", [])
 
     def get_deployment_type(self):
-        return self.obj[constants.deploy_to_key]
+        return self.obj.get(constants.deploy_to_key)
 
     def is_kubernetes_deployment(self):
-        return self.get_deployment_type() in [constants.k8s_kind_deploy_type, constants.k8s_deploy_type]
+        return self.get_deployment_type() in [constants.k8s_kind_deploy_type,
+                                              constants.k8s_deploy_type]
 
     def is_kind_deployment(self):
         return self.get_deployment_type() in [constants.k8s_kind_deploy_type]

stack_orchestrator/deploy/webapp/deploy_webapp_from_registry.py

@@ -26,7 +26,7 @@ import click
 
 from stack_orchestrator.deploy.images import remote_image_exists, add_tags_to_image
 from stack_orchestrator.deploy.webapp import deploy_webapp
-from stack_orchestrator.deploy.webapp.util import (LaconicRegistryClient,
+from stack_orchestrator.deploy.webapp.util import (LaconicRegistryClient, TimedLogger,
                                                    build_container_image, push_container_image,
                                                    file_hash, deploy_to_k8s, publish_deployment,
                                                    hostname_for_deployment_request, generate_hostname_for_app,
@@ -34,7 +34,6 @@ from stack_orchestrator.deploy.webapp.util import (LaconicRegistryClient,
 
 
 def process_app_deployment_request(
-    run_id,
     ctx,
     laconic: LaconicRegistryClient,
     app_deployment_request,
@@ -44,14 +43,18 @@ def process_app_deployment_request(
     deployment_parent_dir,
     kube_config,
     image_registry,
-    force_rebuild=False,
-    log_file=None
+    force_rebuild,
+    logger
 ):
+    logger.log("BEGIN - process_app_deployment_request")
+
     # 1. look up application
     app = laconic.get_record(app_deployment_request.attributes.application, require=True)
+    logger.log(f"Retrieved app record {app_deployment_request.attributes.application}")
 
     # 2. determine dns
     requested_name = hostname_for_deployment_request(app_deployment_request, laconic)
+    logger.log(f"Determined requested name: {requested_name}")
 
     # HACK
     if "." in requested_name:
@@ -69,7 +72,7 @@ def process_app_deployment_request(
             matched_owner = match_owner(app_deployment_request, laconic.get_record(dns_record.attributes.request, require=True))
 
         if matched_owner:
-            print("Matched DnsRecord ownership:", matched_owner)
+            logger.log(f"Matched DnsRecord ownership: {matched_owner}")
         else:
             raise Exception("Unable to confirm ownership of DnsRecord %s for request %s" %
                             (dns_record.id, app_deployment_request.id))
@@ -115,15 +118,21 @@ def process_app_deployment_request(
         shared_tag_exists = remote_image_exists(image_registry, app_image_shared_tag)
         if shared_tag_exists and not force_rebuild:
             # simply add our unique tag to the existing image and we are done
-            print(f"Using existing app image {app_image_shared_tag} for {deployment_container_tag}", file=log_file)
+            logger.log(f"Using existing app image {app_image_shared_tag} for {deployment_container_tag}")
             add_tags_to_image(image_registry, app_image_shared_tag, deployment_container_tag)
+            logger.log("Tag complete")
         else:
             extra_build_args = []  # TODO: pull from request
-            build_container_image(app, deployment_container_tag, extra_build_args, log_file)
-            push_container_image(deployment_dir, log_file)
+            logger.log(f"Building container image {deployment_container_tag}")
+            build_container_image(app, deployment_container_tag, extra_build_args, logger)
+            logger.log("Build complete")
+            logger.log(f"Pushing container image {deployment_container_tag}")
+            push_container_image(deployment_dir, logger)
+            logger.log("Push complete")
             # The build/push commands above will use the unique deployment tag, so now we need to add the shared tag.
-            print(f"Updating app image tag {app_image_shared_tag} from build of {deployment_container_tag}", file=log_file)
+            logger.log(f"Updating app image tag {app_image_shared_tag} from build of {deployment_container_tag}")
             add_tags_to_image(image_registry, deployment_container_tag, app_image_shared_tag)
+            logger.log("Tag complete")
 
     # 7. update config (if needed)
     if not deployment_record or file_hash(deployment_config_file) != deployment_record.attributes.meta.config:
@@ -131,13 +140,13 @@ def process_app_deployment_request(
 
     # 8. update k8s deployment
     if needs_k8s_deploy:
-        print("Deploying to k8s")
         deploy_to_k8s(
             deployment_record,
             deployment_dir,
-            log_file
+            logger
         )
 
+    logger.log("Publishing deployment to registry.")
     publish_deployment(
         laconic,
         app,
@@ -146,8 +155,11 @@ def process_app_deployment_request(
         dns_record,
         dns_crn,
         deployment_dir,
-        app_deployment_request
+        app_deployment_request,
+        logger
     )
+    logger.log("Publication complete.")
+    logger.log("END - process_app_deployment_request")
 
 
 def load_known_requests(filename):
@@ -309,8 +321,9 @@ def command(ctx, kube_config, laconic_config, image_registry, deployment_parent_
                     run_log_file = open(run_log_file_path, "wt")
                     run_reg_client = LaconicRegistryClient(laconic_config, log_file=run_log_file)
 
+                logger = TimedLogger(run_id, run_log_file)
+                logger.log("Processing ...")
                 process_app_deployment_request(
-                    run_id,
                     ctx,
                     run_reg_client,
                     r,
@@ -321,12 +334,14 @@ def command(ctx, kube_config, laconic_config, image_registry, deployment_parent_
                     kube_config,
                     image_registry,
                     force_rebuild,
-                    run_log_file
+                    logger
                 )
                 status = "DEPLOYED"
             except Exception as e:
-                print("ERROR: " + str(e), file=run_log_file)
+                logger.log("ERROR: " + str(e))
             finally:
+                if logger:
+                    logger.log(f"DONE with status {status}", show_step_time=False, show_total_time=True)
                 dump_known_requests(state_file, [r], status)
                 if run_log_file:
                     run_log_file.close()

stack_orchestrator/deploy/webapp/util.py

@@ -13,6 +13,7 @@
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <http:#www.gnu.org/licenses/>.
 
+import datetime
 import hashlib
 import json
 import os
@@ -39,6 +40,25 @@ class AttrDict(dict):
             return v
 
 
+class TimedLogger:
+    def __init__(self, id="", file=None):
+        self.start = datetime.datetime.now()
+        self.last = self.start
+        self.id = id
+        self.file = file
+
+    def log(self, msg, show_step_time=True, show_total_time=False):
+        prefix = f"{datetime.datetime.utcnow()} - {self.id}"
+        if show_step_time:
+            prefix += f" - {datetime.datetime.now() - self.last} (step)"
+        if show_total_time:
+            prefix += f" - {datetime.datetime.now() - self.start} (total)"
+        print(f"{prefix}: {msg}", file=self.file)
+        if self.file:
+            self.file.flush()
+        self.last = datetime.datetime.now()
+
+
 def logged_cmd(log_file, *vargs):
     result = None
     try:
@@ -74,7 +94,7 @@ class LaconicRegistryClient:
         )
 
     def list_records(self, criteria={}, all=False):
-        args = ["laconic", "-c", self.config_file, "cns", "record", "list"]
+        args = ["laconic", "-c", self.config_file, "registry", "record", "list"]
 
         if all:
             args.append("--all")
@@ -120,7 +140,7 @@ class LaconicRegistryClient:
         if name in self.cache.name_or_id:
             return self.cache.name_or_id[name]
 
-        args = ["laconic", "-c", self.config_file, "cns", "name", "resolve", name]
+        args = ["laconic", "-c", self.config_file, "registry", "name", "resolve", name]
 
         parsed = [AttrDict(r) for r in json.loads(logged_cmd(self.log_file, *args))]
         if parsed:
@@ -145,7 +165,7 @@ class LaconicRegistryClient:
             "laconic",
             "-c",
             self.config_file,
-            "cns",
+            "registry",
             "record",
             "get",
             "--id",
@@ -183,7 +203,8 @@ class LaconicRegistryClient:
             print(open(record_fname, 'r').read(), file=self.log_file)
 
             new_record_id = json.loads(
-                logged_cmd(self.log_file, "laconic", "-c", self.config_file, "cns", "record", "publish", "--filename", record_fname)
+                logged_cmd(self.log_file, "laconic", "-c", self.config_file, "registry",
+                           "record", "publish", "--filename", record_fname)
             )["id"]
             for name in names:
                 self.set_name(name, new_record_id)
@@ -192,10 +213,10 @@ class LaconicRegistryClient:
             logged_cmd(self.log_file, "rm", "-rf", tmpdir)
 
     def set_name(self, name, record_id):
-        logged_cmd(self.log_file, "laconic", "-c", self.config_file, "cns", "name", "set", name, record_id)
+        logged_cmd(self.log_file, "laconic", "-c", self.config_file, "registry", "name", "set", name, record_id)
 
     def delete_name(self, name):
-        logged_cmd(self.log_file, "laconic", "-c", self.config_file, "cns", "name", "delete", name)
+        logged_cmd(self.log_file, "laconic", "-c", self.config_file, "registry", "name", "delete", name)
 
 
 def file_hash(filename):
@@ -219,7 +240,7 @@ def determine_base_container(clone_dir, app_type="webapp"):
     return base_container
 
 
-def build_container_image(app_record, tag, extra_build_args=[], log_file=None):
+def build_container_image(app_record, tag, extra_build_args=[], logger=None):
     tmpdir = tempfile.mkdtemp()
 
     try:
@@ -228,31 +249,33 @@ def build_container_image(app_record, tag, extra_build_args=[], log_file=None):
         repo = random.choice(app_record.attributes.repository)
         clone_dir = os.path.join(tmpdir, record_id)
 
-        print(f"Cloning repository {repo} to {clone_dir} ...")
+        logger.log(f"Cloning repository {repo} to {clone_dir} ...")
         if ref:
             # TODO: Determing branch or hash, and use depth 1 if we can.
             git_env = dict(os.environ.copy())
             # Never prompt
             git_env["GIT_TERMINAL_PROMPT"] = "0"
             try:
-                subprocess.check_call(["git", "clone", repo, clone_dir], env=git_env, stdout=log_file, stderr=log_file)
+                subprocess.check_call(["git", "clone", repo, clone_dir], env=git_env, stdout=logger.file, stderr=logger.file)
             except Exception as e:
-                print(f"git clone failed.  Is the repository {repo} private?", file=log_file)
+                logger.log(f"git clone failed.  Is the repository {repo} private?")
                 raise e
             try:
-                subprocess.check_call(["git", "checkout", ref], cwd=clone_dir, env=git_env, stdout=log_file, stderr=log_file)
+                subprocess.check_call(["git", "checkout", ref], cwd=clone_dir, env=git_env, stdout=logger.file, stderr=logger.file)
             except Exception as e:
-                print(f"git checkout failed.  Does ref {ref} exist?", file=log_file)
+                logger.log(f"git checkout failed.  Does ref {ref} exist?")
                 raise e
         else:
-            result = subprocess.run(["git", "clone", "--depth", "1", repo, clone_dir], stdout=log_file, stderr=log_file)
+            result = subprocess.run(["git", "clone", "--depth", "1", repo, clone_dir], stdout=logger.file, stderr=logger.file)
             result.check_returncode()
 
         base_container = determine_base_container(clone_dir, app_record.attributes.app_type)
 
-        print("Building webapp ...")
+        logger.log("Building webapp ...")
         build_command = [
-            sys.argv[0], "build-webapp",
+            sys.argv[0],
+            "--verbose",
+            "build-webapp",
             "--source-repo", clone_dir,
             "--tag", tag,
             "--base-container", base_container
@@ -261,28 +284,31 @@ def build_container_image(app_record, tag, extra_build_args=[], log_file=None):
             build_command.append("--extra-build-args")
             build_command.append(" ".join(extra_build_args))
 
-        result = subprocess.run(build_command, stdout=log_file, stderr=log_file)
+        result = subprocess.run(build_command, stdout=logger.file, stderr=logger.file)
         result.check_returncode()
     finally:
-        logged_cmd(log_file, "rm", "-rf", tmpdir)
+        logged_cmd(logger.file, "rm", "-rf", tmpdir)
 
 
-def push_container_image(deployment_dir, log_file=None):
-    print("Pushing image ...")
+def push_container_image(deployment_dir, logger):
+    logger.log("Pushing images ...")
     result = subprocess.run([sys.argv[0], "deployment", "--dir", deployment_dir, "push-images"],
-                            stdout=log_file, stderr=log_file)
+                            stdout=logger.file, stderr=logger.file)
     result.check_returncode()
+    logger.log("Finished pushing images.")
 
 
-def deploy_to_k8s(deploy_record, deployment_dir, log_file=None):
+def deploy_to_k8s(deploy_record, deployment_dir, logger):
     if not deploy_record:
         command = "up"
     else:
         command = "update"
 
+    logger.log("Deploying to k8s ...")
     result = subprocess.run([sys.argv[0], "deployment", "--dir", deployment_dir, command],
-                            stdout=log_file, stderr=log_file)
+                            stdout=logger.file, stderr=logger.file)
     result.check_returncode()
+    logger.log("Finished deploying to k8s.")
 
 
 def publish_deployment(laconic: LaconicRegistryClient,
@@ -292,7 +318,8 @@ def publish_deployment(laconic: LaconicRegistryClient,
                        dns_record,
                        dns_crn,
                        deployment_dir,
-                       app_deployment_request=None):
+                       app_deployment_request=None,
+                       logger=None):
     if not deploy_record:
         deploy_ver = "0.0.1"
     else:
@@ -322,6 +349,8 @@ def publish_deployment(laconic: LaconicRegistryClient,
     if app_deployment_request:
         new_dns_record["record"]["request"] = app_deployment_request.id
 
+    if logger:
+        logger.log("Publishing DnsRecord.")
     dns_id = laconic.publish(new_dns_record, [dns_crn])
 
     new_deployment_record = {
@@ -341,6 +370,8 @@ def publish_deployment(laconic: LaconicRegistryClient,
     if app_deployment_request:
         new_deployment_record["record"]["request"] = app_deployment_request.id
 
+    if logger:
+        logger.log("Publishing ApplicationDeploymentRecord.")
     deployment_id = laconic.publish(new_deployment_record, [deployment_crn])
     return {"dns": dns_id, "deployment": deployment_id}