WIP

Blind commit to fix laconic CLI calls after rename. (#784 )
`laconic cns` got renamed to `laconic registry` which breaks all the scripts and commands that use it. Reviewed-on: #784 Co-authored-by: Thomas E Lackey <telackey@bozemanpass.com> Co-committed-by: Thomas E Lackey <telackey@bozemanpass.com>
2024-03-28 16:24:36 -04:00 · 2024-03-25 19:09:26 +00:00 · 2024-03-21 07:17:01 +00:00 · 2024-03-13 07:16:15 +00:00 · 2024-03-12 05:32:55 +00:00
20 changed files with 771 additions and 214 deletions
@@ -0,0 +1,271 @@
+# Deploying WebApps from the Laconic Registry to Kubernetes
+
+First, ensure you have `laconicd` and the console running via [this tutorial](./laconicd-with-console.md).
+
+## Setup Kubernetes
+
+If merely requesting deployment, access to the Laconic registry is sufficient, but to *process* deployment requests, Kubernetes configuration is also necessary. The configuration format is the same as used by `kubectl`, and should be places in `/etc/config/kube.yml`
+
+## Setup the Deployer
+
+### Set envs
+
+
+```
+$DEPLOYMENT_DNS_SUFFIX
+$DEPLOYMENT_RECORD_NAMESPACE
+$IMAGE_REGISTRY
+$IMAGE_REGISTRY_CREDS
+$IMAGE_REGISTRY_USER
+
+```
+
+
+```
+laconic-so --stack webapp-deployer-backend setup-repositories
+laconic-so --stack webapp-deployer-backend build-containers
+laconic-so --stack webapp-deployer-backend deploy up
+```
+
+## Setup Your Account
+
+Writing to the Laconic Registry, in order to create records for applications and deployment requests, requires a user key, and a bond with adequate funds available.
+
+### Install the registry CLI
+
+```
+npm config set @cerc-io:registry https://git.vdb.to/api/packages/cerc-io/npm/
+npm install -g @cerc-io/laconic-registry-cli
+```
+
+Alternatively, use the pre-configured container that was setup alongside `laconicd`
+
+### Get your private key
+
+```
+exec into laconicd
+```
+
+### config.yml
+
+```
+services:
+  cns:
+    restEndpoint: http://console.laconic.com:1317
+    gqlEndpoint: http://console.laconic.com:9473/api
+    userKey: 3d4789e88508c6230d973ccea283032d3e3948775dbe02f4f0a80dc6c1f7c8d5
+   # bondId: 3c61577f3c6197ced599e2fc21ccf76f43373004fd29c29f2e560c77f7c4bb6d
+    chainId: laconic_9000-1
+    gas: 550000
+    fees: 200000aphoton
+```
+
+### Create a bond
+
+```
+laconic cns -c config.yml bond create --type aphoton --quantity 1000000000 --gas 200000 --fees 200000aphoton
+```
+
+
+
+
+
+# Creating Deployments
+
+Deploying an application from the registry requires (and generates)
+several records:
+
+1. An `ApplicationRecord` which describes the application itself, including its name, version, and repository.
+2. An `ApplicationDeploymentRequest` which references an`ApplicationRecord`, and provides additional information such as the DNS name and configuration.
+3. A `DnsRecord`, which contains the FQDN of the deployment. The ownership of this record will be checked against any future deployment or removal requests.
+4. An `ApplicationDeploymentRecord` which records the successful processing of the deployment request.
+
+Additionally, since names need to be registered, namespace authorities need to be reserved, for example:
+
+```
+$ laconic -c $LACONIC_CONFIG cns authority reserve my-org-name
+
+$ laconic -c $LACONIC_CONFIG cns authority bond set my-org-name 0e9176d854bc3c20528b6361aab632f0b252a0f69717bf035fa68d1ef7647ba7
+```
+
+## Application Records
+
+The `ApplicationRecord` should, at a minimum, specify the name of the application (`name`), its version (`app_version`), type (`app_type`), one or more repository URLs (`repository`), and the repository reference (eg, branch, tag, hash, etc.) to use (`repository_ref`).
+
+```
+$ cat app.yml
+record:
+  type: ApplicationRecord
+  version: 0.0.4
+  name: "@my-org-name/my-test-webapp"
+  repository:
+    - "https://github.com/my-org-name/my-test-webapp"
+  repository_ref: "v0.1.5"
+  app_version: "0.1.5"
+  app_type: "webapp"
+
+$ laconic -c $LACONIC_CONFIG cns record publish -f app.yml
+bafyreihwvu6ynmk4nfrxg2vdcx2ep3tqry775ksqyehjitj2i4kphhyuky
+```
+
+One or more names should be registered for the application, which deployment requests can reference.
+
+```
+$ laconic -c $LACONIC_CONFIG cns name set "crn://my-org-name/applications/my-test-webapp" bafyreihwvu6ynmk4nfrxg2vdcx2ep3tqry775ksqyehjitj2i4kphhyuky
+
+$ laconic -c $LACONIC_CONFIG cns name set "crn://my-org-name/applications/my-test-webapp@0.1.5" bafyreihwvu6ynmk4nfrxg2vdcx2ep3tqry775ksqyehjitj2i4kphhyuky
+
+$ laconic -c $LACONIC_CONFIG cns record get --id bafyreihwvu6ynmk4nfrxg2vdcx2ep3tqry775ksqyehjitj2i4kphhyuky
+[
+  {
+    "id": "bafyreihwvu6ynmk4nfrxg2vdcx2ep3tqry775ksqyehjitj2i4kphhyuky",
+    "names": [
+      "crn://my-org-name/applications/my-test-webapp",
+      "crn://my-org-name/applications/my-test-webapp@0.1.5",
+    ],
+    "owners": [
+      "2671D38525BDC91A5DF4794EF2059D5771133702"
+    ],
+    "bondId": "0e9176d854bc3c20528b6361aab632f0b252a0f69717bf035fa68d1ef7647ba7",
+    "createTime": "2024-01-12T20:30:33Z",
+    "expiryTime": "2025-01-11T20:30:33Z",
+    "attributes": {
+      "app_type": "webapp",
+      "app_version": "0.1.5",
+      "name": "@my-org-name/my-test-webapp",
+      "repository": [
+        "https://github.com/my-org-name/my-test-webapp"
+      ],
+      "repository_ref": "v0.1.5",
+      "type": "ApplicationRecord",
+      "version": "0.0.4"
+    }
+  }
+]
+```
+
+## Application Deployment Requests
+
+To create the deployment of specific application, and `ApplicationDeploymentRequest` must be published. This request must reference the application to be deployed (`application`) and may optionally provide configuration (`config`) and a DNS name (`dns`). If no DNS name is supplied, one will be generated. A supplied DNS is usually just the short hostname, not the FQDN, since the suffix is supplied by the deployer.
+
+```
+$ cat req.yml
+record:
+  type: ApplicationDeploymentRequest
+  version: 1.0.0
+  name: "my-org-name/my-test-webapp@0.1.5"
+  application: "crn://my-org-name/applications/my-test-webapp@0.1.5"
+  dns: "my-test-app"
+  config:
+    env:
+      CERC_WEBAPP_DEBUG: my_debug_value_here
+
+$ laconic -c $LACONIC_CONFIG cns record publish -f req.yml
+bafyreihtpvwjka5ecjmca46y4dip5gb2h25vvmc7t27d7g4zecngjssvky
+```
+
+It is not necessary to assign a name to the request.
+
+## Building and Deploying
+
+Building and deploying will happen automatically for records published the production registry, but in other environments it can be triggered manually.
+
+Laconic `stack-orchestrator` is used build and launch the application. It will clone the repository, build the application, upload the container to a registry, and launch the instance in Kubernetes, with
+automatic DNS and TLS provisioning.
+
+```
+$ laconic-so deploy-webapp-from-registry \
+    --kube-config $KUBE_CONFIG \
+    --image-registry registry.mydeployer.org/app-registry \
+    --deployment-parent-dir /opt/deployments \
+    --laconic-config $LACONIC_CONFIG \
+    --dns-suffix mydeployer.servesthe.world \
+    --record-namespace-dns crn://my-deployer-org/dns \
+    --record-namespace-deployments crn://my-deployer-org/deployments \
+    --request-id bafyreihtpvwjka5ecjmca46y4dip5gb2h25vvmc7t27d7g4zecngjssvky
+```
+
+When deployment is complete, an `ApplicationDeploymentRecord` will be created:
+
+```
+$ laconic -c $LACONIC_CONFIG cns name resolve crn://my-deployer-org/deployments/my-test-app.mydeployer.servesthe.world
+[
+  {
+    "id": "bafyreiemfmxsue4svzys6tcwsqmhfjeyoo3gp63n7kydi2izbrwuzd4rga",
+    "names": [
+      "crn://my-deployer-org/deployments/my-test-app.laconic.servesthe.world"
+    ],
+    "owners": [
+      "2671D38525BDC91A5DF4794EF2059D5771133702"
+    ],
+    "bondId": "0e9176d854bc3c20528b6361aab632f0b252a0f69717bf035fa68d1ef7647ba7",
+    "createTime": "2024-01-11T20:45:29Z",
+    "expiryTime": "2025-01-10T20:45:29Z",
+    "attributes": {
+      "application": "bafyreibs6y7jhgjlsoxyqlugvkweanp3bi7ippfmhehk2rslcpoaqic2xi",
+      "dns": "bafyreicpac7tar5ua5e42zo7d5zwyp5yv7iord23p3gdox7zdfslrarjle",
+      "meta": {
+        "config": "da39a3ee5e6b4b0d3255bfef95601890afd80709",
+        "so": "82a5f9a3e4924cbb92e28be68759a487"
+      },
+      "name": "@cerc-io/my-test-webapp",
+      "request": "bafyreiex7vmtruiiasra4wvqe7wirl6iuwiso6p3cvb3sjhq55e74zh4ke",
+      "type": "ApplicationDeploymentRecord",
+      "url": "https://my-test-app.mydeployer.servesthe.world",
+      "version": "0.0.1"
+    }
+  }
+]
+```
+
+## Checking Status
+
+The status of the deployment may be checked with `stack-orchestrator`.
+
+```
+$ laconic-so deployment --dir /opt/deployments/my-test-app.snowball.servesthe.world status
+Ingress:
+        Hostname: my-test-app.mydeployer.servesthe.world
+        IP: 204.130.133.199
+        TLS: notBefore: 2024-01-12T23:11:09Z, notAfter: 2024-04-11T23:11:08Z
+
+Pods:
+        default/laconic-9560ffc64512e453-deployment-fb58d756f-l77pm: Running (2024-01-13 00:10:42+00:00)
+```
+
+# Removing Deployments
+
+As with deployment, removal involves publishing a request, which is then fulfilled by deployment processor.
+
+## Application Deployment Removal Requests
+
+```
+$ cat apprm.yml
+record:
+  type: ApplicationDeploymentRemovalRequest
+  version: 1.0.0
+  deployment: bafyreiemfmxsue4svzys6tcwsqmhfjeyoo3gp63n7kydi2izbrwuzd4rga
+
+$ laconic -c $LACONIC_CONFIG cns record publish -f apprm.yml
+bafyreiejnqhsn5ibc3c6pzlsc26co3mt63djdeuntt3gmfuckcntdmisge
+```
+
+## Removal
+
+As with deployment, for records publish to the production service removal should be processed automatically. In other environments, removal can be processed with `stack-orchestrator`.
+
+```
+$ laconic-so undeploy-webapp-from-registry   \
+    --deployment-parent-dir /opt/deployments   \
+    --laconic-config ~/.laconic/local.yml   \
+    --request-id bafyreiejnqhsn5ibc3c6pzlsc26co3mt63djdeuntt3gmfuckcntdmisge
+
+Request bafyreiejnqhsn5ibc3c6pzlsc26co3mt63djdeuntt3gmfuckcntdmisge needs to processed.
+Found 1 unsatisfied request(s) to process.
+Matched deployment ownership: 2671D38525BDC91A5DF4794EF2059D5771133702
+record:
+  type: ApplicationDeploymentRemovalRecord
+  deployment: bafyreiemfmxsue4svzys6tcwsqmhfjeyoo3gp63n7kydi2izbrwuzd4rga
+  request: bafyreiejnqhsn5ibc3c6pzlsc26co3mt63djdeuntt3gmfuckcntdmisge
+  version: 1.0.0
+```
@@ -0,0 +1,76 @@
+version: '3.2'
+
+services:
+  ajna-watcher-db:
+    restart: unless-stopped
+    image: postgres:14-alpine
+    environment:
+      - POSTGRES_USER=vdbm
+      - POSTGRES_MULTIPLE_DATABASES=ajna-watcher,ajna-watcher-job-queue
+      - POSTGRES_EXTENSION=ajna-watcher-job-queue:pgcrypto
+      - POSTGRES_PASSWORD=password
+    volumes:
+      - ../config/postgresql/multiple-postgressql-databases.sh:/docker-entrypoint-initdb.d/multiple-postgressql-databases.sh
+      - ajna_watcher_db_data:/var/lib/postgresql/data
+    ports:
+      - "5432"
+    healthcheck:
+      test: ["CMD", "nc", "-v", "localhost", "5432"]
+      interval: 20s
+      timeout: 5s
+      retries: 15
+      start_period: 10s
+
+  ajna-watcher-job-runner:
+    restart: unless-stopped
+    depends_on:
+      ajna-watcher-db:
+        condition: service_healthy
+    image: cerc/watcher-ajna:local
+    environment:
+      CERC_SCRIPT_DEBUG: ${CERC_SCRIPT_DEBUG}
+      CERC_ETH_RPC_ENDPOINT: ${CERC_ETH_RPC_ENDPOINT}
+    command: ["bash", "./start-job-runner.sh"]
+    volumes:
+      - ../config/watcher-ajna/watcher-config-template.toml:/app/environments/watcher-config-template.toml
+      - ../config/watcher-ajna/start-job-runner.sh:/app/start-job-runner.sh
+    ports:
+      - "9000"
+    healthcheck:
+      test: ["CMD", "nc", "-v", "localhost", "9000"]
+      interval: 20s
+      timeout: 5s
+      retries: 15
+      start_period: 5s
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
+
+  ajna-watcher-server:
+    restart: unless-stopped
+    depends_on:
+      ajna-watcher-db:
+        condition: service_healthy
+      ajna-watcher-job-runner:
+        condition: service_healthy
+    image: cerc/watcher-ajna:local
+    environment:
+      CERC_SCRIPT_DEBUG: ${CERC_SCRIPT_DEBUG}
+      CERC_ETH_RPC_ENDPOINT: ${CERC_ETH_RPC_ENDPOINT}
+    command: ["bash", "./start-server.sh"]
+    volumes:
+      - ../config/watcher-ajna/watcher-config-template.toml:/app/environments/watcher-config-template.toml
+      - ../config/watcher-ajna/start-server.sh:/app/start-server.sh
+    ports:
+      - "3008"
+      - "9001"
+    healthcheck:
+      test: ["CMD", "nc", "-v", "localhost", "3008"]
+      interval: 20s
+      timeout: 5s
+      retries: 15
+      start_period: 5s
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
+
+volumes:
+  ajna_watcher_db_data:
@@ -50,22 +50,6 @@ groups:
              legendFormat: __auto
              range: false
              refId: latest_external
-          - refId: latest_indexed
-            relativeTimeRange:
-              from: 600
-              to: 0
-            datasourceUid: PBFA97CFB590B2093
-            model:
-              datasource:
-                type: prometheus
-                uid: PBFA97CFB590B2093
-              editorMode: code
-              expr: sync_status_block_number{job="azimuth", instance="azimuth", kind="latest_indexed"}
-              hide: false
-              instant: true
-              legendFormat: __auto
-              range: false
-              refId: latest_indexed
          - refId: condition
            relativeTimeRange:
              from: 600
@@ -142,22 +126,6 @@ groups:
              legendFormat: __auto
              range: false
              refId: latest_external
-          - refId: latest_indexed
-            relativeTimeRange:
-              from: 600
-              to: 0
-            datasourceUid: PBFA97CFB590B2093
-            model:
-              datasource:
-                type: prometheus
-                uid: PBFA97CFB590B2093
-              editorMode: code
-              expr: sync_status_block_number{job="azimuth", instance="censures", kind="latest_indexed"}
-              hide: false
-              instant: true
-              legendFormat: __auto
-              range: false
-              refId: latest_indexed
          - refId: condition
            relativeTimeRange:
              from: 600
@@ -234,22 +202,6 @@ groups:
              legendFormat: __auto
              range: false
              refId: latest_external
-          - refId: latest_indexed
-            relativeTimeRange:
-              from: 600
-              to: 0
-            datasourceUid: PBFA97CFB590B2093
-            model:
-              datasource:
-                type: prometheus
-                uid: PBFA97CFB590B2093
-              editorMode: code
-              expr: sync_status_block_number{job="azimuth", instance="claims", kind="latest_indexed"}
-              hide: false
-              instant: true
-              legendFormat: __auto
-              range: false
-              refId: latest_indexed
          - refId: condition
            relativeTimeRange:
              from: 600
@@ -326,22 +278,6 @@ groups:
              legendFormat: __auto
              range: false
              refId: latest_external
-          - refId: latest_indexed
-            relativeTimeRange:
-              from: 600
-              to: 0
-            datasourceUid: PBFA97CFB590B2093
-            model:
-              datasource:
-                type: prometheus
-                uid: PBFA97CFB590B2093
-              editorMode: code
-              expr: sync_status_block_number{job="azimuth", instance="conditional_star_release", kind="latest_indexed"}
-              hide: false
-              instant: true
-              legendFormat: __auto
-              range: false
-              refId: latest_indexed
          - refId: condition
            relativeTimeRange:
              from: 600
@@ -418,22 +354,6 @@ groups:
              legendFormat: __auto
              range: false
              refId: latest_external
-          - refId: latest_indexed
-            relativeTimeRange:
-              from: 600
-              to: 0
-            datasourceUid: PBFA97CFB590B2093
-            model:
-              datasource:
-                type: prometheus
-                uid: PBFA97CFB590B2093
-              editorMode: code
-              expr: sync_status_block_number{job="azimuth", instance="delegated_sending", kind="latest_indexed"}
-              hide: false
-              instant: true
-              legendFormat: __auto
-              range: false
-              refId: latest_indexed
          - refId: condition
            relativeTimeRange:
              from: 600
@@ -510,22 +430,6 @@ groups:
              legendFormat: __auto
              range: false
              refId: latest_external
-          - refId: latest_indexed
-            relativeTimeRange:
-              from: 600
-              to: 0
-            datasourceUid: PBFA97CFB590B2093
-            model:
-              datasource:
-                type: prometheus
-                uid: PBFA97CFB590B2093
-              editorMode: code
-              expr: sync_status_block_number{job="azimuth", instance="ecliptic", kind="latest_indexed"}
-              hide: false
-              instant: true
-              legendFormat: __auto
-              range: false
-              refId: latest_indexed
          - refId: condition
            relativeTimeRange:
              from: 600
@@ -602,22 +506,6 @@ groups:
              legendFormat: __auto
              range: false
              refId: latest_external
-          - refId: latest_indexed
-            relativeTimeRange:
-              from: 600
-              to: 0
-            datasourceUid: PBFA97CFB590B2093
-            model:
-              datasource:
-                type: prometheus
-                uid: PBFA97CFB590B2093
-              editorMode: code
-              expr: sync_status_block_number{job="azimuth", instance="azimuth", kind="latest_indexed"}
-              hide: false
-              instant: true
-              legendFormat: __auto
-              range: false
-              refId: latest_indexed
          - refId: condition
            relativeTimeRange:
              from: 600
@@ -694,22 +582,6 @@ groups:
              legendFormat: __auto
              range: false
              refId: latest_external
-          - refId: latest_indexed
-            relativeTimeRange:
-              from: 600
-              to: 0
-            datasourceUid: PBFA97CFB590B2093
-            model:
-              datasource:
-                type: prometheus
-                uid: PBFA97CFB590B2093
-              editorMode: code
-              expr: sync_status_block_number{job="azimuth", instance="polls", kind="latest_indexed"}
-              hide: false
-              instant: true
-              legendFormat: __auto
-              range: false
-              refId: latest_indexed
          - refId: condition
            relativeTimeRange:
              from: 600
@@ -788,22 +660,6 @@ groups:
              legendFormat: __auto
              range: false
              refId: latest_external
-          - refId: latest_indexed
-            relativeTimeRange:
-              from: 600
-              to: 0
-            datasourceUid: PBFA97CFB590B2093
-            model:
-              datasource:
-                type: prometheus
-                uid: PBFA97CFB590B2093
-              editorMode: code
-              expr: sync_status_block_number{job="sushi", instance="sushiswap", kind="latest_indexed"}
-              hide: false
-              instant: true
-              legendFormat: __auto
-              range: false
-              refId: latest_indexed
          - refId: condition
            relativeTimeRange:
              from: 600
@@ -880,22 +736,6 @@ groups:
              legendFormat: __auto
              range: false
              refId: latest_external
-          - refId: latest_indexed
-            relativeTimeRange:
-              from: 600
-              to: 0
-            datasourceUid: PBFA97CFB590B2093
-            model:
-              datasource:
-                type: prometheus
-                uid: PBFA97CFB590B2093
-              editorMode: code
-              expr: sync_status_block_number{job="sushi", instance="merkl_sushiswap", kind="latest_indexed"}
-              hide: false
-              instant: true
-              legendFormat: __auto
-              range: false
-              refId: latest_indexed
          - refId: condition
            relativeTimeRange:
              from: 600
@@ -0,0 +1,20 @@
+#!/bin/sh
+
+set -e
+if [ -n "$CERC_SCRIPT_DEBUG" ]; then
+  set -x
+fi
+set -u
+
+echo "Using ETH RPC endpoint ${CERC_ETH_RPC_ENDPOINT}"
+
+# Read in the config template TOML file and modify it
+WATCHER_CONFIG_TEMPLATE=$(cat environments/watcher-config-template.toml)
+WATCHER_CONFIG=$(echo "$WATCHER_CONFIG_TEMPLATE" | \
+  sed -E "s|REPLACE_WITH_CERC_ETH_RPC_ENDPOINT|${CERC_ETH_RPC_ENDPOINT}| ")
+
+# Write the modified content to a new file
+echo "$WATCHER_CONFIG" > environments/local.toml
+
+echo "Running job-runner..."
+DEBUG=vulcanize:* exec node --enable-source-maps dist/job-runner.js
@@ -0,0 +1,20 @@
+#!/bin/sh
+
+set -e
+if [ -n "$CERC_SCRIPT_DEBUG" ]; then
+  set -x
+fi
+set -u
+
+echo "Using ETH RPC endpoint ${CERC_ETH_RPC_ENDPOINT}"
+
+# Read in the config template TOML file and modify it
+WATCHER_CONFIG_TEMPLATE=$(cat environments/watcher-config-template.toml)
+WATCHER_CONFIG=$(echo "$WATCHER_CONFIG_TEMPLATE" | \
+  sed -E "s|REPLACE_WITH_CERC_ETH_RPC_ENDPOINT|${CERC_ETH_RPC_ENDPOINT}| ")
+
+# Write the modified content to a new file
+echo "$WATCHER_CONFIG" > environments/local.toml
+
+echo "Running server..."
+DEBUG=vulcanize:* exec node --enable-source-maps dist/server.js
@@ -0,0 +1,98 @@
+[server]
+  host = "0.0.0.0"
+  port = 3008
+  kind = "active"
+  gqlPath = "/graphql"
+
+  # Checkpointing state.
+  checkpointing = true
+
+  # Checkpoint interval in number of blocks.
+  checkpointInterval = 2000
+
+  # Enable state creation
+  # CAUTION: Disable only if state creation is not desired or can be filled subsequently
+  enableState = false
+
+  subgraphPath = "./subgraph-build"
+
+  # Interval to restart wasm instance periodically
+  wasmRestartBlocksInterval = 20
+
+  # Interval in number of blocks at which to clear entities cache.
+  clearEntitiesCacheInterval = 1000
+
+  # Max block range for which to return events in eventsInRange GQL query.
+  # Use -1 for skipping check on block range.
+  maxEventsBlockRange = 1000
+
+  # Flag to specify whether RPC endpoint supports block hash as block tag parameter
+  rpcSupportsBlockHashParam = false
+
+  # GQL cache settings
+  [server.gqlCache]
+    enabled = true
+
+    # Max in-memory cache size (in bytes) (default 8 MB)
+    # maxCacheSize
+
+    # GQL cache-control max-age settings (in seconds)
+    maxAge = 15
+    timeTravelMaxAge = 86400 # 1 day
+
+[metrics]
+  host = "0.0.0.0"
+  port = 9000
+  [metrics.gql]
+    port = 9001
+
+[database]
+  type = "postgres"
+  host = "ajna-watcher-db"
+  port = 5432
+  database = "ajna-watcher"
+  username = "vdbm"
+  password = "password"
+  synchronize = true
+  logging = false
+
+[upstream]
+  [upstream.ethServer]
+    rpcProviderEndpoint = "REPLACE_WITH_CERC_ETH_RPC_ENDPOINT"
+
+    # Boolean flag to specify if rpc-eth-client should be used for RPC endpoint instead of ipld-eth-client (ipld-eth-server GQL client)
+    rpcClient = true
+
+    # Boolean flag to specify if rpcProviderEndpoint is an FEVM RPC endpoint
+    isFEVM = true
+
+    # Boolean flag to filter event logs by contracts
+    filterLogsByAddresses = true
+    # Boolean flag to filter event logs by topics
+    filterLogsByTopics = true
+
+  [upstream.cache]
+    name = "requests"
+    enabled = false
+    deleteOnStart = false
+
+[jobQueue]
+  dbConnectionString = "postgres://vdbm:password@ajna-watcher-db/ajna-watcher-job-queue"
+  maxCompletionLagInSecs = 300
+  jobDelayInMilliSecs = 100
+  eventsInBatch = 50
+  subgraphEventsOrder = true
+  # Filecoin block time: https://docs.filecoin.io/basics/the-blockchain/blocks-and-tipsets#blocktime
+  blockDelayInMilliSecs = 30000
+
+  # Boolean to switch between modes of processing events when starting the server.
+  # Setting to true will fetch filtered events and required blocks in a range of blocks and then process them.
+  # Setting to false will fetch blocks consecutively with its events and then process them (Behaviour is followed in realtime processing near head).
+  useBlockRanges = true
+
+  # Block range in which logs are fetched during historical blocks processing
+  historicalLogsBlockRange = 2000
+
+  # Max block range of historical processing after which it waits for completion of events processing
+  # If set to -1 historical processing does not wait for events processing and completes till latest canonical block
+  historicalMaxFetchAhead = 10000
@@ -5,7 +5,7 @@ if [ -n "$CERC_SCRIPT_DEBUG" ]; then
  set -x
 fi

-registry_command="laconic cns"
+registry_command="laconic registry"
 demo_records_dir="scripts/demo-records"

 # Check we have funds
@@ -0,0 +1,10 @@
+FROM node:18.17.1-alpine3.18
+
+RUN apk --update --no-cache add git python3 alpine-sdk bash curl jq
+
+WORKDIR /app
+
+COPY . .
+
+RUN echo "Installing dependencies and building ajna-watcher-ts" && \
+  yarn && yarn build
@@ -0,0 +1,9 @@
+#!/usr/bin/env bash
+# Build cerc/watcher-ajna
+
+source ${CERC_CONTAINER_BASE_DIR}/build-base.sh
+
+# See: https://stackoverflow.com/a/246128/1701505
+SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
+
+docker build -t cerc/watcher-ajna:local -f ${SCRIPT_DIR}/Dockerfile ${build_command_args} ${CERC_REPO_BASE_DIR}/ajna-watcher-ts
@@ -0,0 +1,118 @@
+# Ajna Watcher
+
+## Setup
+
+Clone required repositories:
+
+```bash
+laconic-so --stack ajna setup-repositories --git-ssh --pull
+```
+
+Build the container images:
+
+```bash
+laconic-so --stack ajna build-containers
+```
+
+## Deploy
+
+Create a spec file for the deployment:
+
+```bash
+laconic-so --stack ajna deploy init --output ajna-spec.yml
+```
+
+### Ports
+
+Edit `network` in the spec file to map container ports to host ports as required:
+
+```yml
+...
+network:
+  ports:
+    ajna-watcher-db:
+     - 15432:5432
+    ajna-watcher-job-runner:
+     - 9000:9000
+    ajna-watcher-server:
+     - 3008:3008
+     - 9001:9001
+```
+
+### Create a deployment
+
+Create a deployment from the spec file:
+
+```bash
+laconic-so --stack ajna deploy create --spec-file ajna-spec.yml --deployment-dir ajna-deployment
+```
+
+### Configuration
+
+Inside deployment directory, open the `config.env` file  and set following env variables:
+
+```bash
+# External Filecoin (ETH RPC) endpoint to point the watcher to
+CERC_ETH_RPC_ENDPOINT=https://example-lotus-endpoint/rpc/v1
+```
+
+### Start the deployment
+
+```bash
+laconic-so deployment --dir ajna-deployment start
+```
+
+* To list down and monitor the running containers:
+
+  ```bash
+  # With status
+  docker ps -a
+
+  # Check logs for a container
+  docker logs -f <CONTAINER_ID>
+  ```
+
+* Open the GQL playground at <http://localhost:3008/graphql>
+
+  ```graphql
+  # Example query
+  query {
+    _meta {
+      block {
+        hash
+        number
+        timestamp
+      }
+      deployment
+      hasIndexingErrors
+    }
+
+    accounts {
+      id
+      txCount
+      tokensDelegated
+      rewardsClaimed
+    }
+  }
+  ```
+
+## Clean up
+
+Stop all the ajna services running in background:
+
+```bash
+# Only stop the docker containers
+laconic-so deployment --dir ajna-deployment stop
+
+# Run 'start' to restart the deployment
+```
+
+To stop all the ajna services and also delete data:
+
+```bash
+# Stop the docker containers
+laconic-so deployment --dir ajna-deployment stop --delete-volumes
+
+# Remove deployment directory (deployment will have to be recreated for a re-run)
+rm -r ajna-deployment
+```
@@ -0,0 +1,9 @@
+version: "1.0"
+name: ajna
+description: "Ajna watcher stack"
+repos:
+  - git.vdb.to/cerc-io/ajna-watcher-ts
+containers:
+  - cerc/watcher-ajna
+pods:
+  - watcher-ajna
@@ -49,7 +49,7 @@ $ laconic-so --stack fixturenet-laconic-loaded deploy logs
 ```
 ## 6. Test with the Registry CLI
 ```
-$ laconic-so --stack fixturenet-laconic-loaded deploy exec cli "laconic cns status"
+$ laconic-so --stack fixturenet-laconic-loaded deploy exec cli "laconic registry status"
 ```
 ## 7. View the laconic console
 Get the URL for the console web app with this command (the port number will be different for each deployment):
@@ -44,5 +44,5 @@ $ laconic-so --stack fixturenet-laconicd deploy logs
 ```
 ## 6. Test with the Registry CLI
 ```
-$ laconic-so --stack fixturenet-laconicd deploy exec cli "laconic cns status"
+$ laconic-so --stack fixturenet-laconicd deploy exec cli "laconic registry status"
 ```
@@ -16,26 +16,55 @@ laconic-so --stack merkl-sushiswap-v3 build-containers

 ## Deploy

-### Configuration
-
-Create and update an env file to be used in the next step:
-
-  ```bash
-  # External Filecoin (ETH RPC) endpoint to point the watcher
-  CERC_ETH_RPC_ENDPOINT=
-  ```
-
-### Deploy the stack
+Create a spec file for the deployment:

 ```bash
-laconic-so --stack merkl-sushiswap-v3 deploy --cluster merkl_sushiswap_v3 --env-file <PATH_TO_ENV_FILE> up
+laconic-so --stack merkl-sushiswap-v3 deploy init --output merkl-sushiswap-v3-spec.yml
+```
+
+### Ports
+
+Edit `network` in the spec file to map container ports to host ports as required:
+
+```
+...
+network:
+  ports:
+    merkl-sushiswap-v3-watcher-db:
+     - '5432'
+    merkl-sushiswap-v3-watcher-job-runner:
+     - 9002:9000
+    merkl-sushiswap-v3-watcher-server:
+     - 127.0.0.1:3007:3008
+     - 9003:9001
+```
+
+### Create a deployment
+
+Create a deployment from the spec file:
+
+```bash
+laconic-so --stack merkl-sushiswap-v3 deploy create --spec-file merkl-sushiswap-v3-spec.yml --deployment-dir merkl-sushiswap-v3-deployment
+```
+
+### Configuration
+
+Inside deployment directory, open the `config.env` file  and set following env variables:
+
+```bash
+# External Filecoin (ETH RPC) endpoint to point the watcher to
+CERC_ETH_RPC_ENDPOINT=https://example-lotus-endpoint/rpc/v1
+```
+
+### Start the deployment
+
+```bash
+laconic-so deployment --dir merkl-sushiswap-v3-deployment start
 ```

 * To list down and monitor the running containers:

  ```bash
-  laconic-so --stack merkl-sushiswap-v3 deploy --cluster merkl_sushiswap_v3 ps
-
  # With status
  docker ps -a

@@ -46,6 +75,7 @@ laconic-so --stack merkl-sushiswap-v3 deploy --cluster merkl_sushiswap_v3 --env-
 * Open the GQL playground at http://localhost:3007/graphql

  ```graphql
+  # Example query
  {
    _meta {
      block {
@@ -54,7 +84,7 @@ laconic-so --stack merkl-sushiswap-v3 deploy --cluster merkl_sushiswap_v3 --env-
      }
      hasIndexingErrors
    }
-    
+
    factories {
      id
      poolCount
@@ -64,18 +94,21 @@ laconic-so --stack merkl-sushiswap-v3 deploy --cluster merkl_sushiswap_v3 --env-

 ## Clean up

-Stop all the services running in background:
+Stop all the merkl-sushiswap-v3 services running in background:

 ```bash
-laconic-so --stack merkl-sushiswap-v3 deploy --cluster merkl_sushiswap_v3 down
+# Only stop the docker containers
+laconic-so deployment --dir merkl-sushiswap-v3-deployment stop
+
+# Run 'start' to restart the deployment
 ```

-Clear volumes created by this stack:
+To stop all the merkl-sushiswap-v3 services and also delete data:

 ```bash
-# List all relevant volumes
-docker volume ls -q --filter "name=merkl_sushiswap_v3"
+# Stop the docker containers
+laconic-so deployment --dir merkl-sushiswap-v3-deployment stop --delete-volumes

-# Remove all the listed volumes
-docker volume rm $(docker volume ls -q --filter "name=merkl_sushiswap_v3")
+# Remove deployment directory (deployment will have to be recreated for a re-run)
+rm -r merkl-sushiswap-v3-deployment
 ```
@@ -2,7 +2,7 @@ version: "1.0"
 name: merkl-sushiswap-v3
 description: "SushiSwap v3 watcher stack"
 repos:
-  - github.com/cerc-io/merkl-sushiswap-v3-watcher-ts@v0.1.6
+  - github.com/cerc-io/merkl-sushiswap-v3-watcher-ts@v0.1.7
 containers:
  - cerc/watcher-merkl-sushiswap-v3
 pods:
@@ -1,7 +1,7 @@
 version: "0.1"
 name: monitoring
 repos:
-  - github.com/cerc-io/watcher-ts@v0.2.79
+  - github.com/cerc-io/watcher-ts@v0.2.81
 containers:
  - cerc/watcher-ts
 pods:
@@ -55,7 +55,7 @@ ports:
 Create deployment:

 ```bash
-laconic-so deploy create --spec-file sushiswap-subgraph-spec.yml --deployment-dir sushiswap-subgraph-deployment
+laconic-so --stack sushiswap-subgraph deploy create --spec-file sushiswap-subgraph-spec.yml --deployment-dir sushiswap-subgraph-deployment
 ```

 ## Start the stack
@@ -16,26 +16,55 @@ laconic-so --stack sushiswap-v3 build-containers

 ## Deploy

-### Configuration
-
-Create and update an env file to be used in the next step:
-
-  ```bash
-  # External Filecoin (ETH RPC) endpoint to point the watcher
-  CERC_ETH_RPC_ENDPOINT=
-  ```
-
-### Deploy the stack
+Create a spec file for the deployment:

 ```bash
-laconic-so --stack sushiswap-v3 deploy --cluster sushiswap_v3 --env-file <PATH_TO_ENV_FILE> up
+laconic-so --stack sushiswap-v3 deploy init --output sushiswap-v3-spec.yml
+```
+
+### Ports
+
+Edit `network` in the spec file to map container ports to host ports as required:
+
+```
+...
+network:
+  ports:
+    sushiswap-v3-watcher-db:
+     - '5432'
+    sushiswap-v3-watcher-job-runner:
+     - 9000:9000
+    sushiswap-v3-watcher-server:
+     - 127.0.0.1:3008:3008
+     - 9001:9001
+```
+
+### Create a deployment
+
+Create a deployment from the spec file:
+
+```bash
+laconic-so --stack sushiswap-v3 deploy create --spec-file sushiswap-v3-spec.yml --deployment-dir sushiswap-v3-deployment
+```
+
+### Configuration
+
+Inside deployment directory, open the `config.env` file  and set following env variables:
+
+```bash
+# External Filecoin (ETH RPC) endpoint to point the watcher to
+CERC_ETH_RPC_ENDPOINT=https://example-lotus-endpoint/rpc/v1
+```
+
+### Start the deployment
+
+```bash
+laconic-so deployment --dir sushiswap-v3-deployment start
 ```

 * To list down and monitor the running containers:

  ```bash
-  laconic-so --stack sushiswap-v3 deploy --cluster sushiswap_v3 ps
-
  # With status
  docker ps -a

@@ -43,20 +72,43 @@ laconic-so --stack sushiswap-v3 deploy --cluster sushiswap_v3 --env-file <PATH_T
  docker logs -f <CONTAINER_ID>
  ```

+* Open the GQL playground at http://localhost:3008/graphql
+
+  ```graphql
+  # Example query
+  {
+    _meta {
+      block {
+        number
+        timestamp
+      }
+      hasIndexingErrors
+    }
+
+    factories {
+      id
+      poolCount
+    }
+  }
+  ```
+
 ## Clean up

-Stop all the services running in background:
+Stop all the sushiswap-v3 services running in background:

 ```bash
-laconic-so --stack sushiswap-v3 deploy --cluster sushiswap_v3 down
+# Only stop the docker containers
+laconic-so deployment --dir sushiswap-v3-deployment stop
+
+# Run 'start' to restart the deployment
 ```

-Clear volumes created by this stack:
+To stop all the sushiswap-v3 services and also delete data:

 ```bash
-# List all relevant volumes
-docker volume ls -q --filter "name=sushiswap_v3"
+# Stop the docker containers
+laconic-so deployment --dir sushiswap-v3-deployment stop --delete-volumes

-# Remove all the listed volumes
-docker volume rm $(docker volume ls -q --filter "name=sushiswap_v3")
+# Remove deployment directory (deployment will have to be recreated for a re-run)
+rm -r sushiswap-v3-deployment
 ```
@@ -2,7 +2,7 @@ version: "1.0"
 name: sushiswap-v3
 description: "SushiSwap v3 watcher stack"
 repos:
-  - github.com/cerc-io/sushiswap-v3-watcher-ts@v0.1.6
+  - github.com/cerc-io/sushiswap-v3-watcher-ts@v0.1.7
 containers:
  - cerc/watcher-sushiswap-v3
 pods:
@@ -94,7 +94,7 @@ class LaconicRegistryClient:
        )

    def list_records(self, criteria={}, all=False):
-        args = ["laconic", "-c", self.config_file, "cns", "record", "list"]
+        args = ["laconic", "-c", self.config_file, "registry", "record", "list"]

        if all:
            args.append("--all")
@@ -140,7 +140,7 @@ class LaconicRegistryClient:
        if name in self.cache.name_or_id:
            return self.cache.name_or_id[name]

-        args = ["laconic", "-c", self.config_file, "cns", "name", "resolve", name]
+        args = ["laconic", "-c", self.config_file, "registry", "name", "resolve", name]

        parsed = [AttrDict(r) for r in json.loads(logged_cmd(self.log_file, *args))]
        if parsed:
@@ -165,7 +165,7 @@ class LaconicRegistryClient:
            "laconic",
            "-c",
            self.config_file,
-            "cns",
+            "registry",
            "record",
            "get",
            "--id",
@@ -203,7 +203,8 @@ class LaconicRegistryClient:
            print(open(record_fname, 'r').read(), file=self.log_file)

            new_record_id = json.loads(
-                logged_cmd(self.log_file, "laconic", "-c", self.config_file, "cns", "record", "publish", "--filename", record_fname)
+                logged_cmd(self.log_file, "laconic", "-c", self.config_file, "registry",
+                           "record", "publish", "--filename", record_fname)
            )["id"]
            for name in names:
                self.set_name(name, new_record_id)
@@ -212,10 +213,10 @@ class LaconicRegistryClient:
            logged_cmd(self.log_file, "rm", "-rf", tmpdir)

    def set_name(self, name, record_id):
-        logged_cmd(self.log_file, "laconic", "-c", self.config_file, "cns", "name", "set", name, record_id)
+        logged_cmd(self.log_file, "laconic", "-c", self.config_file, "registry", "name", "set", name, record_id)

    def delete_name(self, name):
-        logged_cmd(self.log_file, "laconic", "-c", self.config_file, "cns", "name", "delete", name)
+        logged_cmd(self.log_file, "laconic", "-c", self.config_file, "registry", "name", "delete", name)


 def file_hash(filename):
Author	SHA1	Message	Date
zramsay	22febffb09	WIP	2024-03-28 16:24:36 -04:00
telackey	5308ab1e4e	Blind commit to fix laconic CLI calls after rename. (#784 ) Lint Checks / Run linter (push) Successful in 42s Details Publish / Build and publish (push) Successful in 1m25s Details Deploy Test / Run deploy test suite (push) Successful in 5m20s Details Webapp Test / Run webapp test suite (push) Successful in 4m32s Details Smoke Test / Run basic test suite (push) Successful in 5m47s Details `laconic cns` got renamed to `laconic registry` which breaks all the scripts and commands that use it. Reviewed-on: #784 Co-authored-by: Thomas E Lackey <telackey@bozemanpass.com> Co-committed-by: Thomas E Lackey <telackey@bozemanpass.com>	2024-03-25 19:09:26 +00:00
prathamesh	cd50832038	Add a Ajna watcher stack (#781 ) Fixturenet-Laconicd-Test / Run an Laconicd fixturenet test (push) Successful in 11m14s Details Fixturenet-Eth-Plugeth-Arm-Test / Run an Ethereum plugeth fixturenet test (push) Successful in 51m28s Details Fixturenet-Eth-Plugeth-Test / Run an Ethereum plugeth fixturenet test (push) Successful in 51m45s Details K8s Deploy Test / Run deploy test suite on kind/k8s (push) Successful in 8m38s Details Database Test / Run database hosting test on kind/k8s (push) Successful in 11m46s Details Container Registry Test / Run contaier registry hosting test on kind/k8s (push) Successful in 4m57s Details Lint Checks / Run linter (push) Successful in 41s Details Publish / Build and publish (push) Successful in 1m20s Details Deploy Test / Run deploy test suite (push) Successful in 4m36s Details Webapp Test / Run webapp test suite (push) Successful in 4m11s Details Smoke Test / Run basic test suite (push) Successful in 4m29s Details Part of https://www.notion.so/Generate-ajna-finance-subgraph-watcher-with-codegen-5b80ac149b3f449fb138f5d92cc5485e Reviewed-on: #781 Co-authored-by: Prathamesh Musale <prathamesh.musale0@gmail.com> Co-committed-by: Prathamesh Musale <prathamesh.musale0@gmail.com>	2024-03-21 07:17:01 +00:00
prathamesh	aeddc82ebc	Remove latest indexed block value from watcher alerts data (#780 ) Lint Checks / Run linter (push) Successful in 40s Details Publish / Build and publish (push) Successful in 1m21s Details Webapp Test / Run webapp test suite (push) Successful in 4m34s Details Deploy Test / Run deploy test suite (push) Successful in 5m42s Details Smoke Test / Run basic test suite (push) Successful in 4m50s Details Fixturenet-Laconicd-Test / Run an Laconicd fixturenet test (push) Successful in 8m55s Details Fixturenet-Eth-Plugeth-Test / Run an Ethereum plugeth fixturenet test (push) Successful in 50m50s Details Fixturenet-Eth-Plugeth-Arm-Test / Run an Ethereum plugeth fixturenet test (push) Successful in 54m1s Details K8s Deploy Test / Run deploy test suite on kind/k8s (push) Successful in 9m5s Details Database Test / Run database hosting test on kind/k8s (push) Successful in 10m40s Details Container Registry Test / Run contaier registry hosting test on kind/k8s (push) Successful in 4m20s Details Part of https://www.notion.so/Setup-grafana-SO-stack-for-monitoring-watchers-7e23042c296c4de6b8676f1f604aa03c Reviewed-on: #780 Co-authored-by: Prathamesh Musale <prathamesh.musale0@gmail.com> Co-committed-by: Prathamesh Musale <prathamesh.musale0@gmail.com>	2024-03-13 07:16:15 +00:00
prathamesh	17e860d6e4	Update subgraph watcher versions and instructions to use deployments (#775 ) Fixturenet-Laconicd-Test / Run an Laconicd fixturenet test (push) Successful in 9m39s Details Fixturenet-Eth-Plugeth-Arm-Test / Run an Ethereum plugeth fixturenet test (push) Successful in 53m46s Details Fixturenet-Eth-Plugeth-Test / Run an Ethereum plugeth fixturenet test (push) Successful in 55m59s Details K8s Deploy Test / Run deploy test suite on kind/k8s (push) Successful in 8m44s Details Database Test / Run database hosting test on kind/k8s (push) Successful in 10m32s Details Container Registry Test / Run contaier registry hosting test on kind/k8s (push) Successful in 4m19s Details Lint Checks / Run linter (push) Successful in 38s Details Publish / Build and publish (push) Successful in 1m16s Details Webapp Test / Run webapp test suite (push) Successful in 4m10s Details Deploy Test / Run deploy test suite (push) Successful in 5m30s Details Smoke Test / Run basic test suite (push) Successful in 4m53s Details Part of https://www.notion.so/Setup-watchers-on-sandman-34b5514a10634c6fbf3ec338967c871c Reviewed-on: #775 Co-authored-by: Prathamesh Musale <prathamesh.musale0@gmail.com> Co-committed-by: Prathamesh Musale <prathamesh.musale0@gmail.com>	2024-03-12 05:32:55 +00:00