Add panels for GQL metrics to watcher dashboard (#834 )

Part of [Metrics and logging for GQL queries in watcher](https://www.notion.so/Metrics-and-logging-for-GQL-queries-in-watcher-928c692292b140a2a4f52cda9795df5e) Reviewed-on: cerc-io/stack-orchestrator#834 Co-authored-by: Prathamesh Musale <prathamesh.musale0@gmail.com> Co-committed-by: Prathamesh Musale <prathamesh.musale0@gmail.com>
Add dashboard for graph-node subgraphs (#832 )
2024-06-06 11:47:18 +00:00 · 2024-06-04 07:21:27 +00:00 · 2024-05-23 04:12:31 +00:00 · 2024-05-22 19:00:42 +00:00 · 2024-05-22 18:31:57 +00:00 · 2024-05-22 18:18:27 +00:00
64 changed files with 3059 additions and 514 deletions
@@ -2,15 +2,13 @@ name: Fixturenet-Eth-Plugeth-Arm-Test

 on:
  push:
-    branches: 'main'
+    branches: '*'
    paths:
      - '!**'
      - '.gitea/workflows/triggers/fixturenet-eth-plugeth-arm-test'
-  pull_request:
-    paths:
-      - '.gitea/workflows/triggers/fixturenet-eth-plugeth-arm-test'
  schedule: # Note: coordinate with other tests to not overload runners at the same time of day
-    - cron: '0 11 * * *'
+    - cron: '2 14 * * *'
+

 jobs:
  test:
@@ -2,15 +2,14 @@ name: Fixturenet-Eth-Plugeth-Test

 on:
  push:
-    branches: 'main'
-    paths:
-      - '.gitea/workflows/triggers/fixturenet-eth-plugeth-test'
-  pull_request:
+    branches: '*'
    paths:
+      - '!**'
      - '.gitea/workflows/triggers/fixturenet-eth-plugeth-test'
  schedule: # Note: coordinate with other tests to not overload runners at the same time of day
    - cron: '2 14 * * *'

+
 jobs:
  test:
    name: "Run an Ethereum plugeth fixturenet test"
@@ -2,13 +2,12 @@ name: Fixturenet-Eth-Test

 on:
  push:
-    branches: 'main'
-    paths:
-      - '.gitea/workflows/triggers/fixturenet-eth-test'
-  pull_request:
+    branches: '*'
    paths:
+      - '!**'
      - '.gitea/workflows/triggers/fixturenet-eth-test'

+
 jobs:
  test:
    name: "Run an Ethereum fixturenet test"
@@ -10,9 +10,6 @@ on:
    paths-ignore:
      - '.gitea/workflows/triggers/*'

-# Needed until we can incorporate docker startup into the executor container
-env:
-  DOCKER_HOST: unix:///var/run/dind.sock

 jobs:
  test:
@@ -41,10 +38,6 @@ jobs:
        run: ./scripts/create_build_tag_file.sh
      - name: "Build local shiv package"
        run: ./scripts/build_shiv_package.sh
-      - name: Start dockerd # Also needed until we can incorporate into the executor
-        run: |
-          dockerd -H $DOCKER_HOST --userland-proxy=false &
-          sleep 5
      - name: "Run deploy tests"
        run: ./tests/deploy/run-deploy-test.sh
      - name: Notify Vulcanize Slack on CI failure
@@ -10,10 +10,6 @@ on:
    paths-ignore:
      - '.gitea/workflows/triggers/*'

-# Needed until we can incorporate docker startup into the executor container
-env:
-  DOCKER_HOST: unix:///var/run/dind.sock
-
 jobs:
  test:
    name: "Run webapp test suite"
@@ -43,10 +39,6 @@ jobs:
        run: ./scripts/build_shiv_package.sh
      - name: "Install wget" # 20240109 - Only needed until the executors are updated.
        run: apt update && apt install -y wget
-      - name: Start dockerd # Also needed until we can incorporate into the executor
-        run: |
-          dockerd -H $DOCKER_HOST --userland-proxy=false &
-          sleep 5
      - name: "Run webapp tests"
        run: ./tests/webapp-test/run-webapp-test.sh
      - name: Notify Vulcanize Slack on CI failure
@@ -10,9 +10,6 @@ on:
    paths-ignore:
      - '.gitea/workflows/triggers/*'

-# Needed until we can incorporate docker startup into the executor container
-env:
-  DOCKER_HOST: unix:///var/run/dind.sock

 jobs:
  test:
@@ -41,10 +38,6 @@ jobs:
        run: ./scripts/create_build_tag_file.sh
      - name: "Build local shiv package"
        run: ./scripts/build_shiv_package.sh
-      - name: Start dockerd # Also needed until we can incorporate into the executor
-        run: |
-          dockerd -H $DOCKER_HOST --userland-proxy=false &
-          sleep 5
      - name: "Run smoke tests"
        run: ./tests/smoke-test/run-smoke-test.sh
      - name: Notify Vulcanize Slack on CI failure
@@ -1,3 +1,2 @@
 Change this file to trigger running the fixturenet-eth-plugeth-arm-test CI job
-1

@@ -1,4 +1,3 @@
 Change this file to trigger running the fixturenet-eth-plugeth-test CI job
 trigger
 trigger
-1
@@ -1,4 +1,2 @@
 Change this file to trigger running the fixturenet-eth-test CI job
-1
-1

@@ -6,12 +6,20 @@ services:
    restart: always
    environment:
      GF_SERVER_ROOT_URL: ${GF_SERVER_ROOT_URL}
+      CERC_GRAFANA_ALERTS_SUBGRAPH_IDS: ${CERC_GRAFANA_ALERTS_SUBGRAPH_IDS}
    volumes:
      - ../config/monitoring/grafana/provisioning:/etc/grafana/provisioning
      - ../config/monitoring/grafana/dashboards:/etc/grafana/dashboards
+      - ../config/monitoring/update-grafana-alerts-config.sh:/update-grafana-alerts-config.sh
      - grafana_storage:/var/lib/grafana
+    user: root
+    entrypoint: ["bash", "-c"]
+    command: |
+      "/update-grafana-alerts-config.sh && /run.sh"
    ports:
      - "3000"
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
    healthcheck:
      test: ["CMD", "nc", "-vz", "localhost", "3000"]
      interval: 30s
@@ -22,6 +22,7 @@ services:
      GRAPH_ETHEREUM_JSON_RPC_TIMEOUT: ${GRAPH_ETHEREUM_JSON_RPC_TIMEOUT:-180}
      GRAPH_ETHEREUM_REQUEST_RETRIES: ${GRAPH_ETHEREUM_REQUEST_RETRIES:-10}
      GRAPH_ETHEREUM_MAX_BLOCK_RANGE_SIZE: ${GRAPH_ETHEREUM_MAX_BLOCK_RANGE_SIZE:-2000}
+      GRAPH_ETHEREUM_BLOCK_INGESTOR_MAX_CONCURRENT_JSON_RPC_CALLS_FOR_TXN_RECEIPTS: ${GRAPH_ETHEREUM_BLOCK_INGESTOR_MAX_CONCURRENT_JSON_RPC_CALLS_FOR_TXN_RECEIPTS:-1000}
    entrypoint: ["bash", "-c"]
    # Wait for ETH RPC endpoint to be up when running with fixturenet-lotus
    command: |
@@ -31,6 +32,7 @@ services:
      - "8001"
      - "8020"
      - "8030"
+      - "8040"
    healthcheck:
      test: ["CMD", "nc", "-vz", "localhost", "8020"]
      interval: 30s
@@ -28,15 +28,37 @@ services:
    extra_hosts:
      - "host.docker.internal:host-gateway"

-  chain-head-exporter:
+  ethereum-chain-head-exporter:
    image: cerc/watcher-ts:local
    restart: always
    working_dir: /app/packages/cli
    environment:
-      ETH_RPC_ENDPOINT: ${CERC_ETH_RPC_ENDPOINT}
-      FIL_RPC_ENDPOINT: ${CERC_FIL_RPC_ENDPOINT}
+      ETH_RPC_ENDPOINT: ${CERC_ETH_RPC_ENDPOINT:-https://mainnet.infura.io/v3}
      ETH_RPC_API_KEY: ${CERC_INFURA_KEY}
-      PORT: ${CERC_METRICS_PORT}
+    command: ["sh", "-c", "yarn export-metrics:chain-heads"]
+    ports:
+      - '5000'
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
+
+  filecoin-chain-head-exporter:
+    image: cerc/watcher-ts:local
+    restart: always
+    working_dir: /app/packages/cli
+    environment:
+      ETH_RPC_ENDPOINT: ${CERC_FIL_RPC_ENDPOINT:-https://api.node.glif.io/rpc/v1}
+    command: ["sh", "-c", "yarn export-metrics:chain-heads"]
+    ports:
+      - '5000'
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
+
+  graph-node-upstream-head-exporter:
+    image: cerc/watcher-ts:local
+    restart: always
+    working_dir: /app/packages/cli
+    environment:
+      ETH_RPC_ENDPOINT: ${GRAPH_NODE_RPC_ENDPOINT}
    command: ["sh", "-c", "yarn export-metrics:chain-heads"]
    ports:
      - '5000'
@@ -29,7 +29,7 @@ services:
    image: cerc/watcher-ajna:local
    environment:
      CERC_SCRIPT_DEBUG: ${CERC_SCRIPT_DEBUG}
-      CERC_ETH_RPC_ENDPOINT: ${CERC_ETH_RPC_ENDPOINT}
+      CERC_ETH_RPC_ENDPOINTS: ${CERC_ETH_RPC_ENDPOINTS}
    command: ["bash", "./start-job-runner.sh"]
    volumes:
      - ../config/watcher-ajna/watcher-config-template.toml:/app/environments/watcher-config-template.toml
@@ -37,7 +37,7 @@ services:
    ports:
      - "9000"
    healthcheck:
-      test: ["CMD", "nc", "-v", "localhost", "9000"]
+      test: ["CMD", "nc", "-vz", "127.0.0.1", "9000"]
      interval: 20s
      timeout: 5s
      retries: 15
@@ -55,7 +55,7 @@ services:
    image: cerc/watcher-ajna:local
    environment:
      CERC_SCRIPT_DEBUG: ${CERC_SCRIPT_DEBUG}
-      CERC_ETH_RPC_ENDPOINT: ${CERC_ETH_RPC_ENDPOINT}
+      CERC_ETH_RPC_ENDPOINTS: ${CERC_ETH_RPC_ENDPOINTS}
    command: ["bash", "./start-server.sh"]
    volumes:
      - ../config/watcher-ajna/watcher-config-template.toml:/app/environments/watcher-config-template.toml
@@ -64,7 +64,7 @@ services:
      - "3008"
      - "9001"
    healthcheck:
-      test: ["CMD", "nc", "-v", "localhost", "3008"]
+      test: ["CMD", "nc", "-vz", "127.0.0.1", "3008"]
      interval: 20s
      timeout: 5s
      retries: 15
@@ -32,8 +32,8 @@ services:
        condition: service_healthy
    environment:
      CERC_SCRIPT_DEBUG: ${CERC_SCRIPT_DEBUG}
-      CERC_IPLD_ETH_RPC: ${CERC_IPLD_ETH_RPC}
-      CERC_IPLD_ETH_GQL: ${CERC_IPLD_ETH_GQL}
+      CERC_ETH_RPC_ENDPOINTS: ${CERC_ETH_RPC_ENDPOINTS}
+      CERC_IPLD_ETH_GQL_ENDPOINT: ${CERC_IPLD_ETH_GQL_ENDPOINT}
      CERC_HISTORICAL_BLOCK_RANGE: 500
      CONTRACT_ADDRESS: 0x223c067F8CF28ae173EE5CafEa60cA44C335fecB
      CONTRACT_NAME: Azimuth
@@ -47,7 +47,7 @@ services:
    ports:
      - "9000"
    healthcheck:
-      test: ["CMD", "nc", "-vz", "localhost", "9000"]
+      test: ["CMD", "nc", "-vz", "127.0.0.1", "9000"]
      interval: 20s
      timeout: 5s
      retries: 15
@@ -66,8 +66,8 @@ services:
        condition: service_healthy
    environment:
      CERC_SCRIPT_DEBUG: ${CERC_SCRIPT_DEBUG}
-      CERC_IPLD_ETH_RPC: ${CERC_IPLD_ETH_RPC}
-      CERC_IPLD_ETH_GQL: ${CERC_IPLD_ETH_GQL}
+      CERC_ETH_RPC_ENDPOINTS: ${CERC_ETH_RPC_ENDPOINTS}
+      CERC_IPLD_ETH_GQL_ENDPOINT: ${CERC_IPLD_ETH_GQL_ENDPOINT}
    working_dir: /app/packages/azimuth-watcher
    command: "./start-server.sh"
    volumes:
@@ -76,8 +76,9 @@ services:
      - ../config/watcher-azimuth/start-server.sh:/app/packages/azimuth-watcher/start-server.sh
    ports:
      - "3001"
+      - "9001"
    healthcheck:
-      test: ["CMD", "nc", "-vz", "localhost", "3001"]
+      test: ["CMD", "nc", "-vz", "127.0.0.1", "3001"]
      interval: 20s
      timeout: 5s
      retries: 15
@@ -94,8 +95,8 @@ services:
        condition: service_healthy
    environment:
      CERC_SCRIPT_DEBUG: ${CERC_SCRIPT_DEBUG}
-      CERC_IPLD_ETH_RPC: ${CERC_IPLD_ETH_RPC}
-      CERC_IPLD_ETH_GQL: ${CERC_IPLD_ETH_GQL}
+      CERC_ETH_RPC_ENDPOINTS: ${CERC_ETH_RPC_ENDPOINTS}
+      CERC_IPLD_ETH_GQL_ENDPOINT: ${CERC_IPLD_ETH_GQL_ENDPOINT}
      CONTRACT_ADDRESS: 0x325f68d32BdEe6Ed86E7235ff2480e2A433D6189
      CONTRACT_NAME: Censures
      STARTING_BLOCK: 6784954
@@ -108,7 +109,7 @@ services:
    ports:
      - "9002"
    healthcheck:
-      test: ["CMD", "nc", "-vz", "localhost", "9002"]
+      test: ["CMD", "nc", "-vz", "127.0.0.1", "9002"]
      interval: 20s
      timeout: 5s
      retries: 15
@@ -127,8 +128,8 @@ services:
        condition: service_healthy
    environment:
      CERC_SCRIPT_DEBUG: ${CERC_SCRIPT_DEBUG}
-      CERC_IPLD_ETH_RPC: ${CERC_IPLD_ETH_RPC}
-      CERC_IPLD_ETH_GQL: ${CERC_IPLD_ETH_GQL}
+      CERC_ETH_RPC_ENDPOINTS: ${CERC_ETH_RPC_ENDPOINTS}
+      CERC_IPLD_ETH_GQL_ENDPOINT: ${CERC_IPLD_ETH_GQL_ENDPOINT}
    working_dir: /app/packages/censures-watcher
    command: "./start-server.sh"
    volumes:
@@ -137,8 +138,9 @@ services:
      - ../config/watcher-azimuth/start-server.sh:/app/packages/censures-watcher/start-server.sh
    ports:
      - "3002"
+      - "9003"
    healthcheck:
-      test: ["CMD", "nc", "-vz", "localhost", "3002"]
+      test: ["CMD", "nc", "-vz", "127.0.0.1", "3002"]
      interval: 20s
      timeout: 5s
      retries: 15
@@ -155,8 +157,8 @@ services:
        condition: service_healthy
    environment:
      CERC_SCRIPT_DEBUG: ${CERC_SCRIPT_DEBUG}
-      CERC_IPLD_ETH_RPC: ${CERC_IPLD_ETH_RPC}
-      CERC_IPLD_ETH_GQL: ${CERC_IPLD_ETH_GQL}
+      CERC_ETH_RPC_ENDPOINTS: ${CERC_ETH_RPC_ENDPOINTS}
+      CERC_IPLD_ETH_GQL_ENDPOINT: ${CERC_IPLD_ETH_GQL_ENDPOINT}
      CONTRACT_ADDRESS: 0xe7e7f69b34D7d9Bd8d61Fb22C33b22708947971A
      CONTRACT_NAME: Claims
      STARTING_BLOCK: 6784941
@@ -169,7 +171,7 @@ services:
    ports:
      - "9004"
    healthcheck:
-      test: ["CMD", "nc", "-vz", "localhost", "9004"]
+      test: ["CMD", "nc", "-vz", "127.0.0.1", "9004"]
      interval: 20s
      timeout: 5s
      retries: 15
@@ -188,8 +190,8 @@ services:
        condition: service_healthy
    environment:
      CERC_SCRIPT_DEBUG: ${CERC_SCRIPT_DEBUG}
-      CERC_IPLD_ETH_RPC: ${CERC_IPLD_ETH_RPC}
-      CERC_IPLD_ETH_GQL: ${CERC_IPLD_ETH_GQL}
+      CERC_ETH_RPC_ENDPOINTS: ${CERC_ETH_RPC_ENDPOINTS}
+      CERC_IPLD_ETH_GQL_ENDPOINT: ${CERC_IPLD_ETH_GQL_ENDPOINT}
    working_dir: /app/packages/claims-watcher
    command: "./start-server.sh"
    volumes:
@@ -198,8 +200,9 @@ services:
      - ../config/watcher-azimuth/start-server.sh:/app/packages/claims-watcher/start-server.sh
    ports:
      - "3003"
+      - "9005"
    healthcheck:
-      test: ["CMD", "nc", "-vz", "localhost", "3003"]
+      test: ["CMD", "nc", "-vz", "127.0.0.1", "3003"]
      interval: 20s
      timeout: 5s
      retries: 15
@@ -216,8 +219,8 @@ services:
        condition: service_healthy
    environment:
      CERC_SCRIPT_DEBUG: ${CERC_SCRIPT_DEBUG}
-      CERC_IPLD_ETH_RPC: ${CERC_IPLD_ETH_RPC}
-      CERC_IPLD_ETH_GQL: ${CERC_IPLD_ETH_GQL}
+      CERC_ETH_RPC_ENDPOINTS: ${CERC_ETH_RPC_ENDPOINTS}
+      CERC_IPLD_ETH_GQL_ENDPOINT: ${CERC_IPLD_ETH_GQL_ENDPOINT}
      CONTRACT_ADDRESS: 0x8C241098C3D3498Fe1261421633FD57986D74AeA
      CONTRACT_NAME: ConditionalStarRelease
      STARTING_BLOCK: 6828004
@@ -230,7 +233,7 @@ services:
    ports:
      - "9006"
    healthcheck:
-      test: ["CMD", "nc", "-vz", "localhost", "9006"]
+      test: ["CMD", "nc", "-vz", "127.0.0.1", "9006"]
      interval: 20s
      timeout: 5s
      retries: 15
@@ -249,8 +252,8 @@ services:
        condition: service_healthy
    environment:
      CERC_SCRIPT_DEBUG: ${CERC_SCRIPT_DEBUG}
-      CERC_IPLD_ETH_RPC: ${CERC_IPLD_ETH_RPC}
-      CERC_IPLD_ETH_GQL: ${CERC_IPLD_ETH_GQL}
+      CERC_ETH_RPC_ENDPOINTS: ${CERC_ETH_RPC_ENDPOINTS}
+      CERC_IPLD_ETH_GQL_ENDPOINT: ${CERC_IPLD_ETH_GQL_ENDPOINT}
    working_dir: /app/packages/conditional-star-release-watcher
    command: "./start-server.sh"
    volumes:
@@ -259,8 +262,9 @@ services:
      - ../config/watcher-azimuth/start-server.sh:/app/packages/conditional-star-release-watcher/start-server.sh
    ports:
      - "3004"
+      - "9007"
    healthcheck:
-      test: ["CMD", "nc", "-vz", "localhost", "3004"]
+      test: ["CMD", "nc", "-vz", "127.0.0.1", "3004"]
      interval: 20s
      timeout: 5s
      retries: 15
@@ -277,8 +281,8 @@ services:
        condition: service_healthy
    environment:
      CERC_SCRIPT_DEBUG: ${CERC_SCRIPT_DEBUG}
-      CERC_IPLD_ETH_RPC: ${CERC_IPLD_ETH_RPC}
-      CERC_IPLD_ETH_GQL: ${CERC_IPLD_ETH_GQL}
+      CERC_ETH_RPC_ENDPOINTS: ${CERC_ETH_RPC_ENDPOINTS}
+      CERC_IPLD_ETH_GQL_ENDPOINT: ${CERC_IPLD_ETH_GQL_ENDPOINT}
      CONTRACT_ADDRESS: 0xf6b461fE1aD4bd2ce25B23Fe0aff2ac19B3dFA76
      CONTRACT_NAME: DelegatedSending
      STARTING_BLOCK: 6784956
@@ -291,7 +295,7 @@ services:
    ports:
      - "9008"
    healthcheck:
-      test: ["CMD", "nc", "-vz", "localhost", "9008"]
+      test: ["CMD", "nc", "-vz", "127.0.0.1", "9008"]
      interval: 20s
      timeout: 5s
      retries: 15
@@ -310,8 +314,8 @@ services:
        condition: service_healthy
    environment:
      CERC_SCRIPT_DEBUG: ${CERC_SCRIPT_DEBUG}
-      CERC_IPLD_ETH_RPC: ${CERC_IPLD_ETH_RPC}
-      CERC_IPLD_ETH_GQL: ${CERC_IPLD_ETH_GQL}
+      CERC_ETH_RPC_ENDPOINTS: ${CERC_ETH_RPC_ENDPOINTS}
+      CERC_IPLD_ETH_GQL_ENDPOINT: ${CERC_IPLD_ETH_GQL_ENDPOINT}
    working_dir: /app/packages/delegated-sending-watcher
    command: "./start-server.sh"
    volumes:
@@ -320,8 +324,9 @@ services:
      - ../config/watcher-azimuth/start-server.sh:/app/packages/delegated-sending-watcher/start-server.sh
    ports:
      - "3005"
+      - "9009"
    healthcheck:
-      test: ["CMD", "nc", "-vz", "localhost", "3005"]
+      test: ["CMD", "nc", "-vz", "127.0.0.1", "3005"]
      interval: 20s
      timeout: 5s
      retries: 15
@@ -338,8 +343,8 @@ services:
        condition: service_healthy
    environment:
      CERC_SCRIPT_DEBUG: ${CERC_SCRIPT_DEBUG}
-      CERC_IPLD_ETH_RPC: ${CERC_IPLD_ETH_RPC}
-      CERC_IPLD_ETH_GQL: ${CERC_IPLD_ETH_GQL}
+      CERC_ETH_RPC_ENDPOINTS: ${CERC_ETH_RPC_ENDPOINTS}
+      CERC_IPLD_ETH_GQL_ENDPOINT: ${CERC_IPLD_ETH_GQL_ENDPOINT}
      CONTRACT_ADDRESS: 0x33EeCbf908478C10614626A9D304bfe18B78DD73
      CONTRACT_NAME: Ecliptic
      STARTING_BLOCK: 13692129
@@ -352,7 +357,7 @@ services:
    ports:
      - "9010"
    healthcheck:
-      test: ["CMD", "nc", "-vz", "localhost", "9010"]
+      test: ["CMD", "nc", "-vz", "127.0.0.1", "9010"]
      interval: 20s
      timeout: 5s
      retries: 15
@@ -371,8 +376,8 @@ services:
        condition: service_healthy
    environment:
      CERC_SCRIPT_DEBUG: ${CERC_SCRIPT_DEBUG}
-      CERC_IPLD_ETH_RPC: ${CERC_IPLD_ETH_RPC}
-      CERC_IPLD_ETH_GQL: ${CERC_IPLD_ETH_GQL}
+      CERC_ETH_RPC_ENDPOINTS: ${CERC_ETH_RPC_ENDPOINTS}
+      CERC_IPLD_ETH_GQL_ENDPOINT: ${CERC_IPLD_ETH_GQL_ENDPOINT}
    working_dir: /app/packages/ecliptic-watcher
    command: "./start-server.sh"
    volumes:
@@ -381,8 +386,9 @@ services:
      - ../config/watcher-azimuth/start-server.sh:/app/packages/ecliptic-watcher/start-server.sh
    ports:
      - "3006"
+      - "9011"
    healthcheck:
-      test: ["CMD", "nc", "-vz", "localhost", "3006"]
+      test: ["CMD", "nc", "-vz", "127.0.0.1", "3006"]
      interval: 20s
      timeout: 5s
      retries: 15
@@ -399,8 +405,8 @@ services:
        condition: service_healthy
    environment:
      CERC_SCRIPT_DEBUG: ${CERC_SCRIPT_DEBUG}
-      CERC_IPLD_ETH_RPC: ${CERC_IPLD_ETH_RPC}
-      CERC_IPLD_ETH_GQL: ${CERC_IPLD_ETH_GQL}
+      CERC_ETH_RPC_ENDPOINTS: ${CERC_ETH_RPC_ENDPOINTS}
+      CERC_IPLD_ETH_GQL_ENDPOINT: ${CERC_IPLD_ETH_GQL_ENDPOINT}
      CONTRACT_ADDRESS: 0x86cd9cd0992F04231751E3761De45cEceA5d1801
      CONTRACT_NAME: LinearStarRelease
      STARTING_BLOCK: 6784943
@@ -413,7 +419,7 @@ services:
    ports:
      - "9012"
    healthcheck:
-      test: ["CMD", "nc", "-vz", "localhost", "9012"]
+      test: ["CMD", "nc", "-vz", "127.0.0.1", "9012"]
      interval: 20s
      timeout: 5s
      retries: 15
@@ -432,8 +438,8 @@ services:
        condition: service_healthy
    environment:
      CERC_SCRIPT_DEBUG: ${CERC_SCRIPT_DEBUG}
-      CERC_IPLD_ETH_RPC: ${CERC_IPLD_ETH_RPC}
-      CERC_IPLD_ETH_GQL: ${CERC_IPLD_ETH_GQL}
+      CERC_ETH_RPC_ENDPOINTS: ${CERC_ETH_RPC_ENDPOINTS}
+      CERC_IPLD_ETH_GQL_ENDPOINT: ${CERC_IPLD_ETH_GQL_ENDPOINT}
    working_dir: /app/packages/linear-star-release-watcher
    command: "./start-server.sh"
    volumes:
@@ -442,8 +448,9 @@ services:
      - ../config/watcher-azimuth/start-server.sh:/app/packages/linear-star-release-watcher/start-server.sh
    ports:
      - "3007"
+      - "9013"
    healthcheck:
-      test: ["CMD", "nc", "-vz", "localhost", "3007"]
+      test: ["CMD", "nc", "-vz", "127.0.0.1", "3007"]
      interval: 20s
      timeout: 5s
      retries: 15
@@ -460,8 +467,8 @@ services:
        condition: service_healthy
    environment:
      CERC_SCRIPT_DEBUG: ${CERC_SCRIPT_DEBUG}
-      CERC_IPLD_ETH_RPC: ${CERC_IPLD_ETH_RPC}
-      CERC_IPLD_ETH_GQL: ${CERC_IPLD_ETH_GQL}
+      CERC_ETH_RPC_ENDPOINTS: ${CERC_ETH_RPC_ENDPOINTS}
+      CERC_IPLD_ETH_GQL_ENDPOINT: ${CERC_IPLD_ETH_GQL_ENDPOINT}
      CONTRACT_ADDRESS: 0x7fEcaB617c868Bb5996d99D95200D2Fa708218e4
      CONTRACT_NAME: Polls
      STARTING_BLOCK: 6784912
@@ -474,7 +481,7 @@ services:
    ports:
      - "9014"
    healthcheck:
-      test: ["CMD", "nc", "-vz", "localhost", "9014"]
+      test: ["CMD", "nc", "-vz", "127.0.0.1", "9014"]
      interval: 20s
      timeout: 5s
      retries: 15
@@ -493,8 +500,8 @@ services:
        condition: service_healthy
    environment:
      CERC_SCRIPT_DEBUG: ${CERC_SCRIPT_DEBUG}
-      CERC_IPLD_ETH_RPC: ${CERC_IPLD_ETH_RPC}
-      CERC_IPLD_ETH_GQL: ${CERC_IPLD_ETH_GQL}
+      CERC_ETH_RPC_ENDPOINTS: ${CERC_ETH_RPC_ENDPOINTS}
+      CERC_IPLD_ETH_GQL_ENDPOINT: ${CERC_IPLD_ETH_GQL_ENDPOINT}
    working_dir: /app/packages/polls-watcher
    command: "./start-server.sh"
    volumes:
@@ -503,8 +510,9 @@ services:
      - ../config/watcher-azimuth/start-server.sh:/app/packages/polls-watcher/start-server.sh
    ports:
      - "3008"
+      - "9015"
    healthcheck:
-      test: ["CMD", "nc", "-vz", "localhost", "3008"]
+      test: ["CMD", "nc", "-vz", "127.0.0.1", "3008"]
      interval: 20s
      timeout: 5s
      retries: 15
@@ -542,7 +550,7 @@ services:
    ports:
      - "0.0.0.0:4000:4000"
    healthcheck:
-      test: ["CMD", "nc", "-vz", "localhost", "4000"]
+      test: ["CMD", "nc", "-vz", "127.0.0.1", "4000"]
      interval: 20s
      timeout: 5s
      retries: 15
@@ -29,7 +29,7 @@ services:
    image: cerc/watcher-merkl-sushiswap-v3:local
    environment:
      CERC_SCRIPT_DEBUG: ${CERC_SCRIPT_DEBUG}
-      CERC_ETH_RPC_ENDPOINT: ${CERC_ETH_RPC_ENDPOINT}
+      CERC_ETH_RPC_ENDPOINTS: ${CERC_ETH_RPC_ENDPOINTS}
    command: ["bash", "./start-job-runner.sh"]
    volumes:
      - ../config/watcher-merkl-sushiswap-v3/watcher-config-template.toml:/app/environments/watcher-config-template.toml
@@ -37,7 +37,7 @@ services:
    ports:
      - "9002:9000"
    healthcheck:
-      test: ["CMD", "nc", "-v", "localhost", "9000"]
+      test: ["CMD", "nc", "-vz", "127.0.0.1", "9000"]
      interval: 20s
      timeout: 5s
      retries: 15
@@ -55,7 +55,7 @@ services:
    image: cerc/watcher-merkl-sushiswap-v3:local
    environment:
      CERC_SCRIPT_DEBUG: ${CERC_SCRIPT_DEBUG}
-      CERC_ETH_RPC_ENDPOINT: ${CERC_ETH_RPC_ENDPOINT}
+      CERC_ETH_RPC_ENDPOINTS: ${CERC_ETH_RPC_ENDPOINTS}
    command: ["bash", "./start-server.sh"]
    volumes:
      - ../config/watcher-merkl-sushiswap-v3/watcher-config-template.toml:/app/environments/watcher-config-template.toml
@@ -64,7 +64,7 @@ services:
      - "127.0.0.1:3007:3008"
      - "9003:9001"
    healthcheck:
-      test: ["CMD", "nc", "-v", "localhost", "3008"]
+      test: ["CMD", "nc", "-vz", "127.0.0.1", "3008"]
      interval: 20s
      timeout: 5s
      retries: 15
@@ -29,7 +29,7 @@ services:
    image: cerc/watcher-sushiswap-v3:local
    environment:
      CERC_SCRIPT_DEBUG: ${CERC_SCRIPT_DEBUG}
-      CERC_ETH_RPC_ENDPOINT: ${CERC_ETH_RPC_ENDPOINT}
+      CERC_ETH_RPC_ENDPOINTS: ${CERC_ETH_RPC_ENDPOINTS}
    command: ["bash", "./start-job-runner.sh"]
    volumes:
      - ../config/watcher-sushiswap-v3/watcher-config-template.toml:/app/environments/watcher-config-template.toml
@@ -37,7 +37,7 @@ services:
    ports:
      - "9000:9000"
    healthcheck:
-      test: ["CMD", "nc", "-v", "localhost", "9000"]
+      test: ["CMD", "nc", "-vz", "127.0.0.1", "9000"]
      interval: 20s
      timeout: 5s
      retries: 15
@@ -55,7 +55,7 @@ services:
    image: cerc/watcher-sushiswap-v3:local
    environment:
      CERC_SCRIPT_DEBUG: ${CERC_SCRIPT_DEBUG}
-      CERC_ETH_RPC_ENDPOINT: ${CERC_ETH_RPC_ENDPOINT}
+      CERC_ETH_RPC_ENDPOINTS: ${CERC_ETH_RPC_ENDPOINTS}
    command: ["bash", "./start-server.sh"]
    volumes:
      - ../config/watcher-sushiswap-v3/watcher-config-template.toml:/app/environments/watcher-config-template.toml
@@ -64,7 +64,7 @@ services:
      - "127.0.0.1:3008:3008"
      - "9001:9001"
    healthcheck:
-      test: ["CMD", "nc", "-v", "localhost", "3008"]
+      test: ["CMD", "nc", "-vz", "127.0.0.1", "3008"]
      interval: 20s
      timeout: 5s
      retries: 15
@@ -0,0 +1,20 @@
+apiVersion: 1
+
+datasources:
+  - name: Graph Node Postgres
+    type: postgres
+    jsonData:
+      database: graph-node
+      sslmode: 'disable'
+      maxOpenConns: 100
+      maxIdleConns: 100
+      maxIdleConnsAuto: true
+      connMaxLifetime: 14400
+      postgresVersion: 1411 # 903=9.3, 1000=10, 1411=14.11
+      timescaledb: false
+    user: graph-node
+    # # Add URL for graph-node database
+    # url: graph-node-db:5432
+    # # Set password for graph-node database
+    # secureJsonData:
+    #   password: 'password'
@@ -45,7 +45,18 @@ scrape_configs:
    metrics_path: /metrics
    scheme: http
    static_configs:
-      - targets: ['chain-head-exporter:5000']
+      - targets: ['ethereum-chain-head-exporter:5000']
+        labels:
+          instance: 'external'
+          chain: 'ethereum'
+      - targets: ['filecoin-chain-head-exporter:5000']
+        labels:
+          instance: 'external'
+          chain: 'filecoin'
+      - targets: ['graph-node-upstream-head-exporter:5000']
+        labels:
+          instance: 'graph-node'
+          chain: 'filecoin'

  - job_name: 'postgres'
    scrape_interval: 30s
@@ -74,3 +85,11 @@ scrape_configs:
      # - targets: ['example-host:1317']
    params:
      format: ['prometheus']
+
+  - job_name: graph-node
+    metrics_path: /metrics
+    scrape_interval: 30s
+    scheme: http
+    static_configs:
+      # Add graph-node targets to be monitored below
+      # - targets: ['graph-node:8040']
@@ -0,0 +1,64 @@
+apiVersion: 1
+groups:
+  - orgId: 1
+    name: subgraph
+    folder: SubgraphAlerts
+    interval: 30s
+    rules:
+      - uid: b2a9144b-6104-46fc-92b5-352f4e643c4c
+        title: subgraph_head_tracking
+        condition: condition
+        data:
+          - refId: diff
+            relativeTimeRange:
+              from: 600
+              to: 0
+            datasourceUid: PBFA97CFB590B2093
+            model:
+              datasource:
+                  type: prometheus
+                  uid: PBFA97CFB590B2093
+              editorMode: code
+              expr: ethereum_chain_head_number - on(network) group_right deployment_head{deployment=~"REPLACE_WITH_SUBGRAPH_IDS"}
+              instant: true
+              intervalMs: 1000
+              legendFormat: __auto
+              maxDataPoints: 43200
+              range: false
+              refId: diff
+          - refId: condition
+            relativeTimeRange:
+              from: 600
+              to: 0
+            datasourceUid: __expr__
+            model:
+              conditions:
+                  - evaluator:
+                      params:
+                          - 15
+                          - 0
+                      type: gt
+                    operator:
+                      type: and
+                    query:
+                      params: []
+                    reducer:
+                      params: []
+                      type: avg
+                    type: query
+              datasource:
+                  name: Expression
+                  type: __expr__
+                  uid: __expr__
+              expression: diff
+              intervalMs: 1000
+              maxDataPoints: 43200
+              refId: condition
+              type: threshold
+        noDataState: OK
+        execErrState: Alerting
+        for: 5m
+        annotations:
+          summary: Subgraph deployment {{ index $labels "deployment" }} is falling behind head by {{ index $values "diff" }}
+        labels: {}
+        isPaused: false
@@ -0,0 +1,9 @@
+#!/bin/bash
+
+echo Using CERC_GRAFANA_ALERTS_SUBGRAPH_IDS ${CERC_GRAFANA_ALERTS_SUBGRAPH_IDS}
+
+# Replace subgraph ids in subgraph alerting config
+# Note: Requires the grafana container to be run with user root
+if [ -n "$CERC_GRAFANA_ALERTS_SUBGRAPH_IDS" ]; then
+  sed -i "s/REPLACE_WITH_SUBGRAPH_IDS/$CERC_GRAFANA_ALERTS_SUBGRAPH_IDS/g" /etc/grafana/provisioning/alerting/subgraph-alert-rules.yml
+fi
@@ -24,7 +24,7 @@ groups:
                uid: PBFA97CFB590B2093
              disableTextWrap: false
              editorMode: code
-              expr: latest_block_number - on(chain) group_right sync_status_block_number{job="azimuth", instance="azimuth", kind="latest_indexed"}
+              expr: latest_block_number{instance="external"} - on(chain) group_right sync_status_block_number{job="azimuth", instance="azimuth", kind="latest_indexed"}
              fullMetaSearch: false
              includeNullMetadata: true
              instant: true
@@ -100,7 +100,7 @@ groups:
                uid: PBFA97CFB590B2093
              disableTextWrap: false
              editorMode: code
-              expr: latest_block_number - on(chain) group_right sync_status_block_number{job="azimuth", instance="censures", kind="latest_indexed"}
+              expr: latest_block_number{instance="external"} - on(chain) group_right sync_status_block_number{job="azimuth", instance="censures", kind="latest_indexed"}
              fullMetaSearch: false
              includeNullMetadata: true
              instant: true
@@ -176,7 +176,7 @@ groups:
                uid: PBFA97CFB590B2093
              disableTextWrap: false
              editorMode: code
-              expr: latest_block_number - on(chain) group_right sync_status_block_number{job="azimuth", instance="claims", kind="latest_indexed"}
+              expr: latest_block_number{instance="external"} - on(chain) group_right sync_status_block_number{job="azimuth", instance="claims", kind="latest_indexed"}
              fullMetaSearch: false
              includeNullMetadata: true
              instant: true
@@ -252,7 +252,7 @@ groups:
                uid: PBFA97CFB590B2093
              disableTextWrap: false
              editorMode: code
-              expr: latest_block_number - on(chain) group_right sync_status_block_number{job="azimuth", instance="conditional_star_release", kind="latest_indexed"}
+              expr: latest_block_number{instance="external"} - on(chain) group_right sync_status_block_number{job="azimuth", instance="conditional_star_release", kind="latest_indexed"}
              fullMetaSearch: false
              includeNullMetadata: true
              instant: true
@@ -328,7 +328,7 @@ groups:
                uid: PBFA97CFB590B2093
              disableTextWrap: false
              editorMode: code
-              expr: latest_block_number - on(chain) group_right sync_status_block_number{job="azimuth", instance="delegated_sending", kind="latest_indexed"}
+              expr: latest_block_number{instance="external"} - on(chain) group_right sync_status_block_number{job="azimuth", instance="delegated_sending", kind="latest_indexed"}
              fullMetaSearch: false
              includeNullMetadata: true
              instant: true
@@ -404,7 +404,7 @@ groups:
                uid: PBFA97CFB590B2093
              disableTextWrap: false
              editorMode: code
-              expr: latest_block_number - on(chain) group_right sync_status_block_number{job="azimuth", instance="ecliptic", kind="latest_indexed"}
+              expr: latest_block_number{instance="external"} - on(chain) group_right sync_status_block_number{job="azimuth", instance="ecliptic", kind="latest_indexed"}
              fullMetaSearch: false
              includeNullMetadata: true
              instant: true
@@ -480,7 +480,7 @@ groups:
                uid: PBFA97CFB590B2093
              disableTextWrap: false
              editorMode: code
-              expr: latest_block_number - on(chain) group_right sync_status_block_number{job="azimuth", instance="linear_star_release", kind="latest_indexed"}
+              expr: latest_block_number{instance="external"} - on(chain) group_right sync_status_block_number{job="azimuth", instance="linear_star_release", kind="latest_indexed"}
              fullMetaSearch: false
              includeNullMetadata: true
              instant: true
@@ -556,7 +556,7 @@ groups:
                uid: PBFA97CFB590B2093
              disableTextWrap: false
              editorMode: code
-              expr: latest_block_number - on(chain) group_right sync_status_block_number{job="azimuth", instance="polls", kind="latest_indexed"}
+              expr: latest_block_number{instance="external"} - on(chain) group_right sync_status_block_number{job="azimuth", instance="polls", kind="latest_indexed"}
              fullMetaSearch: false
              includeNullMetadata: true
              instant: true
@@ -634,7 +634,7 @@ groups:
                uid: PBFA97CFB590B2093
              disableTextWrap: false
              editorMode: code
-              expr: latest_block_number - on(chain) group_right sync_status_block_number{job="sushi", instance="sushiswap", kind="latest_indexed"}
+              expr: latest_block_number{instance="external"} - on(chain) group_right sync_status_block_number{job="sushi", instance="sushiswap", kind="latest_indexed"}
              fullMetaSearch: false
              includeNullMetadata: true
              instant: true
@@ -710,7 +710,7 @@ groups:
                uid: PBFA97CFB590B2093
              disableTextWrap: false
              editorMode: code
-              expr: latest_block_number - on(chain) group_right sync_status_block_number{job="sushi", instance="merkl_sushiswap", kind="latest_indexed"}
+              expr: latest_block_number{instance="external"} - on(chain) group_right sync_status_block_number{job="sushi", instance="merkl_sushiswap", kind="latest_indexed"}
              fullMetaSearch: false
              includeNullMetadata: true
              instant: true
@@ -788,7 +788,7 @@ groups:
                uid: PBFA97CFB590B2093
              disableTextWrap: false
              editorMode: code
-              expr: latest_block_number - on(chain) group_right sync_status_block_number{job="ajna", instance="ajna", kind="latest_indexed"}
+              expr: latest_block_number{instance="external"} - on(chain) group_right sync_status_block_number{job="ajna", instance="ajna", kind="latest_indexed"}
              fullMetaSearch: false
              includeNullMetadata: true
              instant: true
@@ -6,12 +6,16 @@ if [ -n "$CERC_SCRIPT_DEBUG" ]; then
 fi
 set -u

-echo "Using ETH RPC endpoint ${CERC_ETH_RPC_ENDPOINT}"
+echo "Using ETH RPC endpoints ${CERC_ETH_RPC_ENDPOINTS}"

 # Read in the config template TOML file and modify it
 WATCHER_CONFIG_TEMPLATE=$(cat environments/watcher-config-template.toml)
+
+# Convert the comma-separated list in CERC_ETH_RPC_ENDPOINTS to a JSON array
+RPC_ENDPOINTS_ARRAY=$(echo "$CERC_ETH_RPC_ENDPOINTS" | tr ',' '\n' | awk '{print "\"" $0 "\""}' | paste -sd, - | sed 's/^/[/; s/$/]/')
+
 WATCHER_CONFIG=$(echo "$WATCHER_CONFIG_TEMPLATE" | \
-  sed -E "s|REPLACE_WITH_CERC_ETH_RPC_ENDPOINT|${CERC_ETH_RPC_ENDPOINT}| ")
+  sed -E "s|REPLACE_WITH_CERC_ETH_RPC_ENDPOINTS|${RPC_ENDPOINTS_ARRAY}| ")

 # Write the modified content to a new file
 echo "$WATCHER_CONFIG" > environments/local.toml
@@ -6,12 +6,16 @@ if [ -n "$CERC_SCRIPT_DEBUG" ]; then
 fi
 set -u

-echo "Using ETH RPC endpoint ${CERC_ETH_RPC_ENDPOINT}"
+echo "Using ETH RPC endpoints ${CERC_ETH_RPC_ENDPOINTS}"

 # Read in the config template TOML file and modify it
 WATCHER_CONFIG_TEMPLATE=$(cat environments/watcher-config-template.toml)
+
+# Convert the comma-separated list in CERC_ETH_RPC_ENDPOINTS to a JSON array
+RPC_ENDPOINTS_ARRAY=$(echo "$CERC_ETH_RPC_ENDPOINTS" | tr ',' '\n' | awk '{print "\"" $0 "\""}' | paste -sd, - | sed 's/^/[/; s/$/]/')
+
 WATCHER_CONFIG=$(echo "$WATCHER_CONFIG_TEMPLATE" | \
-  sed -E "s|REPLACE_WITH_CERC_ETH_RPC_ENDPOINT|${CERC_ETH_RPC_ENDPOINT}| ")
+  sed -E "s|REPLACE_WITH_CERC_ETH_RPC_ENDPOINTS|${RPC_ENDPOINTS_ARRAY}| ")

 # Write the modified content to a new file
 echo "$WATCHER_CONFIG" > environments/local.toml
@@ -58,7 +58,7 @@

 [upstream]
  [upstream.ethServer]
-    rpcProviderEndpoint = "REPLACE_WITH_CERC_ETH_RPC_ENDPOINT"
+    rpcProviderEndpoints = REPLACE_WITH_CERC_ETH_RPC_ENDPOINTS

    # Boolean flag to specify if rpc-eth-client should be used for RPC endpoint instead of ipld-eth-client (ipld-eth-server GQL client)
    rpcClient = true
@@ -96,3 +96,6 @@
  # Max block range of historical processing after which it waits for completion of events processing
  # If set to -1 historical processing does not wait for events processing and completes till latest canonical block
  historicalMaxFetchAhead = 10000
+
+  # Max number of retries to fetch new block after which watcher will failover to other RPC endpoints
+  maxNewBlockRetries = 3
@@ -4,16 +4,19 @@ if [ -n "$CERC_SCRIPT_DEBUG" ]; then
  set -x
 fi

-echo "Using IPLD ETH RPC endpoint ${CERC_IPLD_ETH_RPC}"
-echo "Using IPLD GQL endpoint ${CERC_IPLD_ETH_GQL}"
+echo "Using ETH RPC endpoints ${CERC_ETH_RPC_ENDPOINTS}"
+echo "Using IPLD GQL endpoint ${CERC_IPLD_ETH_GQL_ENDPOINT}"
 echo "Using historicalLogsBlockRange ${CERC_HISTORICAL_BLOCK_RANGE:-2000}"

+# Convert the comma-separated list in CERC_ETH_RPC_ENDPOINTS to a JSON array
+RPC_ENDPOINTS_ARRAY=$(echo "$CERC_ETH_RPC_ENDPOINTS" | tr ',' '\n' | awk '{print "\"" $0 "\""}' | paste -sd, - | sed 's/^/[/; s/$/]/')
+
 # Replace env variables in template TOML file
 # Read in the config template TOML file and modify it
 WATCHER_CONFIG_TEMPLATE=$(cat environments/watcher-config-template.toml)
 WATCHER_CONFIG=$(echo "$WATCHER_CONFIG_TEMPLATE" | \
-  sed -E "s|REPLACE_WITH_CERC_IPLD_ETH_RPC|${CERC_IPLD_ETH_RPC}|g; \
-          s|REPLACE_WITH_CERC_IPLD_ETH_GQL|${CERC_IPLD_ETH_GQL}|g; \
+  sed -E "s|REPLACE_WITH_CERC_ETH_RPC_ENDPOINTS|${RPC_ENDPOINTS_ARRAY}|g; \
+          s|REPLACE_WITH_CERC_IPLD_ETH_GQL_ENDPOINT|${CERC_IPLD_ETH_GQL_ENDPOINT}|g; \
          s|REPLACE_WITH_CERC_HISTORICAL_BLOCK_RANGE|${CERC_HISTORICAL_BLOCK_RANGE:-2000}| ")

 # Write the modified content to a new file
@@ -4,16 +4,19 @@ if [ -n "$CERC_SCRIPT_DEBUG" ]; then
  set -x
 fi

-echo "Using IPLD ETH RPC endpoint ${CERC_IPLD_ETH_RPC}"
-echo "Using IPLD GQL endpoint ${CERC_IPLD_ETH_GQL}"
+echo "Using ETH RPC endpoints ${CERC_ETH_RPC_ENDPOINTS}"
+echo "Using IPLD GQL endpoint ${CERC_IPLD_ETH_GQL_ENDPOINT}"
 echo "Using historicalLogsBlockRange ${CERC_HISTORICAL_BLOCK_RANGE:-2000}"

+# Convert the comma-separated list in CERC_ETH_RPC_ENDPOINTS to a JSON array
+RPC_ENDPOINTS_ARRAY=$(echo "$CERC_ETH_RPC_ENDPOINTS" | tr ',' '\n' | awk '{print "\"" $0 "\""}' | paste -sd, - | sed 's/^/[/; s/$/]/')
+
 # Replace env variables in template TOML file
 # Read in the config template TOML file and modify it
 WATCHER_CONFIG_TEMPLATE=$(cat environments/watcher-config-template.toml)
 WATCHER_CONFIG=$(echo "$WATCHER_CONFIG_TEMPLATE" | \
-  sed -E "s|REPLACE_WITH_CERC_IPLD_ETH_RPC|${CERC_IPLD_ETH_RPC}|g; \
-          s|REPLACE_WITH_CERC_IPLD_ETH_GQL|${CERC_IPLD_ETH_GQL}|g; \
+  sed -E "s|REPLACE_WITH_CERC_ETH_RPC_ENDPOINTS|${RPC_ENDPOINTS_ARRAY}|g; \
+          s|REPLACE_WITH_CERC_IPLD_ETH_GQL_ENDPOINT|${CERC_IPLD_ETH_GQL_ENDPOINT}|g; \
          s|REPLACE_WITH_CERC_HISTORICAL_BLOCK_RANGE|${CERC_HISTORICAL_BLOCK_RANGE:-2000}| ")

 # Write the modified content to a new file
@@ -13,8 +13,8 @@

 [upstream]
  [upstream.ethServer]
-    gqlApiEndpoint = "REPLACE_WITH_CERC_IPLD_ETH_GQL"
-    rpcProviderEndpoint = "REPLACE_WITH_CERC_IPLD_ETH_RPC"
+    gqlApiEndpoint = "REPLACE_WITH_CERC_IPLD_ETH_GQL_ENDPOINT"
+    rpcProviderEndpoints = REPLACE_WITH_CERC_ETH_RPC_ENDPOINTS

 [jobQueue]
  historicalLogsBlockRange = REPLACE_WITH_CERC_HISTORICAL_BLOCK_RANGE
@@ -6,12 +6,16 @@ if [ -n "$CERC_SCRIPT_DEBUG" ]; then
 fi
 set -u

-echo "Using ETH RPC endpoint ${CERC_ETH_RPC_ENDPOINT}"
+echo "Using ETH RPC endpoints ${CERC_ETH_RPC_ENDPOINTS}"

 # Read in the config template TOML file and modify it
 WATCHER_CONFIG_TEMPLATE=$(cat environments/watcher-config-template.toml)
+
+# Convert the comma-separated list in CERC_ETH_RPC_ENDPOINTS to a JSON array
+RPC_ENDPOINTS_ARRAY=$(echo "$CERC_ETH_RPC_ENDPOINTS" | tr ',' '\n' | awk '{print "\"" $0 "\""}' | paste -sd, - | sed 's/^/[/; s/$/]/')
+
 WATCHER_CONFIG=$(echo "$WATCHER_CONFIG_TEMPLATE" | \
-  sed -E "s|REPLACE_WITH_CERC_ETH_RPC_ENDPOINT|${CERC_ETH_RPC_ENDPOINT}| ")
+  sed -E "s|REPLACE_WITH_CERC_ETH_RPC_ENDPOINTS|${RPC_ENDPOINTS_ARRAY}| ")

 # Write the modified content to a new file
 echo "$WATCHER_CONFIG" > environments/local.toml
@@ -6,12 +6,16 @@ if [ -n "$CERC_SCRIPT_DEBUG" ]; then
 fi
 set -u

-echo "Using ETH RPC endpoint ${CERC_ETH_RPC_ENDPOINT}"
+echo "Using ETH RPC endpoints ${CERC_ETH_RPC_ENDPOINTS}"

 # Read in the config template TOML file and modify it
 WATCHER_CONFIG_TEMPLATE=$(cat environments/watcher-config-template.toml)
+
+# Convert the comma-separated list in CERC_ETH_RPC_ENDPOINTS to a JSON array
+RPC_ENDPOINTS_ARRAY=$(echo "$CERC_ETH_RPC_ENDPOINTS" | tr ',' '\n' | awk '{print "\"" $0 "\""}' | paste -sd, - | sed 's/^/[/; s/$/]/')
+
 WATCHER_CONFIG=$(echo "$WATCHER_CONFIG_TEMPLATE" | \
-  sed -E "s|REPLACE_WITH_CERC_ETH_RPC_ENDPOINT|${CERC_ETH_RPC_ENDPOINT}| ")
+  sed -E "s|REPLACE_WITH_CERC_ETH_RPC_ENDPOINTS|${RPC_ENDPOINTS_ARRAY}| ")

 # Write the modified content to a new file
 echo "$WATCHER_CONFIG" > environments/local.toml
@@ -58,7 +58,7 @@

 [upstream]
  [upstream.ethServer]
-    rpcProviderEndpoint = "REPLACE_WITH_CERC_ETH_RPC_ENDPOINT"
+    rpcProviderEndpoints = REPLACE_WITH_CERC_ETH_RPC_ENDPOINTS

    # Boolean flag to specify if rpc-eth-client should be used for RPC endpoint instead of ipld-eth-client (ipld-eth-server GQL client)
    rpcClient = true
@@ -69,7 +69,7 @@
    # Boolean flag to filter event logs by contracts
    filterLogsByAddresses = true
    # Boolean flag to filter event logs by topics
-    filterLogsByTopics = false
+    filterLogsByTopics = true

  [upstream.cache]
    name = "requests"
@@ -96,3 +96,6 @@
  # Max block range of historical processing after which it waits for completion of events processing
  # If set to -1 historical processing does not wait for events processing and completes till latest canonical block
  historicalMaxFetchAhead = 10000
+
+  # Max number of retries to fetch new block after which watcher will failover to other RPC endpoints
+  maxNewBlockRetries = 3
@@ -6,12 +6,16 @@ if [ -n "$CERC_SCRIPT_DEBUG" ]; then
 fi
 set -u

-echo "Using ETH RPC endpoint ${CERC_ETH_RPC_ENDPOINT}"
+echo "Using ETH RPC endpoints ${CERC_ETH_RPC_ENDPOINTS}"

 # Read in the config template TOML file and modify it
 WATCHER_CONFIG_TEMPLATE=$(cat environments/watcher-config-template.toml)
+
+# Convert the comma-separated list in CERC_ETH_RPC_ENDPOINTS to a JSON array
+RPC_ENDPOINTS_ARRAY=$(echo "$CERC_ETH_RPC_ENDPOINTS" | tr ',' '\n' | awk '{print "\"" $0 "\""}' | paste -sd, - | sed 's/^/[/; s/$/]/')
+
 WATCHER_CONFIG=$(echo "$WATCHER_CONFIG_TEMPLATE" | \
-  sed -E "s|REPLACE_WITH_CERC_ETH_RPC_ENDPOINT|${CERC_ETH_RPC_ENDPOINT}| ")
+  sed -E "s|REPLACE_WITH_CERC_ETH_RPC_ENDPOINTS|${RPC_ENDPOINTS_ARRAY}| ")

 # Write the modified content to a new file
 echo "$WATCHER_CONFIG" > environments/local.toml
@@ -6,12 +6,16 @@ if [ -n "$CERC_SCRIPT_DEBUG" ]; then
 fi
 set -u

-echo "Using ETH RPC endpoint ${CERC_ETH_RPC_ENDPOINT}"
+echo "Using ETH RPC endpoints ${CERC_ETH_RPC_ENDPOINTS}"

 # Read in the config template TOML file and modify it
 WATCHER_CONFIG_TEMPLATE=$(cat environments/watcher-config-template.toml)
+
+# Convert the comma-separated list in CERC_ETH_RPC_ENDPOINTS to a JSON array
+RPC_ENDPOINTS_ARRAY=$(echo "$CERC_ETH_RPC_ENDPOINTS" | tr ',' '\n' | awk '{print "\"" $0 "\""}' | paste -sd, - | sed 's/^/[/; s/$/]/')
+
 WATCHER_CONFIG=$(echo "$WATCHER_CONFIG_TEMPLATE" | \
-  sed -E "s|REPLACE_WITH_CERC_ETH_RPC_ENDPOINT|${CERC_ETH_RPC_ENDPOINT}| ")
+  sed -E "s|REPLACE_WITH_CERC_ETH_RPC_ENDPOINTS|${RPC_ENDPOINTS_ARRAY}| ")

 # Write the modified content to a new file
 echo "$WATCHER_CONFIG" > environments/local.toml
@@ -58,7 +58,7 @@

 [upstream]
  [upstream.ethServer]
-    rpcProviderEndpoint = "REPLACE_WITH_CERC_ETH_RPC_ENDPOINT"
+    rpcProviderEndpoints = REPLACE_WITH_CERC_ETH_RPC_ENDPOINTS

    # Boolean flag to specify if rpc-eth-client should be used for RPC endpoint instead of ipld-eth-client (ipld-eth-server GQL client)
    rpcClient = true
@@ -96,3 +96,6 @@
  # Max block range of historical processing after which it waits for completion of events processing
  # If set to -1 historical processing does not wait for events processing and completes till latest canonical block
  historicalMaxFetchAhead = 10000
+
+  # Max number of retries to fetch new block after which watcher will failover to other RPC endpoints
+  maxNewBlockRetries = 3
@@ -9,7 +9,7 @@ STATUSES=(
  "beacon phase0"
  "beacon altair"
  "beacon bellatrix pre-merge"
-  "beacon post-merge"
+  "beacon bellatrix merge"
  "block number $MIN_BLOCK_NUM"
 )
 STATUS=0
@@ -69,13 +69,13 @@ while [ $STATUS -lt ${#STATUSES[@]} ]; do
      ;;
    2)
      result=`wget --no-check-certificate --quiet -O - "$LIGHTHOUSE_BASE_URL/eth/v2/beacon/blocks/head" | jq -r '.version'`
-      if [ ! -z "$result" ] && ([ "$result" == "altair" ] || [ "$result" == "bellatrix" ] || [ "$result" == "capella" ] || [ "$result" == "deneb" ]); then
+      if [ ! -z "$result" ] && ([ "$result" == "altair" ] || [ "$result" == "bellatrix" ]); then
        inc_status
      fi
      ;;
    3)
      result=`wget --no-check-certificate --quiet -O - "$LIGHTHOUSE_BASE_URL/eth/v2/beacon/blocks/head" | jq -r '.version'`
-      if [ ! -z "$result" ] && ([ "$result" == "bellatrix" ] || [ "$result" == "capella" ] || [ "$result" == "deneb" ]); then
+      if [ ! -z "$result" ] && [ "$result" == "bellatrix" ]; then
        inc_status
      fi
      ;;
@@ -6,5 +6,10 @@ WORKDIR /app

 COPY . .

+# Get the latest Git commit hash and set in package.json
+RUN COMMIT_HASH=$(git rev-parse HEAD) && \
+  jq --arg commitHash "$COMMIT_HASH" '.commitHash = $commitHash' package.json > tmp.json && \
+  mv tmp.json package.json
+
 RUN echo "Installing dependencies and building ajna-watcher-ts" && \
  yarn && yarn build
@@ -1,11 +1,20 @@
 FROM node:18.16.0-alpine3.16

-RUN apk --update --no-cache add git python3 alpine-sdk
+RUN apk --update --no-cache add git python3 alpine-sdk jq

 WORKDIR /app

 COPY . .

+# Get the latest Git commit hash and set it in package.json of all watchers
+RUN COMMIT_HASH=$(git rev-parse HEAD) && \
+    find . -name 'package.json' -exec sh -c ' \
+    for packageFile; do \
+        jq --arg commitHash "$0" ".commitHash = \$commitHash" "$packageFile" > "$packageFile.tmp" && \
+        mv "$packageFile.tmp" "$packageFile"; \
+    done \
+    ' "$COMMIT_HASH" {} \;
+
 RUN echo "Building azimuth-watcher-ts" && \
    yarn && yarn build

@@ -6,5 +6,10 @@ WORKDIR /app

 COPY . .

+# Get the latest Git commit hash and set in package.json
+RUN COMMIT_HASH=$(git rev-parse HEAD) && \
+  jq --arg commitHash "$COMMIT_HASH" '.commitHash = $commitHash' package.json > tmp.json && \
+  mv tmp.json package.json
+
 RUN echo "Installing dependencies and building merkl-sushiswap-v3-watcher-ts" && \
  yarn && yarn build
@@ -6,5 +6,10 @@ WORKDIR /app

 COPY . .

+# Get the latest Git commit hash and set in package.json
+RUN COMMIT_HASH=$(git rev-parse HEAD) && \
+  jq --arg commitHash "$COMMIT_HASH" '.commitHash = $commitHash' package.json > tmp.json && \
+  mv tmp.json package.json
+
 RUN echo "Installing dependencies and building sushiswap-v3-watcher-ts" && \
  yarn && yarn build
@@ -8,21 +8,27 @@ CERC_WEBAPP_FILES_DIR="${CERC_WEBAPP_FILES_DIR:-/data}"
 CERC_ENABLE_CORS="${CERC_ENABLE_CORS:-false}"
 CERC_SINGLE_PAGE_APP="${CERC_SINGLE_PAGE_APP}"

-if [ -z "${CERC_SINGLE_PAGE_APP}" ]; then
-  if [ 1 -eq $(find "${CERC_WEBAPP_FILES_DIR}" -name '*.html' | wc -l) ] && [ -d "${CERC_WEBAPP_FILES_DIR}/static" ]; then
+if [ -z "${CERC_SINGLE_PAGE_APP}" ]; then 
+  # If there is only one HTML file, assume an SPA.
+  if [ 1 -eq $(find "${CERC_WEBAPP_FILES_DIR}" -name '*.html' | wc -l) ]; then
    CERC_SINGLE_PAGE_APP=true
  else
    CERC_SINGLE_PAGE_APP=false
  fi
 fi

-if [ "true" == "$CERC_ENABLE_CORS" ]; then
+# ${var,,} is a lower-case comparison
+if [ "true" == "${CERC_ENABLE_CORS,,}" ]; then
  CERC_HTTP_EXTRA_ARGS="$CERC_HTTP_EXTRA_ARGS --cors"
 fi

-if [ "true" == "$CERC_SINGLE_PAGE_APP" ]; then
+# ${var,,} is a lower-case comparison
+if [ "true" == "${CERC_SINGLE_PAGE_APP,,}" ]; then
+  echo "Serving content as single-page app.  If this is wrong, set 'CERC_SINGLE_PAGE_APP=false'"
  # Create a catchall redirect back to /
  CERC_HTTP_EXTRA_ARGS="$CERC_HTTP_EXTRA_ARGS --proxy http://localhost:${CERC_LISTEN_PORT}?"
+else
+  echo "Serving content normally.  If this is a single-page app, set 'CERC_SINGLE_PAGE_APP=true'"
 fi

 LACONIC_HOSTED_CONFIG_FILE=${LACONIC_HOSTED_CONFIG_FILE}
@@ -39,4 +45,4 @@ if [ -f "${LACONIC_HOSTED_CONFIG_FILE}" ]; then
 fi

 /scripts/apply-runtime-env.sh ${CERC_WEBAPP_FILES_DIR}
-http-server $CERC_HTTP_EXTRA_ARGS -p ${CERC_LISTEN_PORT} "${CERC_WEBAPP_FILES_DIR}"
+http-server $CERC_HTTP_EXTRA_ARGS -p ${CERC_LISTEN_PORT} "${CERC_WEBAPP_FILES_DIR}"
@@ -53,7 +53,7 @@ Inside deployment directory, open the `config.env` file  and set following env v

 ```bash
 # External Filecoin (ETH RPC) endpoint to point the watcher to
-CERC_ETH_RPC_ENDPOINT=https://example-lotus-endpoint/rpc/v1
+CERC_ETH_RPC_ENDPOINTS=https://example-lotus-endpoint-1/rpc/v1,https://example-lotus-endpoint-2/rpc/v1
 ```

 ### Start the deployment
@@ -2,7 +2,7 @@ version: "1.0"
 name: ajna
 description: "Ajna watcher stack"
 repos:
-  - git.vdb.to/cerc-io/ajna-watcher-ts@v0.1.1
+  - git.vdb.to/cerc-io/ajna-watcher-ts@v0.1.12
 containers:
  - cerc/watcher-ajna
 pods:
@@ -4,7 +4,7 @@ Instructions to setup and deploy Azimuth Watcher stack

 ## Setup

-Prerequisite: `ipld-eth-server` RPC and GQL endpoints
+Prerequisite: External RPC endpoints

 Clone required repositories:

@@ -44,34 +44,42 @@ network:
     - 0.0.0.0:9000:9000
    azimuth-watcher-server:
     - 0.0.0.0:3001:3001
+     - 0.0.0.0:9001:9001
    censures-watcher-job-runner:
     - 0.0.0.0:9002:9002
    censures-watcher-server:
     - 0.0.0.0:3002:3002
+     - 0.0.0.0:9003:9003
    claims-watcher-job-runner:
     - 0.0.0.0:9004:9004
    claims-watcher-server:
     - 0.0.0.0:3003:3003
+     - 0.0.0.0:9005:9005
    conditional-star-release-watcher-job-runner:
     - 0.0.0.0:9006:9006
    conditional-star-release-watcher-server:
     - 0.0.0.0:3004:3004
+     - 0.0.0.0:9007:9007
    delegated-sending-watcher-job-runner:
     - 0.0.0.0:9008:9008
    delegated-sending-watcher-server:
     - 0.0.0.0:3005:3005
+     - 0.0.0.0:9009:9009
    ecliptic-watcher-job-runner:
     - 0.0.0.0:9010:9010
    ecliptic-watcher-server:
     - 0.0.0.0:3006:3006
+     - 0.0.0.0:9011:9011
    linear-star-release-watcher-job-runner:
     - 0.0.0.0:9012:9012
    linear-star-release-watcher-server:
     - 0.0.0.0:3007:3007
+     - 0.0.0.0:9013:9013
    polls-watcher-job-runner:
     - 0.0.0.0:9014:9014
    polls-watcher-server:
     - 0.0.0.0:3008:3008
+     - 0.0.0.0:9015:9015
    gateway-server:
     - 0.0.0.0:4000:4000
 ...
@@ -94,7 +102,7 @@ Inside the deployment directory, open the file `config.env` and add variable to

  ```bash
  # External RPC endpoints
-  CERC_IPLD_ETH_RPC=
+  CERC_ETH_RPC_ENDPOINTS=https://example-rpc-endpoint-1,https://example-rpc-endpoint-2
  ```

 * NOTE: If RPC endpoint is on the host machine, use `host.docker.internal` as the hostname to access the host port, or use the `ip a` command to find the IP address of the `docker0` interface (this will usually be something like `172.17.0.1` or `172.18.0.1`)
@@ -120,4 +128,7 @@ To stop all azimuth services and also delete data:

 ```bash
 laconic-so deployment --dir azimuth-deployment stop --delete-volumes
+
+# Remove deployment directory (deployment will have to be recreated for a re-run)
+rm -r azimuth-deployment
 ```
@@ -1,7 +1,7 @@
 version: "1.0"
 name: azimuth
 repos:
-  - github.com/cerc-io/azimuth-watcher-ts@v0.1.3
+  - github.com/cerc-io/azimuth-watcher-ts@0.1.6
 containers:
  - cerc/watcher-azimuth
 pods:
@@ -1,9 +1,9 @@
 version: "1.0"
 name: erc20-watcher
 repos:
-  - git.vdb.to/cerc-io/go-ethereum@v1.13.14
+  - git.vdb.to/cerc-io/go-ethereum@v1.11.6-statediff-v5
  - git.vdb.to/cerc-io/ipld-eth-db@v5
-  - git.vdb.to/cerc-io/ipld-eth-server@v5
+  - git.vdb.to/cerc-io/ipld-eth-server@v1.11.6-statediff-v5
  - github.com/dboreham/foundry
  - github.com/cerc-io/erc20-watcher-ts
 containers:
@@ -1,9 +1,9 @@
 version: "1.0"
 name: erc721-watcher
 repos:
-  - git.vdb.to/cerc-io/go-ethereum@v1.13.14
+  - git.vdb.to/cerc-io/go-ethereum@v1.11.6-statediff-v5
  - git.vdb.to/cerc-io/ipld-eth-db@v5
-  - git.vdb.to/cerc-io/ipld-eth-server@v5
+  - git.vdb.to/cerc-io/ipld-eth-server@v1.11.6-statediff-v5
  - github.com/cerc-io/watcher-ts
 containers:
  - cerc/go-ethereum
@@ -2,9 +2,9 @@ version: "1.0"
 name: fixturenet-eth-loaded
 description: "Loaded Ethereum Fixturenet"
 repos:
-  - git.vdb.to/cerc-io/go-ethereum@v1.13.14
+  - git.vdb.to/cerc-io/go-ethereum@v1.11.6-statediff-v5
  - git.vdb.to/cerc-io/tx-spammer
-  - git.vdb.to/cerc-io/ipld-eth-server@v5
+  - git.vdb.to/cerc-io/ipld-eth-server@v1.11.6-statediff-v5
  - git.vdb.to/cerc-io/ipld-eth-db@v5
  - git.vdb.to/cerc-io/lighthouse
 containers:
@@ -2,7 +2,7 @@ version: "1.2"
 name: fixturenet-eth-tx
 description: "Ethereum Fixturenet w/ tx-spammer"
 repos:
-  - git.vdb.to/cerc-io/go-ethereum@v1.13.14
+  - git.vdb.to/cerc-io/go-ethereum@v1.11.6-statediff-v5
  - git.vdb.to/cerc-io/tx-spammer
  - git.vdb.to/cerc-io/lighthouse
  - github.com/dboreham/foundry
@@ -2,7 +2,7 @@ version: "1.1"
 name: fixturenet-eth
 description: "Ethereum Fixturenet"
 repos:
-  - git.vdb.to/cerc-io/go-ethereum@v1.13.14
+  - git.vdb.to/cerc-io/go-ethereum@v1.11.6-statediff-v5
  - git.vdb.to/cerc-io/lighthouse
  - github.com/dboreham/foundry
 containers:
@@ -2,7 +2,7 @@ version: "1.0"
 name: fixturenet-optimism
 description: "Optimism Fixturenet"
 repos:
-  - git.vdb.to/cerc-io/go-ethereum@v1.13.14
+  - git.vdb.to/cerc-io/go-ethereum@v1.11.6-statediff-v5
  - git.vdb.to/cerc-io/lighthouse
  - github.com/dboreham/foundry
  - github.com/ethereum-optimism/optimism@op-node/v1.3.0
@@ -3,8 +3,7 @@ name: fixturenet-payments
 description: "Stack to demonstrate payments between various services"
 repos:
  # fixturenet repos
-  - git.vdb.to/cerc-io/plugeth@statediff
-  - git.vdb.to/cerc-io/plugeth-statediff
+  - git.vdb.to/cerc-io/go-ethereum@v1.11.6-statediff-v5
  - git.vdb.to/cerc-io/lighthouse
  - git.vdb.to/cerc-io/ipld-eth-db@v5
  # ipld-eth-server with payments -- to be used by ipld-eth-server-payments pod
@@ -25,12 +24,11 @@ repos:
  - github.com/cerc-io/erc20-watcher-ts
 containers:
  # fixturenet images
-  - cerc/plugeth-statediff
-  - cerc/plugeth
-  - cerc/fixturenet-eth-genesis
-  - cerc/fixturenet-plugeth-plugeth
+  - cerc/go-ethereum
  - cerc/lighthouse
  - cerc/lighthouse-cli
+  - cerc/fixturenet-eth-genesis
+  - cerc/fixturenet-eth-geth
  - cerc/fixturenet-eth-lighthouse
  - cerc/ipld-eth-db
  - cerc/ipld-eth-server
@@ -48,7 +46,7 @@ containers:
  - cerc/ponder
  - cerc/watcher-erc20
 pods:
-  - fixturenet-plugeth
+  - fixturenet-eth
  - ipld-eth-server-payments
  - ipld-eth-db
  - nitro-contracts
@@ -2,7 +2,7 @@ version: "1.0"
 name: fixturenet-pocket
 description: "A single node pocket chain that can serve relays from the geth-1 node in eth-fixturenet"
 repos:
-  - git.vdb.to/cerc-io/go-ethereum@v1.13.14
+  - git.vdb.to/cerc-io/go-ethereum@v1.11.6-statediff-v5
  - git.vdb.to/cerc-io/lighthouse
  - github.com/pokt-network/pocket-core
  - github.com/pokt-network/pocket-core-deployments # contains the dockerfile
@@ -43,16 +43,18 @@ customized by editing the "spec" file generated by `laconic-so deploy init`.
 ```
 $ cat graph-node-spec.yml
 stack: graph-node
-ports:
-  graph-node:
-    - '8000:8000'
-    - '8001'
-    - '8020:8020'
-    - '8030'
-  ipfs:
-    - '8080'
-    - '4001'
-    - '5001:5001'
+network:
+  ports:
+    graph-node:
+      - '8000:8000'
+      - '8001'
+      - '8020:8020'
+      - '8030'
+      - '8040'
+    ipfs:
+      - '8080'
+      - '4001'
+      - '5001:5001'
 ...
 ```

@@ -64,7 +66,7 @@ laconic-so --stack graph-node deploy create --spec-file graph-node-spec.yml --de

 ## Start the stack

-Create an env file with the following values to be set before starting the stack:
+Update `config.env` file inside deployment directory with the following values before starting the stack:

 ```bash
 # Set ETH RPC endpoint the graph node will use
@@ -88,10 +90,13 @@ export GRAPH_ETHEREUM_REQUEST_RETRIES=
 # Maximum number of blocks to scan for triggers in each request (default: 2000)
 export GRAPH_ETHEREUM_MAX_BLOCK_RANGE_SIZE=

+# Maximum number of concurrent requests made against Ethereum for requesting transaction receipts during block ingestion (default: 1000)
+export GRAPH_ETHEREUM_BLOCK_INGESTOR_MAX_CONCURRENT_JSON_RPC_CALLS_FOR_TXN_RECEIPTS=
+
 # Ref: https://git.vdb.to/cerc-io/graph-node/src/branch/master/docs/environment-variables.md
 ```

-Example env file:
+Example `config.env` file:

 ```bash
 export ETH_RPC_HOST=filecoin.chainup.net
@@ -104,12 +109,6 @@ export GRAPH_ETHEREUM_REQUEST_RETRIES=5
 export GRAPH_ETHEREUM_MAX_BLOCK_RANGE_SIZE=50
 ```

-Set the environment variables:
-
-```bash
-source <PATH_TO_ENV_FILE>
-```
-
 Deploy the stack:

 ```bash
@@ -3,7 +3,9 @@ name: graph-node
 description: "Stack for running graph-node"
 repos:
  - github.com/graphprotocol/graph-node
+  - github.com/cerc-io/watcher-ts@v0.2.92
 containers:
  - cerc/graph-node
+  - cerc/watcher-ts
 pods:
  - graph-node
@@ -2,7 +2,7 @@ version: "1.2"
 name: mainnet-eth
 description: "Ethereum Mainnet"
 repos:
-  - git.vdb.to/cerc-io/go-ethereum@v1.13.14
+  - git.vdb.to/cerc-io/go-ethereum@v1.11.6-statediff-v5
  - git.vdb.to/cerc-io/lighthouse
  - github.com/dboreham/foundry
  - git.vdb.to/cerc-io/keycloak-reg-api
@@ -53,7 +53,7 @@ Inside deployment directory, open the `config.env` file  and set following env v

 ```bash
 # External Filecoin (ETH RPC) endpoint to point the watcher to
-CERC_ETH_RPC_ENDPOINT=https://example-lotus-endpoint/rpc/v1
+CERC_ETH_RPC_ENDPOINTS=https://example-lotus-endpoint-1/rpc/v1,https://example-lotus-endpoint-2/rpc/v1
 ```

 ### Start the deployment
@@ -2,7 +2,7 @@ version: "1.0"
 name: merkl-sushiswap-v3
 description: "SushiSwap v3 watcher stack"
 repos:
-  - github.com/cerc-io/merkl-sushiswap-v3-watcher-ts@v0.1.7
+  - github.com/cerc-io/merkl-sushiswap-v3-watcher-ts@v0.1.13
 containers:
  - cerc/watcher-merkl-sushiswap-v3
 pods:
@@ -1,7 +1,7 @@
 version: "1.0"
 name: mobymask-v2
 repos:
-  - git.vdb.to/cerc-io/go-ethereum@v1.13.14
+  - git.vdb.to/cerc-io/go-ethereum@v1.11.6-statediff-v5
  - git.vdb.to/cerc-io/lighthouse
  - github.com/dboreham/foundry
  - github.com/ethereum-optimism/optimism@v1.0.4
@@ -134,6 +134,29 @@ Note: Use `host.docker.internal` as host to access ports on the host machine

 Place the dashboard json files in grafana dashboards config directory (`monitoring-deployment/config/monitoring/grafana/dashboards`) in the deployment folder

+#### Graph Node Config
+
+For graph-node dashboard postgres datasource needs to be setup in `monitoring-deployment/config/monitoring/grafana/provisioning/datasources/graph-node-postgres.yml` (in deployment folder)
+
+```yml
+# graph-node-postgres.yml
+...
+datasources:
+  - name: Graph Node Postgres
+    type: postgres
+    jsonData:
+      # Set name to remote graph-node database name
+      database: graph-node
+      ...
+    # Set user to remote graph-node database username
+    user: graph-node
+    # Add URL for remote graph-node database
+    url: graph-node-db:5432
+    # Set password for graph-node database
+    secureJsonData:
+      password: 'password'
+```
+
 ### Env

 Set the following env variables in the deployment env config file (`monitoring-deployment/config.env`):
@@ -156,6 +179,11 @@ Set the following env variables in the deployment env config file (`monitoring-d
  # Grafana server host URL (used in various links in alerts, etc.)
  # (Optional, default: http://localhost:3000)
  GF_SERVER_ROOT_URL=
+
+  
+  # RPC endpoint used by graph-node for upstream head metric
+  # (Optional, default: https://mainnet.infura.io/v3)
+  GRAPH_NODE_RPC_ENDPOINT=
  ```

 ## Start the stack
@@ -57,35 +57,35 @@ Add the following scrape configs to prometheus config file (`monitoring-watchers
    metrics_path: /metrics
    scheme: http
    static_configs:
-      - targets: ['AZIMUTH_WATCHER_HOST:AZIMUTH_WATCHER_PORT']
+      - targets: ['AZIMUTH_WATCHER_HOST:AZIMUTH_WATCHER_METRICS_PORT', 'AZIMUTH_WATCHER_HOST:AZIMUTH_WATCHER_GQL_METRICS_PORT']
        labels:
          instance: 'azimuth'
          chain: 'ethereum'
-      - targets: ['CENSURES_WATCHER_HOST:CENSURES_WATCHER_PORT']
+      - targets: ['CENSURES_WATCHER_HOST:CENSURES_WATCHER_METRICS_PORT', 'CENSURES_WATCHER_HOST:CENSURES_WATCHER_GQL_METRICS_PORT']
        labels:
          instance: 'censures'
          chain: 'ethereum'
-      - targets: ['CLAIMS_WATCHER_HOST:CLAIMS_WATCHER_PORT']
+      - targets: ['CLAIMS_WATCHER_HOST:CLAIMS_WATCHER_METRICS_PORT', 'CLAIMS_WATCHER_HOST:CLAIMS_WATCHER_GQL_METRICS_PORT']
        labels:
          instance: 'claims'
          chain: 'ethereum'
-      - targets: ['CONDITIONAL_STAR_RELEASE_WATCHER_HOST:CONDITIONAL_STAR_RELEASE_WATCHER_PORT']
+      - targets: ['CONDITIONAL_STAR_RELEASE_WATCHER_HOST:CONDITIONAL_STAR_RELEASE_WATCHER_METRICS_PORT', 'CONDITIONAL_STAR_RELEASE_WATCHER_HOST:CONDITIONAL_STAR_RELEASE_WATCHER_GQL_METRICS_PORT']
        labels:
          instance: 'conditional_star_release'
          chain: 'ethereum'
-      - targets: ['DELEGATED_SENDING_WATCHER_HOST:DELEGATED_SENDING_WATCHER_PORT']
+      - targets: ['DELEGATED_SENDING_WATCHER_HOST:DELEGATED_SENDING_WATCHER_METRICS_PORT', 'DELEGATED_SENDING_WATCHER_HOST:DELEGATED_SENDING_WATCHER_GQL_METRICS_PORT']
        labels:
          instance: 'delegated_sending'
          chain: 'ethereum'
-      - targets: ['ECLIPTIC_WATCHER_HOST:ECLIPTIC_WATCHER_PORT']
+      - targets: ['ECLIPTIC_WATCHER_HOST:ECLIPTIC_WATCHER_METRICS_PORT', 'ECLIPTIC_WATCHER_HOST:ECLIPTIC_WATCHER_GQL_METRICS_PORT']
        labels:
          instance: 'ecliptic'
          chain: 'ethereum'
-      - targets: ['LINEAR_STAR_WATCHER_HOST:LINEAR_STAR_WATCHER_PORT']
+      - targets: ['LINEAR_STAR_WATCHER_HOST:LINEAR_STAR_WATCHER_METRICS_PORT', 'LINEAR_STAR_WATCHER_HOST:LINEAR_STAR_WATCHER_GQL_METRICS_PORT']
        labels:
          instance: 'linear_star_release'
          chain: 'ethereum'
-      - targets: ['POLLS_WATCHER_HOST:POLLS_WATCHER_PORT']
+      - targets: ['POLLS_WATCHER_HOST:POLLS_WATCHER_METRICS_PORT', 'POLLS_WATCHER_HOST:POLLS_WATCHER_GQL_METRICS_PORT']
        labels:
          instance: 'polls'
          chain: 'ethereum'
@@ -95,11 +95,11 @@ Add the following scrape configs to prometheus config file (`monitoring-watchers
    metrics_path: /metrics
    scheme: http
    static_configs:
-      - targets: ['SUSHISWAP_WATCHER_HOST:SUSHISWAP_WATCHER_PORT']
+      - targets: ['SUSHISWAP_WATCHER_HOST:SUSHISWAP_WATCHER_METRICS_PORT', 'SUSHISWAP_WATCHER_HOST:SUSHISWAP_WATCHER_GQL_METRICS_PORT']
        labels:
          instance: 'sushiswap'
          chain: 'filecoin'
-      - targets: ['MERKLE_SUSHISWAP_WATCHER_HOST:MERKLE_SUSHISWAP_WATCHER_PORT']
+      - targets: ['MERKLE_SUSHISWAP_WATCHER_HOST:MERKLE_SUSHISWAP_WATCHER_METRICS_PORT', 'MERKLE_SUSHISWAP_WATCHER_HOST:MERKLE_SUSHISWAP_WATCHER_GQL_METRICS_PORT']
        labels:
          instance: 'merkl_sushiswap'
          chain: 'filecoin'
@@ -109,25 +109,35 @@ Add the following scrape configs to prometheus config file (`monitoring-watchers
    metrics_path: /metrics
    scheme: http
    static_configs:
-      - targets: ['AJNA_WATCHER_HOST:AJNA_WATCHER_PORT']
+      - targets: ['AJNA_WATCHER_HOST:AJNA_WATCHER_METRICS_PORT', 'AJNA_WATCHER_HOST:AJNA_WATCHER_GQL_METRICS_PORT']
        labels:
          instance: 'ajna'
          chain: 'filecoin'
+
+  - job_name: graph-node
+    metrics_path: /metrics
+    scrape_interval: 30s
+    static_configs:
+      - targets: ['GRAPH_NODE_HOST:GRAPH_NODE_HOST_METRICS_PORT']
  ```

 Add scrape config as done above for any additional watcher to add it to the Watchers dashboard.

 ### Grafana alerts config

-Place the pre-configured watcher alerts rules in Grafana provisioning directory:
+Place the pre-configured alerts rules in Grafana provisioning directory:

  ```bash
+  # watcher alert rules
  cp monitoring-watchers-deployment/config/monitoring/watcher-alert-rules.yml monitoring-watchers-deployment/config/monitoring/grafana/provisioning/alerting/
+
+  # subgraph alert rules
+  cp monitoring-watchers-deployment/config/monitoring/subgraph-alert-rules.yml monitoring-watchers-deployment/config/monitoring/grafana/provisioning/alerting/
  ```

 Update the alerting contact points config (`monitoring-watchers-deployment/config/monitoring/grafana/provisioning/alerting/contactpoints.yml`) with desired contact points

-Add corresponding routes to the notification policies config (`monitoring-watchers-deployment/monitoring/grafana/provisioning/alerting/policies.yaml`) with appropriate object-matchers:
+Add corresponding routes to the notification policies config (`monitoring-watchers-deployment/config/monitoring/grafana/provisioning/alerting/policies.yml`) with appropriate object-matchers:

  ```yml
  ...
@@ -135,7 +145,7 @@ Add corresponding routes to the notification policies config (`monitoring-watche
      - receiver: SlackNotifier
      object_matchers:
        # Add matchers below
-        - ['grafana_folder', '=', 'WatcherAlerts']
+        - ['grafana_folder', '=~', 'WatcherAlerts|SubgraphAlerts']
  ```

 ### Env
@@ -149,6 +159,9 @@ Set the following env variables in the deployment env config file (`monitoring-w
  # Grafana server host URL to be used
  # (Optional, default: http://localhost:3000)
  GF_SERVER_ROOT_URL=
+
+  # List of subgraph ids to configure alerts for (separated by |)
+  CERC_GRAFANA_ALERTS_SUBGRAPH_IDS=
  ```

 ## Start the stack
@@ -1,7 +1,7 @@
 version: "0.1"
 name: monitoring
 repos:
-  - github.com/cerc-io/watcher-ts@v0.2.81
+  - github.com/cerc-io/watcher-ts@v0.2.92
 containers:
  - cerc/watcher-ts
 pods:
@@ -53,7 +53,7 @@ Inside deployment directory, open the `config.env` file  and set following env v

 ```bash
 # External Filecoin (ETH RPC) endpoint to point the watcher to
-CERC_ETH_RPC_ENDPOINT=https://example-lotus-endpoint/rpc/v1
+CERC_ETH_RPC_ENDPOINTS=https://example-lotus-endpoint-1/rpc/v1,https://example-lotus-endpoint-2/rpc/v1
 ```

 ### Start the deployment
@@ -2,7 +2,7 @@ version: "1.0"
 name: sushiswap-v3
 description: "SushiSwap v3 watcher stack"
 repos:
-  - github.com/cerc-io/sushiswap-v3-watcher-ts@v0.1.7
+  - github.com/cerc-io/sushiswap-v3-watcher-ts@v0.1.13
 containers:
  - cerc/watcher-sushiswap-v3
 pods:
Author	SHA1	Message	Date
prathamesh	abc0c2423f	Add panels for GQL metrics to watcher dashboard (#834 ) Part of [Metrics and logging for GQL queries in watcher](https://www.notion.so/Metrics-and-logging-for-GQL-queries-in-watcher-928c692292b140a2a4f52cda9795df5e) Reviewed-on: cerc-io/stack-orchestrator#834 Co-authored-by: Prathamesh Musale <prathamesh.musale0@gmail.com> Co-committed-by: Prathamesh Musale <prathamesh.musale0@gmail.com>	2024-06-06 11:47:18 +00:00
nabarun	a322d6eed4	Add dashboard for graph-node subgraphs (#832 ) Part of [Deploy v2 and updated v3 sushiswap subgraphs](https://www.notion.so/Deploy-v2-and-updated-v3-sushiswap-subgraphs-e331945fdeea487c890706fc22c6cc94) - Add param `GRAPH_ETHEREUM_BLOCK_INGESTOR_MAX_CONCURRENT_JSON_RPC_CALLS_FOR_TXN_RECEIPTS` in graph-node stack - <https://github.com/graphprotocol/graph-node/blob/v0.31.0/docs/environment-variables.md#json-rpc-configuration-for-evm-chains> - Add dashboard for subgraphs deployment in graph-node - Show subgraph names in dashboard - Add watcher dashboard panel for showing watcher release version and commit hash Reviewed-on: cerc-io/stack-orchestrator#832 Co-authored-by: Nabarun <nabarun@deepstacksoft.com> Co-committed-by: Nabarun <nabarun@deepstacksoft.com>	2024-06-04 07:21:27 +00:00
prathameshandnabarun	ed8914b8d3	Upgrade watchers and their config (#827 ) Part of [Investigate subgraph watchers lagging behind head](https://www.notion.so/Investigate-subgraph-watchers-lagging-behind-head-01b72294ca8e4f658e4c0e86b36d19e2) Co-authored-by: Nabarun <nabarun@deepstacksoft.com> Reviewed-on: cerc-io/stack-orchestrator#827 Co-authored-by: Prathamesh Musale <prathamesh.musale0@gmail.com> Co-committed-by: Prathamesh Musale <prathamesh.musale0@gmail.com>	2024-05-23 04:12:31 +00:00
telackey	fef7649683	Refactor SPA check. (#831 ) Reviewed-on: cerc-io/stack-orchestrator#831 Co-authored-by: Thomas E Lackey <telackey@bozemanpass.com> Co-committed-by: Thomas E Lackey <telackey@bozemanpass.com>	2024-05-22 19:00:42 +00:00
telackey	579b402f2f	Make the SPA detection even simpler. (#830 ) Reviewed-on: cerc-io/stack-orchestrator#830 Co-authored-by: Thomas E Lackey <telackey@bozemanpass.com> Co-committed-by: Thomas E Lackey <telackey@bozemanpass.com>	2024-05-22 18:31:57 +00:00
telackey	25d0bc8a98	Case insensitive comparison (#829 ) Reviewed-on: cerc-io/stack-orchestrator#829 Co-authored-by: Thomas E Lackey <telackey@bozemanpass.com> Co-committed-by: Thomas E Lackey <telackey@bozemanpass.com>	2024-05-22 18:18:27 +00:00
telackey	855288368c	Add messages around the SPA auto-detect. (#828 ) Tweak the auto-detection logic slightly for single-page apps, and also print the results. Reviewed-on: cerc-io/stack-orchestrator#828 Co-authored-by: Thomas E Lackey <telackey@bozemanpass.com> Co-committed-by: Thomas E Lackey <telackey@bozemanpass.com>	2024-05-22 17:42:28 +00:00
prathamesh	8f2da38183	Add alerts for graph-node subgraphs (#821 ) Part of [Deploy v2 and updated v3 sushiswap subgraphs](https://www.notion.so/Deploy-v2-and-updated-v3-sushiswap-subgraphs-e331945fdeea487c890706fc22c6cc94) Reviewed-on: cerc-io/stack-orchestrator#821 Co-authored-by: Prathamesh Musale <prathamesh.musale0@gmail.com> Co-committed-by: Prathamesh Musale <prathamesh.musale0@gmail.com>	2024-05-17 04:01:41 +00:00
dboreham	254f95e59f	Merge pull request 'Remove legacy commands for docker startup' (#826 ) from dboreham/remove-old-docker-ci into main Reviewed-on: cerc-io/stack-orchestrator#826	2024-05-15 16:09:09 +00:00
dboreham	0acb6ea6bc	Remove legacy commands for docker startup	2024-05-15 09:04:19 -06:00
prathamesh	b9369a13e6	Update watcher dashboard with panels for ETH RPC request failures and durations (#825 ) Part of [Regenerate ajna watcher with updated subgraph config ](https://www.notion.so/Regenerate-ajna-watcher-with-updated-subgraph-config-c9bbecb033024c13a7515c7f1efc3363) Requires [Add metrics to monitor errors and duration for ETH RPC requests](https://github.com/cerc-io/watcher-ts/pull/507) Reviewed-on: cerc-io/stack-orchestrator#825 Co-authored-by: Prathamesh Musale <prathamesh.musale0@gmail.com> Co-committed-by: Prathamesh Musale <prathamesh.musale0@gmail.com>	2024-05-15 13:52:18 +00:00
prathamesh	0b1eb8eb0f	Update ajna-watcher version in ajna stack (#824 ) Part of [Regenerate ajna watcher with updated subgraph config](https://www.notion.so/Regenerate-ajna-watcher-with-updated-subgraph-config-c9bbecb033024c13a7515c7f1efc3363) Reviewed-on: cerc-io/stack-orchestrator#824 Co-authored-by: Prathamesh Musale <prathamesh.musale0@gmail.com> Co-committed-by: Prathamesh Musale <prathamesh.musale0@gmail.com>	2024-05-14 08:27:59 +00:00
nabarun	78092f5793	Update subgraph watcher stacks to configure multiple RPC endpoints (#822 ) Part of [Ability to configure watchers with multiple RPC endpoints](https://www.notion.so/Ability-to-configure-watchers-with-multiple-RPC-endpoints-dc8d3ff4d647404ab718dfd5a4c9035c) Reviewed-on: cerc-io/stack-orchestrator#822 Co-authored-by: Nabarun <nabarun@deepstacksoft.com> Co-committed-by: Nabarun <nabarun@deepstacksoft.com>	2024-05-10 04:58:30 +00:00
prathameshandnabarun	247dbdd2f0	Update subgraph watcher versions for improved `eth_getLogs` calls (#820 ) Part of [Investigate subgraph watchers lagging behind head](https://www.notion.so/Investigate-subgraph-watchers-lagging-behind-head-01b72294ca8e4f658e4c0e86b36d19e2) Co-authored-by: Nabarun <nabarun@deepstacksoft.com> Reviewed-on: cerc-io/stack-orchestrator#820 Co-authored-by: Prathamesh Musale <prathamesh.musale0@gmail.com> Co-committed-by: Prathamesh Musale <prathamesh.musale0@gmail.com>	2024-05-07 04:18:45 +00:00