# monitoring

* Instructions to setup and run a Prometheus server and Grafana dashboard
* Comes with the following built-in exporters / dashboards:
  * Chain Head Exporter - for tracking chain heads given external ETH RPC endpoints
  * Watchers dashboard
  * laconicd dashboard
  * [Prometheus Blackbox](https://grafana.com/grafana/dashboards/7587-prometheus-blackbox-exporter/) - for tracking HTTP endpoints
  * [NodeJS Application Dashboard](https://grafana.com/grafana/dashboards/11159-nodejs-application-dashboard/) - for default NodeJS metrics
  * [PostgreSQL Database](https://grafana.com/grafana/dashboards/9628-postgresql-database/) - for monitoring Postgres dbs
  * [Node Exporter Full](https://grafana.com/grafana/dashboards/1860-node-exporter-full/) - for monitoring system metrics
* See [monitoring-watchers.md](./monitoring-watchers.md) for an example usage of the stack with pre-configured dashboards for watchers

## Setup

Clone required repositories:

```bash
laconic-so --stack monitoring setup-repositories --git-ssh --pull
```

Build the container images:

```bash
laconic-so --stack monitoring build-containers
```

## Create a deployment

First, create a spec file for the deployment, which will map the stack's ports and volumes to the host:

```bash
laconic-so --stack monitoring deploy init --output monitoring-spec.yml
```

### Ports

Edit `network` in spec file to map container ports to same ports in host:

```
...
network:
  ports:
    prometheus:
      - '9090:9090'
    grafana:
      - '3000:3000'
...
```

### Data volumes

Container data volumes are bind-mounted to specified paths in the host filesystem.
The default setup (generated by `laconic-so deploy init`) places the volumes in the `./data` subdirectory of the deployment directory. The default mappings can be customized by editing the "spec" file generated by `laconic-so deploy init`.

---

Once you've made any needed changes to the spec file, create a deployment from it:

```bash
laconic-so --stack monitoring deploy create --spec-file monitoring-spec.yml --deployment-dir monitoring-deployment
```

## Configure

### Prometheus Config

* Add desired scrape configs to prometheus config file (`monitoring-deployment/config/monitoring/prometheus/prometheus.yml`) in the deployment folder; for example:

  ```yml
  ...
  - job_name: <JOB_NAME>
    metrics_path: /metrics/path
    scheme: http
    static_configs:
      - targets: ['<METRICS_ENDPOINT_HOST>:<METRICS_ENDPOINT_PORT>']
  ```

* Node exporter: update the `node` job to add any node-exporter targets to be monitored:

  ```yml
  ...
  - job_name: 'node'
    ...
    static_configs:
      # Add node-exporter targets to be monitored below
      - targets: [example-host:9100]
        labels:
          instance: 'my-host'
  ```

* Blackbox (in-stack exporter): update the `blackbox` job to add any endpoints to be monitored on the Blackbox dashboard:

  ```yml
  ...
  - job_name: 'blackbox'
    ...
    static_configs:
      # Add URLs to be monitored below
      - targets:
        - <HTTP_ENDPOINT_1>
        - <HTTP_ENDPOINT_2>
        - <LACONICD_GQL_ENDPOINT>
  ```

* Postgres (in-stack exporter):
  * Update the `postgres` job to add Postgres db targets to be monitored:

    ```yml
    ...
    - job_name: 'postgres'
      ...
      static_configs:
        # Add DB targets below
        - targets: [example-server:5432]
          labels:
            instance: 'example-db'
    ```
  * Add database credentials to be used in `auth_modules` in the postgres-exporter config file (`monitoring-deployment/config/monitoring/postgres-exporter.yml`)

* laconicd: update the `laconicd` job with a laconicd node's REST endpoint host and port:

  ```yml
  ...
  - job_name: laconicd
  static_configs:
    - targets: ['example-host:1317']
  ...
  ```

Note: Use `host.docker.internal` as host to access ports on the host machine

### Grafana Config

Place the dashboard json files in grafana dashboards config directory (`monitoring-deployment/config/monitoring/grafana/dashboards`) in the deployment folder

#### Graph Node Config

For graph-node dashboard postgres datasource needs to be setup in `monitoring-deployment/config/monitoring/grafana/provisioning/datasources/graph-node-postgres.yml` (in deployment folder)

```yml
# graph-node-postgres.yml
...
datasources:
  - name: Graph Node Postgres
    type: postgres
    jsonData:
      # Set name to remote graph-node database name
      database: graph-node
      ...
    # Set user to remote graph-node database username
    user: graph-node
    # Add URL for remote graph-node database
    url: graph-node-db:5432
    # Set password for graph-node database
    secureJsonData:
      password: 'password'
```

### Env

Set the following env variables in the deployment env config file (`monitoring-deployment/config.env`):

  ```bash
  # For chain-head exporter

  # External ETH RPC endpoint (ethereum)
  # (Optional, default: https://mainnet.infura.io/v3)
  CERC_ETH_RPC_ENDPOINT=

  # Infura key to be used
  # (Optional, used with ETH_RPC_ENDPOINT if provided)
  CERC_INFURA_KEY=

  # External ETH RPC endpoint (filecoin)
  # (Optional, default: https://api.node.glif.io/rpc/v1)
  CERC_FIL_RPC_ENDPOINT=

  # Grafana server host URL (used in various links in alerts, etc.)
  # (Optional, default: http://localhost:3000)
  GF_SERVER_ROOT_URL=

  
  # RPC endpoint used by graph-node for upstream head metric
  # (Optional, default: https://mainnet.infura.io/v3)
  GRAPH_NODE_RPC_ENDPOINT=
  ```

## Start the stack

Start the deployment:

```bash
laconic-so deployment --dir monitoring-deployment start
```

* List and check the health status of all the containers using `docker ps` and wait for them to be `healthy`

* Grafana should now be visible at http://localhost:3000 with configured dashboards

## Clean up

To stop monitoring services running in the background, while preserving data:

```bash
# Only stop the docker containers
laconic-so deployment --dir monitoring-deployment stop

# Run 'start' to restart the deployment
```

To stop monitoring services and also delete data:

```bash
# Stop the docker containers
laconic-so deployment --dir monitoring-deployment stop --delete-volumes

# Remove deployment directory (deployment will have to be recreated for a re-run)
rm -rf monitoring-deployment
```
