From 85452a870f7a6596c5554a55e408c8eaca876c63 Mon Sep 17 00:00:00 2001 From: zramsay Date: Tue, 10 Jan 2023 07:39:29 -0500 Subject: [PATCH] combine demo & readme, graph-test-watcher; start docs/cli --- docs/cli.md | 136 ++++++++++++- packages/erc20-watcher/README.md | 2 +- packages/graph-test-watcher/README.md | 275 +++++++++++++------------- packages/graph-test-watcher/demo.md | 182 ----------------- 4 files changed, 269 insertions(+), 326 deletions(-) delete mode 100644 packages/graph-test-watcher/demo.md diff --git a/docs/cli.md b/docs/cli.md index 27f93087..43d83ee9 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -1,3 +1,137 @@ ## Watcher CLI commands (yarn) -spot to explain core commands and flags so they aren't repeated in every tutorial +* Run the server: + + ```bash + yarn server + ``` + + GQL console: http://localhost:3008/graphql + +* If the watcher is an `active` watcher: + + * Run the job-runner: + + ```bash + yarn job-runner + ``` + + * Run the server: + + ```bash + yarn server + ``` + + GQL console: http://localhost:3008/graphql + + * To watch a contract: + + ```bash + yarn watch:contract --address --kind --checkpoint --starting-block [block-number] + ``` + + * `address`: Address or identifier of the contract to be watched. + * `kind`: Kind of the contract. + * `checkpoint`: Turn checkpointing on (`true` | `false`). + * `starting-block`: Starting block for the contract (default: `1`). + + Examples: + + Watch a contract with its address and checkpointing on: + + ```bash + yarn watch:contract --address 0x1F78641644feB8b64642e833cE4AFE93DD6e7833 --kind ERC20 --checkpoint true + ``` + + Watch a contract with its identifier and checkpointing on: + + ```bash + yarn watch:contract --address MyProtocol --kind protocol --checkpoint true + ``` + + * To fill a block range: + + ```bash + yarn fill --start-block --end-block + ``` + + * `start-block`: Block number to start filling from. + * `end-block`: Block number till which to fill. + + * To create a checkpoint for a contract: + + ```bash + yarn checkpoint create --address --block-hash [block-hash] + ``` + + * `address`: Address or identifier of the contract for which to create a checkpoint. + * `block-hash`: Hash of a block (in the pruned region) at which to create the checkpoint (default: latest canonical block hash). + + * To verify a checkpoint: + + ```bash + yarn checkpoint verify --cid + ``` + + `cid`: CID of the checkpoint for which to verify. + + * To reset the watcher to a previous block number: + + * Reset watcher: + + ```bash + yarn reset watcher --block-number + ``` + + * Reset job-queue: + + ```bash + yarn reset job-queue + ``` + + * Reset state: + + ```bash + yarn reset state --block-number + ``` + + * `block-number`: Block number to which to reset the watcher. + + * To export and import the watcher state: + + * In source watcher, export watcher state: + + ```bash + yarn export-state --export-file [export-file-path] --block-number [snapshot-block-height] + ``` + + * `export-file`: Path of file to which to export the watcher data. + * `block-number`: Block height at which to take snapshot for export. + + * In target watcher, run job-runner: + + ```bash + yarn job-runner + ``` + + * Import watcher state: + + ```bash + yarn import-state --import-file + ``` + + * `import-file`: Path of file from which to import the watcher data. + + * Run server: + + ```bash + yarn server + ``` + + * To inspect a CID: + + ```bash + yarn inspect-cid --cid + ``` + + * `cid`: CID to be inspected. diff --git a/packages/erc20-watcher/README.md b/packages/erc20-watcher/README.md index 49c20be4..a31fb83d 100644 --- a/packages/erc20-watcher/README.md +++ b/packages/erc20-watcher/README.md @@ -31,7 +31,7 @@ yarn server ``` For development or to specify the config file: -```basj +```bash yarn server:dev yarn server -f environments/local.toml ``` diff --git a/packages/graph-test-watcher/README.md b/packages/graph-test-watcher/README.md index 39217475..fbf4aa3d 100644 --- a/packages/graph-test-watcher/README.md +++ b/packages/graph-test-watcher/README.md @@ -2,12 +2,143 @@ ## Setup -* Run the following command to install required packages: +First try the [stack orchestrator](https://github.com/cerc-io/stack-orchestrator) to quickly get started. Advanced users can see [here](/docs/README.md) for instructions on setting up a local environment by hand. + +Run the following command to install required packages: + +```bash +yarn +``` + +## Run + +* In [packages/graph-node](../graph-node/), deploy an `Example` contract: ```bash - yarn + yarn example:deploy ``` +* Set the returned address to the variable `$EXAMPLE_ADDRESS`: + + ```bash + EXAMPLE_ADDRESS= + ``` + +* In [packages/graph-node/test/subgraph/example1/subgraph.yaml](../graph-node/test/subgraph/example1/subgraph.yaml): + + * Set the source address for `Example1` datasource to the `EXAMPLE_ADDRESS`. + * Set the `startBlock` less than or equal to the latest mined block. + +* Build the example subgraph: + + ```bash + yarn build:example + ``` + +* Run the job-runner: + + ```bash + yarn job-runner + ``` + +* Run the watcher: + + ```bash + yarn server + ``` + +* The output from the block handler in the mapping code should be visible in the `job-runner` for each block. + +* Run the following GQL subscription at the graphql endpoint http://127.0.0.1:3008/graphql + + ```graphql + subscription { + onEvent { + event { + __typename + ... on TestEvent { + param1 + param2 + }, + }, + block { + number + hash + } + } + } + ``` + +* In [packages/graph-node](../graph-node/), trigger the `Test` event by calling a example contract method: + + ```bash + yarn example:test --address $EXAMPLE_ADDRESS + ``` + + * A `Test` event shall be visible in the subscription at endpoint. + + * The subgraph entity `Category` should be updated in the database. + + * An auto-generated `diff-staged` entry `State` should be added. + +* Run the query for entity in at the endpoint: + + ```graphql + query { + category( + block: { hash: "EVENT_BLOCK_HASH" }, + id: "1" + ) { + __typename + id + count + name + } + } + ``` + +* Run the `getState` query at the endpoint to get the latest `State` for `EXAMPLE_ADDRESS`: + + ```graphql + query { + getState ( + blockHash: "EVENT_BLOCK_HASH" + contractAddress: "EXAMPLE_ADDRESS" + # kind: "checkpoint" + # kind: "diff" + kind: "diff_staged" + ) { + cid + block { + cid + hash + number + timestamp + parentHash + } + contractAddress + data + } + } + ``` + +* `diff` states get created corresponding to the `diff_staged` states when their respective blocks reach the pruned region. + +* In [packages/graph-test-watcher](./): + + * After the `diff` state has been created, create a `checkpoint`: + + ```bash + yarn checkpoint create --address $EXAMPLE_ADDRESS + ``` + + * A `checkpoint` state should be created at the latest canonical block hash. + + * Run the `getState` query again at the endpoint with the output `blockHash` and kind `checkpoint`. + +* All the `State` entries can be seen in `pg-admin` in table `state`. + + ## Customize * Indexing on an event: @@ -25,143 +156,3 @@ * Edit the custom hook function `createStateCheckpoint` (triggered just before default and CLI checkpoint) in [hooks.ts](./src/hooks.ts) to save the state in a `checkpoint` `State` using the `Indexer` object. * The existing example hooks in [hooks.ts](./src/hooks.ts) are for an `ERC20` contract. - -## Run - -Follow the steps below or follow the [Demo](./demo.md) - -* Run the server: - - ```bash - yarn server - ``` - - GQL console: http://localhost:3008/graphql - -* If the watcher is an `active` watcher: - - * Run the job-runner: - - ```bash - yarn job-runner - ``` - - * Run the server: - - ```bash - yarn server - ``` - - GQL console: http://localhost:3008/graphql - - * To watch a contract: - - ```bash - yarn watch:contract --address --kind --checkpoint --starting-block [block-number] - ``` - - * `address`: Address or identifier of the contract to be watched. - * `kind`: Kind of the contract. - * `checkpoint`: Turn checkpointing on (`true` | `false`). - * `starting-block`: Starting block for the contract (default: `1`). - - Examples: - - Watch a contract with its address and checkpointing on: - - ```bash - yarn watch:contract --address 0x1F78641644feB8b64642e833cE4AFE93DD6e7833 --kind ERC20 --checkpoint true - ``` - - Watch a contract with its identifier and checkpointing on: - - ```bash - yarn watch:contract --address MyProtocol --kind protocol --checkpoint true - ``` - - * To fill a block range: - - ```bash - yarn fill --start-block --end-block - ``` - - * `start-block`: Block number to start filling from. - * `end-block`: Block number till which to fill. - - * To create a checkpoint for a contract: - - ```bash - yarn checkpoint create --address --block-hash [block-hash] - ``` - - * `address`: Address or identifier of the contract for which to create a checkpoint. - * `block-hash`: Hash of a block (in the pruned region) at which to create the checkpoint (default: latest canonical block hash). - - * To verify a checkpoint: - - ```bash - yarn checkpoint verify --cid - ``` - - `cid`: CID of the checkpoint for which to verify. - - * To reset the watcher to a previous block number: - - * Reset watcher: - - ```bash - yarn reset watcher --block-number - ``` - - * Reset job-queue: - - ```bash - yarn reset job-queue - ``` - - * Reset state: - - ```bash - yarn reset state --block-number - ``` - - * `block-number`: Block number to which to reset the watcher. - - * To export and import the watcher state: - - * In source watcher, export watcher state: - - ```bash - yarn export-state --export-file [export-file-path] --block-number [snapshot-block-height] - ``` - - * `export-file`: Path of file to which to export the watcher data. - * `block-number`: Block height at which to take snapshot for export. - - * In target watcher, run job-runner: - - ```bash - yarn job-runner - ``` - - * Import watcher state: - - ```bash - yarn import-state --import-file - ``` - - * `import-file`: Path of file from which to import the watcher data. - - * Run server: - - ```bash - yarn server - ``` - - * To inspect a CID: - - ```bash - yarn inspect-cid --cid - ``` - - * `cid`: CID to be inspected. diff --git a/packages/graph-test-watcher/demo.md b/packages/graph-test-watcher/demo.md deleted file mode 100644 index deb84e0d..00000000 --- a/packages/graph-test-watcher/demo.md +++ /dev/null @@ -1,182 +0,0 @@ -# Demo - -* The following core services need to be running for the demo: - * [ipld-eth-db](https://github.com/cerc-io/ipld-eth-db) - * Version: [v4.2.3-alpha](https://github.com/cerc-io/ipld-eth-db/releases/tag/v4.2.3-alpha) - * [geth](https://github.com/cerc-io/go-ethereum) - * State diffing service should use `ipld-eth-db` for database. - * Version: [v1.10.26-statediff-4.2.2-alpha](https://github.com/cerc-io/go-ethereum/releases/tag/v1.10.26-statediff-4.2.2-alpha) - * Endpoint: http://127.0.0.1:8545 - * [ipld-eth-server](https://github.com/cerc-io/ipld-eth-server) - * Should use `ipld-eth-db` for database. - * Version: [v4.2.3-alpha](https://github.com/cerc-io/ipld-eth-server/releases/tag/v4.2.3-alpha) - * Endpoints: - * GQL: http://127.0.0.1:8082/graphql - * RPC: http://127.0.0.1:8081 - -* Create a postgres12 database for the watcher: - - ```bash - sudo su - postgres - - # If database already exists - # dropdb graph-test-watcher - - createdb graph-test-watcher - ``` - -* Create database for the job queue and enable the `pgcrypto` extension on them (https://github.com/timgit/pg-boss/blob/master/docs/usage.md#intro): - - ```bash - # If database already exists - # dropdb graph-test-watcher-job-queue - - createdb graph-test-watcher-job-queue - ``` - - ``` - postgres@tesla:~$ psql -U postgres -h localhost graph-test-watcher-job-queue - Password for user postgres: - psql (12.7 (Ubuntu 12.7-1.pgdg18.04+1)) - SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, bits: 256, compression: off) - Type "help" for help. - - graph-test-watcher-job-queue=# CREATE EXTENSION pgcrypto; - CREATE EXTENSION - graph-test-watcher-job-queue=# exit - ``` - -* In the [config file](./environments/local.toml) update the `database` connection settings. - -* In `watcher-ts` repo, follow the instructions in [Setup](../../README.md#setup) for installing and building packages. - - ```bash - # After setup - yarn && yarn build - ``` - -* In [packages/graph-node](../graph-node/), deploy an `Example` contract: - - ```bash - yarn example:deploy - ``` - -* Set the returned address to the variable `$EXAMPLE_ADDRESS`: - - ```bash - EXAMPLE_ADDRESS= - ``` - -* In [packages/graph-node/test/subgraph/example1/subgraph.yaml](../graph-node/test/subgraph/example1/subgraph.yaml): - - * Set the source address for `Example1` datasource to the `EXAMPLE_ADDRESS`. - * Set the `startBlock` less than or equal to the latest mined block. - -* Build the example subgraph: - - ```bash - yarn build:example - ``` - -* Run the job-runner: - - ```bash - yarn job-runner - ``` - -* Run the watcher: - - ```bash - yarn server - ``` - -* The output from the block handler in the mapping code should be visible in the `job-runner` for each block. - -* Run the following GQL subscription at the graphql endpoint http://127.0.0.1:3008/graphql - - ```graphql - subscription { - onEvent { - event { - __typename - ... on TestEvent { - param1 - param2 - }, - }, - block { - number - hash - } - } - } - ``` - -* In [packages/graph-node](../graph-node/), trigger the `Test` event by calling a example contract method: - - ```bash - yarn example:test --address $EXAMPLE_ADDRESS - ``` - - * A `Test` event shall be visible in the subscription at endpoint. - - * The subgraph entity `Category` should be updated in the database. - - * An auto-generated `diff-staged` entry `State` should be added. - -* Run the query for entity in at the endpoint: - - ```graphql - query { - category( - block: { hash: "EVENT_BLOCK_HASH" }, - id: "1" - ) { - __typename - id - count - name - } - } - ``` - -* Run the `getState` query at the endpoint to get the latest `State` for `EXAMPLE_ADDRESS`: - - ```graphql - query { - getState ( - blockHash: "EVENT_BLOCK_HASH" - contractAddress: "EXAMPLE_ADDRESS" - # kind: "checkpoint" - # kind: "diff" - kind: "diff_staged" - ) { - cid - block { - cid - hash - number - timestamp - parentHash - } - contractAddress - data - } - } - ``` - -* `diff` states get created corresponding to the `diff_staged` states when their respective blocks reach the pruned region. - -* In [packages/graph-test-watcher](./): - - * After the `diff` state has been created, create a `checkpoint`: - - ```bash - yarn checkpoint create --address $EXAMPLE_ADDRESS - ``` - - * A `checkpoint` state should be created at the latest canonical block hash. - - * Run the `getState` query again at the endpoint with the output `blockHash` and kind `checkpoint`. - -* All the `State` entries can be seen in `pg-admin` in table `state`.