Add a local stack for ERC20 watcher #80

Merged
prathamesh0 merged 8 commits from pm-erc20-watcher-stack into main 2022-12-22 06:03:39 +00:00
12 changed files with 297 additions and 5 deletions

View File

@ -13,6 +13,7 @@ cerc/laconic-cns-cli
cerc/fixturenet-eth-geth
cerc/fixturenet-eth-lighthouse
cerc/watcher-mobymask
cerc/watcher-erc20
cerc/test-container
cerc/eth-probe
cerc/builder-js

View File

@ -11,6 +11,7 @@ laconicd
fixturenet-laconicd
fixturenet-eth
watcher-mobymask
watcher-erc20
test
eth-probe
keycloak

View File

@ -30,4 +30,9 @@ services:
ports:
- "127.0.0.1:8081:8081"
- "127.0.0.1:8082:8082"
healthcheck:
test: ["CMD", "nc", "-v", "localhost", "8081"]
interval: 20s
timeout: 5s
retries: 15
start_period: 5s

View File

@ -0,0 +1,49 @@
version: '3.2'
services:
watcher-db:
restart: unless-stopped
image: postgres:14-alpine
environment:
- POSTGRES_USER=vdbm
- POSTGRES_MULTIPLE_DATABASES=erc20-watcher,erc20-watcher-job-queue
- POSTGRES_EXTENSION=erc20-watcher-job-queue:pgcrypto
- POSTGRES_PASSWORD=password
volumes:
- ../config/postgresql/multiple-postgressql-databases.sh:/docker-entrypoint-initdb.d/multiple-postgressql-databases.sh
- watcher_db_data:/var/lib/postgresql/data
ports:
- "0.0.0.0:15432:5432"
healthcheck:
test: ["CMD", "nc", "-v", "localhost", "5432"]
interval: 20s
timeout: 5s
retries: 15
start_period: 10s
erc20-watcher:
restart: unless-stopped
depends_on:
ipld-eth-server:
condition: service_healthy
watcher-db:
condition: service_healthy
image: cerc/watcher-erc20:local
environment:
- ETH_RPC_URL=http://go-ethereum:8545
command: ["sh", "-c", "yarn server"]
volumes:
- ../config/watcher-erc20/erc20-watcher.toml:/app/packages/erc20-watcher/environments/local.toml
ports:
- "0.0.0.0:3001:3001"
- "0.0.0.0:9001:9001"
healthcheck:
test: ["CMD", "nc", "-v", "localhost", "3001"]
interval: 20s
timeout: 5s
retries: 15
start_period: 5s
volumes:
watcher_db_data:

View File

@ -0,0 +1,41 @@
[server]
host = "0.0.0.0"
port = 3001
mode = "storage"
kind = "lazy"
[metrics]
host = "127.0.0.1"
port = 9000
[metrics.gql]
port = 9001
[database]
type = "postgres"
host = "watcher-db"
port = 5432
database = "erc20-watcher"
username = "vdbm"
password = "password"
synchronize = true
logging = false
maxQueryExecutionTime = 100
[upstream]
[upstream.ethServer]
gqlApiEndpoint = "http://ipld-eth-server:8082/graphql"
rpcProviderEndpoint = "http://ipld-eth-server:8081"
[upstream.cache]
name = "requests"
enabled = false
deleteOnStart = false
[jobQueue]
dbConnectionString = "postgres://vdbm:password@watcher-db:5432/erc20-watcher-job-queue"
maxCompletionLagInSecs = 300
jobDelayInMilliSecs = 100
eventsInBatch = 50
blockDelayInMilliSecs = 2000
prefetchBlocksInMem = true
prefetchBlockCount = 10

View File

@ -37,8 +37,8 @@
[upstream]
[upstream.ethServer]
gqlApiEndpoint = "http://ipld-eth-server:8083/graphql"
rpcProviderEndpoint = "http://ipld-eth-server:8082"
gqlApiEndpoint = "http://ipld-eth-server:8082/graphql"
rpcProviderEndpoint = "http://ipld-eth-server:8081"
blockDelayInMilliSecs = 60000
[upstream.cache]

View File

@ -10,7 +10,7 @@
"petersburgBlock": 0,
"istanbulBlock": 0,
"clique": {
"period": 2,
"period": 5,
"epoch": 3000
}
},

View File

@ -0,0 +1,13 @@
FROM node:16.17.1-alpine3.16
RUN apk --update --no-cache add git python3 alpine-sdk
WORKDIR /app
COPY . .
RUN echo "Building watcher-ts" && \
git checkout v0.2.19 && \
Review

i get error: pathspec 'v0.2.19' did not match any file(s) known to git
even though i see the latest watcher-ts release of v0.2.19 is over an hour ago. checkout works locally, maybe be that the cache docker pulls takes a bit to update?

i get `error: pathspec 'v0.2.19' did not match any file(s) known to git` even though i see the latest watcher-ts release of `v0.2.19` is over an hour ago. checkout works locally, maybe be that the cache docker pulls takes a bit to update?
prathamesh0 commented 2022-12-21 14:51:38 +00:00 (Migrated from github.com)
Review

@zramsay
Thanks for catching; it's actually because we need to pull the repo if it already exists in the dev-root directory.
Should be fixed now.

@zramsay Thanks for catching; it's actually because we need to pull the repo if it already exists in the dev-root directory. Should be fixed now.
Review

awesome, yep --pull did the trick, thanks

awesome, yep `--pull` did the trick, thanks
Review

A special branch should be implemented in laconic-so setup-repositories. I think there is already some support for non-main branches there but it's been a while since I tried it.

A special branch should be implemented in `laconic-so setup-repositories`. I think there is already some support for non-main branches there but it's been a while since I tried it.
Review

oh i see, --pull is only required because this PR isn't merged on to main, makes sense

oh i see, `--pull` is only required because this PR isn't merged on to main, makes sense
prathamesh0 commented 2022-12-22 05:33:08 +00:00 (Migrated from github.com)
Review

I think there is already some support for non-main branches there but it's been a while since I tried it.

@dboreham
I see that checking out to branches is currently supported using the --branches-file option.
However, when I try checking out to the required release tag, it works if the HEAD is pointing to a branch and throws the following error if it's detached (checked out to a release tag for eg.):

(venv) $ laconic-so --verbose setup-repositories --include cerc-io/watcher-ts --pull --branches-file branches.txt
loading branches from: branches.txt
Branches are: ['cerc-io/watcher-ts v0.2.18']
.
.
  File "/home/prathamesh/deepstack/stack-orchestrator/app/setup_repositories.py", line 116, in process_repo
    present_text = f"already exists active branch: {git.Repo(full_filesystem_repo_path).active_branch}" if is_present \
  File "/home/prathamesh/deepstack/stack-orchestrator/venv/lib/python3.10/site-packages/git/repo/base.py", line 865, in active_branch
    return self.head.reference
  File "/home/prathamesh/deepstack/stack-orchestrator/venv/lib/python3.10/site-packages/git/refs/symbolic.py", line 309, in _get_reference
    raise TypeError("%s is a detached symbolic reference as it points to %r" % (self, sha))
TypeError: HEAD is a detached symbolic reference as it points to '9566dff4664f1f9620d2edd6a3ce78a7c6ca40c3'
> I think there is already some support for non-main branches there but it's been a while since I tried it. @dboreham I see that checking out to branches is currently supported using the `--branches-file` option. However, when I try checking out to the required release tag, it works if the `HEAD` is pointing to a branch and throws the following error if it's detached (checked out to a release tag for eg.): ``` (venv) $ laconic-so --verbose setup-repositories --include cerc-io/watcher-ts --pull --branches-file branches.txt loading branches from: branches.txt Branches are: ['cerc-io/watcher-ts v0.2.18'] . . File "/home/prathamesh/deepstack/stack-orchestrator/app/setup_repositories.py", line 116, in process_repo present_text = f"already exists active branch: {git.Repo(full_filesystem_repo_path).active_branch}" if is_present \ File "/home/prathamesh/deepstack/stack-orchestrator/venv/lib/python3.10/site-packages/git/repo/base.py", line 865, in active_branch return self.head.reference File "/home/prathamesh/deepstack/stack-orchestrator/venv/lib/python3.10/site-packages/git/refs/symbolic.py", line 309, in _get_reference raise TypeError("%s is a detached symbolic reference as it points to %r" % (self, sha)) TypeError: HEAD is a detached symbolic reference as it points to '9566dff4664f1f9620d2edd6a3ce78a7c6ca40c3' ```
prathamesh0 commented 2022-12-22 05:36:19 +00:00 (Migrated from github.com)
Review

oh i see, --pull is only required because this PR isn't merged on to main, makes sense

--pull is actually required because the repo we're trying to setup has a new remote branch / tag.

> oh i see, --pull is only required because this PR isn't merged on to main, makes sense `--pull` is actually required because the repo we're trying to setup has a new remote branch / tag.
Review

Hmm...ok. But do we need to support the "already detached" state? Wouldn't the repo have been cloned by stack-orchestrator, hence it wouldn't be detached?

Hmm...ok. But do we need to support the "already detached" state? Wouldn't the repo have been cloned by stack-orchestrator, hence it wouldn't be detached?
prathamesh0 commented 2022-12-22 05:43:11 +00:00 (Migrated from github.com)
Review

hence it wouldn't be detached

It might be in a detached state when we run setup-repositories the next time (re-running the demo for eg.).

> hence it wouldn't be detached It might be in a detached state when we run `setup-repositories` the next time (re-running the demo for eg.).
Review

Ok, this approach is fine until we add proper branch/tag-switching support.

Ok, this approach is fine until we add proper branch/tag-switching support.
yarn && yarn build
WORKDIR /app/packages/erc20-watcher

View File

@ -0,0 +1,7 @@
#!/usr/bin/env bash
# Build cerc/watcher-erc20
# See: https://stackoverflow.com/a/246128/1701505
SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
docker build -t cerc/watcher-erc20:local -f ${SCRIPT_DIR}/Dockerfile ${CERC_REPO_BASE_DIR}/watcher-ts

157
stacks/erc20/README.md Normal file
View File

@ -0,0 +1,157 @@
# ERC20 Watcher
Instructions to deploy a local ERC20 watcher stack (core + watcher) for demonstration and testing purposes using [laconic-stack-orchestrator](../../README.md#setup)
## Setup
* Clone / pull required repositories:
```bash
$ laconic-so setup-repositories --include cerc-io/go-ethereum,cerc-io/ipld-eth-db,cerc-io/ipld-eth-server,cerc-io/watcher-ts --pull
```
* Build the core and watcher container images:
```bash
$ laconic-so build-containers --include cerc/go-ethereum,cerc/go-ethereum-foundry,cerc/ipld-eth-db,cerc/ipld-eth-server,cerc/watcher-erc20
```
This should create the required docker images in the local image registry.
* Deploy the stack:
```bash
$ laconic-so deploy-system --include db,go-ethereum-foundry,ipld-eth-server,watcher-erc20 up
```
## Demo
* Find the watcher container's id using `docker ps` and export it for later use:
```bash
$ export CONTAINER_ID=<CONTAINER_ID>
```
* Deploy an ERC20 token:
```bash
$ docker exec $CONTAINER_ID yarn token:deploy:docker
```
Export the address of the deployed token to a shell variable for later use:
```bash
$ export TOKEN_ADDRESS=<TOKEN_ADDRESS>
```
* Open `http://localhost:3001/graphql` (GraphQL Playground) in a browser window
* Connect MetaMask to `http://localhost:8545` (with chain ID `99`)
* Add the deployed token as an asset in MetaMask and check that the initial balance is zero
* Export your MetaMask account (second account) address to a shell variable for later use:
```bash
$ export RECIPIENT_ADDRESS=<RECIPIENT_ADDRESS>
```
* To get the primary account's address, run:
```bash
$ docker exec $CONTAINER_ID yarn account:docker
```
* To get the current block hash at any time, run:
```bash
$ docker exec $CONTAINER_ID yarn block:latest:docker
```
* Fire a GQL query in the playground to get the name, symbol and total supply of the deployed token:
```graphql
query {
name(
blockHash: "LATEST_BLOCK_HASH"
token: "TOKEN_ADDRESS"
) {
value
proof {
data
}
}
symbol(
blockHash: "LATEST_BLOCK_HASH"
token: "TOKEN_ADDRESS"
) {
value
proof {
data
}
}
totalSupply(
blockHash: "LATEST_BLOCK_HASH"
token: "TOKEN_ADDRESS"
) {
value
proof {
data
}
}
}
```
* Fire the following query to get balances for the primary and the recipient account at the latest block hash:
```graphql
query {
fromBalanceOf: balanceOf(
blockHash: "LATEST_BLOCK_HASH"
token: "TOKEN_ADDRESS",
# primary account having all the balance initially
owner: "PRIMARY_ADDRESS"
) {
value
proof {
data
}
}
toBalanceOf: balanceOf(
blockHash: "LATEST_BLOCK_HASH"
token: "TOKEN_ADDRESS",
owner: "RECIPIENT_ADDRESS"
) {
value
proof {
data
}
}
}
```
* The initial balance for the primary account should be `1000000000000000000000`
* The initial balance for the recipient should be `0`
* Transfer tokens to the recipient account:
```bash
$ docker exec $CONTAINER_ID yarn token:transfer:docker --token $TOKEN_ADDRESS --to $RECIPIENT_ADDRESS --amount 100
```
* Fire the above GQL query again with the latest block hash to get updated balances for the primary (`from`) and the recipient (`to`) account:
* The balance for the primary account should be reduced by the transfer amount (`100`)
* The balance for the recipient account should be equal to the transfer amount (`100`)
* Transfer funds between different accounts using MetaMask and use the playground to query the balance before and after the transfer.
## Clean up
* To stop all the services running in background run:
```bash
$ laconic-so deploy-system --include db,go-ethereum-foundry,ipld-eth-server,watcher-erc20 down
```

18
stacks/erc20/stack.yml Normal file
View File

@ -0,0 +1,18 @@
version: "1.0"
name: erc20-watcher
repos:
- cerc-io/go-ethereum
- cerc-io/ipld-eth-db
- cerc-io/ipld-eth-server
- cerc-io/watcher-ts
containers:
- cerc/go-ethereum
- cerc/go-ethereum-foundry
- cerc/ipld-eth-db
- cerc/ipld-eth-server
- cerc/watcher-erc20
pods:
- go-ethereum-foundry
- db
- ipld-eth-server
- watcher-erc20