cerc-io/stack-orchestrator
17
1
Fork 3
Code Issues 113 Pull Requests 18 Actions Packages Projects Releases 480 Wiki Activity

Add a local stack for ERC20 watcher #80

Merged
prathamesh0 merged 8 commits from pm-erc20-watcher-stack into main 2022-12-22 06:03:39 +00:00
Conversation 1 Commits 8 Files Changed 12 +152
8 changed files with 152 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files
Showing only changes of commit 633a72f5fa - Show all commits

1
app/data/container-image-list.txt
View File

@ -13,6 +13,7 @@ cerc/laconic-cns-cli
cerc/fixturenet-eth-geth
cerc/fixturenet-eth-lighthouse
cerc/watcher-mobymask
cerc/watcher-erc20
cerc/test-container
cerc/eth-probe
cerc/builder-js

1
app/data/pod-list.txt
View File

@ -11,6 +11,7 @@ laconicd
fixturenet-laconicd
fixturenet-eth
watcher-mobymask
watcher-erc20
test
eth-probe
keycloak

49
compose/docker-compose-watcher-erc20.yml Normal file
View File

@ -0,0 +1,49 @@
version: '3.2'
services:
watcher-db:
restart: unless-stopped
image: postgres:14-alpine
environment:
- POSTGRES_USER=vdbm
- POSTGRES_MULTIPLE_DATABASES=erc20-watcher,erc20-watcher-job-queue
- POSTGRES_EXTENSION=erc20-watcher-job-queue:pgcrypto
- POSTGRES_PASSWORD=password
volumes:
- ../config/postgresql/multiple-postgressql-databases.sh:/docker-entrypoint-initdb.d/multiple-postgressql-databases.sh
- watcher_db_data:/var/lib/postgresql/data
ports:
- "0.0.0.0:15432:5432"
healthcheck:
test: ["CMD", "nc", "-v", "localhost", "5432"]
interval: 20s
timeout: 5s
retries: 15
start_period: 10s
erc20-watcher:
restart: unless-stopped
depends_on:
ipld-eth-server:
condition: service_healthy
watcher-db:
condition: service_healthy
image: cerc/watcher-erc20:local
environment:
- ETH_RPC_URL=http://go-ethereum:8545
command: ["sh", "-c", "yarn server"]
volumes:
- ../config/watcher-erc20/erc20-watcher.toml:/app/packages/erc20-watcher/environments/local.toml
ports:
- "0.0.0.0:3001:3001"
- "0.0.0.0:9001:9001"
healthcheck:
test: ["CMD", "nc", "-v", "localhost", "3001"]
interval: 20s
timeout: 5s
retries: 15
start_period: 5s
volumes:
watcher_db_data:

41
config/watcher-erc20/erc20-watcher.toml Normal file
View File

@ -0,0 +1,41 @@
[server]
host = "0.0.0.0"
port = 3001
mode = "storage"
kind = "lazy"
[metrics]
host = "127.0.0.1"
port = 9000
[metrics.gql]
port = 9001
[database]
type = "postgres"
host = "watcher-db"
port = 5432
database = "erc20-watcher"
username = "vdbm"
password = "password"
synchronize = true
logging = false
maxQueryExecutionTime = 100
[upstream]
[upstream.ethServer]
gqlApiEndpoint = "http://ipld-eth-server:8082/graphql"
rpcProviderEndpoint = "http://ipld-eth-server:8081"
[upstream.cache]
name = "requests"
enabled = false
deleteOnStart = false
[jobQueue]
dbConnectionString = "postgres://vdbm:password@watcher-db:5432/erc20-watcher-job-queue"
maxCompletionLagInSecs = 300
jobDelayInMilliSecs = 100
eventsInBatch = 50
blockDelayInMilliSecs = 2000
prefetchBlocksInMem = true
prefetchBlockCount = 10

13
container-build/cerc-watcher-erc20/Dockerfile Normal file
View File

@ -0,0 +1,13 @@
FROM node:16.17.1-alpine3.16
RUN apk --update --no-cache add git python3 alpine-sdk
WORKDIR /app
COPY . .
RUN echo "Building watcher-ts" && \
git checkout v0.2.18 && \
zramsay commented 2022-12-21 14:27:54 +00:00
Review
Copy Link

i get error: pathspec 'v0.2.19' did not match any file(s) known to git
even though i see the latest watcher-ts release of v0.2.19 is over an hour ago. checkout works locally, maybe be that the cache docker pulls takes a bit to update?

i get `error: pathspec 'v0.2.19' did not match any file(s) known to git` even though i see the latest watcher-ts release of `v0.2.19` is over an hour ago. checkout works locally, maybe be that the cache docker pulls takes a bit to update?
prathamesh0 commented 2022-12-21 14:51:38 +00:00 (Migrated from github.com)
Review
Copy Link

@zramsay
Thanks for catching; it's actually because we need to pull the repo if it already exists in the dev-root directory.
Should be fixed now.

@zramsay Thanks for catching; it's actually because we need to pull the repo if it already exists in the dev-root directory. Should be fixed now.
zramsay commented 2022-12-21 14:55:33 +00:00
Review
Copy Link

awesome, yep --pull did the trick, thanks

awesome, yep `--pull` did the trick, thanks
dboreham commented 2022-12-21 15:00:08 +00:00
Review
Copy Link

A special branch should be implemented in laconic-so setup-repositories. I think there is already some support for non-main branches there but it's been a while since I tried it.

A special branch should be implemented in `laconic-so setup-repositories`. I think there is already some support for non-main branches there but it's been a while since I tried it.
zramsay commented 2022-12-21 16:14:16 +00:00
Review
Copy Link

oh i see, --pull is only required because this PR isn't merged on to main, makes sense

oh i see, `--pull` is only required because this PR isn't merged on to main, makes sense
prathamesh0 commented 2022-12-22 05:33:08 +00:00 (Migrated from github.com)
Review
Copy Link

I think there is already some support for non-main branches there but it's been a while since I tried it.

@dboreham
I see that checking out to branches is currently supported using the --branches-file option.
However, when I try checking out to the required release tag, it works if the HEAD is pointing to a branch and throws the following error if it's detached (checked out to a release tag for eg.):

(venv) $ laconic-so --verbose setup-repositories --include cerc-io/watcher-ts --pull --branches-file branches.txt
loading branches from: branches.txt
Branches are: ['cerc-io/watcher-ts v0.2.18']
.
.
  File "/home/prathamesh/deepstack/stack-orchestrator/app/setup_repositories.py", line 116, in process_repo
    present_text = f"already exists active branch: {git.Repo(full_filesystem_repo_path).active_branch}" if is_present \
  File "/home/prathamesh/deepstack/stack-orchestrator/venv/lib/python3.10/site-packages/git/repo/base.py", line 865, in active_branch
    return self.head.reference
  File "/home/prathamesh/deepstack/stack-orchestrator/venv/lib/python3.10/site-packages/git/refs/symbolic.py", line 309, in _get_reference
    raise TypeError("%s is a detached symbolic reference as it points to %r" % (self, sha))
TypeError: HEAD is a detached symbolic reference as it points to '9566dff4664f1f9620d2edd6a3ce78a7c6ca40c3'
> I think there is already some support for non-main branches there but it's been a while since I tried it. @dboreham I see that checking out to branches is currently supported using the `--branches-file` option. However, when I try checking out to the required release tag, it works if the `HEAD` is pointing to a branch and throws the following error if it's detached (checked out to a release tag for eg.): ``` (venv) $ laconic-so --verbose setup-repositories --include cerc-io/watcher-ts --pull --branches-file branches.txt loading branches from: branches.txt Branches are: ['cerc-io/watcher-ts v0.2.18'] . . File "/home/prathamesh/deepstack/stack-orchestrator/app/setup_repositories.py", line 116, in process_repo present_text = f"already exists active branch: {git.Repo(full_filesystem_repo_path).active_branch}" if is_present \ File "/home/prathamesh/deepstack/stack-orchestrator/venv/lib/python3.10/site-packages/git/repo/base.py", line 865, in active_branch return self.head.reference File "/home/prathamesh/deepstack/stack-orchestrator/venv/lib/python3.10/site-packages/git/refs/symbolic.py", line 309, in _get_reference raise TypeError("%s is a detached symbolic reference as it points to %r" % (self, sha)) TypeError: HEAD is a detached symbolic reference as it points to '9566dff4664f1f9620d2edd6a3ce78a7c6ca40c3' ```
prathamesh0 commented 2022-12-22 05:36:19 +00:00 (Migrated from github.com)
Review
Copy Link

oh i see, --pull is only required because this PR isn't merged on to main, makes sense

--pull is actually required because the repo we're trying to setup has a new remote branch / tag.

> oh i see, --pull is only required because this PR isn't merged on to main, makes sense `--pull` is actually required because the repo we're trying to setup has a new remote branch / tag.
dboreham commented 2022-12-22 05:37:11 +00:00
Review
Copy Link

Hmm...ok. But do we need to support the "already detached" state? Wouldn't the repo have been cloned by stack-orchestrator, hence it wouldn't be detached?

Hmm...ok. But do we need to support the "already detached" state? Wouldn't the repo have been cloned by stack-orchestrator, hence it wouldn't be detached?
prathamesh0 commented 2022-12-22 05:43:11 +00:00 (Migrated from github.com)
Review
Copy Link

hence it wouldn't be detached

It might be in a detached state when we run setup-repositories the next time (re-running the demo for eg.).

> hence it wouldn't be detached It might be in a detached state when we run `setup-repositories` the next time (re-running the demo for eg.).
dboreham commented 2022-12-22 05:47:01 +00:00
Review
Copy Link

Ok, this approach is fine until we add proper branch/tag-switching support.

Ok, this approach is fine until we add proper branch/tag-switching support.
yarn && yarn build
WORKDIR /app/packages/erc20-watcher

7
container-build/cerc-watcher-erc20/build.sh Executable file
View File

@ -0,0 +1,7 @@
#!/usr/bin/env bash
# Build cerc/watcher-erc20
# See: https://stackoverflow.com/a/246128/1701505
SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
docker build -t cerc/watcher-erc20:local -f ${SCRIPT_DIR}/Dockerfile ${CERC_REPO_BASE_DIR}/watcher-ts

22
stacks/erc20/README.md Normal file
View File

@ -0,0 +1,22 @@
# ERC20 Watcher
## Setup
### Clone required repositories
```bash
$ laconic-so setup-repositories --include cerc-io/go-ethereum,cerc-io/ipld-eth-db,cerc-io/ipld-eth-server,cerc-io/watcher-ts
```
### Build the core and watcher containers
```bash
$ laconic-so build-containers --include cerc/go-ethereum,cerc/go-ethereum-foundry,cerc/ipld-eth-db,cerc/ipld-eth-server,cerc/watcher-erc20
```
### Deploy the stack
```bash
$ laconic-so deploy-system --include db,go-ethereum-foundry,ipld-eth-server,watcher-erc20 up
```

18
stacks/erc20/stack.yml Normal file
View File

@ -0,0 +1,18 @@
version: "1.0"
name: erc20-watcher
repos:
- cerc-io/go-ethereum
- cerc-io/ipld-eth-db
- cerc-io/ipld-eth-server
- cerc-io/watcher-ts
containers:
- cerc/go-ethereum
- cerc/go-ethereum-foundry
- cerc/ipld-eth-db
- cerc/ipld-eth-server
- cerc/watcher-erc20
pods:
- go-ethereum-foundry
- db
- ipld-eth-server
- watcher-erc20