From ca29e9cf0d887228db37ef4c190800c5fa107e79 Mon Sep 17 00:00:00 2001 From: zramsay Date: Tue, 18 Apr 2023 18:01:54 -0400 Subject: [PATCH 01/20] first pass --- docs/laconicd-fixturenet.md | 113 ++++++++++++++++++++++++++++++++++++ 1 file changed, 113 insertions(+) create mode 100644 docs/laconicd-fixturenet.md diff --git a/docs/laconicd-fixturenet.md b/docs/laconicd-fixturenet.md new file mode 100644 index 00000000..bb60af9b --- /dev/null +++ b/docs/laconicd-fixturenet.md @@ -0,0 +1,113 @@ +# Running a laconicd fixturenet with console + +The following tutorial explains the steps to run a laconicd fixturenet with CLI and web console that displays records in the registry. It is designed as an introduction to Stack Orchestrator and to showcase one component of the Laconic Stack. Prior to Stack Orchestrator, the following 4 repositories had to be cloned and setup manually: + +- https://github.com/cerc-io/laconicd +- https://github.com/cerc-io/laconic-sdk +- https://github.com/cerc-io/laconic-registry-cli +- https://github.com/cerc-io/laconic-console + +Now, with Stack Orchestrator, it is a few quick commands. Additionally, the `docker` and `docker compose` integration on the back-end allows the stack to easily persist, facilitating workflows. + +## Setup laconic-so + +To avoid hiccups on Mac M1/M2 and any local machine nuances that may affect the user experience, this tutorial is focused on using a fresh Digital Ocean (DO) droplet with similar specs: +16 GB Memory / 8 Intel vCPUs / 160 GB Disk. + +1. Login to the droplet as root (either by SSH key or password set in the DO console) + +``` +ssh root@IP +``` + +2. Get the install script, give it executable permissions, and run it: + +``` +curl -o install.sh https://raw.githubusercontent.com/cerc-io/stack-orchestrator/main/scripts/quick-install-ubuntu.sh +``` +``` +chmod +x install.sh +``` +``` +bash install.sh +``` + +3. Confirm docker was installed and activate the changes in `~/.profile`: + +``` +docker run hello-world +``` +``` +source ~/.profile +``` + +4. Verify installation: + +``` +laconic-so version +``` + +## Setup the laconic fixturenet stack + +1. Get the repositories + +``` +laconic-so --stack fixturenet-laconic-loaded setup-repositories --include cerc-io/laconicd,cerc-io/laconic-sdk,cerc-io/laconic-registry-cli,cerc-io/laconic-console +``` + +2. Set this environment variable to the Laconic self-hosted Gitea instance: + +``` +export CERC_NPM_REGISTRY_URL=https://git.vdb.to/api/packages/cerc-io/npm/ +``` + +3. Build the containers: + +``` +laconic-so --stack fixturenet-laconic-loaded build-containers +``` + +It's possible to run into an `ESOCKETTIMEDOUT` error, e.g., `error An unexpected error occurred: "https://registry.yarnpkg.com/@material-ui/icons/-/icons-4.11.3.tgz: ESOCKETTIMEDOUT"`. This may happen even if you have a great internet connection. In that case, re-run the `build-containers` command. + +4. Set this environment variable to your droplet's IP address: + +``` +export LACONIC_HOSTED_ENDPOINT=68.183.195.210 +``` + +5. Deploy the stack: + +``` +laconic-so --stack fixturenet-laconic-loaded deploy up +``` + +6. Check the logs: + +``` +laconic-so --stack fixturenet-laconic-loaded deploy logs +``` + +You'll see output from `laconicd` and the block height should be >1 to confirm it is running: + +``` +laconic-5cd0a80c1442c3044c8b295d26426bae-laconicd-1 | 9:29PM INF indexed block exents height=12 module=txindex server=node +laconic-5cd0a80c1442c3044c8b295d26426bae-laconicd-1 | 9:30PM INF Timed out dur=4976.960115 height=13 module=consensus round=0 server=node step=1 +laconic-5cd0a80c1442c3044c8b295d26426bae-laconicd-1 | 9:30PM INF received proposal module=consensus proposal={"Type":32,"block_id":{"hash":"D26C088A711F912ADB97888C269F628DA33153795621967BE44DCB43C3D03CA4","parts":{"hash":"22411A20B7F14CDA33244420FBDDAF24450C0628C7A06034FF22DAC3699DDCC8","total":1}},"height":13,"pol_round":-1,"round":0,"signature":"DEuqnaQmvyYbUwckttJmgKdpRu6eVm9i+9rQ1pIrV2PidkMNdWRZBLdmNghkIrUzGbW8Xd7UVJxtLRmwRASgBg==","timestamp":"2023-04-18T21:30:01.49450663Z"} server=node +laconic-5cd0a80c1442c3044c8b295d26426bae-laconicd-1 | 9:30PM INF received complete proposal block hash=D26C088A711F912ADB97888C269F628DA33153795621967BE44DCB43C3D03CA4 height=13 module=consensus server=node +laconic-5cd0a80c1442c3044c8b295d26426bae-laconicd-1 | 9:30PM INF finalizing commit of block hash={} height=13 module=consensus num_txs=0 root=1A8CA1AF139CCC80EC007C6321D8A63A46A793386EE2EDF9A5CA0AB2C90728B7 server=node +laconic-5cd0a80c1442c3044c8b295d26426bae-laconicd-1 | 9:30PM INF minted coins from module account amount=2059730459416582643aphoton from=mint module=x/bank +laconic-5cd0a80c1442c3044c8b295d26426bae-laconicd-1 | 9:30PM INF executed block height=13 module=state num_invalid_txs=0 num_valid_txs=0 server=node +laconic-5cd0a80c1442c3044c8b295d26426bae-laconicd-1 | 9:30PM INF commit synced commit=436F6D6D697449447B5B363520313037203630203232372039352038352032303820313334203231392032303520313433203130372031343920313431203139203139322038362031323720362031383520323533203137362031333820313735203135392031383620323334203135382031323120313431203230342037335D3A447D +laconic-5cd0a80c1442c3044c8b295d26426bae-laconicd-1 | 9:30PM INF committed state app_hash=416B3CE35F55D086DBCD8F6B958D13C0567F06B9FDB08AAF9FBAEA9E798DCC49 height=13 module=state num_txs=0 server=node +laconic-5cd0a80c1442c3044c8b295d26426bae-laconicd-1 | 9:30PM INF indexed block exents height=13 module=txindex server=node +``` + +7. Confirm operation of the registry CLI: + +``` +laconic-so --stack fixturenet-laconic-loaded deploy exec cli "laconic cns status" +``` + +## Configure Digital Ocean firewall + +Let's open some ports. In the DO web console, navigate to your droplet's main page. Select the "Networking" tab and scroll down to "Firewall". From 07c2a01a5810d2d446e8465f4db4ce4bdcfc619f Mon Sep 17 00:00:00 2001 From: zramsay Date: Tue, 18 Apr 2023 18:48:52 -0400 Subject: [PATCH 02/20] second pass --- docs/laconicd-fixturenet.md | 72 ++++++++++++++++++++++++++++++++++++- 1 file changed, 71 insertions(+), 1 deletion(-) diff --git a/docs/laconicd-fixturenet.md b/docs/laconicd-fixturenet.md index bb60af9b..b9045970 100644 --- a/docs/laconicd-fixturenet.md +++ b/docs/laconicd-fixturenet.md @@ -110,4 +110,74 @@ laconic-so --stack fixturenet-laconic-loaded deploy exec cli "laconic cns status ## Configure Digital Ocean firewall -Let's open some ports. In the DO web console, navigate to your droplet's main page. Select the "Networking" tab and scroll down to "Firewall". +Let's open some ports. + +1. In the Digital Ocean web console, navigate to your droplet's main page. Select the "Networking" tab and scroll down to "Firewall". + +2. Get the port for the running console: + +``` +echo http://IP:$(laconic-so --stack fixturenet-laconic-loaded deploy port laconic-console 80 | cut -d ':' -f 2) +``` +``` +http://IP:32778 +``` + +3. Go back to the Digital Ocean web console and set an Inbound Rule for Custom TCP of the above port: + +- `32778` in this example, but yours will be different. +- do the same for port `9473` + +Additional ports will need to be opened depending on your application. Ensure you add your droplet to this new Firewall and wait a minute or so for the update to propagate. + +4. Navigate to http://IP:port and ensure laconic-console is functioning as expected: + +- ensure you are connected to `laconicd`; no error message should pop up; +- the wifi symbol in the bottom right should have a green check mark beside it +- navigate to the status tab; it should display similar/identical information +- navigate to the config tab, you'll see something like (with your IP): + +``` +wns + webui 68.183.195.210:9473/console + server 68.183.195.210:9473/api +``` + +## Publish and query a sample record to the registry + +1. The following command will create a bond and publish a record: + +``` +laconic-so --stack fixturenet-laconic-loaded deploy exec cli ./scripts/create-demo-records.sh +``` + +You'll get an output like: + +``` +Balance is: 99998999999999998999600000 +Created bond with id: dd88e8d6f9567b32b28e70552aea4419c5dd3307ebae85a284d1fe38904e301a +Published demo-record-1.yml with id: bafyreierh3xnfivexlscdwubvczmddsnf46uytyfvrbdhkjzztvsz6ruly +``` + +The sample record we deployed looks like: + +``` +TODO +``` + +2. Return to the laconic-console + +- the published record should now be viewable +- explore it for more information +- click on the link that opens the GraphQL console +- the query is pre-loaded, click the button to run it +- inspect the output + +3. Try out additional CLI commands + +- these are documented [here](https://github.com/cerc-io/laconic-registry-cli#readme) and updates are forthcoming +- e.g,: + +``` +laconic-so --stack fixturenet-laconic-loaded deploy exec cli "laconic cns record list" +``` From e5b9b74b4c0b890d6e79627f47139ef0b255e7a2 Mon Sep 17 00:00:00 2001 From: zramsay Date: Tue, 18 Apr 2023 19:54:18 -0400 Subject: [PATCH 03/20] the fix --- docs/laconicd-fixturenet.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/laconicd-fixturenet.md b/docs/laconicd-fixturenet.md index b9045970..66ae1020 100644 --- a/docs/laconicd-fixturenet.md +++ b/docs/laconicd-fixturenet.md @@ -72,7 +72,7 @@ It's possible to run into an `ESOCKETTIMEDOUT` error, e.g., `error An unexpected 4. Set this environment variable to your droplet's IP address: ``` -export LACONIC_HOSTED_ENDPOINT=68.183.195.210 +export LACONIC_HOSTED_ENDPOINT=http://68.183.195.210 ``` 5. Deploy the stack: @@ -139,8 +139,8 @@ Additional ports will need to be opened depending on your application. Ensure yo ``` wns - webui 68.183.195.210:9473/console - server 68.183.195.210:9473/api + webui http://68.183.195.210:9473/console + server http://68.183.195.210:9473/api ``` ## Publish and query a sample record to the registry From 32c0830e777ec3ca9d9b638e4c0d6071cd9d4aaa Mon Sep 17 00:00:00 2001 From: Zach Date: Wed, 19 Apr 2023 21:10:19 -0400 Subject: [PATCH 04/20] Update laconicd-fixturenet.md --- docs/laconicd-fixturenet.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/laconicd-fixturenet.md b/docs/laconicd-fixturenet.md index 66ae1020..eacdf33e 100644 --- a/docs/laconicd-fixturenet.md +++ b/docs/laconicd-fixturenet.md @@ -72,7 +72,7 @@ It's possible to run into an `ESOCKETTIMEDOUT` error, e.g., `error An unexpected 4. Set this environment variable to your droplet's IP address: ``` -export LACONIC_HOSTED_ENDPOINT=http://68.183.195.210 +export LACONIC_HOSTED_ENDPOINT=http:// ``` 5. Deploy the stack: From 7a607c299441126d3c9126dc4efa70b18b63ea48 Mon Sep 17 00:00:00 2001 From: Thomas E Lackey Date: Wed, 26 Apr 2023 00:13:35 -0500 Subject: [PATCH 05/20] Add script for exporting ethdb from fixturenet. (#370) * Add script for exporting ethdb from fixturenet. * Update README * Script --- .../compose/docker-compose-fixturenet-eth.yml | 2 + .../cerc-fixturenet-eth-geth/run-el.sh | 6 +++ .../scripts/export-ethdb.sh | 38 +++++++++++++++++++ .../scripts/status.sh | 26 +++++++++++-- app/data/stacks/fixturenet-eth-tx/README.md | 37 ++++++++++++++++++ app/data/stacks/fixturenet-eth-tx/stack.yml | 15 ++++++++ 6 files changed, 121 insertions(+), 3 deletions(-) create mode 100755 app/data/container-build/cerc-fixturenet-eth-lighthouse/scripts/export-ethdb.sh create mode 100644 app/data/stacks/fixturenet-eth-tx/README.md create mode 100644 app/data/stacks/fixturenet-eth-tx/stack.yml diff --git a/app/data/compose/docker-compose-fixturenet-eth.yml b/app/data/compose/docker-compose-fixturenet-eth.yml index c687b326..59f729b9 100644 --- a/app/data/compose/docker-compose-fixturenet-eth.yml +++ b/app/data/compose/docker-compose-fixturenet-eth.yml @@ -49,6 +49,8 @@ services: timeout: 10s retries: 10 start_period: 3s + environment: + CERC_KEEP_RUNNING_AFTER_GETH_EXIT: "true" env_file: - ../config/fixturenet-eth/fixturenet-eth.env image: cerc/fixturenet-eth-geth:local diff --git a/app/data/container-build/cerc-fixturenet-eth-geth/run-el.sh b/app/data/container-build/cerc-fixturenet-eth-geth/run-el.sh index 526c76d7..edde1ba2 100755 --- a/app/data/container-build/cerc-fixturenet-eth-geth/run-el.sh +++ b/app/data/container-build/cerc-fixturenet-eth-geth/run-el.sh @@ -127,3 +127,9 @@ else fi wait $geth_pid + +if [ "true" == "$CERC_KEEP_RUNNING_AFTER_GETH_EXIT" ]; then + while [ 1 -eq 1 ]; do + sleep 60 + done +fi diff --git a/app/data/container-build/cerc-fixturenet-eth-lighthouse/scripts/export-ethdb.sh b/app/data/container-build/cerc-fixturenet-eth-lighthouse/scripts/export-ethdb.sh new file mode 100755 index 00000000..77198c61 --- /dev/null +++ b/app/data/container-build/cerc-fixturenet-eth-lighthouse/scripts/export-ethdb.sh @@ -0,0 +1,38 @@ +#!/bin/bash + +MY_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd ) + +GETH_EXPORT_MIN_BLOCK=${1:-${GETH_EXPORT_MIN_BLOCK:-1000}} + +# Wait for block. +${MY_DIR}/status.sh $GETH_EXPORT_MIN_BLOCK +if [ $? -ne 0 ]; then + echo "Unable to export ethdb." 1>&2 + exit 1 +fi + +# Stop geth. +echo -n "Exporting ethdb.... " +GETH_CONTAINER=`docker ps -q -f "name=${CERC_SO_COMPOSE_PROJECT}-fixturenet-eth-geth-2-1"` +if [ -z "$GETH_CONTAINER" ]; then + echo "not found" + exit 1 +fi + +docker exec $GETH_CONTAINER sh -c 'rm -rf /root/tmp && mkdir -p /root/tmp/export' +docker exec $GETH_CONTAINER sh -c 'ln -s /opt/testnet/build/el/geth.json /root/tmp/export/genesis.json && ln -s /root/ethdata /root/tmp/export/' +docker exec $GETH_CONTAINER sh -c 'cat /root/tmp/export/genesis.json | jq ".config" > /root/tmp/export/genesis.config.json' + +# Stop geth and zip up ethdb. +docker exec $GETH_CONTAINER sh -c 'curl -s --location "localhost:8545" --header "Content-Type: application/json" --data "{\"jsonrpc\": \"2.0\", \"id\": 1, \"method\": \"eth_getBlockByNumber\", \"params\": [\"0x0\", false]}" > /root/tmp/export/eth_getBlockByNumber_0x0.json' +docker exec $GETH_CONTAINER sh -c 'curl -s --location "localhost:8545" --header "Content-Type: application/json" --data "{\"jsonrpc\": \"2.0\", \"id\": 1, \"method\": \"eth_blockNumber\", \"params\": []}" > /root/tmp/export/eth_blockNumber.json' +docker exec $GETH_CONTAINER sh -c "killall geth && sleep 2 && tar chzf /root/tmp/ethdb.tgz -C /root/tmp/export ." + +# Copy ethdb to host. +GETH_EXPORT_FILE=${2:-${GETH_EXPORT_FILE:-./ethdb.tgz}} +docker cp $GETH_CONTAINER:/root/tmp/ethdb.tgz $GETH_EXPORT_FILE +echo "$GETH_EXPORT_FILE" +docker exec $GETH_CONTAINER sh -c "rm -rf /root/tmp" + +# Restart the container to get geth back up and running. +docker restart $GETH_CONTAINER >/dev/null diff --git a/app/data/container-build/cerc-fixturenet-eth-lighthouse/scripts/status.sh b/app/data/container-build/cerc-fixturenet-eth-lighthouse/scripts/status.sh index c6e65a68..f6d9d27b 100755 --- a/app/data/container-build/cerc-fixturenet-eth-lighthouse/scripts/status.sh +++ b/app/data/container-build/cerc-fixturenet-eth-lighthouse/scripts/status.sh @@ -2,9 +2,10 @@ if [ -n "$CERC_SCRIPT_DEBUG" ]; then set -x fi -STATUSES=("geth to generate DAG" "beacon phase0" "beacon altair" "beacon bellatrix pre-merge" "beacon bellatrix merge") -STATUS=0 +MIN_BLOCK_NUM=${1:-${MIN_BLOCK_NUM:-3}} +STATUSES=("geth to generate DAG" "beacon phase0" "beacon altair" "beacon bellatrix pre-merge" "beacon bellatrix merge" "block number $MIN_BLOCK_NUM") +STATUS=0 LIGHTHOUSE_BASE_URL=${LIGHTHOUSE_BASE_URL} GETH_BASE_URL=${GETH_BASE_URL} @@ -13,18 +14,29 @@ GETH_BASE_URL=${GETH_BASE_URL} # or some execution environment-neutral mechanism. if [ -z "$LIGHTHOUSE_BASE_URL" ]; then LIGHTHOUSE_CONTAINER=`docker ps -q -f "name=fixturenet-eth-lighthouse-1-1"` + if [ -z "$LIGHTHOUSE_CONTAINER" ]; then + echo "Lighthouse container not found." 1>&2 + exit 1 + fi LIGHTHOUSE_PORT=`docker port $LIGHTHOUSE_CONTAINER 8001 | cut -d':' -f2` LIGHTHOUSE_BASE_URL="http://localhost:${LIGHTHOUSE_PORT}" fi if [ -z "$GETH_BASE_URL" ]; then GETH_CONTAINER=`docker ps -q -f "name=fixturenet-eth-geth-1-1"` + if [ -z "$GETH_CONTAINER" ]; then + echo "Lighthouse container not found." 1>&2 + exit 1 + fi GETH_PORT=`docker port $GETH_CONTAINER 8545 | cut -d':' -f2` GETH_BASE_URL="http://localhost:${GETH_PORT}" fi +MARKER="." + function inc_status() { echo " done" + MARKEr="." STATUS=$((STATUS + 1)) if [ $STATUS -lt ${#STATUSES[@]} ]; then echo -n "Waiting for ${STATUSES[$STATUS]}..." @@ -34,7 +46,7 @@ function inc_status() { echo -n "Waiting for ${STATUSES[$STATUS]}..." while [ $STATUS -lt ${#STATUSES[@]} ]; do sleep 1 - echo -n "." + echo -n "$MARKER" case $STATUS in 0) result=`wget --no-check-certificate --quiet -O - --method POST --header 'Content-Type: application/json' \ @@ -67,5 +79,13 @@ while [ $STATUS -lt ${#STATUSES[@]} ]; do inc_status fi ;; + 5) + result=`wget --no-check-certificate --quiet -O - "$LIGHTHOUSE_BASE_URL/eth/v2/beacon/blocks/head" | jq -r '.data.message.body.execution_payload.block_number'` + if [ ! -z "$result" ] && [ $result -gt $MIN_BLOCK_NUM ]; then + inc_status + else + MARKER="$result " + fi + ;; esac done diff --git a/app/data/stacks/fixturenet-eth-tx/README.md b/app/data/stacks/fixturenet-eth-tx/README.md new file mode 100644 index 00000000..9af9ee3d --- /dev/null +++ b/app/data/stacks/fixturenet-eth-tx/README.md @@ -0,0 +1,37 @@ +# fixturenet-eth-tx + +A variation of `fixturenet-eth` that automatically generates transactions using `tx-spammer`. + +See `stacks/fixturenet-eth/README.md` for more information. + +## Containers + +* cerc/go-ethereum +* cerc/lighthouse +* cerc/fixturenet-eth-geth +* cerc/fixturenet-eth-lighthouse +* cerc/tx-spammer + +## Deploy the stack +``` +$ laconic-so --stack fixturenet-eth-tx setup-repositories +$ laconic-so --stack fixturenet-eth-tx build-containers +$ laconic-so --stack fixturenet-eth-tx deploy up +``` + +## Export the ethdb (optional) + +It is easy to export data from the fixturenet for offline processing of the raw ethdb files (eg, by eth-statediff-service) using the `export-ethdb.sh` script. + +For example: + +``` +❯ app/data/container-build/cerc-fixturenet-eth-lighthouse/scripts/export-ethdb.sh 500 +Waiting for geth to generate DAG.... done +Waiting for beacon phase0.... done +Waiting for beacon altair.... done +Waiting for beacon bellatrix pre-merge.... done +Waiting for beacon bellatrix merge.... done +Waiting for block number 500.... done +Exporting ethdb.... ./ethdb.tgz +``` diff --git a/app/data/stacks/fixturenet-eth-tx/stack.yml b/app/data/stacks/fixturenet-eth-tx/stack.yml new file mode 100644 index 00000000..0c3b45a3 --- /dev/null +++ b/app/data/stacks/fixturenet-eth-tx/stack.yml @@ -0,0 +1,15 @@ +version: "1.1" +name: fixturenet-eth-tx +decription: "Ethereum Fixturenet w/ tx-spammer" +repos: + - cerc-io/go-ethereum + - cerc-io/tx-spammer +containers: + - cerc/go-ethereum + - cerc/lighthouse + - cerc/fixturenet-eth-geth + - cerc/fixturenet-eth-lighthouse + - cerc/tx-spammer +pods: + - fixturenet-eth + - tx-spammer From a4ff8f3dcb8ff2b45fc89b9d99d8f354bb14b545 Mon Sep 17 00:00:00 2001 From: Zach Date: Wed, 26 Apr 2023 14:43:32 -0400 Subject: [PATCH 06/20] optimism: on error, wait, then re-run 'deploy up' --- app/data/stacks/fixturenet-optimism/README.md | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/app/data/stacks/fixturenet-optimism/README.md b/app/data/stacks/fixturenet-optimism/README.md index 7664a7c3..02e06a44 100644 --- a/app/data/stacks/fixturenet-optimism/README.md +++ b/app/data/stacks/fixturenet-optimism/README.md @@ -2,7 +2,7 @@ Instructions to setup and deploy an end-to-end L1+L2 stack with [fixturenet-eth](../fixturenet-eth/) (L1) and [Optimism](https://stack.optimism.io) (L2) -We support running just the L2 part of stack, given an external L1 endpoint. Follow [l2-only](./l2-only.md) for the same. +We support running just the L2 part of stack, given an external L1 endpoint. Follow the [L2 only doc](./l2-only.md) for the same. ## Setup @@ -28,6 +28,8 @@ Build the container images: laconic-so --stack fixturenet-optimism build-containers ``` +Note: this will take >10 mins depending on the specs of your machine, and **requires** 16GB of memory or greater. + This should create the required docker images in the local image registry: * `cerc/go-ethereum` * `cerc/lighthouse` @@ -48,12 +50,14 @@ Deploy the stack: laconic-so --stack fixturenet-optimism deploy up ``` -The `fixturenet-optimism-contracts` service may take a while (`~15 mins`) to complete running as it: +If you get the error `service "fixturenet-optimism-contracts" didn't complete successfully: exit 1` with ~25 lines of Traceback, wait 15-20 mins then re-run the command. + +The `fixturenet-optimism-contracts` service takes a while to complete running as it: 1. waits for the 'Merge' to happen on L1 2. waits for a finalized block to exist on L1 (so that it can be taken as a starting block for roll ups) 3. deploys the L1 contracts -To list down and monitor the running containers: +To list and monitor the running containers: ```bash laconic-so --stack fixturenet-optimism deploy ps From 323ca3b2387e0ac656b901405362298895b8a3e8 Mon Sep 17 00:00:00 2001 From: Thomas E Lackey Date: Wed, 26 Apr 2023 23:26:50 -0500 Subject: [PATCH 07/20] Minor script cleanup. (#372) --- .../scripts/export-ethdb.sh | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) diff --git a/app/data/container-build/cerc-fixturenet-eth-lighthouse/scripts/export-ethdb.sh b/app/data/container-build/cerc-fixturenet-eth-lighthouse/scripts/export-ethdb.sh index 77198c61..dd93b219 100755 --- a/app/data/container-build/cerc-fixturenet-eth-lighthouse/scripts/export-ethdb.sh +++ b/app/data/container-build/cerc-fixturenet-eth-lighthouse/scripts/export-ethdb.sh @@ -1,12 +1,21 @@ #!/bin/bash -MY_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd ) +# Exports the complete fixturenet-eth ethdb data to a tarball (default, ./ethdb.tgz), waiting for a minimum +# block height (default 1000) to be reached before exporting. + +# Usage: export-ethdb.sh [min_block_number=1000] [output_file=./ethdb.tgz] + +if [[ -n "$CERC_SCRIPT_DEBUG" ]]; then + set -x +fi + +SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd ) GETH_EXPORT_MIN_BLOCK=${1:-${GETH_EXPORT_MIN_BLOCK:-1000}} # Wait for block. -${MY_DIR}/status.sh $GETH_EXPORT_MIN_BLOCK -if [ $? -ne 0 ]; then +${SCRIPT_DIR}/status.sh $GETH_EXPORT_MIN_BLOCK +if [[ $? -ne 0 ]]; then echo "Unable to export ethdb." 1>&2 exit 1 fi @@ -14,7 +23,7 @@ fi # Stop geth. echo -n "Exporting ethdb.... " GETH_CONTAINER=`docker ps -q -f "name=${CERC_SO_COMPOSE_PROJECT}-fixturenet-eth-geth-2-1"` -if [ -z "$GETH_CONTAINER" ]; then +if [[ -z "$GETH_CONTAINER" ]]; then echo "not found" exit 1 fi From f04b266a24351d89fe60c2f87a0cbcd7e81dc6bf Mon Sep 17 00:00:00 2001 From: prathamesh0 <42446521+prathamesh0@users.noreply.github.com> Date: Thu, 27 Apr 2023 18:26:58 +0530 Subject: [PATCH 08/20] Map op-batcher and op-proposer ports to host (#373) --- app/data/compose/docker-compose-fixturenet-optimism.yml | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/app/data/compose/docker-compose-fixturenet-optimism.yml b/app/data/compose/docker-compose-fixturenet-optimism.yml index c9e17b03..a20fd118 100644 --- a/app/data/compose/docker-compose-fixturenet-optimism.yml +++ b/app/data/compose/docker-compose-fixturenet-optimism.yml @@ -122,6 +122,8 @@ services: # Waits for L1 endpoint to be up before running the batcher command: | "/wait-for-it.sh -h ${CERC_L1_HOST:-$${DEFAULT_CERC_L1_HOST}} -p ${CERC_L1_PORT:-$${DEFAULT_CERC_L1_PORT}} -s -t 60 -- /run-op-batcher.sh" + ports: + - "127.0.0.1:8548:8548" extra_hosts: - "host.docker.internal:host-gateway" @@ -145,6 +147,8 @@ services: # Waits for L1 endpoint to be up before running the proposer command: | "/wait-for-it.sh -h ${CERC_L1_HOST:-$${DEFAULT_CERC_L1_HOST}} -p ${CERC_L1_PORT:-$${DEFAULT_CERC_L1_PORT}} -s -t 60 -- /run-op-proposer.sh" + ports: + - "127.0.0.1:8560:8560" extra_hosts: - "host.docker.internal:host-gateway" From 73419c341a427133161264e9bee5e6082bc918ce Mon Sep 17 00:00:00 2001 From: David Boreham Date: Thu, 27 Apr 2023 10:22:21 -0600 Subject: [PATCH 09/20] Remove > --- app/data/stacks/fixturenet-eth-tx/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/app/data/stacks/fixturenet-eth-tx/README.md b/app/data/stacks/fixturenet-eth-tx/README.md index 9af9ee3d..f6ae483a 100644 --- a/app/data/stacks/fixturenet-eth-tx/README.md +++ b/app/data/stacks/fixturenet-eth-tx/README.md @@ -26,7 +26,7 @@ It is easy to export data from the fixturenet for offline processing of the raw For example: ``` -❯ app/data/container-build/cerc-fixturenet-eth-lighthouse/scripts/export-ethdb.sh 500 +$ app/data/container-build/cerc-fixturenet-eth-lighthouse/scripts/export-ethdb.sh 500 Waiting for geth to generate DAG.... done Waiting for beacon phase0.... done Waiting for beacon altair.... done From af93743974a5ea35f13a65261b53a2581eb36016 Mon Sep 17 00:00:00 2001 From: David Boreham Date: Thu, 27 Apr 2023 09:52:37 -0700 Subject: [PATCH 10/20] Add foundry to the fixturenet-eth-tx stack (#374) --- app/data/stacks/fixturenet-eth-tx/stack.yml | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/app/data/stacks/fixturenet-eth-tx/stack.yml b/app/data/stacks/fixturenet-eth-tx/stack.yml index 0c3b45a3..0a0a6bd6 100644 --- a/app/data/stacks/fixturenet-eth-tx/stack.yml +++ b/app/data/stacks/fixturenet-eth-tx/stack.yml @@ -1,15 +1,18 @@ -version: "1.1" +version: "1.2" name: fixturenet-eth-tx decription: "Ethereum Fixturenet w/ tx-spammer" repos: - cerc-io/go-ethereum - cerc-io/tx-spammer + - dboreham/foundry containers: - cerc/go-ethereum - cerc/lighthouse - cerc/fixturenet-eth-geth - cerc/fixturenet-eth-lighthouse - cerc/tx-spammer + - cerc/foundry pods: - fixturenet-eth + - foundry - tx-spammer From 19e38d2a942af03e80e81138b3f079dfbf4de7fa Mon Sep 17 00:00:00 2001 From: David Boreham Date: Thu, 27 Apr 2023 09:54:11 -0700 Subject: [PATCH 11/20] Remove gerbil builder container from build-support stack (#375) --- app/data/stacks/build-support/stack.yml | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/app/data/stacks/build-support/stack.yml b/app/data/stacks/build-support/stack.yml index e4efb457..e0962515 100644 --- a/app/data/stacks/build-support/stack.yml +++ b/app/data/stacks/build-support/stack.yml @@ -1,6 +1,5 @@ -version: "1.1" +version: "1.2" name: build-support decription: "Build Support Components" containers: - cerc/builder-js - - cerc/builder-gerbil From 993118deb4e4ee668ad0c746fde40317cc37c2f0 Mon Sep 17 00:00:00 2001 From: zramsay Date: Thu, 27 Apr 2023 13:15:39 -0400 Subject: [PATCH 12/20] better direction to stacks --- README.md | 46 ++++++++-------------------------------------- 1 file changed, 8 insertions(+), 38 deletions(-) diff --git a/README.md b/README.md index 0f2ca38e..4f490d8c 100644 --- a/README.md +++ b/README.md @@ -35,58 +35,28 @@ curl -L -o ~/bin/laconic-so https://github.com/cerc-io/stack-orchestrator/releas ``` Give it execute permissions: + ```bash chmod +x ~/bin/laconic-so ``` Ensure `laconic-so` is on the [`PATH`](https://unix.stackexchange.com/a/26059) -Verify operation (your version will probably be different, just check here that you see some version outut and not an error): +Verify operation (your version will probably be different, just check here that you see some version output and not an error): ``` laconic-so version -Version: v1.0.27-7831078 +Version: 1.1.0-7a607c2-202304260513 ``` ## Usage -Three sub-commands: `setup-repositories`, `build-containers` and `deploy-system` are generally run in order. The following is a slim example for standing up the `erc20-watcher`. Go further with the [erc20 watcher demo](/app/data/stacks/erc20) and other pieces of the stack, within the [`stacks` directory](/app/data/stacks). +The various [stacks](/app/data/stacks) each contain instructions for running different stacks based on your use case. For example: -### Setup Repositories - -Clone the set of git repositories necessary to build a system: - -```bash -laconic-so --stack erc20 setup-repositories -``` - -This will default to cloning git reposiories into: `~/cerc` or - if set - the environment variable `CERC_REPO_BASE_DIR` - -### Build Containers - -Build the set of docker container images required to run a system. It takes around 10 minutes to build all the containers from scratch. - -```bash -laconic-so --stack erc20 build-containers -``` - -### Deploy System - -Uses `docker compose` to deploy a system (with most recently built container images). - -```bash -laconic-so --stack erc20 deploy-system up -``` - -Check out he GraphQL playground here: [http://localhost:3002/graphql](http://localhost:3002/graphql) - -See the [erc20 watcher demo](/app/data/stacks/erc20) to continue further. - -### Cleanup - -```bash -laconic-so --stack erc20 deploy-system down -``` +- [self-hosted Gitea](/app/data/stacks/build-support) +- [an Optimism Fixturenet](/app/data/stacks/fixturenet-optimism) +- [laconicd with console and CLI](app/data/stacks/fixturenet-laconic-loaded) +- [kubo (IPFS)](app/data/stacks/kubo) ## Contributing From 97433a7bb562c94e09b0a311324318c65c0c0b84 Mon Sep 17 00:00:00 2001 From: Zach Date: Sat, 29 Apr 2023 14:55:10 -0400 Subject: [PATCH 13/20] rm gerbil from doc --- app/data/stacks/build-support/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/app/data/stacks/build-support/README.md b/app/data/stacks/build-support/README.md index b4090d76..fd85927f 100644 --- a/app/data/stacks/build-support/README.md +++ b/app/data/stacks/build-support/README.md @@ -16,7 +16,7 @@ Leave `CERC_NPM_REGISTRY_URL` un-set to use the local gitea registry. Note: the scheme/gerbil container is excluded as it isn't currently required for the package registry. ``` -$ laconic-so --stack build-support build-containers --exclude cerc/builder-gerbil +$ laconic-so --stack build-support build-containers ``` ### 2. Deploy Gitea Package Registry From cba2345af3485d22a9d315d3cffe618caae470b6 Mon Sep 17 00:00:00 2001 From: prathamesh0 <42446521+prathamesh0@users.noreply.github.com> Date: Tue, 2 May 2023 12:14:48 +0530 Subject: [PATCH 14/20] Avoid persisting lighthouse bootnode ENR between restarts (#377) --- .../compose/docker-compose-fixturenet-eth.yml | 3 --- .../genesis/cl/bootnode.sh | 16 ++++------------ app/data/stacks/fixturenet-eth/README.md | 4 ++-- 3 files changed, 6 insertions(+), 17 deletions(-) diff --git a/app/data/compose/docker-compose-fixturenet-eth.yml b/app/data/compose/docker-compose-fixturenet-eth.yml index 59f729b9..42737f5b 100644 --- a/app/data/compose/docker-compose-fixturenet-eth.yml +++ b/app/data/compose/docker-compose-fixturenet-eth.yml @@ -64,8 +64,6 @@ services: environment: RUN_BOOTNODE: "true" image: cerc/fixturenet-eth-lighthouse:local - volumes: - - fixturenet_eth_bootnode_lighthouse_data:/opt/testnet/build/cl fixturenet-eth-lighthouse-1: hostname: fixturenet-eth-lighthouse-1 @@ -120,6 +118,5 @@ volumes: fixturenet_eth_bootnode_geth_data: fixturenet_eth_geth_1_data: fixturenet_eth_geth_2_data: - fixturenet_eth_bootnode_lighthouse_data: fixturenet_eth_lighthouse_1_data: fixturenet_eth_lighthouse_2_data: diff --git a/app/data/container-build/cerc-fixturenet-eth-lighthouse/genesis/cl/bootnode.sh b/app/data/container-build/cerc-fixturenet-eth-lighthouse/genesis/cl/bootnode.sh index a395f41a..70ecb47c 100755 --- a/app/data/container-build/cerc-fixturenet-eth-lighthouse/genesis/cl/bootnode.sh +++ b/app/data/container-build/cerc-fixturenet-eth-lighthouse/genesis/cl/bootnode.sh @@ -21,22 +21,14 @@ if [ ! -f "$DATADIR/bootnode/enr.dat" ]; then --udp-port $BOOTNODE_PORT \ --tcp-port $BOOTNODE_PORT \ --genesis-fork-version $GENESIS_FORK_VERSION \ - --output-dir $DATADIR/bootnode-temp + --output-dir $DATADIR/bootnode - # Output ENR to a temp dir and mv as "lcli generate-bootnode-enr" will not overwrite an empty dir (mounted volume) - mkdir -p $DATADIR/bootnode - mv $DATADIR/bootnode-temp/* $DATADIR/bootnode - rm -r $DATADIR/bootnode-temp + bootnode_enr=`cat $DATADIR/bootnode/enr.dat` + echo "- $bootnode_enr" > $TESTNET_DIR/boot_enr.yaml - echo "Generated bootnode enr" -else - echo "Found existing bootnode enr" + echo "Generated bootnode enr and written to $TESTNET_DIR/boot_enr.yaml" fi -bootnode_enr=`cat $DATADIR/bootnode/enr.dat` -echo "- $bootnode_enr" > $TESTNET_DIR/boot_enr.yaml -echo "Written bootnode enr to $TESTNET_DIR/boot_enr.yaml" - exec lighthouse boot_node \ --testnet-dir $TESTNET_DIR \ --port $BOOTNODE_PORT \ diff --git a/app/data/stacks/fixturenet-eth/README.md b/app/data/stacks/fixturenet-eth/README.md index c54237aa..7bd96506 100644 --- a/app/data/stacks/fixturenet-eth/README.md +++ b/app/data/stacks/fixturenet-eth/README.md @@ -117,8 +117,8 @@ Clear volumes created by this stack: ```bash # List all relevant volumes -$ docker volume ls -q --filter "name=.*fixturenet_eth_bootnode_geth_data|.*fixturenet_eth_bootnode_lighthouse_data|.*fixturenet_eth_geth_1_data|.*fixturenet_eth_geth_2_data|.*fixturenet_eth_lighthouse_1_data|.*fixturenet_eth_lighthouse_2_data" +$ docker volume ls -q --filter "name=.*fixturenet_eth_bootnode_geth_data|.*fixturenet_eth_geth_1_data|.*fixturenet_eth_geth_2_data|.*fixturenet_eth_lighthouse_1_data|.*fixturenet_eth_lighthouse_2_data" # Remove all the listed volumes -$ docker volume rm $(docker volume ls -q --filter "name=.*fixturenet_eth_bootnode_geth_data|.*fixturenet_eth_bootnode_lighthouse_data|.*fixturenet_eth_geth_1_data|.*fixturenet_eth_geth_2_data|.*fixturenet_eth_lighthouse_1_data|.*fixturenet_eth_lighthouse_2_data") +$ docker volume rm $(docker volume ls -q --filter "name=.*fixturenet_eth_bootnode_geth_data|.*fixturenet_eth_geth_1_data|.*fixturenet_eth_geth_2_data|.*fixturenet_eth_lighthouse_1_data|.*fixturenet_eth_lighthouse_2_data") ``` From ed4f40118fdf14f145c25b1a4d954a87ce1c9143 Mon Sep 17 00:00:00 2001 From: prathamesh0 <42446521+prathamesh0@users.noreply.github.com> Date: Tue, 2 May 2023 13:18:29 +0530 Subject: [PATCH 15/20] Upgrade to use latest lighthouse release (#378) --- .../container-build/cerc-fixturenet-eth-lighthouse/Dockerfile | 2 +- app/data/container-build/cerc-lighthouse/Dockerfile | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/app/data/container-build/cerc-fixturenet-eth-lighthouse/Dockerfile b/app/data/container-build/cerc-fixturenet-eth-lighthouse/Dockerfile index 2295262d..958d2b39 100644 --- a/app/data/container-build/cerc-fixturenet-eth-lighthouse/Dockerfile +++ b/app/data/container-build/cerc-fixturenet-eth-lighthouse/Dockerfile @@ -1,4 +1,4 @@ -FROM sigp/lcli:v3.2.1 AS lcli +FROM sigp/lcli:v4.1.0 AS lcli FROM skylenet/ethereum-genesis-generator@sha256:210353ce7c898686bc5092f16c61220a76d357f51eff9c451e9ad1b9ad03d4d3 AS ethgen FROM cerc/fixturenet-eth-geth:local AS fnetgeth diff --git a/app/data/container-build/cerc-lighthouse/Dockerfile b/app/data/container-build/cerc-lighthouse/Dockerfile index 7d4fe5d8..cab3af06 100644 --- a/app/data/container-build/cerc-lighthouse/Dockerfile +++ b/app/data/container-build/cerc-lighthouse/Dockerfile @@ -1,4 +1,4 @@ -FROM sigp/lighthouse:v4.0.1-modern +FROM sigp/lighthouse:v4.1.0-modern RUN apt-get update; apt-get install bash netcat curl less jq -y; From b23b5ae3bfab43b8d9bfefa2ac388da1ad78bff7 Mon Sep 17 00:00:00 2001 From: David Boreham Date: Tue, 2 May 2023 14:13:48 -0700 Subject: [PATCH 16/20] Fixturenet pocket (#350) * add fixturenet-gaia stack * add fixturenet-pocket * integrate with eth fixturenet * separate out fixturenet-gaia * use pocket-deployments Dockerfile --------- Co-authored-by: iskay Co-authored-by: Ian --- .../docker-compose-fixturenet-pocket.yml | 18 ++ app/data/config/fixturenet-pocket/chains.json | 18 ++ .../fixturenet-pocket/create-fixturenet.sh | 65 +++++ .../config/fixturenet-pocket/genesis.json | 272 ++++++++++++++++++ app/data/container-build/cerc-pocket/build.sh | 3 + app/data/container-image-list.txt | 1 + app/data/pod-list.txt | 1 + app/data/repository-list.txt | 2 + app/data/stacks/fixturenet-pocket/README.md | 59 ++++ app/data/stacks/fixturenet-pocket/stack.yml | 16 ++ 10 files changed, 455 insertions(+) create mode 100644 app/data/compose/docker-compose-fixturenet-pocket.yml create mode 100755 app/data/config/fixturenet-pocket/chains.json create mode 100644 app/data/config/fixturenet-pocket/create-fixturenet.sh create mode 100644 app/data/config/fixturenet-pocket/genesis.json create mode 100755 app/data/container-build/cerc-pocket/build.sh create mode 100644 app/data/stacks/fixturenet-pocket/README.md create mode 100644 app/data/stacks/fixturenet-pocket/stack.yml diff --git a/app/data/compose/docker-compose-fixturenet-pocket.yml b/app/data/compose/docker-compose-fixturenet-pocket.yml new file mode 100644 index 00000000..88d34f91 --- /dev/null +++ b/app/data/compose/docker-compose-fixturenet-pocket.yml @@ -0,0 +1,18 @@ +version: "3.2" +services: + pocket: + restart: unless-stopped + image: cerc/pocket:local + # command: ["sh", "/docker-entrypoint-scripts.d/create-fixturenet.sh"] + entrypoint: ["sh", "/docker-entrypoint-scripts.d/create-fixturenet.sh"] + volumes: + # TODO: look at folding these scripts into the container + - ../config/fixturenet-pocket/create-fixturenet.sh:/docker-entrypoint-scripts.d/create-fixturenet.sh + - ../config/fixturenet-pocket/chains.json:/home/app/pocket-configs/chains.json + - ../config/fixturenet-pocket/genesis.json:/home/app/pocket-configs/genesis.json + ports: + - "8081:8081" # pocket relay rpc +networks: + net1: + name: fixturenet-eth_default + external: true diff --git a/app/data/config/fixturenet-pocket/chains.json b/app/data/config/fixturenet-pocket/chains.json new file mode 100755 index 00000000..3045620a --- /dev/null +++ b/app/data/config/fixturenet-pocket/chains.json @@ -0,0 +1,18 @@ +[ + { + "id": "0001", + "url": "http://127.0.0.1:8081/", + "basic_auth": { + "username": "", + "password": "" + } + }, + { + "id": "0021", + "url": "http://fixturenet-eth-geth-1:8545/", + "basic_auth": { + "username": "", + "password": "" + } + } +] diff --git a/app/data/config/fixturenet-pocket/create-fixturenet.sh b/app/data/config/fixturenet-pocket/create-fixturenet.sh new file mode 100644 index 00000000..4696749f --- /dev/null +++ b/app/data/config/fixturenet-pocket/create-fixturenet.sh @@ -0,0 +1,65 @@ +#!/bin/bash + +# TODO: we should have a mechanism to bundle it inside the container rather than link from here +# at deploy time. + +CHAINID="pocketlocal-1" +MONIKER="localtestnet" +SERVICE_URL="http://127.0.0.1:8081" +PASSWORD="mypassword" # wallet password, required by cli + +# check if jq is installed; install if necessary +# command -v jq > /dev/null 2>&1 || { echo >&2 "jq not installed. More info: https://stedolan.github.io/jq/download/"; exit 1; } +if ! command -v jq > /dev/null 2>&1; then + echo "jq not installed, downloading..." + mkdir -p /home/app/bin + wget -O /home/app/bin/jq https://github.com/stedolan/jq/releases/download/jq-1.6/jq-linux64 + chmod +x /home/app/bin/jq + export PATH=$PATH:/home/app/bin +fi + +# remove existing daemon and client +rm -rf ~/.pocket* + +# create a wallet with password "mypassword" and save the address for later +address=$(pocket accounts create --pwd $PASSWORD | awk '/Address:/ {print $2}') + +# set this address as the validator address for the node +pocket accounts set-validator $address --pwd $PASSWORD + +# save the public key for later +pubkey=$(pocket accounts show $address | awk '/Public Key:/ {print $3}') + +# set node's moniker +echo $(pocket util print-configs) | jq '.tendermint_config.Moniker = "'"$MONIKER"'"' | jq . > $HOME/.pocket/config/config.json + +# pocket mainnet has block time of 15 minutes, set closer to 1 minute instead +cat $HOME/.pocket/config/config.json | jq '.tendermint_config.Consensus.TimeoutPropose = 8000000000' | jq . > $HOME/.pocket/config/tmp_config.json && mv $HOME/.pocket/config/tmp_config.json $HOME/.pocket/config/config.json +cat $HOME/.pocket/config/config.json | jq '.tendermint_config.Consensus.TimeoutProposeDelta = 600000000' | jq . > $HOME/.pocket/config/tmp_config.json && mv $HOME/.pocket/config/tmp_config.json $HOME/.pocket/config/config.json +cat $HOME/.pocket/config/config.json | jq '.tendermint_config.Consensus.TimeoutPrevote = 4000000000' | jq . > $HOME/.pocket/config/tmp_config.json && mv $HOME/.pocket/config/tmp_config.json $HOME/.pocket/config/config.json +cat $HOME/.pocket/config/config.json | jq '.tendermint_config.Consensus.TimeoutPrevoteDelta = 600000000' | jq . > $HOME/.pocket/config/tmp_config.json && mv $HOME/.pocket/config/tmp_config.json $HOME/.pocket/config/config.json +cat $HOME/.pocket/config/config.json | jq '.tendermint_config.Consensus.TimeoutPrecommit = 4000000000' | jq . > $HOME/.pocket/config/tmp_config.json && mv $HOME/.pocket/config/tmp_config.json $HOME/.pocket/config/config.json +cat $HOME/.pocket/config/config.json | jq '.tendermint_config.Consensus.TimeoutPrecommitDelta = 6000000006' | jq . > $HOME/.pocket/config/tmp_config.json && mv $HOME/.pocket/config/tmp_config.json $HOME/.pocket/config/config.json +cat $HOME/.pocket/config/config.json | jq '.tendermint_config.Consensus.TimeoutCommit = 52000000000' | jq . > $HOME/.pocket/config/tmp_config.json && mv $HOME/.pocket/config/tmp_config.json $HOME/.pocket/config/config.json +cat $HOME/.pocket/config/config.json | jq '.tendermint_config.Consensus.CreateEmptyBlocksInterval = 60000000000' | jq . > $HOME/.pocket/config/tmp_config.json && mv $HOME/.pocket/config/tmp_config.json $HOME/.pocket/config/config.json +cat $HOME/.pocket/config/config.json | jq '.tendermint_config.Consensus.PeerGossipSleepDuration = 2000000000' | jq . > $HOME/.pocket/config/tmp_config.json && mv $HOME/.pocket/config/tmp_config.json $HOME/.pocket/config/config.json +cat $HOME/.pocket/config/config.json | jq '.tendermint_config.Consensus.PeerQueryMaj23SleepDuration = 1200000000' | jq . > $HOME/.pocket/config/tmp_config.json && mv $HOME/.pocket/config/tmp_config.json $HOME/.pocket/config/config.json + +# include genesis.json and chains.json +cp $HOME/pocket-configs/genesis.json $HOME/.pocket/config/genesis.json +cp $HOME/pocket-configs/chains.json $HOME/.pocket/config/chains.json + +# set chain-id and add node to genesis.json as a validator +cat $HOME/.pocket/config/genesis.json | jq '.chain_id="'"$CHAINID"'"' > $HOME/.pocket/config/tmp_genesis.json && mv $HOME/.pocket/config/tmp_genesis.json $HOME/.pocket/config/genesis.json +cat $HOME/.pocket/config/genesis.json | jq '.app_state.auth.accounts[0].value.address="'"$address"'"' > $HOME/.pocket/config/tmp_genesis.json && mv $HOME/.pocket/config/tmp_genesis.json $HOME/.pocket/config/genesis.json +cat $HOME/.pocket/config/genesis.json | jq '.app_state.auth.accounts[0].value.public_key.value="'"$pubkey"'"' > $HOME/.pocket/config/tmp_genesis.json && mv $HOME/.pocket/config/tmp_genesis.json $HOME/.pocket/config/genesis.json +cat $HOME/.pocket/config/genesis.json | jq '.app_state.pos.validators[0].address="'"$address"'"' > $HOME/.pocket/config/tmp_genesis.json && mv $HOME/.pocket/config/tmp_genesis.json $HOME/.pocket/config/genesis.json +cat $HOME/.pocket/config/genesis.json | jq '.app_state.pos.validators[0].public_key="'"$pubkey"'"' > $HOME/.pocket/config/tmp_genesis.json && mv $HOME/.pocket/config/tmp_genesis.json $HOME/.pocket/config/genesis.json +cat $HOME/.pocket/config/genesis.json | jq '.app_state.pos.validators[0].service_url="'"$SERVICE_URL"'"' > $HOME/.pocket/config/tmp_genesis.json && mv $HOME/.pocket/config/tmp_genesis.json $HOME/.pocket/config/genesis.json + +# if [[ $1 == "pending" ]]; then +# echo "pending mode is on, please wait for the first block committed." +# fi + +# Start the node +pocket start --simulateRelay diff --git a/app/data/config/fixturenet-pocket/genesis.json b/app/data/config/fixturenet-pocket/genesis.json new file mode 100644 index 00000000..4d92d5a6 --- /dev/null +++ b/app/data/config/fixturenet-pocket/genesis.json @@ -0,0 +1,272 @@ +{ + "genesis_time": "2020-07-28T15:00:00.000000Z", + "chain_id": "testnet", + "consensus_params": { + "block": { + "max_bytes": "4000000", + "max_gas": "-1", + "time_iota_ms": "1" + }, + "evidence": { + "max_age": "120000000000" + }, + "validator": { + "pub_key_types": [ + "ed25519" + ] + } + }, + "app_hash": "", + "app_state": { + "application": { + "params": { + "unstaking_time": "1814000000000000", + "max_applications": "9223372036854775807", + "app_stake_minimum": "1000000", + "base_relays_per_pokt": "167", + "stability_adjustment": "0", + "participation_rate_on": false, + "maximum_chains": "15" + }, + "applications": [], + "exported": false + }, + "auth": { + "params": { + "max_memo_characters": "75", + "tx_sig_limit": "8", + "fee_multipliers": { + "fee_multiplier": [], + "default": "1" + } + }, + "accounts": [ + { + "type": "posmint/Account", + "value": { + "address": "!validator-address", + "coins": [ + { + "amount": "0", + "denom": "upokt" + } + ], + "public_key": { + "type": "crypto/ed25519_public_key", + "value": "!validator-pubkey" + } + } + } + ], + "supply": [] + }, + "gov": { + "params": { + "acl": [ + { + "acl_key": "application/ApplicationStakeMinimum", + "address": "a83172b67b5ffbfcb8acb95acc0fd0466a9d4bc4" + }, + { + "acl_key": "application/AppUnstakingTime", + "address": "a83172b67b5ffbfcb8acb95acc0fd0466a9d4bc4" + }, + { + "acl_key": "application/BaseRelaysPerPOKT", + "address": "a83172b67b5ffbfcb8acb95acc0fd0466a9d4bc4" + }, + { + "acl_key": "application/MaxApplications", + "address": "a83172b67b5ffbfcb8acb95acc0fd0466a9d4bc4" + }, + { + "acl_key": "application/MaximumChains", + "address": "a83172b67b5ffbfcb8acb95acc0fd0466a9d4bc4" + }, + { + "acl_key": "application/ParticipationRateOn", + "address": "a83172b67b5ffbfcb8acb95acc0fd0466a9d4bc4" + }, + { + "acl_key": "application/StabilityAdjustment", + "address": "a83172b67b5ffbfcb8acb95acc0fd0466a9d4bc4" + }, + { + "acl_key": "auth/MaxMemoCharacters", + "address": "a83172b67b5ffbfcb8acb95acc0fd0466a9d4bc4" + }, + { + "acl_key": "auth/TxSigLimit", + "address": "a83172b67b5ffbfcb8acb95acc0fd0466a9d4bc4" + }, + { + "acl_key": "gov/acl", + "address": "a83172b67b5ffbfcb8acb95acc0fd0466a9d4bc4" + }, + { + "acl_key": "gov/daoOwner", + "address": "a83172b67b5ffbfcb8acb95acc0fd0466a9d4bc4" + }, + { + "acl_key": "gov/upgrade", + "address": "a83172b67b5ffbfcb8acb95acc0fd0466a9d4bc4" + }, + { + "acl_key": "pocketcore/ClaimExpiration", + "address": "a83172b67b5ffbfcb8acb95acc0fd0466a9d4bc4" + }, + { + "acl_key": "auth/FeeMultipliers", + "address": "a83172b67b5ffbfcb8acb95acc0fd0466a9d4bc4" + }, + { + "acl_key": "pocketcore/ReplayAttackBurnMultiplier", + "address": "a83172b67b5ffbfcb8acb95acc0fd0466a9d4bc4" + }, + { + "acl_key": "pos/ProposerPercentage", + "address": "a83172b67b5ffbfcb8acb95acc0fd0466a9d4bc4" + }, + { + "acl_key": "pocketcore/ClaimSubmissionWindow", + "address": "a83172b67b5ffbfcb8acb95acc0fd0466a9d4bc4" + }, + { + "acl_key": "pocketcore/MinimumNumberOfProofs", + "address": "a83172b67b5ffbfcb8acb95acc0fd0466a9d4bc4" + }, + { + "acl_key": "pocketcore/SessionNodeCount", + "address": "a83172b67b5ffbfcb8acb95acc0fd0466a9d4bc4" + }, + { + "acl_key": "pocketcore/SupportedBlockchains", + "address": "a83172b67b5ffbfcb8acb95acc0fd0466a9d4bc4" + }, + { + "acl_key": "pos/BlocksPerSession", + "address": "a83172b67b5ffbfcb8acb95acc0fd0466a9d4bc4" + }, + { + "acl_key": "pos/DAOAllocation", + "address": "a83172b67b5ffbfcb8acb95acc0fd0466a9d4bc4" + }, + { + "acl_key": "pos/DowntimeJailDuration", + "address": "a83172b67b5ffbfcb8acb95acc0fd0466a9d4bc4" + }, + { + "acl_key": "pos/MaxEvidenceAge", + "address": "a83172b67b5ffbfcb8acb95acc0fd0466a9d4bc4" + }, + { + "acl_key": "pos/MaximumChains", + "address": "a83172b67b5ffbfcb8acb95acc0fd0466a9d4bc4" + }, + { + "acl_key": "pos/MaxJailedBlocks", + "address": "a83172b67b5ffbfcb8acb95acc0fd0466a9d4bc4" + }, + { + "acl_key": "pos/MaxValidators", + "address": "a83172b67b5ffbfcb8acb95acc0fd0466a9d4bc4" + }, + { + "acl_key": "pos/MinSignedPerWindow", + "address": "a83172b67b5ffbfcb8acb95acc0fd0466a9d4bc4" + }, + { + "acl_key": "pos/RelaysToTokensMultiplier", + "address": "a83172b67b5ffbfcb8acb95acc0fd0466a9d4bc4" + }, + { + "acl_key": "pos/SignedBlocksWindow", + "address": "a83172b67b5ffbfcb8acb95acc0fd0466a9d4bc4" + }, + { + "acl_key": "pos/SlashFractionDoubleSign", + "address": "a83172b67b5ffbfcb8acb95acc0fd0466a9d4bc4" + }, + { + "acl_key": "pos/SlashFractionDowntime", + "address": "a83172b67b5ffbfcb8acb95acc0fd0466a9d4bc4" + }, + { + "acl_key": "pos/StakeDenom", + "address": "a83172b67b5ffbfcb8acb95acc0fd0466a9d4bc4" + }, + { + "acl_key": "pos/StakeMinimum", + "address": "a83172b67b5ffbfcb8acb95acc0fd0466a9d4bc4" + }, + { + "acl_key": "pos/UnstakingTime", + "address": "a83172b67b5ffbfcb8acb95acc0fd0466a9d4bc4" + } + ], + "dao_owner": "a83172b67b5ffbfcb8acb95acc0fd0466a9d4bc4", + "upgrade": { + "Height": "0", + "Version": "0" + } + }, + "DAO_Tokens": "50000000000000" + }, + "pos": { + "params": { + "relays_to_tokens_multiplier": "10000", + "unstaking_time": "1814000000000000", + "max_validators": "5000", + "stake_denom": "upokt", + "stake_minimum": "15000000000", + "session_block_frequency": "4", + "dao_allocation": "10", + "proposer_allocation": "1", + "maximum_chains": "15", + "max_jailed_blocks": "37960", + "max_evidence_age": "120000000000", + "signed_blocks_window": "10", + "min_signed_per_window": "0.60", + "downtime_jail_duration": "3600000000000", + "slash_fraction_double_sign": "0.05", + "slash_fraction_downtime": "0.000001" + }, + "prevState_total_power": "0", + "prevState_validator_powers": null, + "validators": [ + { + "address": "!validator-address", + "public_key": "!validator-pubkey", + "jailed": false, + "status": 2, + "tokens": "5000000000000", + "service_url": "!validator-url", + "chains": [ + "0001", + "0021" + ], + "unstaking_time": "2021-05-15T00:00:00Z" + } + ], + "exported": false, + "signing_infos": {}, + "missed_blocks": {}, + "previous_proposer": "" + }, + "pocketcore": { + "params": { + "session_node_count": "5", + "proof_waiting_period": "3", + "supported_blockchains": [ + "0001", + "0021" + ], + "claim_expiration": "120", + "replay_attack_burn_multiplier": "3", + "minimum_number_of_proofs": "10" + }, + "receipts": null, + "claims": null + } + } +} \ No newline at end of file diff --git a/app/data/container-build/cerc-pocket/build.sh b/app/data/container-build/cerc-pocket/build.sh new file mode 100755 index 00000000..1f59a8f0 --- /dev/null +++ b/app/data/container-build/cerc-pocket/build.sh @@ -0,0 +1,3 @@ +#!/usr/bin/env bash +# Build cerc/pocket +docker build -t cerc/pocket:local ${CERC_REPO_BASE_DIR}/pocket-core-deployments/docker diff --git a/app/data/container-image-list.txt b/app/data/container-image-list.txt index 0caea48a..0fe2be5c 100644 --- a/app/data/container-image-list.txt +++ b/app/data/container-image-list.txt @@ -36,3 +36,4 @@ cerc/optimism-l2geth cerc/optimism-op-batcher cerc/optimism-op-node cerc/optimism-op-proposer +cerc/pocket diff --git a/app/data/pod-list.txt b/app/data/pod-list.txt index f24c9ed0..a3ae8f44 100644 --- a/app/data/pod-list.txt +++ b/app/data/pod-list.txt @@ -24,3 +24,4 @@ tx-spammer kubo foundry fixturenet-optimism +fixturenet-pocket diff --git a/app/data/repository-list.txt b/app/data/repository-list.txt index 0d808d68..438b9bbb 100644 --- a/app/data/repository-list.txt +++ b/app/data/repository-list.txt @@ -27,3 +27,5 @@ lirewine/sdk telackey/act_runner ethereum-optimism/op-geth ethereum-optimism/optimism +pokt-network/pocket-core +pokt-network/pocket-core-deployments diff --git a/app/data/stacks/fixturenet-pocket/README.md b/app/data/stacks/fixturenet-pocket/README.md new file mode 100644 index 00000000..b29520c8 --- /dev/null +++ b/app/data/stacks/fixturenet-pocket/README.md @@ -0,0 +1,59 @@ +# Pocket Fixturenet + +Instructions for deploying a local single-node Pocket chain alongside a geth + lighthouse blockchain "fixturenet" for development and testing purposes using laconic-stack-orchestrator. + +## 1. Build Laconic Stack Orchestrator +Build this fork of Laconic Stack Orchestrator which includes the fixturenet-pocket stack: +``` +$ scripts/build_shiv_package.sh +$ cd package +$ mv laconic-so-{version} /usr/local/bin/laconic-so # Or move laconic-so to ~/bin or your favorite on-path directory +``` + +## 2. Clone required repositories +``` +$ laconic-so --stack fixturenet-pocket setup-repositories +``` +## 3. Build the stack's containers +``` +$ laconic-so --stack fixturenet-pocket build-containers +``` +## 4. Deploy the stack +``` +$ laconic-so --stack fixturenet-pocket deploy up +``` +It may take up to 10 minutes for the Eth Fixturenet to fully come online and start producing blocks. +## 5. Check status +**Eth Fixturenet:** +``` +$ laconic-so --stack fixturenet-pocket deploy exec fixturenet-eth-bootnode-lighthouse /scripts/status-internal.sh +Waiting for geth to generate DAG.... done +Waiting for beacon phase0.... done +Waiting for beacon altair.... done +Waiting for beacon bellatrix pre-merge.... done +Waiting for beacon bellatrix merge.... done +``` +**Pocket node:** +``` +$ laconic-so --stack fixturenet-pocket deploy exec pocket "pocket query height" +2023/04/20 08:07:46 Initializing Pocket Datadir +2023/04/20 08:07:46 datadir = /home/app/.pocket +http://localhost:8081/v1/query/height +{ + "height": 4 +} +``` +or +``` +$ laconic-so --stack fixturenet-pocket deploy logs pocket +``` +## 6. Send a relay request to Pocket node +The Pocket node serves relay requests at `http://localhost:8081/v1/client/sim` +**Example request:** +``` +$ curl -X POST --data '{"relay_network_id":"0021","payload":{"data":"{\"jsonrpc\": \"2.0\",\"id\": 1,\"method\": \"eth_blockNumber\",\"params\": []}","method":"POST","path":"","headers":{}}}' http://localhost:8081/v1/client/sim +``` +**Response:** +``` +"{\"jsonrpc\":\"2.0\",\"id\":1,\"result\":\"0x6fe\"}\n" +``` diff --git a/app/data/stacks/fixturenet-pocket/stack.yml b/app/data/stacks/fixturenet-pocket/stack.yml new file mode 100644 index 00000000..c5491f9a --- /dev/null +++ b/app/data/stacks/fixturenet-pocket/stack.yml @@ -0,0 +1,16 @@ +version: "1.0" +name: fixturenet-pocket +description: "A single node pocket chain that can serve relays from the geth-1 node in eth-fixturenet" +repos: + - cerc-io/go-ethereum + - pokt-network/pocket-core + - pokt-network/pocket-core-deployments # contains the dockerfile +containers: + - cerc/go-ethereum + - cerc/lighthouse + - cerc/fixturenet-eth-geth + - cerc/fixturenet-eth-lighthouse + - cerc/pocket +pods: + - fixturenet-pocket + - fixturenet-eth From 5a94aed7f7e150381c0a49a3bcc5b43763333151 Mon Sep 17 00:00:00 2001 From: Nabarun Gogoi Date: Thu, 4 May 2023 15:35:04 +0530 Subject: [PATCH 17/20] Add stack for azimuth watchers with gateway-server (#379) * Setup gateway-server with watchers * Add js script to merge toml config files * Remove individual watcher configs * Add all azimuth watchers in stack * Fix toml-js install * Use env variables for ipld-eth-server endpoints * Checkout to version tag in azimuth-watcher-ts repo --- .../docker-compose-watcher-azimuth.yml | 304 ++++++++++++++++++ .../watcher-azimuth/gateway-watchers.json | 34 ++ app/data/config/watcher-azimuth/merge-toml.js | 31 ++ .../config/watcher-azimuth/start-server.sh | 27 ++ .../watcher-config-template.toml | 14 + .../config/watcher-azimuth/watcher-params.env | 5 + .../cerc-watcher-azimuth/Dockerfile | 13 + .../cerc-watcher-azimuth/build.sh | 9 + app/data/container-image-list.txt | 1 + app/data/pod-list.txt | 1 + app/data/repository-list.txt | 1 + app/data/stacks/azimuth/README.md | 72 +++++ app/data/stacks/azimuth/stack.yml | 8 + 13 files changed, 520 insertions(+) create mode 100644 app/data/compose/docker-compose-watcher-azimuth.yml create mode 100644 app/data/config/watcher-azimuth/gateway-watchers.json create mode 100644 app/data/config/watcher-azimuth/merge-toml.js create mode 100755 app/data/config/watcher-azimuth/start-server.sh create mode 100644 app/data/config/watcher-azimuth/watcher-config-template.toml create mode 100644 app/data/config/watcher-azimuth/watcher-params.env create mode 100644 app/data/container-build/cerc-watcher-azimuth/Dockerfile create mode 100755 app/data/container-build/cerc-watcher-azimuth/build.sh create mode 100644 app/data/stacks/azimuth/README.md create mode 100644 app/data/stacks/azimuth/stack.yml diff --git a/app/data/compose/docker-compose-watcher-azimuth.yml b/app/data/compose/docker-compose-watcher-azimuth.yml new file mode 100644 index 00000000..327c77fc --- /dev/null +++ b/app/data/compose/docker-compose-watcher-azimuth.yml @@ -0,0 +1,304 @@ +version: '3.2' + +services: + # Starts the PostgreSQL database for watchers + watcher-db: + restart: unless-stopped + image: postgres:14-alpine + environment: + - POSTGRES_USER=vdbm + - POSTGRES_MULTIPLE_DATABASES=azimuth-watcher,azimuth-watcher-job-queue,censures-watcher,censures-watcher-job-queue,claims-watcher,claims-watcher-job-queue,conditional-star-release-watcher,conditional-star-release-watcher-job-queue,delegated-sending-watcher,delegated-sending-watcher-job-queue,ecliptic-watcher,ecliptic-watcher-job-queue,linear-star-release-watcher,linear-star-release-watcher-job-queue,polls-watcher,polls-watcher-job-queue + - POSTGRES_EXTENSION=azimuth-watcher-job-queue:pgcrypto,censures-watcher-job-queue:pgcrypto,claims-watcher-job-queue:pgcrypto,conditional-star-release-watcher-job-queue:pgcrypto,delegated-sending-watcher-job-queue:pgcrypto,ecliptic-watcher-job-queue:pgcrypto,linear-star-release-watcher-job-queue:pgcrypto,polls-watcher-job-queue:pgcrypto, + - POSTGRES_PASSWORD=password + volumes: + - ../config/postgresql/multiple-postgressql-databases.sh:/docker-entrypoint-initdb.d/multiple-postgressql-databases.sh + - watcher_db_data:/var/lib/postgresql/data + ports: + - "0.0.0.0:15432:5432" + healthcheck: + test: ["CMD", "nc", "-v", "localhost", "5432"] + interval: 20s + timeout: 5s + retries: 15 + start_period: 10s + + # Starts the azimuth-watcher server + azimuth-watcher-server: + image: cerc/watcher-azimuth:local + restart: unless-stopped + depends_on: + watcher-db: + condition: service_healthy + env_file: + - ../config/watcher-azimuth/watcher-params.env + environment: + CERC_SCRIPT_DEBUG: ${CERC_SCRIPT_DEBUG} + CERC_IPLD_ETH_RPC: ${CERC_IPLD_ETH_RPC} + CERC_IPLD_ETH_GQL: ${CERC_IPLD_ETH_GQL} + working_dir: /app/packages/azimuth-watcher + command: "./start-server.sh" + volumes: + - ../config/watcher-azimuth/watcher-config-template.toml:/app/packages/azimuth-watcher/environments/watcher-config-template.toml + - ../config/watcher-azimuth/merge-toml.js:/app/packages/azimuth-watcher/merge-toml.js + - ../config/watcher-azimuth/start-server.sh:/app/packages/azimuth-watcher/start-server.sh + ports: + - "3001" + healthcheck: + test: ["CMD", "nc", "-vz", "localhost", "3001"] + interval: 20s + timeout: 5s + retries: 15 + start_period: 5s + extra_hosts: + - "host.docker.internal:host-gateway" + + # Starts the censures-watcher server + censures-watcher-server: + image: cerc/watcher-azimuth:local + restart: unless-stopped + depends_on: + watcher-db: + condition: service_healthy + env_file: + - ../config/watcher-azimuth/watcher-params.env + environment: + CERC_SCRIPT_DEBUG: ${CERC_SCRIPT_DEBUG} + CERC_IPLD_ETH_RPC: ${CERC_IPLD_ETH_RPC} + CERC_IPLD_ETH_GQL: ${CERC_IPLD_ETH_GQL} + working_dir: /app/packages/censures-watcher + command: "./start-server.sh" + volumes: + - ../config/watcher-azimuth/watcher-config-template.toml:/app/packages/censures-watcher/environments/watcher-config-template.toml + - ../config/watcher-azimuth/merge-toml.js:/app/packages/censures-watcher/merge-toml.js + - ../config/watcher-azimuth/start-server.sh:/app/packages/censures-watcher/start-server.sh + ports: + - "3002" + healthcheck: + test: ["CMD", "nc", "-vz", "localhost", "3002"] + interval: 20s + timeout: 5s + retries: 15 + start_period: 5s + extra_hosts: + - "host.docker.internal:host-gateway" + + # Starts the claims-watcher server + claims-watcher-server: + image: cerc/watcher-azimuth:local + restart: unless-stopped + depends_on: + watcher-db: + condition: service_healthy + env_file: + - ../config/watcher-azimuth/watcher-params.env + environment: + CERC_SCRIPT_DEBUG: ${CERC_SCRIPT_DEBUG} + CERC_IPLD_ETH_RPC: ${CERC_IPLD_ETH_RPC} + CERC_IPLD_ETH_GQL: ${CERC_IPLD_ETH_GQL} + working_dir: /app/packages/claims-watcher + command: "./start-server.sh" + volumes: + - ../config/watcher-azimuth/watcher-config-template.toml:/app/packages/claims-watcher/environments/watcher-config-template.toml + - ../config/watcher-azimuth/merge-toml.js:/app/packages/claims-watcher/merge-toml.js + - ../config/watcher-azimuth/start-server.sh:/app/packages/claims-watcher/start-server.sh + ports: + - "3003" + healthcheck: + test: ["CMD", "nc", "-vz", "localhost", "3003"] + interval: 20s + timeout: 5s + retries: 15 + start_period: 5s + extra_hosts: + - "host.docker.internal:host-gateway" + + # Starts the conditional-star-release-watcher server + conditional-star-release-watcher-server: + image: cerc/watcher-azimuth:local + restart: unless-stopped + depends_on: + watcher-db: + condition: service_healthy + env_file: + - ../config/watcher-azimuth/watcher-params.env + environment: + CERC_SCRIPT_DEBUG: ${CERC_SCRIPT_DEBUG} + CERC_IPLD_ETH_RPC: ${CERC_IPLD_ETH_RPC} + CERC_IPLD_ETH_GQL: ${CERC_IPLD_ETH_GQL} + working_dir: /app/packages/conditional-star-release-watcher + command: "./start-server.sh" + volumes: + - ../config/watcher-azimuth/watcher-config-template.toml:/app/packages/conditional-star-release-watcher/environments/watcher-config-template.toml + - ../config/watcher-azimuth/merge-toml.js:/app/packages/conditional-star-release-watcher/merge-toml.js + - ../config/watcher-azimuth/start-server.sh:/app/packages/conditional-star-release-watcher/start-server.sh + ports: + - "3004" + healthcheck: + test: ["CMD", "nc", "-vz", "localhost", "3004"] + interval: 20s + timeout: 5s + retries: 15 + start_period: 5s + extra_hosts: + - "host.docker.internal:host-gateway" + + # Starts the delegated-sending-watcher server + delegated-sending-watcher-server: + image: cerc/watcher-azimuth:local + restart: unless-stopped + depends_on: + watcher-db: + condition: service_healthy + env_file: + - ../config/watcher-azimuth/watcher-params.env + environment: + CERC_SCRIPT_DEBUG: ${CERC_SCRIPT_DEBUG} + CERC_IPLD_ETH_RPC: ${CERC_IPLD_ETH_RPC} + CERC_IPLD_ETH_GQL: ${CERC_IPLD_ETH_GQL} + working_dir: /app/packages/delegated-sending-watcher + command: "./start-server.sh" + volumes: + - ../config/watcher-azimuth/watcher-config-template.toml:/app/packages/delegated-sending-watcher/environments/watcher-config-template.toml + - ../config/watcher-azimuth/merge-toml.js:/app/packages/delegated-sending-watcher/merge-toml.js + - ../config/watcher-azimuth/start-server.sh:/app/packages/delegated-sending-watcher/start-server.sh + ports: + - "3005" + healthcheck: + test: ["CMD", "nc", "-vz", "localhost", "3005"] + interval: 20s + timeout: 5s + retries: 15 + start_period: 5s + extra_hosts: + - "host.docker.internal:host-gateway" + + # Starts the ecliptic-watcher server + ecliptic-watcher-server: + image: cerc/watcher-azimuth:local + restart: unless-stopped + depends_on: + watcher-db: + condition: service_healthy + env_file: + - ../config/watcher-azimuth/watcher-params.env + environment: + CERC_SCRIPT_DEBUG: ${CERC_SCRIPT_DEBUG} + CERC_IPLD_ETH_RPC: ${CERC_IPLD_ETH_RPC} + CERC_IPLD_ETH_GQL: ${CERC_IPLD_ETH_GQL} + working_dir: /app/packages/ecliptic-watcher + command: "./start-server.sh" + volumes: + - ../config/watcher-azimuth/watcher-config-template.toml:/app/packages/ecliptic-watcher/environments/watcher-config-template.toml + - ../config/watcher-azimuth/merge-toml.js:/app/packages/ecliptic-watcher/merge-toml.js + - ../config/watcher-azimuth/start-server.sh:/app/packages/ecliptic-watcher/start-server.sh + ports: + - "3006" + healthcheck: + test: ["CMD", "nc", "-vz", "localhost", "3006"] + interval: 20s + timeout: 5s + retries: 15 + start_period: 5s + extra_hosts: + - "host.docker.internal:host-gateway" + + # Starts the linear-star-release-watcher server + linear-star-release-watcher-server: + image: cerc/watcher-azimuth:local + restart: unless-stopped + depends_on: + watcher-db: + condition: service_healthy + env_file: + - ../config/watcher-azimuth/watcher-params.env + environment: + CERC_SCRIPT_DEBUG: ${CERC_SCRIPT_DEBUG} + CERC_IPLD_ETH_RPC: ${CERC_IPLD_ETH_RPC} + CERC_IPLD_ETH_GQL: ${CERC_IPLD_ETH_GQL} + working_dir: /app/packages/linear-star-release-watcher + command: "./start-server.sh" + volumes: + - ../config/watcher-azimuth/watcher-config-template.toml:/app/packages/linear-star-release-watcher/environments/watcher-config-template.toml + - ../config/watcher-azimuth/merge-toml.js:/app/packages/linear-star-release-watcher/merge-toml.js + - ../config/watcher-azimuth/start-server.sh:/app/packages/linear-star-release-watcher/start-server.sh + ports: + - "3007" + healthcheck: + test: ["CMD", "nc", "-vz", "localhost", "3007"] + interval: 20s + timeout: 5s + retries: 15 + start_period: 5s + extra_hosts: + - "host.docker.internal:host-gateway" + + # Starts the polls-watcher server + polls-watcher-server: + image: cerc/watcher-azimuth:local + restart: unless-stopped + depends_on: + watcher-db: + condition: service_healthy + env_file: + - ../config/watcher-azimuth/watcher-params.env + environment: + CERC_SCRIPT_DEBUG: ${CERC_SCRIPT_DEBUG} + CERC_IPLD_ETH_RPC: ${CERC_IPLD_ETH_RPC} + CERC_IPLD_ETH_GQL: ${CERC_IPLD_ETH_GQL} + working_dir: /app/packages/polls-watcher + command: "./start-server.sh" + volumes: + - ../config/watcher-azimuth/watcher-config-template.toml:/app/packages/polls-watcher/environments/watcher-config-template.toml + - ../config/watcher-azimuth/merge-toml.js:/app/packages/polls-watcher/merge-toml.js + - ../config/watcher-azimuth/start-server.sh:/app/packages/polls-watcher/start-server.sh + ports: + - "3008" + healthcheck: + test: ["CMD", "nc", "-vz", "localhost", "3008"] + interval: 20s + timeout: 5s + retries: 15 + start_period: 5s + extra_hosts: + - "host.docker.internal:host-gateway" + + # Starts the gateway-server for proxying queries + gateway-server: + image: cerc/watcher-azimuth:local + restart: unless-stopped + depends_on: + azimuth-watcher-server: + condition: service_healthy + censures-watcher-server: + condition: service_healthy + claims-watcher-server: + condition: service_healthy + conditional-star-release-watcher-server: + condition: service_healthy + delegated-sending-watcher-server: + condition: service_healthy + ecliptic-watcher-server: + condition: service_healthy + linear-star-release-watcher-server: + condition: service_healthy + polls-watcher-server: + condition: service_healthy + environment: + CERC_SCRIPT_DEBUG: ${CERC_SCRIPT_DEBUG} + working_dir: /app/packages/gateway-server + command: "yarn server" + volumes: + - ../config/watcher-azimuth/gateway-watchers.json:/app/packages/gateway-server/dist/watchers.json + ports: + - "0.0.0.0:4000:4000" + healthcheck: + test: ["CMD", "nc", "-vz", "localhost", "4000"] + interval: 20s + timeout: 5s + retries: 15 + start_period: 5s + extra_hosts: + - "host.docker.internal:host-gateway" + +volumes: + watcher_db_data: diff --git a/app/data/config/watcher-azimuth/gateway-watchers.json b/app/data/config/watcher-azimuth/gateway-watchers.json new file mode 100644 index 00000000..40687556 --- /dev/null +++ b/app/data/config/watcher-azimuth/gateway-watchers.json @@ -0,0 +1,34 @@ +[ + { + "endpoint": "http://azimuth-watcher-server:3001/graphql", + "prefix": "azimuth" + }, + { + "endpoint": "http://censures-watcher-server:3002/graphql", + "prefix": "censures" + }, + { + "endpoint": "http://claims-watcher-server:3003/graphql", + "prefix": "claims" + }, + { + "endpoint": "http://conditional-star-release-watcher-server:3004/graphql", + "prefix": "conditionalStarRelease" + }, + { + "endpoint": "http://delegated-sending-watcher-server:3005/graphql", + "prefix": "delegatedSending" + }, + { + "endpoint": "http://ecliptic-watcher-server:3006/graphql", + "prefix": "ecliptic" + }, + { + "endpoint": "http://linear-star-release-watcher-server:3007/graphql", + "prefix": "linearStarRelease" + }, + { + "endpoint": "http://polls-watcher-server:3008/graphql", + "prefix": "polls" + } +] diff --git a/app/data/config/watcher-azimuth/merge-toml.js b/app/data/config/watcher-azimuth/merge-toml.js new file mode 100644 index 00000000..9224baec --- /dev/null +++ b/app/data/config/watcher-azimuth/merge-toml.js @@ -0,0 +1,31 @@ +const fs = require('fs'); +const tomlJS = require('toml-js'); +const toml = require('toml'); +const { merge } = require('lodash') + +const main = () => { + const overrideConfigString = fs.readFileSync('environments/watcher-config.toml', 'utf-8'); + const configString = fs.readFileSync('environments/local.toml', 'utf-8'); + const overrideConfig = toml.parse(overrideConfigString) + const config = toml.parse(configString) + + // Merge configs + const updatedConfig = merge(config, overrideConfig); + + // Form dbConnectionString for jobQueue DB + const parts = config.jobQueue.dbConnectionString.split("://"); + const credsAndDB = parts[1].split("@"); + const creds = credsAndDB[0].split(":"); + creds[0] = overrideConfig.database.username; + creds[1] = overrideConfig.database.password; + credsAndDB[0] = creds.join(":"); + const dbName = credsAndDB[1].split("/")[1] + credsAndDB[1] = [overrideConfig.database.host, dbName].join("/"); + parts[1] = credsAndDB.join("@"); + + updatedConfig.jobQueue.dbConnectionString = parts.join("://"); + + fs.writeFileSync('environments/local.toml', tomlJS.dump(updatedConfig), 'utf-8'); +} + +main(); diff --git a/app/data/config/watcher-azimuth/start-server.sh b/app/data/config/watcher-azimuth/start-server.sh new file mode 100755 index 00000000..c84c58d0 --- /dev/null +++ b/app/data/config/watcher-azimuth/start-server.sh @@ -0,0 +1,27 @@ +#!/bin/sh +set -e +if [ -n "$CERC_SCRIPT_DEBUG" ]; then + set -x +fi + +CERC_IPLD_ETH_RPC="${CERC_IPLD_ETH_RPC:-${DEFAULT_CERC_IPLD_ETH_RPC}}" +CERC_IPLD_ETH_GQL="${CERC_IPLD_ETH_GQL:-${DEFAULT_CERC_IPLD_ETH_GQL}}" + +echo "Using IPLD ETH RPC endpoint ${CERC_IPLD_ETH_RPC}" +echo "Using IPLD GQL endpoint ${CERC_IPLD_ETH_GQL}" + +# Replace env variables in template TOML file +# Read in the config template TOML file and modify it +WATCHER_CONFIG_TEMPLATE=$(cat environments/watcher-config-template.toml) +WATCHER_CONFIG=$(echo "$WATCHER_CONFIG_TEMPLATE" | \ + sed -E "s|REPLACE_WITH_CERC_IPLD_ETH_RPC|${CERC_IPLD_ETH_RPC}|g; \ + s|REPLACE_WITH_CERC_IPLD_ETH_GQL|${CERC_IPLD_ETH_GQL}| ") + +# Write the modified content to a new file +echo "$WATCHER_CONFIG" > environments/watcher-config.toml + +# Merge SO watcher config with existing config file +node merge-toml.js + +echo 'yarn server' +yarn server diff --git a/app/data/config/watcher-azimuth/watcher-config-template.toml b/app/data/config/watcher-azimuth/watcher-config-template.toml new file mode 100644 index 00000000..1a4616fc --- /dev/null +++ b/app/data/config/watcher-azimuth/watcher-config-template.toml @@ -0,0 +1,14 @@ +[server] + host = "0.0.0.0" + maxSimultaneousRequests = -1 + +[database] + host = "watcher-db" + port = 5432 + username = "vdbm" + password = "password" + +[upstream] + [upstream.ethServer] + gqlApiEndpoint = "REPLACE_WITH_CERC_IPLD_ETH_GQL" + rpcProviderEndpoint = "REPLACE_WITH_CERC_IPLD_ETH_RPC" diff --git a/app/data/config/watcher-azimuth/watcher-params.env b/app/data/config/watcher-azimuth/watcher-params.env new file mode 100644 index 00000000..8fcdc2d6 --- /dev/null +++ b/app/data/config/watcher-azimuth/watcher-params.env @@ -0,0 +1,5 @@ +# Defaults + +# ipld-eth-server endpoints +DEFAULT_CERC_IPLD_ETH_RPC= +DEFAULT_CERC_IPLD_ETH_GQL= diff --git a/app/data/container-build/cerc-watcher-azimuth/Dockerfile b/app/data/container-build/cerc-watcher-azimuth/Dockerfile new file mode 100644 index 00000000..eaf717aa --- /dev/null +++ b/app/data/container-build/cerc-watcher-azimuth/Dockerfile @@ -0,0 +1,13 @@ +FROM node:18.16.0-alpine3.16 + +RUN apk --update --no-cache add git python3 alpine-sdk + +WORKDIR /app + +COPY . . + +RUN echo "Building azimuth-watcher-ts" && \ + yarn && yarn build + +RUN echo "Install toml-js to update watcher config files" && \ + yarn add --dev --ignore-workspace-root-check toml-js diff --git a/app/data/container-build/cerc-watcher-azimuth/build.sh b/app/data/container-build/cerc-watcher-azimuth/build.sh new file mode 100755 index 00000000..522f985b --- /dev/null +++ b/app/data/container-build/cerc-watcher-azimuth/build.sh @@ -0,0 +1,9 @@ +#!/usr/bin/env bash +# Build cerc/watcher-azimuth + +source ${CERC_CONTAINER_BASE_DIR}/build-base.sh + +# See: https://stackoverflow.com/a/246128/1701505 +SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd ) + +docker build -t cerc/watcher-azimuth:local -f ${SCRIPT_DIR}/Dockerfile ${build_command_args} ${CERC_REPO_BASE_DIR}/azimuth-watcher-ts diff --git a/app/data/container-image-list.txt b/app/data/container-image-list.txt index 0fe2be5c..ba0eaf23 100644 --- a/app/data/container-image-list.txt +++ b/app/data/container-image-list.txt @@ -37,3 +37,4 @@ cerc/optimism-op-batcher cerc/optimism-op-node cerc/optimism-op-proposer cerc/pocket +cerc/watcher-azimuth diff --git a/app/data/pod-list.txt b/app/data/pod-list.txt index a3ae8f44..34d66ba6 100644 --- a/app/data/pod-list.txt +++ b/app/data/pod-list.txt @@ -25,3 +25,4 @@ kubo foundry fixturenet-optimism fixturenet-pocket +watcher-azimuth diff --git a/app/data/repository-list.txt b/app/data/repository-list.txt index 438b9bbb..7035de2a 100644 --- a/app/data/repository-list.txt +++ b/app/data/repository-list.txt @@ -29,3 +29,4 @@ ethereum-optimism/op-geth ethereum-optimism/optimism pokt-network/pocket-core pokt-network/pocket-core-deployments +cerc-io/azimuth-watcher-ts diff --git a/app/data/stacks/azimuth/README.md b/app/data/stacks/azimuth/README.md new file mode 100644 index 00000000..d3e377da --- /dev/null +++ b/app/data/stacks/azimuth/README.md @@ -0,0 +1,72 @@ +# Azimuth Watcher + +Instructions to setup and deploy Azimuth Watcher stack + +## Setup + +Prerequisite: ipld-eth-server RPC and GQL endpoints + +Clone required repositories: + +```bash +laconic-so --stack azimuth setup-repositories +``` + +NOTE: If the repository already exists and checked out to a different version, `setup-repositories` command will throw an error. +For getting around this, the `azimuth-watcher-ts` repository can be removed and then run the command. + +Checkout to the required versions and branches in repos + +```bash +# azimuth-watcher-ts +cd ~/cerc/azimuth-watcher-ts +git checkout v0.1.0 +``` + +Build the container images: + +```bash +laconic-so --stack azimuth build-containers +``` + +This should create the required docker images in the local image registry. + +### Configuration + +* Create and update an env file to be used in the next step: + + ```bash + # External ipld-eth-server endpoints + CERC_IPLD_ETH_RPC= + CERC_IPLD_ETH_GQL= + ``` + +* NOTE: If ipld-eth-server is running on the host machine, use `host.docker.internal` as the hostname to access host ports + +### Deploy the stack + +* Deploy the containers: + + ```bash + laconic-so --stack azimuth deploy-system --env-file up + ``` + +* List and check the health status of all the containers using `docker ps` and wait for them to be `healthy` + +## Clean up + +Stop all the services running in background run: + +```bash +laconic-so --stack azimuth deploy-system down +``` + +Clear volumes created by this stack: + +```bash +# List all relevant volumes +docker volume ls -q --filter "name=.*watcher_db_data" + +# Remove all the listed volumes +docker volume rm $(docker volume ls -q --filter "name=.*watcher_db_data") +``` diff --git a/app/data/stacks/azimuth/stack.yml b/app/data/stacks/azimuth/stack.yml new file mode 100644 index 00000000..fc3c8473 --- /dev/null +++ b/app/data/stacks/azimuth/stack.yml @@ -0,0 +1,8 @@ +version: "1.0" +name: azimuth +repos: + - cerc-io/azimuth-watcher-ts +containers: + - cerc/watcher-azimuth +pods: + - watcher-azimuth From f1bc8aa4e13fa4111991ed132930dba9e13562a8 Mon Sep 17 00:00:00 2001 From: Marten O'Grady <35665595+Escape613@users.noreply.github.com> Date: Thu, 4 May 2023 09:16:14 -0400 Subject: [PATCH 18/20] Update CONTRIBUTING.md Missing ")" in Step #1 of INSTALL --- docs/CONTRIBUTING.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/CONTRIBUTING.md b/docs/CONTRIBUTING.md index 633fc30f..89d9a748 100644 --- a/docs/CONTRIBUTING.md +++ b/docs/CONTRIBUTING.md @@ -26,7 +26,7 @@ In addition to the pre-requisites listed in the [README](/README.md), the follow 1. Clone this repository: ``` - $ git clone (https://github.com/cerc-io/stack-orchestrator.git + $ git clone https://github.com/cerc-io/stack-orchestrator.git ``` 2. Enter the project directory: From bce604e4bb2823632adf809a1cadd228e5c017c3 Mon Sep 17 00:00:00 2001 From: Marten O'Grady <35665595+Escape613@users.noreply.github.com> Date: Thu, 4 May 2023 09:47:59 -0400 Subject: [PATCH 19/20] Update CONTRIBUTING.md (#383) Wrong output for Step 3 in Build A ZipApp. Fixed it to what I just experienced while smoke testing. --- docs/CONTRIBUTING.md | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/docs/CONTRIBUTING.md b/docs/CONTRIBUTING.md index 89d9a748..39b294fb 100644 --- a/docs/CONTRIBUTING.md +++ b/docs/CONTRIBUTING.md @@ -87,20 +87,27 @@ Use shiv to build a single file Python executable zip archive of laconic-so: ``` $ cp stack-orchetrator/laconic-so ~/bin $ laconic-so - Usage: python -m laconic-so [OPTIONS] COMMAND [ARGS]... + Usage: laconic-so [OPTIONS] COMMAND [ARGS]... Laconic Stack Orchestrator Options: + --stack TEXT specify a stack to build/deploy --quiet --verbose --dry-run - -h, --help Show this message and exit. + --local-stack + --debug + --continue-on-error + -h, --help Show this message and exit. Commands: build-containers build the set of containers required for a complete... + build-npms build the set of npm packages required for a... + deploy deploy a stack deploy-system deploy a stack setup-repositories git clone the set of repositories required to build... + version print tool version ``` For cutting releases, use the [shiv build script](/scripts/build_shiv_package.sh). From b678a3ecb4b318ce26a39fd42f3ed160fd55041e Mon Sep 17 00:00:00 2001 From: Nabarun Gogoi Date: Fri, 5 May 2023 13:32:19 +0530 Subject: [PATCH 20/20] Add environment variables for multiaddrs blacklist (#381) * Add env variable for web apps config denyMultiaddrs * Add watcher config option for blacklisted multiaddrs * Update package versions * Use provided domain for relay multiaddr in peer config * Change delimeter while replacing deny multiaddrs list --------- Co-authored-by: prathamesh0 --- app/data/compose/docker-compose-mobymask-app.yml | 2 ++ app/data/compose/docker-compose-peer-test-app.yml | 1 + .../compose/docker-compose-watcher-mobymask-v2.yml | 1 + .../config/watcher-mobymask-v2/mobymask-app-start.sh | 2 ++ .../config/watcher-mobymask-v2/mobymask-params.env | 3 +++ app/data/config/watcher-mobymask-v2/start-server.sh | 10 +++++++++- .../config/watcher-mobymask-v2/test-app-config.json | 1 + app/data/config/watcher-mobymask-v2/test-app-start.sh | 2 ++ .../watcher-mobymask-v2/watcher-config-template.toml | 2 ++ app/data/container-build/cerc-mobymask-ui/Dockerfile | 4 ++-- .../cerc-mobymask-ui/apply-webapp-config.sh | 2 +- app/data/container-build/cerc-react-peer/Dockerfile | 2 +- .../cerc-react-peer/apply-webapp-config.sh | 2 +- app/data/stacks/mobymask-v2/README.md | 4 ++-- app/data/stacks/mobymask-v2/mobymask-only.md | 11 +++++++---- .../stacks/mobymask-v2/watcher-p2p-network/watcher.md | 4 ++-- app/data/stacks/mobymask-v2/web-apps.md | 3 +++ 17 files changed, 42 insertions(+), 14 deletions(-) diff --git a/app/data/compose/docker-compose-mobymask-app.yml b/app/data/compose/docker-compose-mobymask-app.yml index d43e6b44..7d41264a 100644 --- a/app/data/compose/docker-compose-mobymask-app.yml +++ b/app/data/compose/docker-compose-mobymask-app.yml @@ -13,6 +13,7 @@ services: CERC_DEPLOYED_CONTRACT: ${CERC_DEPLOYED_CONTRACT} CERC_APP_WATCHER_URL: ${CERC_APP_WATCHER_URL} CERC_RELAY_NODES: ${CERC_RELAY_NODES} + CERC_DENY_MULTIADDRS: ${CERC_DENY_MULTIADDRS} CERC_BUILD_DIR: "@cerc-io/mobymask-ui/build" working_dir: /scripts command: ["sh", "mobymask-app-start.sh"] @@ -44,6 +45,7 @@ services: CERC_DEPLOYED_CONTRACT: ${CERC_DEPLOYED_CONTRACT} CERC_APP_WATCHER_URL: ${CERC_APP_WATCHER_URL} CERC_RELAY_NODES: ${CERC_RELAY_NODES} + CERC_DENY_MULTIADDRS: ${CERC_DENY_MULTIADDRS} CERC_BUILD_DIR: "@cerc-io/mobymask-ui-lxdao/build" working_dir: /scripts command: ["sh", "mobymask-app-start.sh"] diff --git a/app/data/compose/docker-compose-peer-test-app.yml b/app/data/compose/docker-compose-peer-test-app.yml index f2a22675..f1f5e475 100644 --- a/app/data/compose/docker-compose-peer-test-app.yml +++ b/app/data/compose/docker-compose-peer-test-app.yml @@ -10,6 +10,7 @@ services: environment: CERC_SCRIPT_DEBUG: ${CERC_SCRIPT_DEBUG} CERC_RELAY_NODES: ${CERC_RELAY_NODES} + CERC_DENY_MULTIADDRS: ${CERC_DENY_MULTIADDRS} command: ["sh", "test-app-start.sh"] volumes: - ../config/wait-for-it.sh:/scripts/wait-for-it.sh diff --git a/app/data/compose/docker-compose-watcher-mobymask-v2.yml b/app/data/compose/docker-compose-watcher-mobymask-v2.yml index 36c5f3f9..0c743670 100644 --- a/app/data/compose/docker-compose-watcher-mobymask-v2.yml +++ b/app/data/compose/docker-compose-watcher-mobymask-v2.yml @@ -83,6 +83,7 @@ services: CERC_L1_ACCOUNTS_CSV_URL: ${CERC_L1_ACCOUNTS_CSV_URL} CERC_PRIVATE_KEY_PEER: ${CERC_PRIVATE_KEY_PEER} CERC_RELAY_PEERS: ${CERC_RELAY_PEERS} + CERC_DENY_MULTIADDRS: ${CERC_DENY_MULTIADDRS} CERC_RELAY_ANNOUNCE_DOMAIN: ${CERC_RELAY_ANNOUNCE_DOMAIN} CERC_ENABLE_PEER_L2_TXS: ${CERC_ENABLE_PEER_L2_TXS} CERC_DEPLOYED_CONTRACT: ${CERC_DEPLOYED_CONTRACT} diff --git a/app/data/config/watcher-mobymask-v2/mobymask-app-start.sh b/app/data/config/watcher-mobymask-v2/mobymask-app-start.sh index 9f343340..6eeb5c1a 100755 --- a/app/data/config/watcher-mobymask-v2/mobymask-app-start.sh +++ b/app/data/config/watcher-mobymask-v2/mobymask-app-start.sh @@ -7,6 +7,7 @@ fi CERC_CHAIN_ID="${CERC_CHAIN_ID:-${DEFAULT_CERC_CHAIN_ID}}" CERC_DEPLOYED_CONTRACT="${CERC_DEPLOYED_CONTRACT:-${DEFAULT_CERC_DEPLOYED_CONTRACT}}" CERC_RELAY_NODES="${CERC_RELAY_NODES:-${DEFAULT_CERC_RELAY_NODES}}" +CERC_DENY_MULTIADDRS="${CERC_DENY_MULTIADDRS:-${DEFAULT_CERC_DENY_MULTIADDRS}}" CERC_APP_WATCHER_URL="${CERC_APP_WATCHER_URL:-${DEFAULT_CERC_APP_WATCHER_URL}}" # If not set (or []), check the mounted volume for relay peer id @@ -37,5 +38,6 @@ yq -n ".address = env(CERC_DEPLOYED_CONTRACT)" > /config/config.yml yq ".watcherUrl = env(CERC_APP_WATCHER_URL)" -i /config/config.yml yq ".chainId = env(CERC_CHAIN_ID)" -i /config/config.yml yq ".relayNodes = strenv(CERC_RELAY_NODES)" -i /config/config.yml +yq ".denyMultiaddrs = strenv(CERC_DENY_MULTIADDRS)" -i /config/config.yml /scripts/start-serving-app.sh diff --git a/app/data/config/watcher-mobymask-v2/mobymask-params.env b/app/data/config/watcher-mobymask-v2/mobymask-params.env index 6d1bf063..39d55016 100644 --- a/app/data/config/watcher-mobymask-v2/mobymask-params.env +++ b/app/data/config/watcher-mobymask-v2/mobymask-params.env @@ -24,3 +24,6 @@ DEFAULT_CERC_CHAIN_ID=42069 # Set of relay nodes to be used by web-apps DEFAULT_CERC_RELAY_NODES=[] + +# Set of multiaddrs to be avoided while dialling +DEFAULT_CERC_DENY_MULTIADDRS=[] diff --git a/app/data/config/watcher-mobymask-v2/start-server.sh b/app/data/config/watcher-mobymask-v2/start-server.sh index b46a7c14..eab8bac2 100755 --- a/app/data/config/watcher-mobymask-v2/start-server.sh +++ b/app/data/config/watcher-mobymask-v2/start-server.sh @@ -8,13 +8,20 @@ CERC_L2_GETH_RPC="${CERC_L2_GETH_RPC:-${DEFAULT_CERC_L2_GETH_RPC}}" CERC_L1_ACCOUNTS_CSV_URL="${CERC_L1_ACCOUNTS_CSV_URL:-${DEFAULT_CERC_L1_ACCOUNTS_CSV_URL}}" CERC_RELAY_PEERS="${CERC_RELAY_PEERS:-${DEFAULT_CERC_RELAY_PEERS}}" +CERC_DENY_MULTIADDRS="${CERC_DENY_MULTIADDRS:-${DEFAULT_CERC_DENY_MULTIADDRS}}" CERC_RELAY_ANNOUNCE_DOMAIN="${CERC_RELAY_ANNOUNCE_DOMAIN:-${DEFAULT_CERC_RELAY_ANNOUNCE_DOMAIN}}" CERC_ENABLE_PEER_L2_TXS="${CERC_ENABLE_PEER_L2_TXS:-${DEFAULT_CERC_ENABLE_PEER_L2_TXS}}" CERC_DEPLOYED_CONTRACT="${CERC_DEPLOYED_CONTRACT:-${DEFAULT_CERC_DEPLOYED_CONTRACT}}" echo "Using L2 RPC endpoint ${CERC_L2_GETH_RPC}" -CERC_RELAY_MULTIADDR="/dns4/mobymask-watcher-server/tcp/9090/ws/p2p/$(jq -r '.id' /app/peers/relay-id.json)" +# Use public domain for relay multiaddr in peer config if specified +# Otherwise, use the docker container's host IP +if [ -n "$CERC_RELAY_ANNOUNCE_DOMAIN" ]; then + CERC_RELAY_MULTIADDR="/dns4/${CERC_RELAY_ANNOUNCE_DOMAIN}/tcp/443/wss/p2p/$(jq -r '.id' /app/peers/relay-id.json)" +else + CERC_RELAY_MULTIADDR="/dns4/mobymask-watcher-server/tcp/9090/ws/p2p/$(jq -r '.id' /app/peers/relay-id.json)" +fi # Use contract address from environment variable or set from config.json in mounted volume if [ -n "$CERC_DEPLOYED_CONTRACT" ]; then @@ -42,6 +49,7 @@ fi WATCHER_CONFIG_TEMPLATE=$(cat environments/watcher-config-template.toml) WATCHER_CONFIG=$(echo "$WATCHER_CONFIG_TEMPLATE" | \ sed -E "s|REPLACE_WITH_CERC_RELAY_PEERS|${CERC_RELAY_PEERS}|g; \ + s|REPLACE_WITH_CERC_DENY_MULTIADDRS|${CERC_DENY_MULTIADDRS}|g; \ s/REPLACE_WITH_CERC_RELAY_ANNOUNCE_DOMAIN/${CERC_RELAY_ANNOUNCE_DOMAIN}/g; \ s|REPLACE_WITH_CERC_RELAY_MULTIADDR|${CERC_RELAY_MULTIADDR}|g; \ s/REPLACE_WITH_CERC_ENABLE_PEER_L2_TXS/${CERC_ENABLE_PEER_L2_TXS}/g; \ diff --git a/app/data/config/watcher-mobymask-v2/test-app-config.json b/app/data/config/watcher-mobymask-v2/test-app-config.json index cce15fd8..a0baf9c6 100644 --- a/app/data/config/watcher-mobymask-v2/test-app-config.json +++ b/app/data/config/watcher-mobymask-v2/test-app-config.json @@ -1,6 +1,7 @@ { "relayNodes": [], "peer": { + "denyMultiaddrs": [], "enableDebugInfo": true } } diff --git a/app/data/config/watcher-mobymask-v2/test-app-start.sh b/app/data/config/watcher-mobymask-v2/test-app-start.sh index 42e4397d..aeeb02d3 100755 --- a/app/data/config/watcher-mobymask-v2/test-app-start.sh +++ b/app/data/config/watcher-mobymask-v2/test-app-start.sh @@ -5,6 +5,7 @@ if [ -n "$CERC_SCRIPT_DEBUG" ]; then fi CERC_RELAY_NODES="${CERC_RELAY_NODES:-${DEFAULT_CERC_RELAY_NODES}}" +CERC_DENY_MULTIADDRS="${CERC_DENY_MULTIADDRS:-${DEFAULT_CERC_DENY_MULTIADDRS}}" # If not set (or []), check the mounted volume for relay peer id if [ -z "$CERC_RELAY_NODES" ] || [ "$CERC_RELAY_NODES" = "[]" ]; then @@ -16,5 +17,6 @@ echo "Using CERC_RELAY_NODES $CERC_RELAY_NODES" # Use yq to create config.yml with environment variables yq -n ".relayNodes = strenv(CERC_RELAY_NODES)" > /config/config.yml +yq ".denyMultiaddrs = strenv(CERC_DENY_MULTIADDRS)" -i /config/config.yml /scripts/start-serving-app.sh diff --git a/app/data/config/watcher-mobymask-v2/watcher-config-template.toml b/app/data/config/watcher-mobymask-v2/watcher-config-template.toml index e6ce0750..5a2c7ce4 100644 --- a/app/data/config/watcher-mobymask-v2/watcher-config-template.toml +++ b/app/data/config/watcher-mobymask-v2/watcher-config-template.toml @@ -27,6 +27,7 @@ host = "0.0.0.0" port = 9090 relayPeers = REPLACE_WITH_CERC_RELAY_PEERS + denyMultiaddrs = REPLACE_WITH_CERC_DENY_MULTIADDRS peerIdFile = './peers/relay-id.json' announce = 'REPLACE_WITH_CERC_RELAY_ANNOUNCE_DOMAIN' enableDebugInfo = true @@ -34,6 +35,7 @@ [server.p2p.peer] relayMultiaddr = 'REPLACE_WITH_CERC_RELAY_MULTIADDR' pubSubTopic = 'mobymask' + denyMultiaddrs = REPLACE_WITH_CERC_DENY_MULTIADDRS peerIdFile = './peers/peer-id.json' enableDebugInfo = true enableL2Txs = REPLACE_WITH_CERC_ENABLE_PEER_L2_TXS diff --git a/app/data/container-build/cerc-mobymask-ui/Dockerfile b/app/data/container-build/cerc-mobymask-ui/Dockerfile index 651d718f..9d4868d2 100644 --- a/app/data/container-build/cerc-mobymask-ui/Dockerfile +++ b/app/data/container-build/cerc-mobymask-ui/Dockerfile @@ -50,9 +50,9 @@ RUN yarn global add http-server # Globally install both versions of the payload web app package # Install old version of MobyMask web app -RUN yarn global add @cerc-io/mobymask-ui@0.1.3 +RUN yarn global add @cerc-io/mobymask-ui@0.1.4 # Install the LXDAO version of MobyMask web app -RUN yarn global add @cerc-io/mobymask-ui-lxdao@npm:@cerc-io/mobymask-ui@0.1.3-lxdao-0.1.1 +RUN yarn global add @cerc-io/mobymask-ui-lxdao@npm:@cerc-io/mobymask-ui@0.1.4-lxdao-0.1.1 # Expose port for http EXPOSE 80 diff --git a/app/data/container-build/cerc-mobymask-ui/apply-webapp-config.sh b/app/data/container-build/cerc-mobymask-ui/apply-webapp-config.sh index 9f32cd23..2779b3a1 100755 --- a/app/data/container-build/cerc-mobymask-ui/apply-webapp-config.sh +++ b/app/data/container-build/cerc-mobymask-ui/apply-webapp-config.sh @@ -33,7 +33,7 @@ do echo "Substituting: ${template_string_to_replace} = ${template_value_to_substitute}" # TODO: Pass keys to be replaced without double quotes - if [[ "$template_string_to_replace" =~ ^${config_prefix}_(relayNodes|chainId)$ ]]; then + if [[ "$template_string_to_replace" =~ ^${config_prefix}_(relayNodes|chainId|denyMultiaddrs)$ ]]; then find ${webapp_files_dir} -type f -exec sed -i 's#"'"${template_string_to_replace}"'"#'"${template_value_to_substitute}"'#g' {} + else # Note: we do not escape our strings, on the expectation they do not container the '#' char. diff --git a/app/data/container-build/cerc-react-peer/Dockerfile b/app/data/container-build/cerc-react-peer/Dockerfile index 191f986b..50670e03 100644 --- a/app/data/container-build/cerc-react-peer/Dockerfile +++ b/app/data/container-build/cerc-react-peer/Dockerfile @@ -21,7 +21,7 @@ RUN mkdir -p /config RUN yarn global add http-server # Globally install the payload web app package -RUN yarn global add @cerc-io/test-app@0.2.33 +RUN yarn global add @cerc-io/test-app@0.2.34 # Expose port for http EXPOSE 80 diff --git a/app/data/container-build/cerc-react-peer/apply-webapp-config.sh b/app/data/container-build/cerc-react-peer/apply-webapp-config.sh index a7f0a28e..59dee869 100755 --- a/app/data/container-build/cerc-react-peer/apply-webapp-config.sh +++ b/app/data/container-build/cerc-react-peer/apply-webapp-config.sh @@ -33,7 +33,7 @@ do echo "Substituting: ${template_string_to_replace} = ${template_value_to_substitute}" # TODO: Pass keys to be replaced without double quotes - if [[ "$template_string_to_replace" == "${config_prefix}_relayNodes" ]]; then + if [[ "$template_string_to_replace" =~ ^${config_prefix}_(relayNodes|denyMultiaddrs)$ ]]; then find ${webapp_files_dir} -type f -exec sed -i 's#"'"${template_string_to_replace}"'"#'"${template_value_to_substitute}"'#g' {} + else # Note: we do not escape our strings, on the expectation they do not container the '#' char. diff --git a/app/data/stacks/mobymask-v2/README.md b/app/data/stacks/mobymask-v2/README.md index 32fdbf60..880b6a35 100644 --- a/app/data/stacks/mobymask-v2/README.md +++ b/app/data/stacks/mobymask-v2/README.md @@ -23,11 +23,11 @@ Checkout to the required versions and branches in repos ```bash # watcher-ts cd ~/cerc/watcher-ts -git checkout v0.2.39 +git checkout v0.2.41 # mobymask-v2-watcher-ts cd ~/cerc/mobymask-v2-watcher-ts -git checkout v0.1.0 +git checkout v0.1.1 # MobyMask cd ~/cerc/MobyMask diff --git a/app/data/stacks/mobymask-v2/mobymask-only.md b/app/data/stacks/mobymask-v2/mobymask-only.md index 4cfb4ecf..01ce0754 100644 --- a/app/data/stacks/mobymask-v2/mobymask-only.md +++ b/app/data/stacks/mobymask-v2/mobymask-only.md @@ -19,11 +19,11 @@ Checkout to the required versions and branches in repos: ```bash # watcher-ts cd ~/cerc/watcher-ts -git checkout v0.2.39 +git checkout v0.2.41 # mobymask-v2-watcher-ts cd ~/cerc/mobymask-v2-watcher-ts -git checkout v0.1.0 +git checkout v0.1.1 # MobyMask cd ~/cerc/MobyMask @@ -67,11 +67,14 @@ Create and update an env file to be used in the next step ([defaults](../../conf # (used for generating a root invite link after deploying the contract) CERC_MOBYMASK_APP_BASE_URI="http://127.0.0.1:3002/#" + # (Optional) Domain to be used in the relay node's announce address + CERC_RELAY_ANNOUNCE_DOMAIN= + # (Optional) Set of relay peers to connect to from the relay node CERC_RELAY_PEERS=[] - # (Optional) Domain to be used in the relay node's announce address - CERC_RELAY_ANNOUNCE_DOMAIN= + # (Optional) Set of multiaddrs to be avoided while dialling + CERC_DENY_MULTIADDRS=[] # Set to false for disabling watcher peer to send txs to L2 CERC_ENABLE_PEER_L2_TXS=true diff --git a/app/data/stacks/mobymask-v2/watcher-p2p-network/watcher.md b/app/data/stacks/mobymask-v2/watcher-p2p-network/watcher.md index 74d95b92..4641726e 100644 --- a/app/data/stacks/mobymask-v2/watcher-p2p-network/watcher.md +++ b/app/data/stacks/mobymask-v2/watcher-p2p-network/watcher.md @@ -35,11 +35,11 @@ Checkout to the required versions and branches in repos: ```bash # watcher-ts cd ~/cerc/watcher-ts - git checkout v0.2.39 + git checkout v0.2.41 # mobymask-v2-watcher-ts cd ~/cerc/mobymask-v2-watcher-ts - git checkout v0.1.0 + git checkout v0.1.1 # MobyMask cd ~/cerc/MobyMask diff --git a/app/data/stacks/mobymask-v2/web-apps.md b/app/data/stacks/mobymask-v2/web-apps.md index 2eb037f6..d1570c93 100644 --- a/app/data/stacks/mobymask-v2/web-apps.md +++ b/app/data/stacks/mobymask-v2/web-apps.md @@ -26,6 +26,9 @@ Create and update an env file to be used in the next step ([defaults](../../conf # Eg. CERC_RELAY_NODES=["/dns4/example.com/tcp/443/wss/p2p/12D3KooWGHmDDCc93XUWL16FMcTPCGu2zFaMkf67k8HZ4gdQbRDr"] CERC_RELAY_NODES=[] + # Set of multiaddrs to be avoided while dialling + CERC_DENY_MULTIADDRS=[] + # Also add if running MobyMask app: # Watcher endpoint used by the app for GQL queries