Gitea deployment (#568)

* First part of deployments for external repos

* Generate deployment dir

* Create empty config file

* Copy script files into deployment

* Run scripts in deployment

* Refactor

* Integrate external plugins

* Remove debug output

app/data/compose/docker-compose-fixturenet-optimism.yml

@@ -70,6 +70,7 @@ services:
     command: "/run-op-geth.sh"
     ports:
       - "0.0.0.0:8545:8545"
+      - "0.0.0.0:8546:8546"
     healthcheck:
       test: ["CMD", "nc", "-vz", "localhost:8545"]
       interval: 30s

app/data/compose/docker-compose-go-nitro.yml

@@ -44,7 +44,6 @@ services:
           condition: service_healthy
       command: ["bash", "-c", "tail -f /dev/null"]
 
-
 volumes:
   go_nitro_data:
   nitro_deployment:

app/data/config/go-nitro/run-nitro-node.sh

@@ -1,6 +1,5 @@
 #!/bin/bash
 
-set -e
 if [ -n "$CERC_SCRIPT_DEBUG" ]; then
   set -x
 fi
@@ -30,6 +29,21 @@ fi
 
 echo "Running Nitro node"
 
-# TODO Wait for chain endpoint to come up
+# Wait till chain endpoint is available
+retry_interval=5
+while true; do
+  # Assuming CERC_NITRO_CHAIN_URL is of format <ws|http>://host:port
+  ws_host=$(echo "$CERC_NITRO_CHAIN_URL" | awk -F '://' '{print $2}' | cut -d ':' -f 1)
+  ws_port=$(echo "$CERC_NITRO_CHAIN_URL" | awk -F '://' '{print $2}' | cut -d ':' -f 2)
+  nc -z -w 1 "$ws_host" "$ws_port"
+
+  if [ $? -eq 0 ]; then
+    echo "Chain endpoint is available"
+    break
+  fi
+
+  echo "Chain endpoint not yet available, retrying in $retry_interval seconds..."
+  sleep $retry_interval
+done
 
 ./nitro -chainurl ${CERC_NITRO_CHAIN_URL} -msgport 3005 -rpcport 4005 -wsmsgport 5005 -pk ${CERC_NITRO_PK} -chainpk ${CERC_NITRO_CHAIN_PK} -naaddress ${NA_ADDRESS} -vpaaddress ${VPA_ADDRESS} -caaddress ${CA_ADDRESS} -usedurablestore ${CERC_NITRO_USE_DURABLE_STORE} -durablestorefolder ${CERC_NITRO_DURABLE_STORE_FOLDER}

app/data/container-image-list.txt

@@ -14,6 +14,7 @@ cerc/laconic-registry-cli
 cerc/laconic-console-host
 cerc/fixturenet-eth-geth
 cerc/fixturenet-eth-lighthouse
+cerc/fixturenet-eth-genesis
 cerc/watcher-ts
 cerc/watcher-mobymask
 cerc/watcher-erc20
@@ -55,3 +56,4 @@ cerc/go-nitro
 cerc/nitro-contracts
 cerc/mobymask-snap
 cerc/ponder
+cerc/nitro-rpc-client

app/data/stacks/fixturenet-payments/README.md

@@ -1,5 +1,7 @@
 # fixturenet-payments
 
+Instructions to setup and deploy an end-to-end fixturenet-payments stack
+
 ## Setup
 
 Clone required repositories:

app/data/stacks/fixturenet-payments/demo.md

@@ -1,14 +1,25 @@
 # Demo
 
-## MobyMask Watcher
+Stack components:
+* `ipld-eth-db` database for statediffed data
+* Local geth + lighthouse blockchain "fixturenet" running in statediffing mode
+* `ipld-eth-server` which runs an ETH RPC API and a GQL server; serves data from `ipld-eth-db`
+* A go-nitro deployment acting as the Nitro node for `ipld-eth-server`
+* A modified reverse payment proxy server (based on the one from go-nitro) that proxies requests to `ipld-eth-server`'s RPC endpoint; it talks to `ipld-eth-server`'s Nitro node to accept and validate payments required for configured RPC requests
+* A MobyMask v3 watcher that pays the `ipld-eth-server` for ETH RPC requests
+* A MobyMask v3 app that pays the watcher for reads (GQL queries) and writes
+* An example ERC20 Ponder app that pays the `ipld-eth-server` for ETH RPC requests
 
-* Check watcher logs and wait for the payment channel to be created with upstream go-nitro node:
+## Setup
+
+* On starting the stack, MobyMask watcher creates a payment channel with the `ipld-eth-server`'s Nitro node. Check watcher logs and wait for the same:
 
   ```bash
   docker logs -f $(docker ps -aq --filter name="mobymask-watcher-server")
 
   # Expected output:
-  # vulcanize:server Using rpcProviderEndpoint as chain URL for Nitro node +0ms
+  # vulcanize:server Peer ID: 12D3KooWKLqLWU82VU7jmsmQMruRvZWhoBoVsf1UHchM5Nuq9ymY
+  # vulcanize:server Using chain URL http://fixturenet-eth-geth-1:8546 for Nitro node
   # ...
   # ts-nitro:util:nitro Ledger channel created with id 0x65703ccdfacab09ac35367bdbe6c5a337e7a6651aad526807607b1c59b28bc1e
   # ...
@@ -16,7 +27,9 @@
   # vulcanize:server Starting server... +0ms
   ```
 
-* Export the payment channel id to a variable:
+* Keep the above command running to keep track of incoming payments and GQL requests from the MobyMask app
+
+* In another terminal, export the payment channel id to a variable:
 
   ```bash
   export WATCHER_UPSTREAM_PAYMENT_CHANNEL=<PAYMENT_CHANNEL_ID>
@@ -41,43 +54,46 @@
   # }
   ```
 
-## MobyMask App
-
-* Track the reverse payment proxy container logs in a terminal:
+* In another terminal, check the reverse payment proxy server's logs to keep track of incoming payments and RPC requests:
 
   ```bash
   docker logs -f $(docker ps -aq --filter name="nitro-reverse-payment-proxy")
   ```
 
-* Get the geth node’s port mapped to host:
+* MetaMask flask wallet setup for running the MobyMask app:
 
-  ```bash
-  docker port payments-fixturenet-eth-geth-1-1 8545
-  ```
+  * Get the geth node’s port mapped to host:
 
-* In MetaMask Flask, add a custom network with the following settings:
+    ```bash
+    docker port payments-fixturenet-eth-geth-1-1 8545
+    ```
 
-  ```bash
-  # New RPC URL
-  http://127.0.0.1:<GETH_PORT>
+  * In MetaMask, add a custom network with the following settings:
 
-  # Chain ID
-  1212
+    ```bash
+    # Network name
+    Local fixturenet
 
-  # Currency symbol
-  ETH
-  ```
+    # New RPC URL
+    http://127.0.0.1:<GETH_PORT>
 
-* Import the faucet account in MetaMask and fund an additional account for usage in the app:
+    # Chain ID
+    1212
 
-  ```bash
-  # Faucet PK
-  # 0x570b909da9669b2f35a0b1ac70b8358516d55ae1b5b3710e95e9a94395090597
+    # Currency symbol
+    ETH
+    ```
 
-  # Clear activity tab for the accounts on chain restart
-  ```
+  * Import a faucet account with the following private key:
 
-* Get the generated root invite link for the app from MobyMask contract deployment container logs:
+    ```bash
+    # Faucet PK
+    # 0x570b909da9669b2f35a0b1ac70b8358516d55ae1b5b3710e95e9a94395090597
+    ```
+
+  * Create an additional account for usage in the app; fund it from the faucet account
+
+* Get the generated root invite link for MobyMask from contract deployment container logs:
 
   ```bash
   docker logs -f $(docker ps -aq --filter name="mobymask-1")
@@ -89,9 +105,13 @@
   # http://127.0.0.1:3004/#/members?invitation=<INVITATION>
   ```
 
-* Open app in a browser using the invite link
+## Run
 
-* Run the following in the browser console to enable logs:
+### MobyMask App
+
+* Open app in a browser (where MetaMask was setup) using the invite link
+
+* Run the following in browser console to enable logs:
 
   ```bash
   localStorage.debug = 'ts-nitro:*'
@@ -104,19 +124,66 @@
   * Click on `Connect Wallet` to connect to MetaMask (make sure that the newly funded account is active)
   * Click on `Connect Snap` to install/connect snap
 
-* Perform `DIRECT FUND` with the preset amount
+* Perform `DIRECT FUND` with the preset amount and wait for the MetaMask confirmation prompt to appear; confirm the transaction and wait for a ledger channel to be created with the watcher
 
-* Perform `VIRTUAL FUND` with amount set to `10000`
+* Perform `VIRTUAL FUND` with amount set to `10000` and wait for a payment channel to be created with the watcher
 
 * Perform phisher status check queries now that a payment channel is created:
-  * Check the watcher logs for received payments
-  * Check the payment proxy server logs for charged RPC requests (`eth_getBlockByHash`, `eth_getBlockByNumber`, `eth_getStorageAt`) made from watcher to upstream ETH server
 
-* Change the amount besides `PAY` button to `>=100` for phisher reports next
+  * Check the watcher logs for received payments along with the GQL queries:
 
-* Perform a phisher report and check the watcher logs for received payments; the RPC mutation request is sent to geth node and is not charged
+    ```bash
+    # Expected output:
+    # ...
+    # laconic:payments Serving a paid query for 0x86804299822212c070178B5135Ba6DdAcFC357D3
+    # vulcanize:resolver isPhisher 0x98ae4f9e9d01cc892adfe6871e1db0287039e0c183d3b5bb31d724228c114744 0x2B6AFbd4F479cE4101Df722cF4E05F941523EaD9 TWT:ash1
+    # vulcanize:indexer isPhisher: db miss, fetching from upstream server
+    # laconic:payments Making RPC call: eth_chainId
+    # laconic:payments Making RPC call: eth_getBlockByHash
+    # laconic:payments Making RPC call: eth_chainId
+    # laconic:payments Making RPC call: eth_getStorageAt
+    ```
 
-* Check the watcher - eth-server payment channel status after a few requests:
+  * The watcher makes several ETH RPC requests to `ipld-eth-server` to fetch data required for satisfying the GQL request(s); check the payment proxy server logs for charged RPC requests (`eth_getBlockByHash`, `eth_getBlockByNumber`, `eth_getStorageAt`):
+
+    ```bash
+    # Expected output:
+    # ...
+    # {"time":"2023-10-06T06:46:52.769009314Z","level":"DEBUG","msg":"Serving RPC request","method":"eth_chainId"}
+    # {"time":"2023-10-06T06:46:52.773006426Z","level":"DEBUG","msg":"Serving RPC request","method":"eth_getBlockByNumber"}
+    # {"time":"2023-10-06T06:46:52.811142054Z","level":"DEBUG","msg":"Request cost","cost-per-byte":1,"response-length":1480,"cost":1480,"method":"eth_getBlockByNumber"}
+    # {"time":"2023-10-06T06:46:52.811418494Z","level":"DEBUG","msg":"sent message","address":"0xAAA6628Ec44A8a742987EF3A114dDFE2D4F7aDCE","method":"receive_voucher"}
+    # {"time":"2023-10-06T06:46:52.812557482Z","level":"DEBUG","msg":"Received voucher","delta":5000}
+    # ...
+    # {"time":"2023-10-06T06:46:52.87525215Z","level":"DEBUG","msg":"Serving RPC request","method":"eth_getStorageAt"}
+    # {"time":"2023-10-06T06:46:52.882859654Z","level":"DEBUG","msg":"Request cost","cost-per-byte":1,"response-length":104,"cost":104,"method":"eth_getStorageAt"}
+    # {"time":"2023-10-06T06:46:52.882946485Z","level":"DEBUG","msg":"sent message","address":"0xAAA6628Ec44A8a742987EF3A114dDFE2D4F7aDCE","method":"receive_voucher"}
+    # {"time":"2023-10-06T06:46:52.884012641Z","level":"DEBUG","msg":"Received voucher","delta":5000}
+    # {"time":"2023-10-06T06:46:52.884032961Z","level":"DEBUG","msg":"Destination request","url":"http://ipld-eth-server:8081/"}
+    ```
+
+* Change the amount besides `PAY` button in debug panel to `>=100` for phisher reports next
+
+* Perform a phisher report and check the watcher logs for received payments:
+
+  ```bash
+  # Expected output:
+  # ...
+  # vulcanize:libp2p-utils [6:50:2] Received a message on mobymask P2P network from peer: 12D3KooWRkxV9SX8uTUZYkbRjai4Fsn7yavB61J5TMnksixsabsP
+  # ts-nitro:engine {"msg":"Received message","_msg":{"to":"0xBBB676","from":"0x868042","payloadSummaries":[],"proposalSummaries":[],"payments":[{"amount":200,"channelId":"0x557153d729cf3323c0bdb40a36b245f98c2d4562933ba2182c9d61c5cfeda948"}],"rejectedObjectives":[]}}
+  # laconic:payments Received a payment voucher of 100 from 0x86804299822212c070178B5135Ba6DdAcFC357D3
+  # vulcanize:libp2p-utils Payment received for a mutation request from 0x86804299822212c070178B5135Ba6DdAcFC357D3
+  # vulcanize:libp2p-utils Transaction receipt for invoke message {
+  #   to: '0x2B6AFbd4F479cE4101Df722cF4E05F941523EaD9',
+  #   blockNumber: 232,
+  #   blockHash: '0x6a188722c102662ea48af3786fe9db0d4b6c7ab7b27473eb0e628cf95746a244',
+  #   transactionHash: '0x6521205db8a905b3222adc2b6855f9b2abc72580624d299bec2a35bcba173efa',
+  #   effectiveGasPrice: '1500000007',
+  #   gasUsed: '113355'
+  # }
+  ```
+
+* Check the watcher - ipld-eth-server payment channel status after a few requests:
 
   ```bash
   docker exec payments-nitro-rpc-client-1 npm exec -c "nitro-rpc-client get-payment-channel $WATCHER_UPSTREAM_PAYMENT_CHANNEL -h go-nitro -p 4005"
@@ -135,7 +202,7 @@
   # }
   ```
 
-## Ponder App
+### ERC20 Ponder App
 
 * Run the ponder app in it's container:
 
@@ -150,13 +217,33 @@
   # 09:59:14.329 INFO  payment    Using payment channel 0x10f049519bc3f862e2b26e974be8666886228f30ea54aab06e2f23718afffab0
   ```
 
+* On starting the Ponder app, it creates a payment channel with the `ipld-eth-server`'s Nitro node and then starts the historical sync service
+
+* The sync service makes several ETH RPC requests to the `ipld-eth-server` to fetch required data; check the payment proxy server logs for charged RPC requests (`eth_getBlockByNumber`, `eth_getLogs`)
+
+  ```bash
+  # Expected output:
+  # ...
+  # {"time":"2023-10-06T06:51:45.214478402Z","level":"DEBUG","msg":"Serving RPC request","method":"eth_getBlockByNumber"}
+  # {"time":"2023-10-06T06:51:45.22251171Z","level":"DEBUG","msg":"Request cost","cost-per-byte":1,"response-length":576,"cost":576,"method":"eth_getBlockByNumber"}
+  # {"time":"2023-10-06T06:51:45.222641963Z","level":"DEBUG","msg":"sent message","address":"0xAAA6628Ec44A8a742987EF3A114dDFE2D4F7aDCE","method":"receive_voucher"}
+  # {"time":"2023-10-06T06:51:45.224042391Z","level":"DEBUG","msg":"Received voucher","delta":5000}
+  # {"time":"2023-10-06T06:51:45.224061411Z","level":"DEBUG","msg":"Destination request","url":"http://ipld-eth-server:8081/"}
+  # {"time":"2023-10-06T06:51:45.242064953Z","level":"DEBUG","msg":"Serving RPC request","method":"eth_getLogs"}
+  # {"time":"2023-10-06T06:51:45.249118517Z","level":"DEBUG","msg":"Request cost","cost-per-byte":1,"response-length":61,"cost":61,"method":"eth_getLogs"}
+  # {"time":"2023-10-06T06:51:45.249189892Z","level":"DEBUG","msg":"sent message","address":"0xAAA6628Ec44A8a742987EF3A114dDFE2D4F7aDCE","method":"receive_voucher"}
+  # {"time":"2023-10-06T06:51:45.249743149Z","level":"DEBUG","msg":"Received voucher","delta":5000}
+  # {"time":"2023-10-06T06:51:45.249760631Z","level":"DEBUG","msg":"Destination request","url":"http://ipld-eth-server:8081/"}
+  # ...
+  ```
+
 * Export the payment channel id to a variable:
 
   ```bash
   export PONDER_UPSTREAM_PAYMENT_CHANNEL=<PAYMENT_CHANNEL_ID>
   ```
 
-* Check the ponder - eth-server payment channel status:
+* Check the ponder - ipld-eth-server payment channel status:
 
   ```bash
   docker exec payments-nitro-rpc-client-1 npm exec -c "nitro-rpc-client get-payment-channel $PONDER_UPSTREAM_PAYMENT_CHANNEL -h go-nitro -p 4005"
@@ -175,23 +262,14 @@
   # }
   ```
 
-* Check reverse payment proxy server logs for charged RPC requests made from ponder app to upstream ETH server:
-
-  ```bash
-  # Expected output:
-  # ...
-  # {"time":"2023-09-28T09:59:14.499841999Z","level":"DEBUG","msg":"Request cost","cost-per-byte":1,"response-length":61,"cost":61}
-  # {"time":"2023-09-28T09:59:14.500060006Z","level":"DEBUG","msg":"sent message","address":"0xAAA6628Ec44A8a742987EF3A114dDFE2D4F7aDCE","method":"receive_voucher"}
-  # {"time":"2023-09-28T09:59:14.501221898Z","level":"DEBUG","msg":"Received voucher","delta":5000}
-  # {"time":"2023-09-28T09:59:14.501245984Z","level":"DEBUG","msg":"Destination request","url":"http://ipld-eth-server:8081/?method=eth_getLogs"}
-  ```
-
 ## Clean Up
 
-* In the MobyMask app, perform `VIRTUAL DEFUND` and `DIRECT DEFUND` (in this order) for closing the payment channel created with watcher
+* In the MobyMask app, perform `VIRTUAL DEFUND` and `DIRECT DEFUND` (in order) for closing the payment channel created with watcher
 
 * Run the following in the browser console to delete the Nitro node's data:
 
   ```bash
   await clearNodeStorage()
   ```
+
+* On a fresh restart, clear activity tab data in MetaMask for concerned accounts

app/data/stacks/fixturenet-payments/stack.yml

@@ -9,14 +9,14 @@ repos:
   - git.vdb.to/cerc-io/ipld-eth-server@v1.11.6-statediff-v5
   # nitro repos
   - github.com/cerc-io/ts-nitro@v0.1.13
-  - github.com/cerc-io/go-nitro@ts-interop # TODO: Update after fixes
+  - github.com/cerc-io/go-nitro@v0.1.1-ts-port-0.1.5
   # mobymask watcher repos
   - github.com/cerc-io/watcher-ts@v0.2.63
   - github.com/cerc-io/mobymask-v2-watcher-ts@v0.2.2
   - github.com/cerc-io/MobyMask@v0.1.3
   # mobymask app repos
   - github.com/cerc-io/mobymask-snap
-  - github.com/cerc-io/mobymask-ui@v0.2.0
+  - github.com/cerc-io/mobymask-ui@v0.2.1
   # ponder repo
   - github.com/cerc-io/ponder@laconic
 containers:

app/data/stacks/mobymask-v3/stack.yml

@@ -2,11 +2,11 @@ version: "1.0"
 description: "MobyMask v3 stack"
 name: mobymask-v3
 repos:
-  - github.com/cerc-io/ts-nitrov0.1.12
-  - github.com/cerc-io/watcher-ts@v0.2.57
-  - github.com/cerc-io/mobymask-v2-watcher-ts@v3 # TODO: Update after fixes
+  - github.com/cerc-io/ts-nitro@v0.1.13
+  - github.com/cerc-io/watcher-ts@v0.2.63
+  - github.com/cerc-io/mobymask-v2-watcher-ts@v0.2.2
   - github.com/cerc-io/MobyMask@v0.1.3
-  - github.com/cerc-io/mobymask-ui@v0.2.0
+  - github.com/cerc-io/mobymask-ui@v0.2.1
 containers:
   - cerc/nitro-contracts
   - cerc/watcher-ts

app/data/stacks/mobymask-v3/watcher.md

@@ -23,7 +23,7 @@ laconic-so --stack mobymask-v3 build-containers --exclude cerc/mobymask-ui
 Create and update an env file to be used in the next step ([defaults](../../config/watcher-mobymask-v3/mobymask-params.env)):
 
   ```bash
-  # External ETH RPC endpoint (L2 Optimism geth)
+  # External ETH RPC endpoint for contract(s) deployment
   CERC_ETH_RPC_ENDPOINT=
 
   # External ETH RPC endpoint used for queries in the watcher
@@ -32,6 +32,9 @@ Create and update an env file to be used in the next step ([defaults](../../conf
   # External ETH RPC endpoint used for mutations in the watcher
   CERC_ETH_RPC_MUTATION_ENDPOINT=
 
+  # External ETH endpoint used by watcher's Nitro node
+  CERC_NITRO_CHAIN_URL=
+
   # Specify the an account PK for contract deployment
   CERC_PRIVATE_KEY_DEPLOYER=
 

app/data/stacks/package-registry/stack.yml

@@ -2,7 +2,7 @@ version: "1.1"
 name: package-registry
 description: "Local Package Registry"
 repos:
-  - github.com/cerc-io/hosting
+  - git.vdb.to/cerc-io/hosting
   - gitea.com/gitea/act_runner
 containers:
   - cerc/act-runner

app/deploy.py

@@ -20,13 +20,12 @@ import copy
 import os
 import sys
 from dataclasses import dataclass
-from decouple import config
 from importlib import resources
 import subprocess
 from python_on_whales import DockerClient, DockerException
 import click
 from pathlib import Path
-from app.util import include_exclude_check, get_parsed_stack_config, global_options2
+from app.util import include_exclude_check, get_parsed_stack_config, global_options2, get_dev_root_path
 from app.deploy_types import ClusterContext, DeployCommandContext
 from app.deployment_create import create as deployment_create
 from app.deployment_create import init as deployment_init
@@ -235,17 +234,15 @@ def _make_runtime_env(ctx):
 # stack has to be either PathLike pointing to a stack yml file, or a string with the name of a known stack
 def _make_cluster_context(ctx, stack, include, exclude, cluster, env_file):
 
-    if ctx.local_stack:
-        dev_root_path = os.getcwd()[0:os.getcwd().rindex("stack-orchestrator")]
-        print(f'Local stack dev_root_path (CERC_REPO_BASE_DIR) overridden to: {dev_root_path}')
-    else:
-        dev_root_path = os.path.expanduser(config("CERC_REPO_BASE_DIR", default="~/cerc"))
+    dev_root_path = get_dev_root_path(ctx)
 
     # TODO: huge hack, fix this
     # If the caller passed a path for the stack file, then we know that we can get the compose files
     # from the same directory
+    deployment = False
     if isinstance(stack, os.PathLike):
         compose_dir = stack.parent.joinpath("compose")
+        deployment = True
     else:
         # See: https://stackoverflow.com/questions/25389095/python-get-path-of-root-project-structure
         compose_dir = Path(__file__).absolute().parent.joinpath("data", "compose")
@@ -296,14 +293,24 @@ def _make_cluster_context(ctx, stack, include, exclude, cluster, env_file):
             if pod_repository is None or pod_repository == "internal":
                 compose_file_name = os.path.join(compose_dir, f"docker-compose-{pod_path}.yml")
             else:
-                pod_root_dir = os.path.join(dev_root_path, pod_repository.split("/")[-1], pod["path"])
-                compose_file_name = os.path.join(pod_root_dir, "docker-compose.yml")
-                pod_pre_start_command = pod["pre_start_command"]
-                pod_post_start_command = pod["post_start_command"]
-                if pod_pre_start_command is not None:
-                    pre_start_commands.append(os.path.join(pod_root_dir, pod_pre_start_command))
-                if pod_post_start_command is not None:
-                    post_start_commands.append(os.path.join(pod_root_dir, pod_post_start_command))
+                if deployment:
+                    compose_file_name = os.path.join(compose_dir, "docker-compose.yml")
+                    pod_pre_start_command = pod["pre_start_command"]
+                    pod_post_start_command = pod["post_start_command"]
+                    script_dir = compose_dir.parent.joinpath("pods", pod_name, "scripts")
+                    if pod_pre_start_command is not None:
+                        pre_start_commands.append(os.path.join(script_dir, pod_pre_start_command))
+                    if pod_post_start_command is not None:
+                        post_start_commands.append(os.path.join(script_dir, pod_post_start_command))
+                else:
+                    pod_root_dir = os.path.join(dev_root_path, pod_repository.split("/")[-1], pod["path"])
+                    compose_file_name = os.path.join(pod_root_dir, "docker-compose.yml")
+                    pod_pre_start_command = pod["pre_start_command"]
+                    pod_post_start_command = pod["post_start_command"]
+                    if pod_pre_start_command is not None:
+                        pre_start_commands.append(os.path.join(pod_root_dir, pod_pre_start_command))
+                    if pod_post_start_command is not None:
+                        post_start_commands.append(os.path.join(pod_root_dir, pod_post_start_command))
             compose_files.append(compose_file_name)
         else:
             if ctx.verbose:

app/deploy_util.py

@@ -16,14 +16,14 @@
 import os
 from typing import List
 from app.deploy_types import DeployCommandContext, VolumeMapping
-from app.util import get_parsed_stack_config, get_yaml, get_compose_file_dir
+from app.util import get_parsed_stack_config, get_yaml, get_compose_file_dir, get_pod_list
 
 
 def _container_image_from_service(stack: str, service: str):
     # Parse the compose files looking for the image name of the specified service
     image_name = None
     parsed_stack = get_parsed_stack_config(stack)
-    pods = parsed_stack["pods"]
+    pods = get_pod_list(parsed_stack)
     yaml = get_yaml()
     for pod in pods:
         pod_file_path = os.path.join(get_compose_file_dir(), f"docker-compose-{pod}.yml")

app/deployment_create.py

@@ -17,12 +17,13 @@ import click
 from importlib import util
 import os
 from pathlib import Path
+from typing import List
 import random
-from shutil import copyfile, copytree
+from shutil import copy, copyfile, copytree
 import sys
-from app.util import get_stack_file_path, get_parsed_deployment_spec, get_parsed_stack_config, global_options, get_yaml
-from app.util import get_compose_file_dir
-from app.deploy_types import DeploymentContext, LaconicStackSetupCommand
+from app.util import (get_stack_file_path, get_parsed_deployment_spec, get_parsed_stack_config, global_options, get_yaml,
+                      get_pod_list, get_pod_file_path, pod_has_scripts, get_pod_script_paths, get_plugin_code_path)
+from app.deploy_types import DeploymentContext, DeployCommandContext, LaconicStackSetupCommand
 
 
 def _make_default_deployment_dir():
@@ -32,10 +33,10 @@ def _make_default_deployment_dir():
 def _get_ports(stack):
     ports = {}
     parsed_stack = get_parsed_stack_config(stack)
-    pods = parsed_stack["pods"]
+    pods = get_pod_list(parsed_stack)
     yaml = get_yaml()
     for pod in pods:
-        pod_file_path = os.path.join(get_compose_file_dir(), f"docker-compose-{pod}.yml")
+        pod_file_path = get_pod_file_path(parsed_stack, pod)
         parsed_pod_file = yaml.load(open(pod_file_path, "r"))
         if "services" in parsed_pod_file:
             for svc_name, svc in parsed_pod_file["services"].items():
@@ -49,10 +50,10 @@ def _get_named_volumes(stack):
     # Parse the compose files looking for named volumes
     named_volumes = []
     parsed_stack = get_parsed_stack_config(stack)
-    pods = parsed_stack["pods"]
+    pods = get_pod_list(parsed_stack)
     yaml = get_yaml()
     for pod in pods:
-        pod_file_path = os.path.join(get_compose_file_dir(), f"docker-compose-{pod}.yml")
+        pod_file_path = get_pod_file_path(parsed_stack, pod)
         parsed_pod_file = yaml.load(open(pod_file_path, "r"))
         if "volumes" in parsed_pod_file:
             volumes = parsed_pod_file["volumes"]
@@ -105,11 +106,16 @@ def _fixup_pod_file(pod, spec, compose_dir):
                 pod["services"][container_name]["ports"] = container_ports
 
 
+def _commands_plugin_path(ctx: DeployCommandContext):
+    plugin_path = get_plugin_code_path(ctx.stack)
+    return plugin_path.joinpath("deploy", "commands.py")
+
+
 def call_stack_deploy_init(deploy_command_context):
     # Link with the python file in the stack
     # Call a function in it
     # If no function found, return None
-    python_file_path = get_stack_file_path(deploy_command_context.stack).parent.joinpath("deploy", "commands.py")
+    python_file_path = _commands_plugin_path(deploy_command_context)
     if python_file_path.exists():
         spec = util.spec_from_file_location("commands", python_file_path)
         imported_stack = util.module_from_spec(spec)
@@ -124,7 +130,8 @@ def call_stack_deploy_setup(deploy_command_context, parameters: LaconicStackSetu
     # Link with the python file in the stack
     # Call a function in it
     # If no function found, return None
-    python_file_path = get_stack_file_path(deploy_command_context.stack).parent.joinpath("deploy", "commands.py")
+    python_file_path = _commands_plugin_path(deploy_command_context)
+    print(f"Path: {python_file_path}")
     if python_file_path.exists():
         spec = util.spec_from_file_location("commands", python_file_path)
         imported_stack = util.module_from_spec(spec)
@@ -139,7 +146,7 @@ def call_stack_deploy_create(deployment_context, extra_args):
     # Link with the python file in the stack
     # Call a function in it
     # If no function found, return None
-    python_file_path = get_stack_file_path(deployment_context.command_context.stack).parent.joinpath("deploy", "commands.py")
+    python_file_path = _commands_plugin_path(deployment_context.command_context)
     if python_file_path.exists():
         spec = util.spec_from_file_location("commands", python_file_path)
         imported_stack = util.module_from_spec(spec)
@@ -263,14 +270,21 @@ def init(ctx, config, output, map_ports_to_host):
 
 def _write_config_file(spec_file: Path, config_env_file: Path):
     spec_content = get_parsed_deployment_spec(spec_file)
-    if spec_content["config"]:
-        config_vars = spec_content["config"]
-        if config_vars:
-            with open(config_env_file, "w") as output_file:
+    # Note: we want to write an empty file even if we have no config variables
+    with open(config_env_file, "w") as output_file:
+        if "config" in spec_content and spec_content["config"]:
+            config_vars = spec_content["config"]
+            if config_vars:
                 for variable_name, variable_value in config_vars.items():
                     output_file.write(f"{variable_name}={variable_value}\n")
 
 
+def _copy_files_to_directory(file_paths: List[Path], directory: Path):
+    for path in file_paths:
+        # Using copy to preserve the execute bit
+        copy(path, os.path.join(directory, os.path.basename(path)))
+
+
 @click.command()
 @click.option("--spec-file", required=True, help="Spec file to use to create this deployment")
 @click.option("--deployment-dir", help="Create deployment files in this directory")
@@ -298,15 +312,19 @@ def create(ctx, spec_file, deployment_dir, network_dir, initial_peers):
     # Copy any config varibles from the spec file into an env file suitable for compose
     _write_config_file(spec_file, os.path.join(deployment_dir, "config.env"))
     # Copy the pod files into the deployment dir, fixing up content
-    pods = parsed_stack['pods']
+    pods = get_pod_list(parsed_stack)
     destination_compose_dir = os.path.join(deployment_dir, "compose")
     os.mkdir(destination_compose_dir)
+    destination_pods_dir = os.path.join(deployment_dir, "pods")
+    os.mkdir(destination_pods_dir)
     data_dir = Path(__file__).absolute().parent.joinpath("data")
     yaml = get_yaml()
     for pod in pods:
-        pod_file_path = os.path.join(get_compose_file_dir(), f"docker-compose-{pod}.yml")
+        pod_file_path = get_pod_file_path(parsed_stack, pod)
         parsed_pod_file = yaml.load(open(pod_file_path, "r"))
         extra_config_dirs = _find_extra_config_dirs(parsed_pod_file, pod)
+        destination_pod_dir = os.path.join(destination_pods_dir, pod)
+        os.mkdir(destination_pod_dir)
         if global_options(ctx).debug:
             print(f"extra config dirs: {extra_config_dirs}")
         _fixup_pod_file(parsed_pod_file, parsed_spec, destination_compose_dir)
@@ -322,6 +340,12 @@ def create(ctx, spec_file, deployment_dir, network_dir, initial_peers):
                 # If the same config dir appears in multiple pods, it may already have been copied
                 if not os.path.exists(destination_config_dir):
                     copytree(source_config_dir, destination_config_dir)
+        # Copy the script files for the pod, if any
+        if pod_has_scripts(parsed_stack, pod):
+            destination_script_dir = os.path.join(destination_pod_dir, "scripts")
+            os.mkdir(destination_script_dir)
+            script_paths = get_pod_script_paths(parsed_stack, pod)
+            _copy_files_to_directory(script_paths, destination_script_dir)
     # Delegate to the stack's Python code
     # The deploy create command doesn't require a --stack argument so we need to insert the
     # stack member here.

app/util.py

@@ -13,6 +13,7 @@
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <http:#www.gnu.org/licenses/>.
 
+from decouple import config
 import os.path
 import sys
 import ruamel.yaml
@@ -37,6 +38,16 @@ def get_stack_file_path(stack):
     return stack_file_path
 
 
+def get_dev_root_path(ctx):
+    if ctx and ctx.local_stack:
+        # TODO: This code probably doesn't work
+        dev_root_path = os.getcwd()[0:os.getcwd().rindex("stack-orchestrator")]
+        print(f'Local stack dev_root_path (CERC_REPO_BASE_DIR) overridden to: {dev_root_path}')
+    else:
+        dev_root_path = os.path.expanduser(config("CERC_REPO_BASE_DIR", default="~/cerc"))
+    return dev_root_path
+
+
 # Caller can pass either the name of a stack, or a path to a stack file
 def get_parsed_stack_config(stack):
     stack_file_path = stack if isinstance(stack, os.PathLike) else get_stack_file_path(stack)
@@ -56,6 +67,68 @@ def get_parsed_stack_config(stack):
         sys.exit(1)
 
 
+def get_pod_list(parsed_stack):
+    # Handle both old and new format
+    pods = parsed_stack["pods"]
+    if type(pods[0]) is str:
+        result = pods
+    else:
+        result = []
+        for pod in pods:
+            result.append(pod["name"])
+    return result
+
+
+def get_plugin_code_path(stack):
+    parsed_stack = get_parsed_stack_config(stack)
+    pods = parsed_stack["pods"]
+    # TODO: Hack
+    pod = pods[0]
+    if type(pod) is str:
+        result = get_stack_file_path(stack).parent
+    else:
+        pod_root_dir = os.path.join(get_dev_root_path(None), pod["repository"].split("/")[-1], pod["path"])
+        result = Path(os.path.join(pod_root_dir, "stack"))
+    return result
+
+
+def get_pod_file_path(parsed_stack, pod_name: str):
+    pods = parsed_stack["pods"]
+    if type(pods[0]) is str:
+        result = os.path.join(get_compose_file_dir(), f"docker-compose-{pod_name}.yml")
+    else:
+        for pod in pods:
+            if pod["name"] == pod_name:
+                pod_root_dir = os.path.join(get_dev_root_path(None), pod["repository"].split("/")[-1], pod["path"])
+                result = os.path.join(pod_root_dir, "docker-compose.yml")
+    return result
+
+
+def get_pod_script_paths(parsed_stack, pod_name: str):
+    pods = parsed_stack["pods"]
+    result = []
+    if not type(pods[0]) is str:
+        for pod in pods:
+            if pod["name"] == pod_name:
+                pod_root_dir = os.path.join(get_dev_root_path(None), pod["repository"].split("/")[-1], pod["path"])
+                if "pre_start_command" in pod:
+                    result.append(os.path.join(pod_root_dir, pod["pre_start_command"]))
+                if "post_start_command" in pod:
+                    result.append(os.path.join(pod_root_dir, pod["post_start_command"]))
+    return result
+
+
+def pod_has_scripts(parsed_stack, pod_name: str):
+    pods = parsed_stack["pods"]
+    if type(pods[0]) is str:
+        result = False
+    else:
+        for pod in pods:
+            if pod["name"] == pod_name:
+                result = "pre_start_command" in pod or "post_start_command" in pod
+    return result
+
+
 def get_compose_file_dir():
     # TODO: refactor to use common code with deploy command
     # See: https://stackoverflow.com/questions/25389095/python-get-path-of-root-project-structure