Fix default webapp port number. (#740 )

Reviewed-on: cerc-io/stack-orchestrator#740 Co-authored-by: Thomas E Lackey <telackey@bozemanpass.com> Co-committed-by: Thomas E Lackey <telackey@bozemanpass.com>
Add support for annotations and labels in spec. (#739 )
2024-02-09 01:20:41 +00:00 · 2024-02-09 00:11:07 +00:00 · 2024-02-08 20:38:41 +00:00 · 2024-02-08 20:30:47 +00:00 · 2024-02-08 20:13:12 +00:00 · 2024-02-08 19:52:49 +00:00
42 changed files with 660 additions and 218 deletions
@@ -0,0 +1,21 @@
+name: Lint Checks
+
+on:
+  pull_request:
+    branches: '*'
+  push:
+    branches: '*'
+
+jobs:
+  test:
+    name: "Run linter"
+    runs-on: ubuntu-latest
+    steps:
+      - name: "Clone project repository"
+        uses: actions/checkout@v3
+      - name: "Install Python"
+        uses: actions/setup-python@v4
+        with:
+          python-version: '3.8'
+      - name : "Run flake8"
+        uses: py-actions/flake8@v2
@@ -1 +1,2 @@
 Change this file to trigger running the test-k8s-deploy CI job
+Trigger test on PR branch
@@ -29,10 +29,10 @@ chmod +x ~/.docker/cli-plugins/docker-compose
 Next decide on a directory where you would like to put the stack-orchestrator program. Typically this would be 
 a "user" binary directory such as `~/bin` or perhaps `/usr/local/laconic` or possibly just the current working directory.

-Now, having selected that directory, download the latest release from [this page](https://github.com/cerc-io/stack-orchestrator/tags) into it (we're using `~/bin` below for concreteness but edit to suit if you selected a different directory). Also be sure that the destination directory exists and is writable:
+Now, having selected that directory, download the latest release from [this page](https://git.vdb.to/cerc-io/stack-orchestrator/tags) into it (we're using `~/bin` below for concreteness but edit to suit if you selected a different directory). Also be sure that the destination directory exists and is writable:

 ```bash
-curl -L -o ~/bin/laconic-so https://github.com/cerc-io/stack-orchestrator/releases/latest/download/laconic-so
+curl -L -o ~/bin/laconic-so https://git.vdb.to/cerc-io/stack-orchestrator/releases/download/latest/laconic-so
 ```

 Give it execute permissions:
@@ -52,7 +52,7 @@ Version: 1.1.0-7a607c2-202304260513
 Save the distribution url to `~/.laconic-so/config.yml`:
 ```bash
 mkdir ~/.laconic-so
-echo "distribution-url: https://github.com/cerc-io/stack-orchestrator/releases/latest/download/laconic-so" >  ~/.laconic-so/config.yml
+echo "distribution-url: https://git.vdb.to/cerc-io/stack-orchestrator/releases/download/latest/laconic-so" >  ~/.laconic-so/config.yml
 ```

 ### Update
@@ -26,7 +26,7 @@ In addition to the pre-requisites listed in the [README](/README.md), the follow

 1. Clone this repository:
   ```
-   $ git clone https://github.com/cerc-io/stack-orchestrator.git
+   $ git clone https://git.vdb.to/cerc-io/stack-orchestrator.git
   ```

 2. Enter the project directory:
@@ -1,10 +1,10 @@
 # Adding a new stack

-See [this PR](https://github.com/cerc-io/stack-orchestrator/pull/434) for an example of how to currently add a minimal stack to stack orchestrator. The [reth stack](https://github.com/cerc-io/stack-orchestrator/pull/435) is another good example.
+See [this PR](https://git.vdb.to/cerc-io/stack-orchestrator/pull/434) for an example of how to currently add a minimal stack to stack orchestrator. The [reth stack](https://git.vdb.to/cerc-io/stack-orchestrator/pull/435) is another good example.

 For external developers, we recommend forking this repo and adding your stack directly to your fork. This initially requires running in "developer mode" as described [here](/docs/CONTRIBUTING.md). Check out the [Namada stack](https://github.com/vknowable/stack-orchestrator/blob/main/app/data/stacks/public-namada/digitalocean_quickstart.md) from Knowable to see how that is done.

-Core to the feature completeness of stack orchestrator is to [decouple the tool functionality from payload](https://github.com/cerc-io/stack-orchestrator/issues/315) which will no longer require forking to add a stack.
+Core to the feature completeness of stack orchestrator is to [decouple the tool functionality from payload](https://git.vdb.to/cerc-io/stack-orchestrator/issues/315) which will no longer require forking to add a stack.

 ## Example

@@ -1,6 +1,6 @@
 # Specification

-Note: this page is out of date (but still useful) - it will no longer be useful once stacks are [decoupled from the tool functionality](https://github.com/cerc-io/stack-orchestrator/issues/315).
+Note: this page is out of date (but still useful) - it will no longer be useful once stacks are [decoupled from the tool functionality](https://git.vdb.to/cerc-io/stack-orchestrator/issues/315).

 ## Implementation

@@ -10,3 +10,4 @@ pydantic==1.10.9
 tomli==2.0.1
 validators==0.22.0
 kubernetes>=28.1.0
+humanfriendly>=10.0
@@ -41,4 +41,4 @@ runcmd:
  - apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
  - systemctl enable docker
  - systemctl start docker
-  - git clone https://github.com/cerc-io/stack-orchestrator.git /home/ubuntu/stack-orchestrator
+  - git clone https://git.vdb.to/cerc-io/stack-orchestrator.git /home/ubuntu/stack-orchestrator
@@ -31,5 +31,5 @@ runcmd:
  - apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
  - systemctl enable docker
  - systemctl start docker
-  - curl -L -o /usr/local/bin/laconic-so https://github.com/cerc-io/stack-orchestrator/releases/latest/download/laconic-so
+  - curl -L -o /usr/local/bin/laconic-so https://git.vdb.to/cerc-io/stack-orchestrator/releases/download/latest/laconic-so
  - chmod +x /usr/local/bin/laconic-so
@@ -137,7 +137,7 @@ fi
 echo "**************************************************************************************"
 echo "Installing laconic-so"
 # install latest `laconic-so`
-distribution_url=https://github.com/cerc-io/stack-orchestrator/releases/latest/download/laconic-so
+distribution_url=https://git.vdb.to/cerc-io/stack-orchestrator/releases/download/latest/laconic-so
 install_filename=${install_dir}/laconic-so
 mkdir -p  ${install_dir}
 curl -L -o ${install_filename} ${distribution_url}
@@ -13,7 +13,7 @@ setup(
    description='Orchestrates deployment of the Laconic stack',
    long_description=long_description,
    long_description_content_type="text/markdown",
-    url='https://github.com/cerc-io/stack-orchestrator',
+    url='https://git.vdb.to/cerc-io/stack-orchestrator',
    py_modules=['stack_orchestrator'],
    packages=find_packages(),
    install_requires=[requirements],
@@ -0,0 +1,20 @@
+version: "3.2"
+
+services:
+  mars:
+    image: cerc/mars:local
+    restart: always
+    ports:
+      - "3000:3000"
+    environment:
+      - URL_OSMOSIS_GQL=https://osmosis-node.marsprotocol.io/GGSFGSFGFG34/osmosis-hive-front/graphql
+      - URL_OSMOSIS_REST=https://lcd-osmosis.blockapsis.com
+      - URL_OSMOSIS_RPC=https://rpc-osmosis.blockapsis.com
+      - URL_NEUTRON_GQL=https://neutron.rpc.p2p.world/qgrnU6PsQZA8F9S5Fb8Fn3tV3kXmMBl2M9bcc9jWLjQy8p/hive/graphql
+      - URL_NEUTRON_REST=https://rest-kralum.neutron-1.neutron.org
+      - URL_NEUTRON_RPC=https://rpc-kralum.neutron-1.neutron.org
+      - URL_NEUTRON_TEST_GQL=https://testnet-neutron-gql.marsprotocol.io/graphql
+      - URL_NEUTRON_TEST_REST=https://rest-palvus.pion-1.ntrn.tech
+      - URL_NEUTRON_TEST_RPC=https://rpc-palvus.pion-1.ntrn.tech
+      - WALLET_CONNECT_ID=0x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0x
+
@@ -0,0 +1,8 @@
+version: "3.2"
+
+services:
+  ping-pub:
+    image: cerc/ping-pub:local
+    restart: always
+    ports:
+      - "5173:5173"
@@ -5,10 +5,13 @@ services:
    environment:
      CERC_SCRIPT_DEBUG: ${CERC_SCRIPT_DEBUG}
      CERC_TEST_PARAM_1: ${CERC_TEST_PARAM_1:-FAILED}
+      CERC_TEST_PARAM_2: "CERC_TEST_PARAM_2_VALUE"
    volumes:
      - test-data:/data
+      - test-config:/config:ro
    ports:
      - "80"

 volumes:
  test-data:
+  test-config:
@@ -5,4 +5,4 @@ services:
    environment:
      CERC_SCRIPT_DEBUG: ${CERC_SCRIPT_DEBUG}
    ports:
-      - "3000"
+      - "80"
@@ -0,0 +1,4 @@
+#!/usr/bin/env bash
+# Build the mars image
+source ${CERC_CONTAINER_BASE_DIR}/build-base.sh
+docker build -t cerc/mars:local -f ${CERC_REPO_BASE_DIR}/mars-interface/Dockerfile ${build_command_args} ${CERC_REPO_BASE_DIR}/mars-interface
@@ -0,0 +1,5 @@
+#!/usr/bin/env bash
+# Build the ping pub image
+source ${CERC_CONTAINER_BASE_DIR}/build-base.sh
+
+docker build -t cerc/ping-pub:local ${build_command_args} -f $CERC_REPO_BASE_DIR/explorer/Dockerfile $CERC_REPO_BASE_DIR/explorer
@@ -17,5 +17,23 @@ fi
 if [ -n "$CERC_TEST_PARAM_1" ]; then
  echo "Test-param-1: ${CERC_TEST_PARAM_1}"
 fi
+if [ -n "$CERC_TEST_PARAM_2" ]; then
+  echo "Test-param-2: ${CERC_TEST_PARAM_2}"
+fi
+
+if [ -d "/config" ]; then
+  echo "/config: EXISTS"
+  for f in /config/*; do
+    if [[ -f "$f" ]] || [[ -L "$f" ]]; then
+      echo "$f:"
+      cat "$f"
+      echo ""
+      echo ""
+    fi
+  done
+else
+  echo "/config: does NOT EXIST"
+fi
+
 # Run nginx which will block here forever
 /usr/sbin/nginx -g "daemon off;"
@@ -1,6 +1,6 @@
 # fixturenet-eth

-Instructions for deploying a local a geth + lighthouse blockchain "fixturenet" for development and testing purposes using laconic-stack-orchestrator (the installation of which is covered [here](https://github.com/cerc-io/stack-orchestrator)):
+Instructions for deploying a local a geth + lighthouse blockchain "fixturenet" for development and testing purposes using laconic-stack-orchestrator (the installation of which is covered [here](https://git.vdb.to/cerc-io/stack-orchestrator)):

 ## Clone required repositories

@@ -7,11 +7,11 @@ Instructions for deploying a local Laconic blockchain "fixturenet" for developme
 **Note:** For building some NPMs, access to the @lirewine repositories is required. If you don't have access, see [this tutorial](/docs/laconicd-fixturenet.md) to run this stack

 ## 1. Install Laconic Stack Orchestrator
-Installation is covered in detail [here](https://github.com/cerc-io/stack-orchestrator#user-mode) but if you're on Linux and already have docker installed it should be as simple as:
+Installation is covered in detail [here](https://git.vdb.to/cerc-io/stack-orchestrator#user-mode) but if you're on Linux and already have docker installed it should be as simple as:
 ```
 $ mkdir my-working-dir
 $ cd my-working-dir
-$ curl -L -o ./laconic-so https://github.com/cerc-io/stack-orchestrator/releases/latest/download/laconic-so
+$ curl -L -o ./laconic-so https://git.vdb.to/cerc-io/stack-orchestrator/releases/download/latest/laconic-so
 $ chmod +x ./laconic-so
 $ export PATH=$PATH:$(pwd)  # Or move laconic-so to ~/bin or your favorite on-path directory
 ```
@@ -3,11 +3,11 @@
 Instructions for deploying a local Laconic blockchain "fixturenet" for development and testing purposes using laconic-stack-orchestrator.

 ## 1. Install Laconic Stack Orchestrator
-Installation is covered in detail [here](https://github.com/cerc-io/stack-orchestrator#user-mode) but if you're on Linux and already have docker installed it should be as simple as:
+Installation is covered in detail [here](https://git.vdb.to/cerc-io/stack-orchestrator#user-mode) but if you're on Linux and already have docker installed it should be as simple as:
 ```
 $ mkdir my-working-dir
 $ cd my-working-dir
-$ curl -L -o ./laconic-so https://github.com/cerc-io/stack-orchestrator/releases/latest/download/laconic-so
+$ curl -L -o ./laconic-so https://git.vdb.to/cerc-io/stack-orchestrator/releases/download/latest/laconic-so
 $ chmod +x ./laconic-so
 $ export PATH=$PATH:$(pwd)  # Or move laconic-so to ~/bin or your favorite on-path directory
 ```
@@ -0,0 +1,16 @@
+# mars
+
+On a fresh Digital Ocean droplet with Ubuntu:
+
+```
+git clone https://github.com/cerc-io/stack-orchestrator
+cd stack-orchestrator
+./scripts/quick-install-linux.sh
+```
+Read and follow the instructions output from the above output to complete installation, then:
+
+```
+laconic-so --stack mars setup-repositories
+laconic-so --stack mars build-containers
+laconic-so --stack mars deploy up
+```
@@ -0,0 +1,8 @@
+version: "0.1"
+name: mars
+repos:
+  - github.com/cerc-io/mars-interface
+containers:
+  - cerc/mars
+pods:
+  - mars
@@ -4,7 +4,7 @@ The MobyMask watcher is a Laconic Network component that provides efficient acce

 ## Deploy the MobyMask Watcher

-The instructions below show how to deploy a MobyMask watcher using laconic-stack-orchestrator (the installation of which is covered [here](https://github.com/cerc-io/stack-orchestrator#install)).
+The instructions below show how to deploy a MobyMask watcher using laconic-stack-orchestrator (the installation of which is covered [here](https://git.vdb.to/cerc-io/stack-orchestrator#install)).

 This deployment expects that ipld-eth-server's endpoints are available on the local machine at http://ipld-eth-server.example.com:8083/graphql and http://ipld-eth-server.example.com:8082. More advanced configurations are supported by modifying the watcher's [config file](../../config/watcher-mobymask/mobymask-watcher.toml).

@@ -0,0 +1,10 @@
+# ping-pub
+Experimental block explorer for laconic
+
+```
+laconic-so --stack ping-pub setup-repositories
+laconic-so --stack ping-pub build-containers
+laconic-so --stack ping-pub deploy init --output ping-pub-spec.yml --map-ports-to-host localhost-same
+laconic-so --stack ping-pub deploy create --spec-file ping-pub-spec.yml --deployment-dir pp-deployment
+laconic-so deployment --dir pp-deployment start
+```
@@ -0,0 +1,9 @@
+version: "0.1"
+name: ping-pub
+repos:
+  # fork, but only for config & Dockerfile reasonsb
+  - github.com/LaconicNetwork/explorer@laconic
+containers:
+  - cerc/ping-pub
+pods:
+  - ping-pub
@@ -2,10 +2,10 @@ version: "1.0"
 name: webapp-deployer-backend
 description: "Deployer for webapps"
 repos:
-  - git.vdb.to:telackey/webapp-deployment-status-api
+  - git.vdb.to/telackey/webapp-deployment-status-api
 containers:
  - cerc/webapp-deployer-backend
 pods:
  - name: webapp-deployer-backend
-    repository: git.vdb.to:telackey/webapp-deployment-status-api
+    repository: git.vdb.to/telackey/webapp-deployment-status-api
    path: ./
@@ -17,6 +17,7 @@ from pathlib import Path
 from python_on_whales import DockerClient, DockerException
 from stack_orchestrator.deploy.deployer import Deployer, DeployerException, DeployerConfigGenerator
 from stack_orchestrator.deploy.deployment_context import DeploymentContext
+from stack_orchestrator.opts import opts


 class DockerDeployer(Deployer):
@@ -29,60 +30,69 @@ class DockerDeployer(Deployer):
        self.type = type

    def up(self, detach, services):
-        try:
-            return self.docker.compose.up(detach=detach, services=services)
-        except DockerException as e:
-            raise DeployerException(e)
+        if not opts.o.dry_run:
+            try:
+                return self.docker.compose.up(detach=detach, services=services)
+            except DockerException as e:
+                raise DeployerException(e)

    def down(self, timeout, volumes):
-        try:
-            return self.docker.compose.down(timeout=timeout, volumes=volumes)
-        except DockerException as e:
-            raise DeployerException(e)
+        if not opts.o.dry_run:
+            try:
+                return self.docker.compose.down(timeout=timeout, volumes=volumes)
+            except DockerException as e:
+                raise DeployerException(e)

    def update(self):
-        try:
-            return self.docker.compose.restart()
-        except DockerException as e:
-            raise DeployerException(e)
+        if not opts.o.dry_run:
+            try:
+                return self.docker.compose.restart()
+            except DockerException as e:
+                raise DeployerException(e)

    def status(self):
-        try:
-            for p in self.docker.compose.ps():
-                print(f"{p.name}\t{p.state.status}")
-        except DockerException as e:
-            raise DeployerException(e)
+        if not opts.o.dry_run:
+            try:
+                for p in self.docker.compose.ps():
+                    print(f"{p.name}\t{p.state.status}")
+            except DockerException as e:
+                raise DeployerException(e)

    def ps(self):
-        try:
-            return self.docker.compose.ps()
-        except DockerException as e:
-            raise DeployerException(e)
+        if not opts.o.dry_run:
+            try:
+                return self.docker.compose.ps()
+            except DockerException as e:
+                raise DeployerException(e)

    def port(self, service, private_port):
-        try:
-            return self.docker.compose.port(service=service, private_port=private_port)
-        except DockerException as e:
-            raise DeployerException(e)
+        if not opts.o.dry_run:
+            try:
+                return self.docker.compose.port(service=service, private_port=private_port)
+            except DockerException as e:
+                raise DeployerException(e)

    def execute(self, service, command, tty, envs):
-        try:
-            return self.docker.compose.execute(service=service, command=command, tty=tty, envs=envs)
-        except DockerException as e:
-            raise DeployerException(e)
+        if not opts.o.dry_run:
+            try:
+                return self.docker.compose.execute(service=service, command=command, tty=tty, envs=envs)
+            except DockerException as e:
+                raise DeployerException(e)

    def logs(self, services, tail, follow, stream):
-        try:
-            return self.docker.compose.logs(services=services, tail=tail, follow=follow, stream=stream)
-        except DockerException as e:
-            raise DeployerException(e)
+        if not opts.o.dry_run:
+            try:
+                return self.docker.compose.logs(services=services, tail=tail, follow=follow, stream=stream)
+            except DockerException as e:
+                raise DeployerException(e)

    def run(self, image: str, command=None, user=None, volumes=None, entrypoint=None, env={}, ports=[], detach=False):
-        try:
-            return self.docker.run(image=image, command=command, user=user, volumes=volumes,
-                                   entrypoint=entrypoint, envs=env, detach=detach, publish=ports, publish_all=len(ports) == 0)
-        except DockerException as e:
-            raise DeployerException(e)
+        if not opts.o.dry_run:
+            try:
+                return self.docker.run(image=image, command=command, user=user, volumes=volumes,
+                                       entrypoint=entrypoint, envs=env, detach=detach, publish=ports, publish_all=len(ports) == 0)
+            except DockerException as e:
+                raise DeployerException(e)


 class DockerDeployerConfigGenerator(DeployerConfigGenerator):
@@ -85,54 +85,39 @@ def create_deploy_context(
 def up_operation(ctx, services_list, stay_attached=False):
    global_context = ctx.parent.parent.obj
    deploy_context = ctx.obj
-    if not global_context.dry_run:
-        cluster_context = deploy_context.cluster_context
-        container_exec_env = _make_runtime_env(global_context)
-        for attr, value in container_exec_env.items():
-            os.environ[attr] = value
-        if global_context.verbose:
-            print(f"Running compose up with container_exec_env: {container_exec_env}, extra_args: {services_list}")
-        for pre_start_command in cluster_context.pre_start_commands:
-            _run_command(global_context, cluster_context.cluster, pre_start_command)
-        deploy_context.deployer.up(detach=not stay_attached, services=services_list)
-        for post_start_command in cluster_context.post_start_commands:
-            _run_command(global_context, cluster_context.cluster, post_start_command)
-        _orchestrate_cluster_config(global_context, cluster_context.config, deploy_context.deployer, container_exec_env)
+    cluster_context = deploy_context.cluster_context
+    container_exec_env = _make_runtime_env(global_context)
+    for attr, value in container_exec_env.items():
+        os.environ[attr] = value
+    if global_context.verbose:
+        print(f"Running compose up with container_exec_env: {container_exec_env}, extra_args: {services_list}")
+    for pre_start_command in cluster_context.pre_start_commands:
+        _run_command(global_context, cluster_context.cluster, pre_start_command)
+    deploy_context.deployer.up(detach=not stay_attached, services=services_list)
+    for post_start_command in cluster_context.post_start_commands:
+        _run_command(global_context, cluster_context.cluster, post_start_command)
+    _orchestrate_cluster_config(global_context, cluster_context.config, deploy_context.deployer, container_exec_env)


 def down_operation(ctx, delete_volumes, extra_args_list):
-    global_context = ctx.parent.parent.obj
-    if not global_context.dry_run:
-        if global_context.verbose:
-            print("Running compose down")
-        timeout_arg = None
-        if extra_args_list:
-            timeout_arg = extra_args_list[0]
-        # Specify shutdown timeout (default 10s) to give services enough time to shutdown gracefully
-        ctx.obj.deployer.down(timeout=timeout_arg, volumes=delete_volumes)
+    timeout_arg = None
+    if extra_args_list:
+        timeout_arg = extra_args_list[0]
+    # Specify shutdown timeout (default 10s) to give services enough time to shutdown gracefully
+    ctx.obj.deployer.down(timeout=timeout_arg, volumes=delete_volumes)


 def status_operation(ctx):
-    global_context = ctx.parent.parent.obj
-    if not global_context.dry_run:
-        if global_context.verbose:
-            print("Running compose status")
-        ctx.obj.deployer.status()
+    ctx.obj.deployer.status()


 def update_operation(ctx):
-    global_context = ctx.parent.parent.obj
-    if not global_context.dry_run:
-        if global_context.verbose:
-            print("Running compose update")
-        ctx.obj.deployer.update()
+    ctx.obj.deployer.update()


 def ps_operation(ctx):
    global_context = ctx.parent.parent.obj
    if not global_context.dry_run:
-        if global_context.verbose:
-            print("Running compose ps")
        container_list = ctx.obj.deployer.ps()
        if len(container_list) > 0:
            print("Running containers:")
@@ -187,15 +172,11 @@ def exec_operation(ctx, extra_args):


 def logs_operation(ctx, tail: int, follow: bool, extra_args: str):
-    global_context = ctx.parent.parent.obj
    extra_args_list = list(extra_args) or None
-    if not global_context.dry_run:
-        if global_context.verbose:
-            print("Running compose logs")
-        services_list = extra_args_list if extra_args_list is not None else []
-        logs_stream = ctx.obj.deployer.logs(services=services_list, tail=tail, follow=follow, stream=True)
-        for stream_type, stream_content in logs_stream:
-            print(stream_content.decode("utf-8"), end="")
+    services_list = extra_args_list if extra_args_list is not None else []
+    logs_stream = ctx.obj.deployer.logs(services=services_list, tail=tail, follow=follow, stream=True)
+    for stream_type, stream_content in logs_stream:
+        print(stream_content.decode("utf-8"), end="")


@command.command()
@@ -463,7 +444,7 @@ def _orchestrate_cluster_config(ctx, cluster_config, deployer, container_exec_en
                                                              tty=False,
                                                              envs=container_exec_env)
                        waiting_for_data = False
-                    if ctx.debug:
+                    if ctx.debug and not waiting_for_data:
                        print(f"destination output: {destination_output}")


@@ -18,11 +18,11 @@ from stack_orchestrator.deploy.k8s.deploy_k8s import K8sDeployer, K8sDeployerCon
 from stack_orchestrator.deploy.compose.deploy_docker import DockerDeployer, DockerDeployerConfigGenerator


-def getDeployerConfigGenerator(type: str):
+def getDeployerConfigGenerator(type: str, deployment_context):
    if type == "compose" or type is None:
        return DockerDeployerConfigGenerator(type)
    elif type == constants.k8s_deploy_type or type == constants.k8s_kind_deploy_type:
-        return K8sDeployerConfigGenerator(type)
+        return K8sDeployerConfigGenerator(type, deployment_context)
    else:
        print(f"ERROR: deploy-to {type} is not valid")

@@ -487,7 +487,7 @@ def create_operation(deployment_command_context, spec_file, deployment_dir, netw
    deployment_context = DeploymentContext()
    deployment_context.init(deployment_dir_path)
    # Call the deployer to generate any deployer-specific files (e.g. for kind)
-    deployer_config_generator = getDeployerConfigGenerator(deployment_type)
+    deployer_config_generator = getDeployerConfigGenerator(deployment_type, deployment_context)
    # TODO: make deployment_dir_path a Path above
    deployer_config_generator.generate(deployment_dir_path)
    call_stack_deploy_create(deployment_context, [network_dir, initial_peers, deployment_command_context])
@@ -22,12 +22,41 @@ from stack_orchestrator.opts import opts
 from stack_orchestrator.util import env_var_map_from_file
 from stack_orchestrator.deploy.k8s.helpers import named_volumes_from_pod_files, volume_mounts_for_service, volumes_for_pod_files
 from stack_orchestrator.deploy.k8s.helpers import get_node_pv_mount_path
-from stack_orchestrator.deploy.k8s.helpers import envs_from_environment_variables_map
+from stack_orchestrator.deploy.k8s.helpers import envs_from_environment_variables_map, envs_from_compose_file, merge_envs
 from stack_orchestrator.deploy.deploy_util import parsed_pod_files_map_from_file_names, images_for_deployment
 from stack_orchestrator.deploy.deploy_types import DeployEnvVars
-from stack_orchestrator.deploy.spec import Spec
+from stack_orchestrator.deploy.spec import Spec, Resources, ResourceLimits
 from stack_orchestrator.deploy.images import remote_tag_for_image

+DEFAULT_VOLUME_RESOURCES = Resources({
+    "reservations": {"storage": "2Gi"}
+})
+
+DEFAULT_CONTAINER_RESOURCES = Resources({
+    "reservations": {"cpus": "0.1", "memory": "200M"},
+    "limits": {"cpus": "1.0", "memory": "2000M"},
+})
+
+
+def to_k8s_resource_requirements(resources: Resources) -> client.V1ResourceRequirements:
+    def to_dict(limits: ResourceLimits):
+        if not limits:
+            return None
+
+        ret = {}
+        if limits.cpus:
+            ret["cpu"] = str(limits.cpus)
+        if limits.memory:
+            ret["memory"] = f"{int(limits.memory / (1000 * 1000))}M"
+        if limits.storage:
+            ret["storage"] = f"{int(limits.storage / (1000 * 1000))}M"
+        return ret
+
+    return client.V1ResourceRequirements(
+        requests=to_dict(resources.reservations),
+        limits=to_dict(resources.limits)
+    )
+

 class ClusterInfo:
    parsed_pod_yaml_map: Any
@@ -135,9 +164,13 @@ class ClusterInfo:
        result = []
        spec_volumes = self.spec.get_volumes()
        named_volumes = named_volumes_from_pod_files(self.parsed_pod_yaml_map)
+        resources = self.spec.get_volume_resources()
+        if not resources:
+            resources = DEFAULT_VOLUME_RESOURCES
        if opts.o.debug:
            print(f"Spec Volumes: {spec_volumes}")
            print(f"Named Volumes: {named_volumes}")
+            print(f"Resources: {resources}")
        for volume_name in spec_volumes:
            if volume_name not in named_volumes:
                if opts.o.debug:
@@ -146,9 +179,7 @@ class ClusterInfo:
            spec = client.V1PersistentVolumeClaimSpec(
                access_modes=["ReadWriteOnce"],
                storage_class_name="manual",
-                resources=client.V1ResourceRequirements(
-                    requests={"storage": "2Gi"}
-                ),
+                resources=to_k8s_resource_requirements(resources),
                volume_name=f"{self.app_name}-{volume_name}"
            )
            pvc = client.V1PersistentVolumeClaim(
@@ -192,6 +223,9 @@ class ClusterInfo:
        result = []
        spec_volumes = self.spec.get_volumes()
        named_volumes = named_volumes_from_pod_files(self.parsed_pod_yaml_map)
+        resources = self.spec.get_volume_resources()
+        if not resources:
+            resources = DEFAULT_VOLUME_RESOURCES
        for volume_name in spec_volumes:
            if volume_name not in named_volumes:
                if opts.o.debug:
@@ -200,7 +234,7 @@ class ClusterInfo:
            spec = client.V1PersistentVolumeSpec(
                storage_class_name="manual",
                access_modes=["ReadWriteOnce"],
-                capacity={"storage": "2Gi"},
+                capacity=to_k8s_resource_requirements(resources).requests,
                host_path=client.V1HostPathVolumeSource(path=get_node_pv_mount_path(volume_name))
            )
            pv = client.V1PersistentVolume(
@@ -214,6 +248,9 @@ class ClusterInfo:
    # TODO: put things like image pull policy into an object-scope struct
    def get_deployment(self, image_pull_policy: str = None):
        containers = []
+        resources = self.spec.get_container_resources()
+        if not resources:
+            resources = DEFAULT_CONTAINER_RESOURCES
        for pod_name in self.parsed_pod_yaml_map:
            pod = self.parsed_pod_yaml_map[pod_name]
            services = pod["services"]
@@ -226,6 +263,13 @@ class ClusterInfo:
                    if opts.o.debug:
                        print(f"image: {image}")
                        print(f"service port: {port}")
+                merged_envs = merge_envs(
+                    envs_from_compose_file(
+                        service_info["environment"]), self.environment_variables.map
+                ) if "environment" in service_info else self.environment_variables.map
+                envs = envs_from_environment_variables_map(merged_envs)
+                if opts.o.debug:
+                    print(f"Merged envs: {envs}")
                # Re-write the image tag for remote deployment
                image_to_use = remote_tag_for_image(
                    image, self.spec.get_image_registry()) if self.spec.get_image_registry() is not None else image
@@ -234,19 +278,40 @@ class ClusterInfo:
                    name=container_name,
                    image=image_to_use,
                    image_pull_policy=image_pull_policy,
-                    env=envs_from_environment_variables_map(self.environment_variables.map),
+                    env=envs,
                    ports=[client.V1ContainerPort(container_port=port)],
                    volume_mounts=volume_mounts,
-                    resources=client.V1ResourceRequirements(
-                        requests={"cpu": "100m", "memory": "200Mi"},
-                        limits={"cpu": "1000m", "memory": "2000Mi"},
+                    security_context=client.V1SecurityContext(
+                        privileged=self.spec.get_privileged(),
+                        capabilities=client.V1Capabilities(
+                            add=self.spec.get_capabilities()
+                        ) if self.spec.get_capabilities() else None
                    ),
+                    resources=to_k8s_resource_requirements(resources),
                )
                containers.append(container)
        volumes = volumes_for_pod_files(self.parsed_pod_yaml_map, self.spec, self.app_name)
        image_pull_secrets = [client.V1LocalObjectReference(name="laconic-registry")]
+
+        annotations = None
+        labels = {"app": self.app_name}
+
+        if self.spec.get_annotations():
+            annotations = {}
+            for key, value in self.spec.get_annotations().items():
+                for service_name in services:
+                    annotations[key.replace("{name}", service_name)] = value
+
+        if self.spec.get_labels():
+            for key, value in self.spec.get_labels().items():
+                for service_name in services:
+                    labels[key.replace("{name}", service_name)] = value
+
        template = client.V1PodTemplateSpec(
-            metadata=client.V1ObjectMeta(labels={"app": self.app_name}),
+            metadata=client.V1ObjectMeta(
+                annotations=annotations,
+                labels=labels
+            ),
            spec=client.V1PodSpec(containers=containers, image_pull_secrets=image_pull_secrets, volumes=volumes),
        )
        spec = client.V1DeploymentSpec(
@@ -81,69 +81,84 @@ class K8sDeployer(Deployer):
        self.apps_api = client.AppsV1Api()
        self.custom_obj_api = client.CustomObjectsApi()

-    def up(self, detach, services):
-
-        if self.is_kind():
-            # Create the kind cluster
-            create_cluster(self.kind_cluster_name, self.deployment_dir.joinpath(constants.kind_config_filename))
-            # Ensure the referenced containers are copied into kind
-            load_images_into_kind(self.kind_cluster_name, self.cluster_info.image_set)
-        self.connect_api()
-
+    def _create_volume_data(self):
        # Create the host-path-mounted PVs for this deployment
        pvs = self.cluster_info.get_pvs()
        for pv in pvs:
            if opts.o.debug:
                print(f"Sending this pv: {pv}")
-            pv_resp = self.core_api.create_persistent_volume(body=pv)
-            if opts.o.debug:
-                print("PVs created:")
-                print(f"{pv_resp}")
+            if not opts.o.dry_run:
+                pv_resp = self.core_api.create_persistent_volume(body=pv)
+                if opts.o.debug:
+                    print("PVs created:")
+                    print(f"{pv_resp}")

        # Figure out the PVCs for this deployment
        pvcs = self.cluster_info.get_pvcs()
        for pvc in pvcs:
            if opts.o.debug:
                print(f"Sending this pvc: {pvc}")
-            pvc_resp = self.core_api.create_namespaced_persistent_volume_claim(body=pvc, namespace=self.k8s_namespace)
-            if opts.o.debug:
-                print("PVCs created:")
-                print(f"{pvc_resp}")
+
+            if not opts.o.dry_run:
+                pvc_resp = self.core_api.create_namespaced_persistent_volume_claim(body=pvc, namespace=self.k8s_namespace)
+                if opts.o.debug:
+                    print("PVCs created:")
+                    print(f"{pvc_resp}")

        # Figure out the ConfigMaps for this deployment
        config_maps = self.cluster_info.get_configmaps()
        for cfg_map in config_maps:
            if opts.o.debug:
                print(f"Sending this ConfigMap: {cfg_map}")
-            cfg_rsp = self.core_api.create_namespaced_config_map(
-                body=cfg_map,
-                namespace=self.k8s_namespace
-            )
-            if opts.o.debug:
-                print("ConfigMap created:")
-                print(f"{cfg_rsp}")
+            if not opts.o.dry_run:
+                cfg_rsp = self.core_api.create_namespaced_config_map(
+                    body=cfg_map,
+                    namespace=self.k8s_namespace
+                )
+                if opts.o.debug:
+                    print("ConfigMap created:")
+                    print(f"{cfg_rsp}")

+    def _create_deployment(self):
        # Process compose files into a Deployment
        deployment = self.cluster_info.get_deployment(image_pull_policy=None if self.is_kind() else "Always")
        # Create the k8s objects
        if opts.o.debug:
            print(f"Sending this deployment: {deployment}")
-        deployment_resp = self.apps_api.create_namespaced_deployment(
-            body=deployment, namespace=self.k8s_namespace
-        )
-        if opts.o.debug:
-            print("Deployment created:")
-            print(f"{deployment_resp.metadata.namespace} {deployment_resp.metadata.name} \
-                  {deployment_resp.metadata.generation} {deployment_resp.spec.template.spec.containers[0].image}")
+        if not opts.o.dry_run:
+            deployment_resp = self.apps_api.create_namespaced_deployment(
+                body=deployment, namespace=self.k8s_namespace
+            )
+            if opts.o.debug:
+                print("Deployment created:")
+                print(f"{deployment_resp.metadata.namespace} {deployment_resp.metadata.name} \
+                    {deployment_resp.metadata.generation} {deployment_resp.spec.template.spec.containers[0].image}")

        service: client.V1Service = self.cluster_info.get_service()
-        service_resp = self.core_api.create_namespaced_service(
-            namespace=self.k8s_namespace,
-            body=service
-        )
        if opts.o.debug:
-            print("Service created:")
-            print(f"{service_resp}")
+            print(f"Sending this service: {service}")
+        if not opts.o.dry_run:
+            service_resp = self.core_api.create_namespaced_service(
+                namespace=self.k8s_namespace,
+                body=service
+            )
+            if opts.o.debug:
+                print("Service created:")
+                print(f"{service_resp}")
+
+    def up(self, detach, services):
+        if not opts.o.dry_run:
+            if self.is_kind():
+                # Create the kind cluster
+                create_cluster(self.kind_cluster_name, self.deployment_dir.joinpath(constants.kind_config_filename))
+                # Ensure the referenced containers are copied into kind
+                load_images_into_kind(self.kind_cluster_name, self.cluster_info.image_set)
+            self.connect_api()
+        else:
+            print("Dry run mode enabled, skipping k8s API connect")
+
+        self._create_volume_data()
+        self._create_deployment()

        if not self.is_kind():
            ingress: client.V1Ingress = self.cluster_info.get_ingress()
@@ -151,13 +166,14 @@ class K8sDeployer(Deployer):
            if ingress:
                if opts.o.debug:
                    print(f"Sending this ingress: {ingress}")
-                ingress_resp = self.networking_api.create_namespaced_ingress(
-                    namespace=self.k8s_namespace,
-                    body=ingress
-                )
-                if opts.o.debug:
-                    print("Ingress created:")
-                    print(f"{ingress_resp}")
+                if not opts.o.dry_run:
+                    ingress_resp = self.networking_api.create_namespaced_ingress(
+                        namespace=self.k8s_namespace,
+                        body=ingress
+                    )
+                    if opts.o.debug:
+                        print("Ingress created:")
+                        print(f"{ingress_resp}")
            else:
                if opts.o.debug:
                    print("No ingress configured")
@@ -390,8 +406,9 @@ class K8sDeployer(Deployer):
 class K8sDeployerConfigGenerator(DeployerConfigGenerator):
    type: str

-    def __init__(self, type: str) -> None:
+    def __init__(self, type: str, deployment_context) -> None:
        self.type = type
+        self.deployment_context = deployment_context
        super().__init__()

    def generate(self, deployment_dir: Path):
@@ -399,7 +416,7 @@ class K8sDeployerConfigGenerator(DeployerConfigGenerator):
        if self.type == "k8s-kind":
            # Check the file isn't already there
            # Get the config file contents
-            content = generate_kind_config(deployment_dir)
+            content = generate_kind_config(deployment_dir, self.deployment_context)
            if opts.o.debug:
                print(f"kind config is: {content}")
            config_file = deployment_dir.joinpath(constants.kind_config_filename)
@@ -17,6 +17,7 @@ from kubernetes import client
 import os
 from pathlib import Path
 import subprocess
+import re
 from typing import Set, Mapping, List

 from stack_orchestrator.opts import opts
@@ -97,11 +98,17 @@ def volume_mounts_for_service(parsed_pod_files, service):
                    if "volumes" in service_obj:
                        volumes = service_obj["volumes"]
                        for mount_string in volumes:
-                            # Looks like: test-data:/data
-                            parts = mount_string.split(":")
-                            volume_name = parts[0]
-                            mount_path = parts[1]
-                            mount_options = parts[2] if len(parts) == 3 else None
+                            # Looks like: test-data:/data or test-data:/data:ro or test-data:/data:rw
+                            if opts.o.debug:
+                                print(f"mount_string: {mount_string}")
+                            mount_split = mount_string.split(":")
+                            volume_name = mount_split[0]
+                            mount_path = mount_split[1]
+                            mount_options = mount_split[2] if len(mount_split) == 3 else None
+                            if opts.o.debug:
+                                print(f"volumne_name: {volume_name}")
+                                print(f"mount path: {mount_path}")
+                                print(f"mount options: {mount_options}")
                            volume_device = client.V1VolumeMount(
                                mount_path=mount_path, name=volume_name, read_only="ro" == mount_options)
                            result.append(volume_device)
@@ -134,8 +141,9 @@ def _get_host_paths_for_volumes(parsed_pod_files):
            volumes = parsed_pod_file["volumes"]
            for volume_name in volumes.keys():
                volume_definition = volumes[volume_name]
-                host_path = volume_definition["driver_opts"]["device"]
-                result[volume_name] = host_path
+                if volume_definition and "driver_opts" in volume_definition:
+                    host_path = volume_definition["driver_opts"]["device"]
+                    result[volume_name] = host_path
    return result


@@ -147,7 +155,7 @@ def _make_absolute_host_path(data_mount_path: Path, deployment_dir: Path) -> Pat
        return Path.cwd().joinpath(deployment_dir.joinpath("compose").joinpath(data_mount_path)).resolve()


-def _generate_kind_mounts(parsed_pod_files, deployment_dir):
+def _generate_kind_mounts(parsed_pod_files, deployment_dir, deployment_context):
    volume_definitions = []
    volume_host_path_map = _get_host_paths_for_volumes(parsed_pod_files)
    # Note these paths are relative to the location of the pod files (at present)
@@ -162,12 +170,21 @@ def _generate_kind_mounts(parsed_pod_files, deployment_dir):
                if "volumes" in service_obj:
                    volumes = service_obj["volumes"]
                    for mount_string in volumes:
-                        # Looks like: test-data:/data
-                        (volume_name, mount_path) = mount_string.split(":")
-                        volume_definitions.append(
-                            f"  - hostPath: {_make_absolute_host_path(volume_host_path_map[volume_name], deployment_dir)}\n"
-                            f"    containerPath: {get_node_pv_mount_path(volume_name)}"
-                        )
+                        # Looks like: test-data:/data or test-data:/data:ro or test-data:/data:rw
+                        if opts.o.debug:
+                            print(f"mount_string: {mount_string}")
+                        mount_split = mount_string.split(":")
+                        volume_name = mount_split[0]
+                        mount_path = mount_split[1]
+                        if opts.o.debug:
+                            print(f"volumne_name: {volume_name}")
+                            print(f"map: {volume_host_path_map}")
+                            print(f"mount path: {mount_path}")
+                        if volume_name not in deployment_context.spec.get_configmaps():
+                            volume_definitions.append(
+                                f"  - hostPath: {_make_absolute_host_path(volume_host_path_map[volume_name], deployment_dir)}\n"
+                                f"    containerPath: {get_node_pv_mount_path(volume_name)}\n"
+                            )
    return (
        "" if len(volume_definitions) == 0 else (
            "  extraMounts:\n"
@@ -189,7 +206,7 @@ def _generate_kind_port_mappings(parsed_pod_files):
                    for port_string in ports:
                        # TODO handle the complex cases
                        # Looks like: 80 or something more complicated
-                        port_definitions.append(f"  - containerPort: {port_string}\n    hostPort: {port_string}")
+                        port_definitions.append(f"  - containerPort: {port_string}\n    hostPort: {port_string}\n")
    return (
        "" if len(port_definitions) == 0 else (
            "  extraPortMappings:\n"
@@ -198,6 +215,33 @@ def _generate_kind_port_mappings(parsed_pod_files):
    )


+# Note: this makes any duplicate definition in b overwrite a
+def merge_envs(a: Mapping[str, str], b: Mapping[str, str]) -> Mapping[str, str]:
+    result = {**a, **b}
+    return result
+
+
+def _expand_shell_vars(raw_val: str) -> str:
+    # could be: <string> or ${<env-var-name>} or ${<env-var-name>:-<default-value>}
+    # TODO: implement support for variable substitution and default values
+    # if raw_val is like ${<something>} print a warning and substitute an empty string
+    # otherwise return raw_val
+    match = re.search(r"^\$\{(.*)\}$", raw_val)
+    if match:
+        print(f"WARNING: found unimplemented environment variable substitution: {raw_val}")
+    else:
+        return raw_val
+
+
+# TODO: handle the case where the same env var is defined in multiple places
+def envs_from_compose_file(compose_file_envs: Mapping[str, str]) -> Mapping[str, str]:
+    result = {}
+    for env_var, env_val in compose_file_envs.items():
+        expanded_env_val = _expand_shell_vars(env_val)
+        result.update({env_var: expanded_env_val})
+    return result
+
+
 def envs_from_environment_variables_map(map: Mapping[str, str]) -> List[client.V1EnvVar]:
    result = []
    for env_var, env_val in map.items():
@@ -223,13 +267,13 @@ def envs_from_environment_variables_map(map: Mapping[str, str]) -> List[client.V
 #  extraMounts:
 #  - hostPath: /path/to/my/files
 #    containerPath: /files
-def generate_kind_config(deployment_dir: Path):
+def generate_kind_config(deployment_dir: Path, deployment_context):
    compose_file_dir = deployment_dir.joinpath("compose")
    # TODO: this should come from the stack file, not this way
    pod_files = [p for p in compose_file_dir.iterdir() if p.is_file()]
    parsed_pod_files_map = parsed_pod_files_map_from_file_names(pod_files)
    port_mappings_yml = _generate_kind_port_mappings(parsed_pod_files_map)
-    mounts_yml = _generate_kind_mounts(parsed_pod_files_map, deployment_dir)
+    mounts_yml = _generate_kind_mounts(parsed_pod_files_map, deployment_dir, deployment_context)
    return (
        "kind: Cluster\n"
        "apiVersion: kind.x-k8s.io/v1alpha4\n"
@@ -13,12 +13,60 @@
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <http:#www.gnu.org/licenses/>.

-from pathlib import Path
 import typing
+import humanfriendly
+
+from pathlib import Path
+
 from stack_orchestrator.util import get_yaml
 from stack_orchestrator import constants


+class ResourceLimits:
+    cpus: float = None
+    memory: int = None
+    storage: int = None
+
+    def __init__(self, obj={}):
+        if "cpus" in obj:
+            self.cpus = float(obj["cpus"])
+        if "memory" in obj:
+            self.memory = humanfriendly.parse_size(obj["memory"])
+        if "storage" in obj:
+            self.storage = humanfriendly.parse_size(obj["storage"])
+
+    def __len__(self):
+        return len(self.__dict__)
+
+    def __iter__(self):
+        for k in self.__dict__:
+            yield k, self.__dict__[k]
+
+    def __repr__(self):
+        return str(self.__dict__)
+
+
+class Resources:
+    limits: ResourceLimits = None
+    reservations: ResourceLimits = None
+
+    def __init__(self, obj={}):
+        if "reservations" in obj:
+            self.reservations = ResourceLimits(obj["reservations"])
+        if "limits" in obj:
+            self.limits = ResourceLimits(obj["limits"])
+
+    def __len__(self):
+        return len(self.__dict__)
+
+    def __iter__(self):
+        for k in self.__dict__:
+            yield k, self.__dict__[k]
+
+    def __repr__(self):
+        return str(self.__dict__)
+
+
 class Spec:

    obj: typing.Any
@@ -47,8 +95,26 @@ class Spec:
                if self.obj and "configmaps" in self.obj
                else {})

+    def get_container_resources(self):
+        return Resources(self.obj.get("resources", {}).get("containers", {}))
+
+    def get_volume_resources(self):
+        return Resources(self.obj.get("resources", {}).get("volumes", {}))
+
    def get_http_proxy(self):
        return (self.obj[constants.network_key][constants.http_proxy_key]
                if self.obj and constants.network_key in self.obj
                and constants.http_proxy_key in self.obj[constants.network_key]
                else None)
+
+    def get_annotations(self):
+        return self.obj.get("annotations", {})
+
+    def get_labels(self):
+        return self.obj.get("labels", {})
+
+    def get_privileged(self):
+        return "true" == str(self.obj.get("security", {}).get("privileged", "false")).lower()
+
+    def get_capabilities(self):
+        return self.obj.get("security", {}).get("capabilities", [])
@@ -44,7 +44,7 @@ def _fixup_url_spec(spec_file_name: str, url: str):
    - host-name: {parsed_url.hostname}
      routes:
        - path: '{parsed_url.path if parsed_url.path else "/"}'
-          proxy-to: webapp:3000
+          proxy-to: webapp:80
    '''
    spec_file_path = Path(spec_file_name)
    with open(spec_file_path) as rfile:
@@ -19,6 +19,8 @@ import shlex
 import shutil
 import sys
 import tempfile
+import time
+import uuid

 import click

@@ -27,7 +29,7 @@ from stack_orchestrator.deploy.webapp.util import (LaconicRegistryClient,
                                                   build_container_image, push_container_image,
                                                   file_hash, deploy_to_k8s, publish_deployment,
                                                   hostname_for_deployment_request, generate_hostname_for_app,
-                                                   match_owner)
+                                                   match_owner, skip_by_tag)


 def process_app_deployment_request(
@@ -39,8 +41,19 @@ def process_app_deployment_request(
    dns_suffix,
    deployment_parent_dir,
    kube_config,
-    image_registry
+    image_registry,
+    log_parent_dir
 ):
+    run_id = f"{app_deployment_request.id}-{str(time.time()).split('.')[0]}-{str(uuid.uuid4()).split('-')[0]}"
+    log_file = None
+    if log_parent_dir:
+        log_dir = os.path.join(log_parent_dir, app_deployment_request.id)
+        if not os.path.exists(log_dir):
+            os.mkdir(log_dir)
+        log_file_path = os.path.join(log_dir, f"{run_id}.log")
+        print(f"Directing build logs to: {log_file_path}")
+        log_file = open(log_file_path, "wt")
+
    # 1. look up application
    app = laconic.get_record(app_deployment_request.attributes.application, require=True)

@@ -102,8 +115,10 @@ def process_app_deployment_request(
    needs_k8s_deploy = False
    # 6. build container (if needed)
    if not deployment_record or deployment_record.attributes.application != app.id:
-        build_container_image(app, deployment_container_tag)
-        push_container_image(deployment_dir)
+        # TODO: pull from request
+        extra_build_args = []
+        build_container_image(app, deployment_container_tag, extra_build_args, log_file)
+        push_container_image(deployment_dir, log_file)
        needs_k8s_deploy = True

    # 7. update config (if needed)
@@ -116,6 +131,7 @@ def process_app_deployment_request(
        deploy_to_k8s(
            deployment_record,
            deployment_dir,
+            log_file
        )

    publish_deployment(
@@ -162,10 +178,14 @@ def dump_known_requests(filename, requests, status="SEEN"):
@click.option("--record-namespace-dns", help="eg, crn://laconic/dns")
@click.option("--record-namespace-deployments", help="eg, crn://laconic/deployments")
@click.option("--dry-run", help="Don't do anything, just report what would be done.", is_flag=True)
+@click.option("--include-tags", help="Only include requests with matching tags (comma-separated).", default="")
+@click.option("--exclude-tags", help="Exclude requests with matching tags (comma-separated).", default="")
+@click.option("--log-dir", help="Output build/deployment logs to directory.", default=None)
@click.pass_context
-def command(ctx, kube_config, laconic_config, image_registry, deployment_parent_dir,
+def command(ctx, kube_config, laconic_config, image_registry, deployment_parent_dir,  # noqa: C901
            request_id, discover, state_file, only_update_state,
-            dns_suffix, record_namespace_dns, record_namespace_deployments, dry_run):
+            dns_suffix, record_namespace_dns, record_namespace_deployments, dry_run,
+            include_tags, exclude_tags, log_dir):
    if request_id and discover:
        print("Cannot specify both --request-id and --discover", file=sys.stderr)
        sys.exit(2)
@@ -183,6 +203,10 @@ def command(ctx, kube_config, laconic_config, image_registry, deployment_parent_
            print("--dns-suffix, --record-namespace-dns, and --record-namespace-deployments are all required", file=sys.stderr)
            sys.exit(2)

+    # Split CSV and clean up values.
+    include_tags = [tag.strip() for tag in include_tags.split(",") if tag]
+    exclude_tags = [tag.strip() for tag in exclude_tags.split(",") if tag]
+
    laconic = LaconicRegistryClient(laconic_config)

    # Find deployment requests.
@@ -204,6 +228,7 @@ def command(ctx, kube_config, laconic_config, image_registry, deployment_parent_
    requests.sort(key=lambda r: r.createTime)
    requests.reverse()
    requests_by_name = {}
+    skipped_by_name = {}
    for r in requests:
        # TODO: Do this _after_ filtering deployments and cancellations to minimize round trips.
        app = laconic.get_record(r.attributes.application)
@@ -216,17 +241,20 @@ def command(ctx, kube_config, laconic_config, image_registry, deployment_parent_
            requested_name = generate_hostname_for_app(app)
            print("Generating name %s for request %s." % (requested_name, r.id))

-        if requested_name not in requests_by_name:
-            print(
-                "Found request %s to run application %s on %s."
-                % (r.id, r.attributes.application, requested_name)
-            )
-            requests_by_name[requested_name] = r
-        else:
-            print(
-                "Ignoring request %s, it is superseded by %s."
-                % (r.id, requests_by_name[requested_name].id)
-            )
+        if requested_name in skipped_by_name or requested_name in requests_by_name:
+            print("Ignoring request %s, it has been superseded." % r.id)
+            continue
+
+        if skip_by_tag(r, include_tags, exclude_tags):
+            print("Skipping request %s, filtered by tag (include %s, exclude %s, present %s)" % (r.id,
+                                                                                                 include_tags,
+                                                                                                 exclude_tags,
+                                                                                                 r.attributes.tags))
+            skipped_by_name[requested_name] = r
+            continue
+
+        print("Found request %s to run application %s on %s." % (r.id, r.attributes.application, requested_name))
+        requests_by_name[requested_name] = r

    # Find deployments.
    deployments = laconic.app_deployments()
@@ -273,7 +301,8 @@ def command(ctx, kube_config, laconic_config, image_registry, deployment_parent_
                    dns_suffix,
                    os.path.abspath(deployment_parent_dir),
                    kube_config,
-                    image_registry
+                    image_registry,
+                    log_dir
                )
                status = "DEPLOYED"
            finally:
@@ -20,7 +20,7 @@ import sys

 import click

-from stack_orchestrator.deploy.webapp.util import LaconicRegistryClient, match_owner
+from stack_orchestrator.deploy.webapp.util import LaconicRegistryClient, match_owner, skip_by_tag


 def process_app_removal_request(ctx,
@@ -107,10 +107,12 @@ def dump_known_requests(filename, requests):
@click.option("--delete-names/--preserve-names", help="Delete all names associated with removed deployments.", default=True)
@click.option("--delete-volumes/--preserve-volumes", default=True, help="delete data volumes")
@click.option("--dry-run", help="Don't do anything, just report what would be done.", is_flag=True)
+@click.option("--include-tags", help="Only include requests with matching tags (comma-separated).", default="")
+@click.option("--exclude-tags", help="Exclude requests with matching tags (comma-separated).", default="")
@click.pass_context
 def command(ctx, laconic_config, deployment_parent_dir,
            request_id, discover, state_file, only_update_state,
-            delete_names, delete_volumes, dry_run):
+            delete_names, delete_volumes, dry_run, include_tags, exclude_tags):
    if request_id and discover:
        print("Cannot specify both --request-id and --discover", file=sys.stderr)
        sys.exit(2)
@@ -123,6 +125,10 @@ def command(ctx, laconic_config, deployment_parent_dir,
        print("--only-update-state requires --state-file", file=sys.stderr)
        sys.exit(2)

+    # Split CSV and clean up values.
+    include_tags = [tag.strip() for tag in include_tags.split(",") if tag]
+    exclude_tags = [tag.strip() for tag in exclude_tags.split(",") if tag]
+
    laconic = LaconicRegistryClient(laconic_config)

    # Find deployment removal requests.
@@ -141,6 +147,7 @@ def command(ctx, laconic_config, deployment_parent_dir,

    previous_requests = load_known_requests(state_file)
    requests.sort(key=lambda r: r.createTime)
+    requests.reverse()

    # Find deployments.
    deployments = {}
@@ -155,10 +162,22 @@ def command(ctx, laconic_config, deployment_parent_dir,
            # TODO: should we handle CRNs?
            removals_by_deployment[r.attributes.deployment] = r

-    requests_to_execute = []
+    one_per_deployment = {}
    for r in requests:
        if not r.attributes.deployment:
            print(f"Skipping removal request {r.id} since it was a cancellation.")
+        elif r.attributes.deployment in one_per_deployment:
+            print(f"Skipping removal request {r.id} since it was superseded.")
+        else:
+            one_per_deployment[r.attributes.deployment] = r
+
+    requests_to_execute = []
+    for r in one_per_deployment.values():
+        if skip_by_tag(r, include_tags, exclude_tags):
+            print("Skipping removal request %s, filtered by tag (include %s, exclude %s, present %s)" % (r.id,
+                                                                                                         include_tags,
+                                                                                                         exclude_tags,
+                                                                                                         r.attributes.tags))
        elif r.id in removals_by_request:
            print(f"Found satisfied request for {r.id} at {removals_by_request[r.id].id}")
        elif r.attributes.deployment in removals_by_deployment:
@@ -212,7 +212,7 @@ def determine_base_container(clone_dir, app_type="webapp"):
    return base_container


-def build_container_image(app_record, tag, extra_build_args=[]):
+def build_container_image(app_record, tag, extra_build_args=[], log_file=None):
    tmpdir = tempfile.mkdtemp()

    try:
@@ -227,10 +227,10 @@ def build_container_image(app_record, tag, extra_build_args=[]):
            git_env = dict(os.environ.copy())
            # Never prompt
            git_env["GIT_TERMINAL_PROMPT"] = "0"
-            subprocess.check_call(["git", "clone", repo, clone_dir], env=git_env)
-            subprocess.check_call(["git", "checkout", ref], cwd=clone_dir, env=git_env)
+            subprocess.check_call(["git", "clone", repo, clone_dir], env=git_env, stdout=log_file, stderr=log_file)
+            subprocess.check_call(["git", "checkout", ref], cwd=clone_dir, env=git_env, stdout=log_file, stderr=log_file)
        else:
-            result = subprocess.run(["git", "clone", "--depth", "1", repo, clone_dir])
+            result = subprocess.run(["git", "clone", "--depth", "1", repo, clone_dir], stdout=log_file, stderr=log_file)
            result.check_returncode()

        base_container = determine_base_container(clone_dir, app_record.attributes.app_type)
@@ -246,25 +246,27 @@ def build_container_image(app_record, tag, extra_build_args=[]):
            build_command.append("--extra-build-args")
            build_command.append(" ".join(extra_build_args))

-        result = subprocess.run(build_command)
+        result = subprocess.run(build_command, stdout=log_file, stderr=log_file)
        result.check_returncode()
    finally:
        cmd("rm", "-rf", tmpdir)


-def push_container_image(deployment_dir):
+def push_container_image(deployment_dir, log_file=None):
    print("Pushing image ...")
-    result = subprocess.run([sys.argv[0], "deployment", "--dir", deployment_dir, "push-images"])
+    result = subprocess.run([sys.argv[0], "deployment", "--dir", deployment_dir, "push-images"],
+                            stdout=log_file, stderr=log_file)
    result.check_returncode()


-def deploy_to_k8s(deploy_record, deployment_dir):
+def deploy_to_k8s(deploy_record, deployment_dir, log_file=None):
    if not deploy_record:
        command = "up"
    else:
        command = "update"

-    result = subprocess.run([sys.argv[0], "deployment", "--dir", deployment_dir, command])
+    result = subprocess.run([sys.argv[0], "deployment", "--dir", deployment_dir, command],
+                            stdout=log_file, stderr=log_file)
    result.check_returncode()


@@ -349,3 +351,17 @@ def generate_hostname_for_app(app):
    else:
        m.update(app.attributes.repository.encode())
    return "%s-%s" % (last_part, m.hexdigest()[0:10])
+
+
+def skip_by_tag(r, include_tags, exclude_tags):
+    for tag in exclude_tags:
+        if r.attributes.tags and tag in r.attributes.tags:
+            return True
+
+    if include_tags:
+        for tag in include_tags:
+            if r.attributes.tags and tag in r.attributes.tags:
+                return False
+        return True
+
+    return False
@@ -19,7 +19,7 @@ import sys
 import ruamel.yaml
 from pathlib import Path
 from dotenv import dotenv_values
-from typing import Mapping
+from typing import Mapping, Set, List


 def include_exclude_check(s, include, exclude):
@@ -81,17 +81,17 @@ def get_pod_list(parsed_stack):
    return result


-def get_plugin_code_paths(stack):
+def get_plugin_code_paths(stack) -> List[Path]:
    parsed_stack = get_parsed_stack_config(stack)
    pods = parsed_stack["pods"]
-    result = []
+    result: Set[Path] = set()
    for pod in pods:
        if type(pod) is str:
-            result.append(get_stack_file_path(stack).parent)
+            result.add(get_stack_file_path(stack).parent)
        else:
            pod_root_dir = os.path.join(get_dev_root_path(None), pod["repository"].split("/")[-1], pod["path"])
-            result.append(Path(os.path.join(pod_root_dir, "stack")))
-    return result
+            result.add(Path(os.path.join(pod_root_dir, "stack")))
+    return list(result)


 def get_pod_file_path(parsed_stack, pod_name: str):
@@ -139,6 +139,13 @@ def get_compose_file_dir():
    return source_compose_dir


+def get_config_file_dir():
+    # TODO: refactor to use common code with deploy command
+    data_dir = Path(__file__).absolute().parent.joinpath("data")
+    source_config_dir = data_dir.joinpath("config")
+    return source_config_dir
+
+
 def get_parsed_deployment_spec(spec_file):
    spec_file_path = Path(spec_file)
    try:
@@ -6,6 +6,12 @@ fi
 # Dump environment variables for debugging
 echo "Environment variables:"
 env
+
+delete_cluster_exit () {
+    $TEST_TARGET_SO deployment --dir $test_deployment_dir stop --delete-volumes
+    exit 1
+}
+
 # Test basic stack-orchestrator deploy
 echo "Running stack-orchestrator deploy test"
 # Bit of a hack, test the most recent package
@@ -106,6 +112,10 @@ if [ ! "$create_file_content" == "create-command-output-data"  ]; then
    echo "deploy create test: FAILED"
    exit 1
 fi
+
+# Add a config file to be picked up by the ConfigMap before starting.
+echo "dbfc7a4d-44a7-416d-b5f3-29842cc47650" > $test_deployment_dir/data/test-config/test_config
+
 echo "deploy create output file test: passed"
 # Try to start the deployment
 $TEST_TARGET_SO deployment --dir $test_deployment_dir start
@@ -124,6 +134,37 @@ else
    echo "deployment config test: FAILED"
    exit 1
 fi
+# Check the config variable CERC_TEST_PARAM_2 was passed correctly from the compose file
+if [[ "$log_output_3" == *"Test-param-2: CERC_TEST_PARAM_2_VALUE"* ]]; then
+    echo "deployment compose config test: passed"
+else
+    echo "deployment compose config test: FAILED"
+    exit 1
+fi
+
+# Check that the ConfigMap is mounted and contains the expected content.
+log_output_4=$( $TEST_TARGET_SO deployment --dir $test_deployment_dir logs )
+if [[ "$log_output_4" == *"/config/test_config:"* ]] && [[ "$log_output_4" == *"dbfc7a4d-44a7-416d-b5f3-29842cc47650"* ]]; then
+    echo "deployment ConfigMap test: passed"
+else
+    echo "deployment ConfigMap test: FAILED"
+    delete_cluster_exit
+fi
+
+# Stop then start again and check the volume was preserved
+$TEST_TARGET_SO deployment --dir $test_deployment_dir stop
+# Sleep a bit just in case
+# sleep for longer to check if that's why the subsequent create cluster fails
+sleep 20
+$TEST_TARGET_SO deployment --dir $test_deployment_dir start
+log_output_5=$( $TEST_TARGET_SO deployment --dir $test_deployment_dir logs )
+if [[ "$log_output_5" == *"Filesystem is old"* ]]; then
+    echo "Retain volumes test: passed"
+else
+    echo "Retain volumes test: FAILED"
+    delete_cluster_exit
+fi
+
 # Stop and clean up
 $TEST_TARGET_SO deployment --dir $test_deployment_dir stop --delete-volumes
 echo "Test passed"
@@ -97,6 +97,10 @@ if [ ! "$create_file_content" == "create-command-output-data"  ]; then
    echo "deploy create test: FAILED"
    exit 1
 fi
+
+# Add a config file to be picked up by the ConfigMap before starting.
+echo "dbfc7a4d-44a7-416d-b5f3-29842cc47650" > $test_deployment_dir/data/test-config/test_config
+
 echo "deploy create output file test: passed"
 # Try to start the deployment
 $TEST_TARGET_SO deployment --dir $test_deployment_dir start
@@ -110,6 +114,7 @@ else
    echo "deployment logs test: FAILED"
    delete_cluster_exit
 fi
+
 # Check the config variable CERC_TEST_PARAM_1 was passed correctly
 if [[ "$log_output_3" == *"Test-param-1: PASSED"* ]]; then
    echo "deployment config test: passed"
@@ -117,6 +122,24 @@ else
    echo "deployment config test: FAILED"
    delete_cluster_exit
 fi
+
+# Check the config variable CERC_TEST_PARAM_2 was passed correctly from the compose file
+if [[ "$log_output_3" == *"Test-param-2: CERC_TEST_PARAM_2_VALUE"* ]]; then
+    echo "deployment compose config test: passed"
+else
+    echo "deployment compose config test: FAILED"
+    exit 1
+fi
+
+# Check that the ConfigMap is mounted and contains the expected content.
+log_output_4=$( $TEST_TARGET_SO deployment --dir $test_deployment_dir logs )
+if [[ "$log_output_4" == *"/config/test_config:"* ]] && [[ "$log_output_4" == *"dbfc7a4d-44a7-416d-b5f3-29842cc47650"* ]]; then
+    echo "deployment ConfigMap test: passed"
+else
+    echo "deployment ConfigMap test: FAILED"
+    delete_cluster_exit
+fi
+
 # Stop then start again and check the volume was preserved
 $TEST_TARGET_SO deployment --dir $test_deployment_dir stop
 # Sleep a bit just in case
@@ -125,8 +148,8 @@ sleep 20
 $TEST_TARGET_SO deployment --dir $test_deployment_dir start
 wait_for_pods_started
 wait_for_log_output
-log_output_4=$( $TEST_TARGET_SO deployment --dir $test_deployment_dir logs )
-if [[ "$log_output_4" == *"Filesystem is old"* ]]; then
+log_output_5=$( $TEST_TARGET_SO deployment --dir $test_deployment_dir logs )
+if [[ "$log_output_5" == *"Filesystem is old"* ]]; then
    echo "Retain volumes test: passed"
 else
    echo "Retain volumes test: FAILED"
Author	SHA1	Message	Date
telackey	c9444591f5	Fix default webapp port number. (#740 ) Reviewed-on: cerc-io/stack-orchestrator#740 Co-authored-by: Thomas E Lackey <telackey@bozemanpass.com> Co-committed-by: Thomas E Lackey <telackey@bozemanpass.com>	2024-02-09 01:20:41 +00:00
telackey	903f3b10e2	Add support for annotations and labels in spec. (#739 ) ``` stack: webapp-deployer-backend deploy-to: k8s annotations: foo.bar.annot/{name}: baz labels: a.b.c/{name}.blah: "value" ``` Reviewed-on: cerc-io/stack-orchestrator#739 Co-authored-by: Thomas E Lackey <telackey@bozemanpass.com> Co-committed-by: Thomas E Lackey <telackey@bozemanpass.com>	2024-02-09 00:11:07 +00:00
telackey	72ed2eb91a	Fix bad test in tag check. (#738 ) Reviewed-on: cerc-io/stack-orchestrator#738 Co-authored-by: Thomas E Lackey <telackey@bozemanpass.com> Co-committed-by: Thomas E Lackey <telackey@bozemanpass.com>	2024-02-08 20:38:41 +00:00
dboreham	2104eb5f30	Merge pull request 'Add Mars stack' (#725 ) from zach/mars into main Reviewed-on: cerc-io/stack-orchestrator#725 Reviewed-by: David Boreham <dboreham@noreply.git.vdb.to>	2024-02-08 20:30:47 +00:00
zramsay zramsay dboreham	afd6be3b13	Ping pub (#663 ) for #170, revives #190 uses https://github.com/LaconicNetwork/explorer/pull/1 Co-authored-by: zramsay <zach@bluecollarcoding.ca> Co-authored-by: David Boreham <david@bozemanpass.com> Reviewed-on: cerc-io/stack-orchestrator#663 Co-authored-by: zramsay <zramsay@noreply.git.vdb.to> Co-committed-by: zramsay <zramsay@noreply.git.vdb.to>	2024-02-08 20:13:12 +00:00
dboreham	f914baa913	Merge branch 'main' into zach/mars	2024-02-08 19:52:49 +00:00
dboreham	8be1e684e8	Process environment variables defined in compose files (#736 ) Reviewed-on: cerc-io/stack-orchestrator#736 Co-authored-by: David Boreham <david@bozemanpass.com> Co-committed-by: David Boreham <david@bozemanpass.com>	2024-02-08 19:41:57 +00:00
telackey	5d16251ce9	Merge pull request 'Add resource limit options to spec.' (#735 ) from telackey/limits into main Reviewed-on: cerc-io/stack-orchestrator#735 Reviewed-by: David Boreham <dboreham@noreply.git.vdb.to>	2024-02-08 16:52:49 +00:00
telackey	3309782439	Refactor	2024-02-08 00:47:46 -06:00
telackey	4b3b3478e7	Switch to Docker-style limits	2024-02-08 00:43:41 -06:00
telackey	2a9955055c	debug	2024-02-07 16:56:35 -06:00
telackey	8964e1c0fe	Add resource limit options to spec.	2024-02-07 16:48:02 -06:00
telackey	d2ebb81d77	Tags for undeploy (#734 ) ``` --include-tags TEXT Only include requests with matching tags (comma-separated). --exclude-tags TEXT Exclude requests with matching tags (comma- separated). ``` Reviewed-on: cerc-io/stack-orchestrator#734 Co-authored-by: Thomas E Lackey <telackey@bozemanpass.com> Co-committed-by: Thomas E Lackey <telackey@bozemanpass.com>	2024-02-07 21:45:16 +00:00
telackey	4a981d8d2e	Fix repo URL (#733 ) Needs a '/' (http) not ':' (ssh). Reviewed-on: cerc-io/stack-orchestrator#733 Co-authored-by: Thomas E Lackey <telackey@bozemanpass.com> Co-committed-by: Thomas E Lackey <telackey@bozemanpass.com>	2024-02-07 18:52:07 +00:00
telackey	88a0236ca9	Add the ability to filter deployment requests by tag. (#730 ) Reviewed-on: cerc-io/stack-orchestrator#730	2024-02-07 03:12:40 +00:00
telackey	937b983ec9	Update links from github.com to git.vdb.to (#732 ) Update links and references to github.com to git.vdb.to. Also enable the flake8 lint action in gitea. Reviewed-on: cerc-io/stack-orchestrator#732 Co-authored-by: Thomas E Lackey <telackey@bozemanpass.com> Co-committed-by: Thomas E Lackey <telackey@bozemanpass.com>	2024-02-06 22:42:44 +00:00
dboreham	bfbcfb7904	Volume processing fixes (#729 )	2024-02-06 12:32:10 -07:00
dboreham	3d5ececba5	Remove duplicate plugin paths and resulting extraneous error message (#728 )	2024-02-06 11:59:37 -07:00
dboreham	6848fc33cf	Implement dry run support for k8s deploy (#727 )	2024-02-06 07:07:56 -07:00
telackeyanddboreham	36bb068983	Add ConfigMap test. (#726 ) * Add ConfigMap test. * eof * Minor tweak * Trigger test --------- Co-authored-by: David Boreham <david@bozemanpass.com>	2024-02-05 14:15:11 -06:00
Zach	098567625a	Create README.md	2024-01-30 17:47:56 -05:00
zramsay	23ee3e19b7	mars: add env vars to docker-compose	2024-01-29 22:44:55 +00:00
zramsay	2d764fc7d0	basic mars stack	2024-01-29 16:00:58 +00:00