Allow relative volume paths for k8s-kind deployments

For k8s-kind, relative paths (e.g., ./data/rpc-config) are resolved to
$DEPLOYMENT_DIR/path by _make_absolute_host_path() during kind config
generation. This provides Docker Host persistence that survives cluster
restarts.

Previously, validation threw an exception before paths could be resolved,
making it impossible to use relative paths for persistent storage.

Changes:
- deployment_create.py: Skip relative path check for k8s-kind
- cluster_info.py: Allow relative paths to reach PV generation
- docs/deployment_patterns.md: Document volume persistence patterns

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

CLAUDE.md

@@ -8,6 +8,7 @@ NEVER assume your hypotheses are true without evidence
 
 ALWAYS clearly state when something is a hypothesis
 ALWAYS use evidence from the systems your interacting with to support your claims and hypotheses
+ALWAYS run `pre-commit run --all-files` before committing changes
 
 ## Key Principles
 

docs/cli.md

@@ -65,3 +65,71 @@ Force full rebuild of packages:
 ```
 $ laconic-so build-npms --include <package-name> --force-rebuild
 ```
+
+## deploy
+
+The `deploy` command group manages persistent deployments. The general workflow is `deploy init` to generate a spec file, then `deploy create` to create a deployment directory from the spec, then runtime commands like `deploy up` and `deploy down`.
+
+### deploy init
+
+Generate a deployment spec file from a stack definition:
+```
+$ laconic-so --stack <stack-name> deploy init --output <spec-file>
+```
+
+Options:
+- `--output` (required): write spec file here
+- `--config`: provide config variables for the deployment
+- `--config-file`: provide config variables in a file
+- `--kube-config`: provide a config file for a k8s deployment
+- `--image-registry`: provide a container image registry url for this k8s cluster
+- `--map-ports-to-host`: map ports to the host (`any-variable-random`, `localhost-same`, `any-same`, `localhost-fixed-random`, `any-fixed-random`)
+
+### deploy create
+
+Create a deployment directory from a spec file:
+```
+$ laconic-so --stack <stack-name> deploy create --spec-file <spec-file> --deployment-dir <dir>
+```
+
+Update an existing deployment in-place (preserving data volumes and env file):
+```
+$ laconic-so --stack <stack-name> deploy create --spec-file <spec-file> --deployment-dir <dir> --update
+```
+
+Options:
+- `--spec-file` (required): spec file to use
+- `--deployment-dir`: target directory for deployment files
+- `--update`: update an existing deployment directory, preserving data volumes and env file. Changed files are backed up with a `.bak` suffix. The deployment's `config.env` and `deployment.yml` are also preserved.
+- `--network-dir`: network configuration supplied in this directory
+- `--initial-peers`: initial set of persistent peers
+
+### deploy up
+
+Start a deployment:
+```
+$ laconic-so deployment --dir <deployment-dir> up
+```
+
+### deploy down
+
+Stop a deployment:
+```
+$ laconic-so deployment --dir <deployment-dir> down
+```
+Use `--delete-volumes` to also remove data volumes.
+
+### deploy ps
+
+Show running services:
+```
+$ laconic-so deployment --dir <deployment-dir> ps
+```
+
+### deploy logs
+
+View service logs:
+```
+$ laconic-so deployment --dir <deployment-dir> logs
+```
+Use `-f` to follow and `-n <count>` to tail.

docs/deployment_patterns.md

@@ -0,0 +1,114 @@
+# Deployment Patterns
+
+## GitOps Pattern
+
+For production deployments, we recommend a GitOps approach where your deployment configuration is tracked in version control.
+
+### Overview
+
+- **spec.yml is your source of truth**: Maintain it in your operator repository
+- **Don't regenerate on every restart**: Run `deploy init` once, then customize and commit
+- **Use restart for updates**: The restart command respects your git-tracked spec.yml
+
+### Workflow
+
+1. **Initial setup**: Run `deploy init` once to generate a spec.yml template
+2. **Customize and commit**: Edit spec.yml with your configuration (hostnames, resources, etc.) and commit to your operator repo
+3. **Deploy from git**: Use the committed spec.yml for deployments
+4. **Update via git**: Make changes in git, then restart to apply
+
+```bash
+# Initial setup (run once)
+laconic-so --stack my-stack deploy init --output spec.yml
+
+# Customize for your environment
+vim spec.yml  # Set hostname, resources, etc.
+
+# Commit to your operator repository
+git add spec.yml
+git commit -m "Add my-stack deployment configuration"
+git push
+
+# On deployment server: deploy from git-tracked spec
+laconic-so deploy create \
+  --spec-file /path/to/operator-repo/spec.yml \
+  --deployment-dir my-deployment
+
+laconic-so deployment --dir my-deployment start
+```
+
+### Updating Deployments
+
+When you need to update a deployment:
+
+```bash
+# 1. Make changes in your operator repo
+vim /path/to/operator-repo/spec.yml
+git commit -am "Update configuration"
+git push
+
+# 2. On deployment server: pull and restart
+cd /path/to/operator-repo && git pull
+laconic-so deployment --dir my-deployment restart
+```
+
+The `restart` command:
+- Pulls latest code from the stack repository
+- Uses your git-tracked spec.yml (does NOT regenerate from defaults)
+- Syncs the deployment directory
+- Restarts services
+
+### Anti-patterns
+
+**Don't do this:**
+```bash
+# BAD: Regenerating spec on every deployment
+laconic-so --stack my-stack deploy init --output spec.yml
+laconic-so deploy create --spec-file spec.yml ...
+```
+
+This overwrites your customizations with defaults from the stack's `commands.py`.
+
+**Do this instead:**
+```bash
+# GOOD: Use your git-tracked spec
+git pull  # Get latest spec.yml from your operator repo
+laconic-so deployment --dir my-deployment restart
+```
+
+## Volume Persistence in k8s-kind
+
+k8s-kind has 3 storage layers:
+
+- **Docker Host**: The physical server running Docker
+- **Kind Node**: A Docker container simulating a k8s node
+- **Pod Container**: Your workload
+
+For k8s-kind, volumes with paths are mounted from Docker Host → Kind Node → Pod via extraMounts.
+
+| spec.yml volume | Storage Location | Survives Pod Restart | Survives Cluster Restart |
+|-----------------|------------------|---------------------|-------------------------|
+| `vol:` (empty)  | Kind Node PVC    | ✅ | ❌ |
+| `vol: ./data/x` | Docker Host      | ✅ | ✅ |
+| `vol: /abs/path`| Docker Host      | ✅ | ✅ |
+
+**Recommendation**: Always use paths for data you want to keep. Relative paths
+(e.g., `./data/rpc-config`) resolve to `$DEPLOYMENT_DIR/data/rpc-config` on the
+Docker Host.
+
+### Example
+
+```yaml
+# In spec.yml
+volumes:
+  rpc-config: ./data/rpc-config  # Persists to $DEPLOYMENT_DIR/data/rpc-config
+  chain-data: ./data/chain       # Persists to $DEPLOYMENT_DIR/data/chain
+  temp-cache:                    # Empty = Kind Node PVC (lost on cluster delete)
+```
+
+### The Antipattern
+
+Empty-path volumes appear persistent because they survive pod restarts (data lives
+in Kind Node container). However, this data is lost when the kind cluster is
+recreated. This "false persistence" has caused data loss when operators assumed
+their data was safe.

stack_orchestrator/constants.py

@@ -44,3 +44,4 @@ unlimited_memlock_key = "unlimited-memlock"
 runtime_class_key = "runtime-class"
 high_memlock_runtime = "high-memlock"
 high_memlock_spec_filename = "high-memlock-spec.json"
+acme_email_key = "acme-email"

stack_orchestrator/data/compose/docker-compose-test.yml

@@ -8,6 +8,8 @@ services:
       CERC_TEST_PARAM_2: "CERC_TEST_PARAM_2_VALUE"
       CERC_TEST_PARAM_3: ${CERC_TEST_PARAM_3:-FAILED}
     volumes:
+      - ../config/test/script.sh:/opt/run.sh
+      - ../config/test/settings.env:/opt/settings.env
       - test-data-bind:/data
       - test-data-auto:/data2
       - test-config:/config:ro

stack_orchestrator/data/config/test/script.sh

@@ -0,0 +1,3 @@
+#!/bin/sh
+
+echo "Hello"

stack_orchestrator/data/config/test/settings.env

@@ -0,0 +1 @@
+ANSWER=42

stack_orchestrator/data/container-build/cerc-test-container/Dockerfile

@@ -1,9 +1,6 @@
-FROM ubuntu:latest
+FROM alpine:latest
 
-RUN apt-get update && export DEBIAN_FRONTEND=noninteractive && export DEBCONF_NOWARNINGS="yes" && \
+RUN apk add --no-cache nginx
-    apt-get install -y software-properties-common && \
-    apt-get install -y nginx && \
-    apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
 
 EXPOSE 80
 

stack_orchestrator/data/container-build/cerc-test-container/run.sh

@@ -1,4 +1,4 @@
-#!/usr/bin/env bash
+#!/usr/bin/env sh
 set -e
 
 if [ -n "$CERC_SCRIPT_DEBUG" ]; then
@@ -8,14 +8,14 @@ fi
 echo "Test container starting"
 
 DATA_DEVICE=$(df | grep "/data$" | awk '{ print $1 }')
-if [[ -n "$DATA_DEVICE" ]]; then
+if [ -n "$DATA_DEVICE" ]; then
   echo "/data: MOUNTED dev=${DATA_DEVICE}"
 else
   echo "/data: not mounted"
 fi
 
 DATA2_DEVICE=$(df | grep "/data2$" | awk '{ print $1 }')
-if [[ -n "$DATA_DEVICE" ]]; then
+if [ -n "$DATA_DEVICE" ]; then
   echo "/data2: MOUNTED dev=${DATA2_DEVICE}"
 else
   echo "/data2: not mounted"
@@ -23,7 +23,7 @@ fi
 
 # Test if the container's filesystem is old (run previously) or new
 for d in /data /data2; do
-  if [[ -f "$d/exists" ]];
+  if [ -f "$d/exists" ];
   then
       TIMESTAMP=`cat $d/exists`
       echo "$d filesystem is old, created: $TIMESTAMP"
@@ -52,7 +52,7 @@ fi
 if [ -d "/config" ]; then
   echo "/config: EXISTS"
   for f in /config/*; do
-    if [[ -f "$f" ]] || [[ -L "$f" ]]; then
+    if [ -f "$f" ] || [ -L "$f" ]; then
       echo "$f:"
       cat "$f"
       echo ""
@@ -64,4 +64,4 @@ else
 fi
 
 # Run nginx which will block here forever
-/usr/sbin/nginx -g "daemon off;"
+nginx -g "daemon off;"

stack_orchestrator/data/k8s/components/ingress/ingress-caddy-kind-deploy.yaml

@@ -93,6 +93,7 @@ rules:
       - get
       - create
       - update
+      - delete
 ---
 apiVersion: rbac.authorization.k8s.io/v1
 kind: ClusterRoleBinding

stack_orchestrator/deploy/deployment.py

@@ -15,7 +15,9 @@
 
 import click
 from pathlib import Path
+import subprocess
 import sys
+import time
 from stack_orchestrator import constants
 from stack_orchestrator.deploy.images import push_images_operation
 from stack_orchestrator.deploy.deploy import (
@@ -228,3 +230,176 @@ def run_job(ctx, job_name, helm_release):
 
     ctx.obj = make_deploy_context(ctx)
     run_job_operation(ctx, job_name, helm_release)
+
+
+@command.command()
+@click.option("--stack-path", help="Path to stack git repo (overrides stored path)")
+@click.option(
+    "--spec-file", help="Path to GitOps spec.yml in repo (e.g., deployment/spec.yml)"
+)
+@click.option("--config-file", help="Config file to pass to deploy init")
+@click.option(
+    "--force",
+    is_flag=True,
+    default=False,
+    help="Skip DNS verification",
+)
+@click.option(
+    "--expected-ip",
+    help="Expected IP for DNS verification (if different from egress)",
+)
+@click.pass_context
+def restart(ctx, stack_path, spec_file, config_file, force, expected_ip):
+    """Pull latest code and restart deployment using git-tracked spec.
+
+    GitOps workflow:
+    1. Operator maintains spec.yml in their git repository
+    2. This command pulls latest code (including updated spec.yml)
+    3. If hostname changed, verifies DNS routes to this server
+    4. Syncs deployment directory with the git-tracked spec
+    5. Stops and restarts the deployment
+
+    Data volumes are always preserved. The cluster is never destroyed.
+
+    Stack source resolution (in order):
+    1. --stack-path argument (if provided)
+    2. stack-source field in deployment.yml (if stored)
+    3. Error if neither available
+
+    Note: spec.yml should be maintained in git, not regenerated from
+    commands.py on each restart. Use 'deploy init' only for initial
+    spec generation, then customize and commit to your operator repo.
+    """
+    from stack_orchestrator.util import get_yaml, get_parsed_deployment_spec
+    from stack_orchestrator.deploy.deployment_create import create_operation
+    from stack_orchestrator.deploy.dns_probe import verify_dns_via_probe
+
+    deployment_context: DeploymentContext = ctx.obj
+
+    # Get current spec info (before git pull)
+    current_spec = deployment_context.spec
+    current_http_proxy = current_spec.get_http_proxy()
+    current_hostname = (
+        current_http_proxy[0]["host-name"] if current_http_proxy else None
+    )
+
+    # Resolve stack source path
+    if stack_path:
+        stack_source = Path(stack_path).resolve()
+    else:
+        # Try to get from deployment.yml
+        deployment_file = (
+            deployment_context.deployment_dir / constants.deployment_file_name
+        )
+        deployment_data = get_yaml().load(open(deployment_file))
+        stack_source_str = deployment_data.get("stack-source")
+        if not stack_source_str:
+            print(
+                "Error: No stack-source in deployment.yml and --stack-path not provided"
+            )
+            print("Use --stack-path to specify the stack git repository location")
+            sys.exit(1)
+        stack_source = Path(stack_source_str)
+
+    if not stack_source.exists():
+        print(f"Error: Stack source path does not exist: {stack_source}")
+        sys.exit(1)
+
+    print("=== Deployment Restart ===")
+    print(f"Deployment dir: {deployment_context.deployment_dir}")
+    print(f"Stack source: {stack_source}")
+    print(f"Current hostname: {current_hostname}")
+
+    # Step 1: Git pull (brings in updated spec.yml from operator's repo)
+    print("\n[1/4] Pulling latest code from stack repository...")
+    git_result = subprocess.run(
+        ["git", "pull"], cwd=stack_source, capture_output=True, text=True
+    )
+    if git_result.returncode != 0:
+        print(f"Git pull failed: {git_result.stderr}")
+        sys.exit(1)
+    print(f"Git pull: {git_result.stdout.strip()}")
+
+    # Determine spec file location
+    # Priority: --spec-file argument > repo's deployment/spec.yml > deployment dir
+    # Stack path is like: repo/stack_orchestrator/data/stacks/stack-name
+    # So repo root is 4 parents up
+    repo_root = stack_source.parent.parent.parent.parent
+    if spec_file:
+        # Spec file relative to repo root
+        spec_file_path = repo_root / spec_file
+    else:
+        # Try standard GitOps location in repo
+        gitops_spec = repo_root / "deployment" / "spec.yml"
+        if gitops_spec.exists():
+            spec_file_path = gitops_spec
+        else:
+            # Fall back to deployment directory
+            spec_file_path = deployment_context.deployment_dir / "spec.yml"
+
+    if not spec_file_path.exists():
+        print(f"Error: spec.yml not found at {spec_file_path}")
+        print("For GitOps, add spec.yml to your repo at deployment/spec.yml")
+        print("Or specify --spec-file with path relative to repo root")
+        sys.exit(1)
+
+    print(f"Using spec: {spec_file_path}")
+
+    # Parse spec to check for hostname changes
+    new_spec_obj = get_parsed_deployment_spec(str(spec_file_path))
+    new_http_proxy = new_spec_obj.get("network", {}).get("http-proxy", [])
+    new_hostname = new_http_proxy[0]["host-name"] if new_http_proxy else None
+
+    print(f"Spec hostname: {new_hostname}")
+
+    # Step 2: DNS verification (only if hostname changed)
+    if new_hostname and new_hostname != current_hostname:
+        print(f"\n[2/4] Hostname changed: {current_hostname} -> {new_hostname}")
+        if force:
+            print("DNS verification skipped (--force)")
+        else:
+            print("Verifying DNS via probe...")
+            if not verify_dns_via_probe(new_hostname):
+                print(f"\nDNS verification failed for {new_hostname}")
+                print("Ensure DNS is configured before restarting.")
+                print("Use --force to skip this check.")
+                sys.exit(1)
+    else:
+        print("\n[2/4] Hostname unchanged, skipping DNS verification")
+
+    # Step 3: Sync deployment directory with spec
+    print("\n[3/4] Syncing deployment directory...")
+    deploy_ctx = make_deploy_context(ctx)
+    create_operation(
+        deployment_command_context=deploy_ctx,
+        spec_file=str(spec_file_path),
+        deployment_dir=str(deployment_context.deployment_dir),
+        update=True,
+        network_dir=None,
+        initial_peers=None,
+    )
+
+    # Reload deployment context with updated spec
+    deployment_context.init(deployment_context.deployment_dir)
+    ctx.obj = deployment_context
+
+    # Stop deployment
+    print("\n[4/4] Restarting deployment...")
+    ctx.obj = make_deploy_context(ctx)
+    down_operation(
+        ctx, delete_volumes=False, extra_args_list=[], skip_cluster_management=True
+    )
+
+    # Brief pause to ensure clean shutdown
+    time.sleep(5)
+
+    # Start deployment
+    up_operation(
+        ctx, services_list=None, stay_attached=False, skip_cluster_management=True
+    )
+
+    print("\n=== Restart Complete ===")
+    print("Deployment restarted with git-tracked configuration.")
+    if new_hostname and new_hostname != current_hostname:
+        print(f"\nNew hostname: {new_hostname}")
+        print("Caddy will automatically provision TLS certificate.")

stack_orchestrator/deploy/deployment_create.py

@@ -17,11 +17,14 @@ import click
 from importlib import util
 import os
 from pathlib import Path
-from typing import List
+from typing import List, Optional
 import random
-from shutil import copy, copyfile, copytree
+from shutil import copy, copyfile, copytree, rmtree
 from secrets import token_hex
 import sys
+import filecmp
+import tempfile
+
 from stack_orchestrator import constants
 from stack_orchestrator.opts import opts
 from stack_orchestrator.util import (
@@ -465,7 +468,10 @@ def init_operation(
             else:
                 volume_descriptors[named_volume] = f"./data/{named_volume}"
         if volume_descriptors:
-            spec_file_content["volumes"] = volume_descriptors
+            # Merge with existing volumes from stack init()
+            # init() volumes take precedence over compose defaults
+            orig_volumes = spec_file_content.get("volumes", {})
+            spec_file_content["volumes"] = {**volume_descriptors, **orig_volumes}
         if configmap_descriptors:
             spec_file_content["configmaps"] = configmap_descriptors
 
@@ -501,11 +507,14 @@ def _copy_files_to_directory(file_paths: List[Path], directory: Path):
         copy(path, os.path.join(directory, os.path.basename(path)))
 
 
-def _create_deployment_file(deployment_dir: Path):
+def _create_deployment_file(deployment_dir: Path, stack_source: Optional[Path] = None):
     deployment_file_path = deployment_dir.joinpath(constants.deployment_file_name)
     cluster = f"{constants.cluster_name_prefix}{token_hex(8)}"
+    deployment_content = {constants.cluster_id_key: cluster}
+    if stack_source:
+        deployment_content["stack-source"] = str(stack_source)
     with open(deployment_file_path, "w") as output_file:
-        output_file.write(f"{constants.cluster_id_key}: {cluster}\n")
+        get_yaml().dump(deployment_content, output_file)
 
 
 def _check_volume_definitions(spec):
@@ -513,10 +522,14 @@ def _check_volume_definitions(spec):
         for volume_name, volume_path in spec.get_volumes().items():
             if volume_path:
                 if not os.path.isabs(volume_path):
-                    raise Exception(
+                    # For k8s-kind: allow relative paths, they'll be resolved
-                        f"Relative path {volume_path} for volume {volume_name} not "
+                    # by _make_absolute_host_path() during kind config generation
-                        f"supported for deployment type {spec.get_deployment_type()}"
+                    if not spec.is_kind_deployment():
-                    )
+                        deploy_type = spec.get_deployment_type()
+                        raise Exception(
+                            f"Relative path {volume_path} for volume "
+                            f"{volume_name} not supported for {deploy_type}"
+                        )
 
 
 @click.command()
@@ -524,6 +537,12 @@ def _check_volume_definitions(spec):
     "--spec-file", required=True, help="Spec file to use to create this deployment"
 )
 @click.option("--deployment-dir", help="Create deployment files in this directory")
+@click.option(
+    "--update",
+    is_flag=True,
+    default=False,
+    help="Update existing deployment directory, preserving data volumes and env file",
+)
 @click.option(
     "--helm-chart",
     is_flag=True,
@@ -536,13 +555,21 @@ def _check_volume_definitions(spec):
 @click.argument("extra_args", nargs=-1, type=click.UNPROCESSED)
 @click.pass_context
 def create(
-    ctx, spec_file, deployment_dir, helm_chart, network_dir, initial_peers, extra_args
+    ctx,
+    spec_file,
+    deployment_dir,
+    update,
+    helm_chart,
+    network_dir,
+    initial_peers,
+    extra_args,
 ):
     deployment_command_context = ctx.obj
     return create_operation(
         deployment_command_context,
         spec_file,
         deployment_dir,
+        update,
         helm_chart,
         network_dir,
         initial_peers,
@@ -556,6 +583,7 @@ def create_operation(
     deployment_command_context,
     spec_file,
     deployment_dir,
+    update=False,
     helm_chart=False,
     network_dir=None,
     initial_peers=None,
@@ -568,23 +596,23 @@ def create_operation(
     stack_name = parsed_spec["stack"]
     deployment_type = parsed_spec[constants.deploy_to_key]
 
-    stack_file = get_stack_path(stack_name).joinpath(constants.stack_file_name)
-    parsed_stack = get_parsed_stack_config(stack_name)
     if opts.o.debug:
         print(f"parsed spec: {parsed_spec}")
+
     if deployment_dir is None:
         deployment_dir_path = _make_default_deployment_dir()
     else:
         deployment_dir_path = Path(deployment_dir)
-    if deployment_dir_path.exists():
-        error_exit(f"{deployment_dir_path} already exists")
-    os.mkdir(deployment_dir_path)
-    # Copy spec file and the stack file into the deployment dir
-    copyfile(spec_file, deployment_dir_path.joinpath(constants.spec_file_name))
-    copyfile(stack_file, deployment_dir_path.joinpath(constants.stack_file_name))
 
-    # Create deployment.yml with cluster-id
+    if deployment_dir_path.exists():
-    _create_deployment_file(deployment_dir_path)
+        if not update:
+            error_exit(f"{deployment_dir_path} already exists")
+        if opts.o.debug:
+            print(f"Updating existing deployment at {deployment_dir_path}")
+    else:
+        if update:
+            error_exit(f"--update requires that {deployment_dir_path} already exists")
+        os.mkdir(deployment_dir_path)
 
     # Branch to Helm chart generation flow if --helm-chart flag is set
     if deployment_type == "k8s" and helm_chart:
@@ -595,104 +623,48 @@ def create_operation(
         generate_helm_chart(stack_name, spec_file, deployment_dir_path)
         return  # Exit early for helm chart generation
 
-    # Existing deployment flow continues unchanged
+    # Resolve stack source path for restart capability
-    # Copy any config varibles from the spec file into an env file suitable for compose
+    stack_source = get_stack_path(stack_name)
-    _write_config_file(
-        spec_file, deployment_dir_path.joinpath(constants.config_file_name)
-    )
-    # Copy any k8s config file into the deployment dir
-    if deployment_type == "k8s":
-        _write_kube_config_file(
-            Path(parsed_spec[constants.kube_config_key]),
-            deployment_dir_path.joinpath(constants.kube_config_filename),
-        )
-    # Copy the pod files into the deployment dir, fixing up content
-    pods = get_pod_list(parsed_stack)
-    destination_compose_dir = deployment_dir_path.joinpath("compose")
-    os.mkdir(destination_compose_dir)
-    destination_pods_dir = deployment_dir_path.joinpath("pods")
-    os.mkdir(destination_pods_dir)
-    yaml = get_yaml()
-    for pod in pods:
-        pod_file_path = get_pod_file_path(stack_name, parsed_stack, pod)
-        if pod_file_path is None:
-            continue
-        parsed_pod_file = yaml.load(open(pod_file_path, "r"))
-        extra_config_dirs = _find_extra_config_dirs(parsed_pod_file, pod)
-        destination_pod_dir = destination_pods_dir.joinpath(pod)
-        os.mkdir(destination_pod_dir)
-        if opts.o.debug:
-            print(f"extra config dirs: {extra_config_dirs}")
-        _fixup_pod_file(parsed_pod_file, parsed_spec, destination_compose_dir)
-        with open(
-            destination_compose_dir.joinpath("docker-compose-%s.yml" % pod), "w"
-        ) as output_file:
-            yaml.dump(parsed_pod_file, output_file)
-        # Copy the config files for the pod, if any
-        config_dirs = {pod}
-        config_dirs = config_dirs.union(extra_config_dirs)
-        for config_dir in config_dirs:
-            source_config_dir = resolve_config_dir(stack_name, config_dir)
-            if os.path.exists(source_config_dir):
-                destination_config_dir = deployment_dir_path.joinpath(
-                    "config", config_dir
-                )
-                # If the same config dir appears in multiple pods, it may already have
-                # been copied
-                if not os.path.exists(destination_config_dir):
-                    copytree(source_config_dir, destination_config_dir)
-        # Copy the script files for the pod, if any
-        if pod_has_scripts(parsed_stack, pod):
-            destination_script_dir = destination_pod_dir.joinpath("scripts")
-            os.mkdir(destination_script_dir)
-            script_paths = get_pod_script_paths(parsed_stack, pod)
-            _copy_files_to_directory(script_paths, destination_script_dir)
-        if parsed_spec.is_kubernetes_deployment():
-            for configmap in parsed_spec.get_configmaps():
-                source_config_dir = resolve_config_dir(stack_name, configmap)
-                if os.path.exists(source_config_dir):
-                    destination_config_dir = deployment_dir_path.joinpath(
-                        "configmaps", configmap
-                    )
-                    copytree(
-                        source_config_dir, destination_config_dir, dirs_exist_ok=True
-                    )
-        else:
-            # TODO: We should probably only do this if the volume is marked :ro.
-            for volume_name, volume_path in parsed_spec.get_volumes().items():
-                source_config_dir = resolve_config_dir(stack_name, volume_name)
-                # Only copy if the source exists and is _not_ empty.
-                if os.path.exists(source_config_dir) and os.listdir(source_config_dir):
-                    destination_config_dir = deployment_dir_path.joinpath(volume_path)
-                    # Only copy if the destination exists and _is_ empty.
-                    if os.path.exists(destination_config_dir) and not os.listdir(
-                        destination_config_dir
-                    ):
-                        copytree(
-                            source_config_dir,
-                            destination_config_dir,
-                            dirs_exist_ok=True,
-                        )
 
-    # Copy the job files into the deployment dir (for Docker deployments)
+    if update:
-    jobs = get_job_list(parsed_stack)
+        # Sync mode: write to temp dir, then copy to deployment dir with backups
-    if jobs and not parsed_spec.is_kubernetes_deployment():
+        temp_dir = Path(tempfile.mkdtemp(prefix="deployment-sync-"))
-        destination_compose_jobs_dir = deployment_dir_path.joinpath("compose-jobs")
+        try:
-        os.mkdir(destination_compose_jobs_dir)
+            # Write deployment files to temp dir
-        for job in jobs:
+            # (skip deployment.yml to preserve cluster ID)
-            job_file_path = get_job_file_path(stack_name, parsed_stack, job)
+            _write_deployment_files(
-            if job_file_path and job_file_path.exists():
+                temp_dir,
-                parsed_job_file = yaml.load(open(job_file_path, "r"))
+                Path(spec_file),
-                _fixup_pod_file(parsed_job_file, parsed_spec, destination_compose_dir)
+                parsed_spec,
-                with open(
+                stack_name,
-                    destination_compose_jobs_dir.joinpath(
+                deployment_type,
-                        "docker-compose-%s.yml" % job
+                include_deployment_file=False,
-                    ),
+                stack_source=stack_source,
-                    "w",
+            )
-                ) as output_file:
+
-                    yaml.dump(parsed_job_file, output_file)
+            # Copy from temp to deployment dir, excluding data volumes
-                if opts.o.debug:
+            # and backing up changed files.
-                    print(f"Copied job compose file: {job}")
+            # Exclude data/* to avoid touching user data volumes.
+            # Exclude config file to preserve deployment settings
+            # (XXX breaks passing config vars from spec)
+            exclude_patterns = ["data", "data/*", constants.config_file_name]
+            _safe_copy_tree(
+                temp_dir, deployment_dir_path, exclude_patterns=exclude_patterns
+            )
+        finally:
+            # Clean up temp dir
+            rmtree(temp_dir)
+    else:
+        # Normal mode: write directly to deployment dir
+        _write_deployment_files(
+            deployment_dir_path,
+            Path(spec_file),
+            parsed_spec,
+            stack_name,
+            deployment_type,
+            include_deployment_file=True,
+            stack_source=stack_source,
+        )
 
     # Delegate to the stack's Python code
     # The deploy create command doesn't require a --stack argument so we need
@@ -712,6 +684,184 @@ def create_operation(
     )
 
 
+def _safe_copy_tree(src: Path, dst: Path, exclude_patterns: Optional[List[str]] = None):
+    """
+    Recursively copy a directory tree, backing up changed files with .bak suffix.
+
+    :param src: Source directory
+    :param dst: Destination directory
+    :param exclude_patterns: List of path patterns to exclude (relative to src)
+    """
+    if exclude_patterns is None:
+        exclude_patterns = []
+
+    def should_exclude(path: Path) -> bool:
+        """Check if path matches any exclude pattern."""
+        rel_path = path.relative_to(src)
+        for pattern in exclude_patterns:
+            if rel_path.match(pattern):
+                return True
+        return False
+
+    def safe_copy_file(src_file: Path, dst_file: Path):
+        """Copy file, backing up destination if it differs."""
+        if (
+            dst_file.exists()
+            and not dst_file.is_dir()
+            and not filecmp.cmp(src_file, dst_file)
+        ):
+            os.rename(dst_file, f"{dst_file}.bak")
+        copy(src_file, dst_file)
+
+    # Walk the source tree
+    for src_path in src.rglob("*"):
+        if should_exclude(src_path):
+            continue
+
+        rel_path = src_path.relative_to(src)
+        dst_path = dst / rel_path
+
+        if src_path.is_dir():
+            dst_path.mkdir(parents=True, exist_ok=True)
+        else:
+            dst_path.parent.mkdir(parents=True, exist_ok=True)
+            safe_copy_file(src_path, dst_path)
+
+
+def _write_deployment_files(
+    target_dir: Path,
+    spec_file: Path,
+    parsed_spec: Spec,
+    stack_name: str,
+    deployment_type: str,
+    include_deployment_file: bool = True,
+    stack_source: Optional[Path] = None,
+):
+    """
+    Write deployment files to target directory.
+
+    :param target_dir: Directory to write files to
+    :param spec_file: Path to spec file
+    :param parsed_spec: Parsed spec object
+    :param stack_name: Name of stack
+    :param deployment_type: Type of deployment
+    :param include_deployment_file: Whether to create deployment.yml (skip for update)
+    :param stack_source: Path to stack source (git repo) for restart capability
+    """
+    stack_file = get_stack_path(stack_name).joinpath(constants.stack_file_name)
+    parsed_stack = get_parsed_stack_config(stack_name)
+
+    # Copy spec file and the stack file into the target dir
+    copyfile(spec_file, target_dir.joinpath(constants.spec_file_name))
+    copyfile(stack_file, target_dir.joinpath(constants.stack_file_name))
+
+    # Create deployment file if requested
+    if include_deployment_file:
+        _create_deployment_file(target_dir, stack_source=stack_source)
+
+    # Copy any config variables from the spec file into an env file suitable for compose
+    _write_config_file(spec_file, target_dir.joinpath(constants.config_file_name))
+
+    # Copy any k8s config file into the target dir
+    if deployment_type == "k8s":
+        _write_kube_config_file(
+            Path(parsed_spec[constants.kube_config_key]),
+            target_dir.joinpath(constants.kube_config_filename),
+        )
+
+    # Copy the pod files into the target dir, fixing up content
+    pods = get_pod_list(parsed_stack)
+    destination_compose_dir = target_dir.joinpath("compose")
+    os.makedirs(destination_compose_dir, exist_ok=True)
+    destination_pods_dir = target_dir.joinpath("pods")
+    os.makedirs(destination_pods_dir, exist_ok=True)
+    yaml = get_yaml()
+
+    for pod in pods:
+        pod_file_path = get_pod_file_path(stack_name, parsed_stack, pod)
+        if pod_file_path is None:
+            continue
+        parsed_pod_file = yaml.load(open(pod_file_path, "r"))
+        extra_config_dirs = _find_extra_config_dirs(parsed_pod_file, pod)
+        destination_pod_dir = destination_pods_dir.joinpath(pod)
+        os.makedirs(destination_pod_dir, exist_ok=True)
+        if opts.o.debug:
+            print(f"extra config dirs: {extra_config_dirs}")
+        _fixup_pod_file(parsed_pod_file, parsed_spec, destination_compose_dir)
+        with open(
+            destination_compose_dir.joinpath("docker-compose-%s.yml" % pod), "w"
+        ) as output_file:
+            yaml.dump(parsed_pod_file, output_file)
+
+        # Copy the config files for the pod, if any
+        config_dirs = {pod}
+        config_dirs = config_dirs.union(extra_config_dirs)
+        for config_dir in config_dirs:
+            source_config_dir = resolve_config_dir(stack_name, config_dir)
+            if os.path.exists(source_config_dir):
+                destination_config_dir = target_dir.joinpath("config", config_dir)
+                copytree(source_config_dir, destination_config_dir, dirs_exist_ok=True)
+
+        # Copy the script files for the pod, if any
+        if pod_has_scripts(parsed_stack, pod):
+            destination_script_dir = destination_pod_dir.joinpath("scripts")
+            os.makedirs(destination_script_dir, exist_ok=True)
+            script_paths = get_pod_script_paths(parsed_stack, pod)
+            _copy_files_to_directory(script_paths, destination_script_dir)
+
+        if parsed_spec.is_kubernetes_deployment():
+            for configmap in parsed_spec.get_configmaps():
+                source_config_dir = resolve_config_dir(stack_name, configmap)
+                if os.path.exists(source_config_dir):
+                    destination_config_dir = target_dir.joinpath(
+                        "configmaps", configmap
+                    )
+                    copytree(
+                        source_config_dir, destination_config_dir, dirs_exist_ok=True
+                    )
+        else:
+            # TODO:
+            # This is odd - looks up config dir that matches a volume name,
+            # then copies as a mount dir?
+            # AFAICT not used by or relevant to any existing stack - roy
+
+            # TODO: We should probably only do this if the volume is marked :ro.
+            for volume_name, volume_path in parsed_spec.get_volumes().items():
+                source_config_dir = resolve_config_dir(stack_name, volume_name)
+                # Only copy if the source exists and is _not_ empty.
+                if os.path.exists(source_config_dir) and os.listdir(source_config_dir):
+                    destination_config_dir = target_dir.joinpath(volume_path)
+                    # Only copy if the destination exists and _is_ empty.
+                    if os.path.exists(destination_config_dir) and not os.listdir(
+                        destination_config_dir
+                    ):
+                        copytree(
+                            source_config_dir,
+                            destination_config_dir,
+                            dirs_exist_ok=True,
+                        )
+
+    # Copy the job files into the target dir (for Docker deployments)
+    jobs = get_job_list(parsed_stack)
+    if jobs and not parsed_spec.is_kubernetes_deployment():
+        destination_compose_jobs_dir = target_dir.joinpath("compose-jobs")
+        os.makedirs(destination_compose_jobs_dir, exist_ok=True)
+        for job in jobs:
+            job_file_path = get_job_file_path(stack_name, parsed_stack, job)
+            if job_file_path and job_file_path.exists():
+                parsed_job_file = yaml.load(open(job_file_path, "r"))
+                _fixup_pod_file(parsed_job_file, parsed_spec, destination_compose_dir)
+                with open(
+                    destination_compose_jobs_dir.joinpath(
+                        "docker-compose-%s.yml" % job
+                    ),
+                    "w",
+                ) as output_file:
+                    yaml.dump(parsed_job_file, output_file)
+                if opts.o.debug:
+                    print(f"Copied job compose file: {job}")
+
+
 # TODO: this code should be in the stack .py files but
 # we haven't yet figured out how to integrate click across
 # the plugin boundary

stack_orchestrator/deploy/dns_probe.py

@@ -0,0 +1,159 @@
+# Copyright © 2024 Vulcanize
+# SPDX-License-Identifier: AGPL-3.0
+
+"""DNS verification via temporary ingress probe."""
+
+import secrets
+import socket
+import time
+from typing import Optional
+import requests
+from kubernetes import client
+
+
+def get_server_egress_ip() -> str:
+    """Get this server's public egress IP via ipify."""
+    response = requests.get("https://api.ipify.org", timeout=10)
+    response.raise_for_status()
+    return response.text.strip()
+
+
+def resolve_hostname(hostname: str) -> list[str]:
+    """Resolve hostname to list of IP addresses."""
+    try:
+        _, _, ips = socket.gethostbyname_ex(hostname)
+        return ips
+    except socket.gaierror:
+        return []
+
+
+def verify_dns_simple(hostname: str, expected_ip: Optional[str] = None) -> bool:
+    """Simple DNS verification - check hostname resolves to expected IP.
+
+    If expected_ip not provided, uses server's egress IP.
+    Returns True if hostname resolves to expected IP.
+    """
+    resolved_ips = resolve_hostname(hostname)
+    if not resolved_ips:
+        print(f"DNS FAIL: {hostname} does not resolve")
+        return False
+
+    if expected_ip is None:
+        expected_ip = get_server_egress_ip()
+
+    if expected_ip in resolved_ips:
+        print(f"DNS OK: {hostname} -> {resolved_ips} (includes {expected_ip})")
+        return True
+    else:
+        print(f"DNS WARN: {hostname} -> {resolved_ips} (expected {expected_ip})")
+        return False
+
+
+def create_probe_ingress(hostname: str, namespace: str = "default") -> str:
+    """Create a temporary ingress for DNS probing.
+
+    Returns the probe token that the ingress will respond with.
+    """
+    token = secrets.token_hex(16)
+
+    networking_api = client.NetworkingV1Api()
+
+    # Create a simple ingress that Caddy will pick up
+    ingress = client.V1Ingress(
+        metadata=client.V1ObjectMeta(
+            name="laconic-dns-probe",
+            annotations={
+                "kubernetes.io/ingress.class": "caddy",
+                "laconic.com/probe-token": token,
+            },
+        ),
+        spec=client.V1IngressSpec(
+            rules=[
+                client.V1IngressRule(
+                    host=hostname,
+                    http=client.V1HTTPIngressRuleValue(
+                        paths=[
+                            client.V1HTTPIngressPath(
+                                path="/.well-known/laconic-probe",
+                                path_type="Exact",
+                                backend=client.V1IngressBackend(
+                                    service=client.V1IngressServiceBackend(
+                                        name="caddy-ingress-controller",
+                                        port=client.V1ServiceBackendPort(number=80),
+                                    )
+                                ),
+                            )
+                        ]
+                    ),
+                )
+            ]
+        ),
+    )
+
+    networking_api.create_namespaced_ingress(namespace=namespace, body=ingress)
+    return token
+
+
+def delete_probe_ingress(namespace: str = "default"):
+    """Delete the temporary probe ingress."""
+    networking_api = client.NetworkingV1Api()
+    try:
+        networking_api.delete_namespaced_ingress(
+            name="laconic-dns-probe", namespace=namespace
+        )
+    except client.exceptions.ApiException:
+        pass  # Ignore if already deleted
+
+
+def verify_dns_via_probe(
+    hostname: str, namespace: str = "default", timeout: int = 30, poll_interval: int = 2
+) -> bool:
+    """Verify DNS by creating temp ingress and probing it.
+
+    This definitively proves that traffic to the hostname reaches this cluster.
+
+    Args:
+        hostname: The hostname to verify
+        namespace: Kubernetes namespace for probe ingress
+        timeout: Total seconds to wait for probe to succeed
+        poll_interval: Seconds between probe attempts
+
+    Returns:
+        True if probe succeeds, False otherwise
+    """
+    # First check DNS resolves at all
+    if not resolve_hostname(hostname):
+        print(f"DNS FAIL: {hostname} does not resolve")
+        return False
+
+    print(f"Creating probe ingress for {hostname}...")
+    create_probe_ingress(hostname, namespace)
+
+    try:
+        # Wait for Caddy to pick up the ingress
+        time.sleep(3)
+
+        # Poll until success or timeout
+        probe_url = f"http://{hostname}/.well-known/laconic-probe"
+        start_time = time.time()
+        last_error = None
+
+        while time.time() - start_time < timeout:
+            try:
+                response = requests.get(probe_url, timeout=5)
+                # For now, just verify we get a response from this cluster
+                # A more robust check would verify a unique token
+                if response.status_code < 500:
+                    print(f"DNS PROBE OK: {hostname} routes to this cluster")
+                    return True
+            except requests.RequestException as e:
+                last_error = e
+
+            time.sleep(poll_interval)
+
+        print(f"DNS PROBE FAIL: {hostname} - {last_error}")
+        return False
+
+    finally:
+        print("Cleaning up probe ingress...")
+        delete_probe_ingress(namespace)

stack_orchestrator/deploy/k8s/cluster_info.py

@@ -352,11 +352,15 @@ class ClusterInfo:
                 continue
 
             if not os.path.isabs(volume_path):
-                print(
+                # For k8s-kind, allow relative paths:
-                    f"WARNING: {volume_name}:{volume_path} is not absolute, "
+                # - PV uses /mnt/{volume_name} (path inside kind node)
-                    "cannot bind volume."
+                # - extraMounts resolve the relative path to Docker Host
-                )
+                if not self.spec.is_kind_deployment():
-                continue
+                    print(
+                        f"WARNING: {volume_name}:{volume_path} is not absolute, "
+                        "cannot bind volume."
+                    )
+                    continue
 
             if self.spec.is_kind_deployment():
                 host_path = client.V1HostPathVolumeSource(

stack_orchestrator/deploy/k8s/deploy_k8s.py

@@ -301,7 +301,7 @@ class K8sDeployer(Deployer):
             self.connect_api()
             if self.is_kind() and not self.skip_cluster_management:
                 # Configure ingress controller (not installed by default in kind)
-                install_ingress_for_kind()
+                install_ingress_for_kind(self.cluster_info.spec.get_acme_email())
                 # Wait for ingress to start
                 # (deployment provisioning will fail unless this is done)
                 wait_for_ingress_in_kind()

stack_orchestrator/deploy/k8s/helpers.py

@@ -27,6 +27,48 @@ from stack_orchestrator.deploy.deployer import DeployerException
 from stack_orchestrator import constants
 
 
+def is_host_path_mount(volume_name: str) -> bool:
+    """Check if a volume name is a host path mount (starts with /, ., or ~)."""
+    return volume_name.startswith(("/", ".", "~"))
+
+
+def sanitize_host_path_to_volume_name(host_path: str) -> str:
+    """Convert a host path to a valid k8s volume name.
+
+    K8s volume names must be lowercase, alphanumeric, with - allowed.
+    E.g., '../config/test/script.sh' -> 'host-path-config-test-script-sh'
+    """
+    # Remove leading ./ or ../
+    clean_path = re.sub(r"^\.+/", "", host_path)
+    # Replace path separators and dots with hyphens
+    name = re.sub(r"[/.]", "-", clean_path)
+    # Remove any non-alphanumeric characters except hyphens
+    name = re.sub(r"[^a-zA-Z0-9-]", "", name)
+    # Convert to lowercase
+    name = name.lower()
+    # Remove leading/trailing hyphens and collapse multiple hyphens
+    name = re.sub(r"-+", "-", name).strip("-")
+    # Prefix with 'host-path-' to distinguish from named volumes
+    return f"host-path-{name}"
+
+
+def resolve_host_path_for_kind(host_path: str, deployment_dir: Path) -> Path:
+    """Resolve a host path mount (relative to compose file) to absolute path.
+
+    Compose files are in deployment_dir/compose/, so '../config/foo'
+    resolves to deployment_dir/config/foo.
+    """
+    # The path is relative to the compose directory
+    compose_dir = deployment_dir.joinpath("compose")
+    resolved = compose_dir.joinpath(host_path).resolve()
+    return resolved
+
+
+def get_kind_host_path_mount_path(sanitized_name: str) -> str:
+    """Get the path inside the kind node where a host path mount will be available."""
+    return f"/mnt/{sanitized_name}"
+
+
 def get_kind_cluster():
     """Get an existing kind cluster, if any.
 
@@ -54,7 +96,177 @@ def _run_command(command: str):
     return result
 
 
+def _get_etcd_host_path_from_kind_config(config_file: str) -> Optional[str]:
+    """Extract etcd host path from kind config extraMounts."""
+    import yaml
+
+    try:
+        with open(config_file, "r") as f:
+            config = yaml.safe_load(f)
+    except Exception:
+        return None
+
+    nodes = config.get("nodes", [])
+    for node in nodes:
+        extra_mounts = node.get("extraMounts", [])
+        for mount in extra_mounts:
+            if mount.get("containerPath") == "/var/lib/etcd":
+                return mount.get("hostPath")
+    return None
+
+
+def _clean_etcd_keeping_certs(etcd_path: str) -> bool:
+    """Clean persisted etcd, keeping only TLS certificates.
+
+    When etcd is persisted and a cluster is recreated, kind tries to install
+    resources fresh but they already exist. Instead of trying to delete
+    specific stale resources (blacklist), we keep only the valuable data
+    (caddy TLS certs) and delete everything else (whitelist approach).
+
+    The etcd image is distroless (no shell), so we extract the statically-linked
+    etcdctl binary and run it from alpine which has shell support.
+
+    Returns True if cleanup succeeded, False if no action needed or failed.
+    """
+    db_path = Path(etcd_path) / "member" / "snap" / "db"
+    # Check existence using docker since etcd dir is root-owned
+    check_cmd = (
+        f"docker run --rm -v {etcd_path}:/etcd:ro alpine:3.19 "
+        "test -f /etcd/member/snap/db"
+    )
+    check_result = subprocess.run(check_cmd, shell=True, capture_output=True)
+    if check_result.returncode != 0:
+        if opts.o.debug:
+            print(f"No etcd snapshot at {db_path}, skipping cleanup")
+        return False
+
+    if opts.o.debug:
+        print(f"Cleaning persisted etcd at {etcd_path}, keeping only TLS certs")
+
+    etcd_image = "gcr.io/etcd-development/etcd:v3.5.9"
+    temp_dir = "/tmp/laconic-etcd-cleanup"
+
+    # Whitelist: prefixes to KEEP - everything else gets deleted
+    keep_prefixes = "/registry/secrets/caddy-system"
+
+    # The etcd image is distroless (no shell). We extract the statically-linked
+    # etcdctl binary and run it from alpine which has shell + jq support.
+    cleanup_script = f"""
+        set -e
+        ALPINE_IMAGE="alpine:3.19"
+
+        # Cleanup previous runs
+        docker rm -f laconic-etcd-cleanup 2>/dev/null || true
+        docker rm -f etcd-extract 2>/dev/null || true
+        docker run --rm -v /tmp:/tmp $ALPINE_IMAGE rm -rf {temp_dir}
+
+        # Create temp dir
+        docker run --rm -v /tmp:/tmp $ALPINE_IMAGE mkdir -p {temp_dir}
+
+        # Extract etcdctl binary (it's statically linked)
+        docker create --name etcd-extract {etcd_image}
+        docker cp etcd-extract:/usr/local/bin/etcdctl /tmp/etcdctl-bin
+        docker rm etcd-extract
+        docker run --rm -v /tmp/etcdctl-bin:/src:ro -v {temp_dir}:/dst $ALPINE_IMAGE \
+            sh -c "cp /src /dst/etcdctl && chmod +x /dst/etcdctl"
+
+        # Copy db to temp location
+        docker run --rm \
+            -v {etcd_path}:/etcd:ro \
+            -v {temp_dir}:/tmp-work \
+            $ALPINE_IMAGE cp /etcd/member/snap/db /tmp-work/etcd-snapshot.db
+
+        # Restore snapshot
+        docker run --rm -v {temp_dir}:/work {etcd_image} \
+            etcdutl snapshot restore /work/etcd-snapshot.db \
+                --data-dir=/work/etcd-data --skip-hash-check 2>/dev/null
+
+        # Start temp etcd (runs the etcd binary, no shell needed)
+        docker run -d --name laconic-etcd-cleanup \
+            -v {temp_dir}/etcd-data:/etcd-data \
+            -v {temp_dir}:/backup \
+            {etcd_image} etcd \
+                --data-dir=/etcd-data \
+                --listen-client-urls=http://0.0.0.0:2379 \
+                --advertise-client-urls=http://localhost:2379
+
+        sleep 3
+
+        # Use alpine with extracted etcdctl to run commands (alpine has shell + jq)
+        # Export caddy secrets
+        docker run --rm \
+            -v {temp_dir}:/backup \
+            --network container:laconic-etcd-cleanup \
+            $ALPINE_IMAGE sh -c \
+            '/backup/etcdctl get --prefix "{keep_prefixes}" -w json \
+                > /backup/kept.json 2>/dev/null || echo "{{}}" > /backup/kept.json'
+
+        # Delete ALL registry keys
+        docker run --rm \
+            -v {temp_dir}:/backup \
+            --network container:laconic-etcd-cleanup \
+            $ALPINE_IMAGE /backup/etcdctl del --prefix /registry
+
+        # Restore kept keys using jq
+        docker run --rm \
+            -v {temp_dir}:/backup \
+            --network container:laconic-etcd-cleanup \
+            $ALPINE_IMAGE sh -c '
+                apk add --no-cache jq >/dev/null 2>&1
+                jq -r ".kvs[] | @base64" /backup/kept.json 2>/dev/null | \
+                while read encoded; do
+                    key=$(echo $encoded | base64 -d | jq -r ".key" | base64 -d)
+                    val=$(echo $encoded | base64 -d | jq -r ".value" | base64 -d)
+                    echo "$val" | /backup/etcdctl put "$key"
+                done
+            ' || true
+
+        # Save cleaned snapshot
+        docker exec laconic-etcd-cleanup \
+            etcdctl snapshot save /etcd-data/cleaned-snapshot.db
+
+        docker stop laconic-etcd-cleanup
+        docker rm laconic-etcd-cleanup
+
+        # Restore to temp location first to verify it works
+        docker run --rm \
+            -v {temp_dir}/etcd-data/cleaned-snapshot.db:/data/db:ro \
+            -v {temp_dir}:/restore \
+            {etcd_image} \
+            etcdutl snapshot restore /data/db --data-dir=/restore/new-etcd \
+            --skip-hash-check 2>/dev/null
+
+        # Create timestamped backup of original (kept forever)
+        TIMESTAMP=$(date +%Y%m%d-%H%M%S)
+        docker run --rm -v {etcd_path}:/etcd $ALPINE_IMAGE \
+            cp -a /etcd/member /etcd/member.backup-$TIMESTAMP
+
+        # Replace original with cleaned version
+        docker run --rm -v {etcd_path}:/etcd -v {temp_dir}:/tmp-work $ALPINE_IMAGE \
+            sh -c "rm -rf /etcd/member && mv /tmp-work/new-etcd/member /etcd/member"
+
+        # Cleanup temp files (but NOT the timestamped backup in etcd_path)
+        docker run --rm -v /tmp:/tmp $ALPINE_IMAGE rm -rf {temp_dir}
+        rm -f /tmp/etcdctl-bin
+    """
+
+    result = subprocess.run(cleanup_script, shell=True, capture_output=True, text=True)
+    if result.returncode != 0:
+        if opts.o.debug:
+            print(f"Warning: etcd cleanup failed: {result.stderr}")
+        return False
+
+    if opts.o.debug:
+        print("Cleaned etcd, kept only TLS certificates")
+    return True
+
+
 def create_cluster(name: str, config_file: str):
+    # Clean persisted etcd, keeping only TLS certificates
+    etcd_path = _get_etcd_host_path_from_kind_config(config_file)
+    if etcd_path:
+        _clean_etcd_keeping_certs(etcd_path)
+
     result = _run_command(f"kind create cluster --name {name} --config {config_file}")
     if result.returncode != 0:
         raise DeployerException(f"kind create cluster failed: {result}")
@@ -90,7 +302,7 @@ def wait_for_ingress_in_kind():
     error_exit("ERROR: Timed out waiting for Caddy ingress to become ready")
 
 
-def install_ingress_for_kind():
+def install_ingress_for_kind(acme_email: str = ""):
     api_client = client.ApiClient()
     ingress_install = os.path.abspath(
         get_k8s_dir().joinpath(
@@ -101,6 +313,21 @@ def install_ingress_for_kind():
         print("Installing Caddy ingress controller in kind cluster")
     utils.create_from_yaml(api_client, yaml_file=ingress_install)
 
+    # Patch ConfigMap with acme email if provided
+    if acme_email:
+        core_v1 = client.CoreV1Api()
+        configmap = core_v1.read_namespaced_config_map(
+            name="caddy-ingress-controller-configmap", namespace="caddy-system"
+        )
+        configmap.data["email"] = acme_email
+        core_v1.patch_namespaced_config_map(
+            name="caddy-ingress-controller-configmap",
+            namespace="caddy-system",
+            body=configmap,
+        )
+        if opts.o.debug:
+            print(f"Patched Caddy ConfigMap with email: {acme_email}")
+
 
 def load_images_into_kind(kind_cluster_name: str, image_set: Set[str]):
     for image in image_set:
@@ -177,6 +404,7 @@ def volume_mounts_for_service(parsed_pod_files, service):
                         for mount_string in volumes:
                             # Looks like: test-data:/data
                             # or test-data:/data:ro or test-data:/data:rw
+                            # or ../config/file.sh:/opt/file.sh (host path mount)
                             if opts.o.debug:
                                 print(f"mount_string: {mount_string}")
                             mount_split = mount_string.split(":")
@@ -185,13 +413,21 @@ def volume_mounts_for_service(parsed_pod_files, service):
                             mount_options = (
                                 mount_split[2] if len(mount_split) == 3 else None
                             )
+                            # For host path mounts, use sanitized name
+                            if is_host_path_mount(volume_name):
+                                k8s_volume_name = sanitize_host_path_to_volume_name(
+                                    volume_name
+                                )
+                            else:
+                                k8s_volume_name = volume_name
                             if opts.o.debug:
                                 print(f"volume_name: {volume_name}")
+                                print(f"k8s_volume_name: {k8s_volume_name}")
                                 print(f"mount path: {mount_path}")
                                 print(f"mount options: {mount_options}")
                             volume_device = client.V1VolumeMount(
                                 mount_path=mount_path,
-                                name=volume_name,
+                                name=k8s_volume_name,
                                 read_only="ro" == mount_options,
                             )
                             result.append(volume_device)
@@ -200,8 +436,12 @@ def volume_mounts_for_service(parsed_pod_files, service):
 
 def volumes_for_pod_files(parsed_pod_files, spec, app_name):
     result = []
+    seen_host_path_volumes = set()  # Track host path volumes to avoid duplicates
+
     for pod in parsed_pod_files:
         parsed_pod_file = parsed_pod_files[pod]
+
+        # Handle named volumes from top-level volumes section
         if "volumes" in parsed_pod_file:
             volumes = parsed_pod_file["volumes"]
             for volume_name in volumes.keys():
@@ -220,6 +460,35 @@ def volumes_for_pod_files(parsed_pod_files, spec, app_name):
                         name=volume_name, persistent_volume_claim=claim
                     )
                     result.append(volume)
+
+        # Handle host path mounts from service volumes
+        if "services" in parsed_pod_file:
+            services = parsed_pod_file["services"]
+            for service_name in services:
+                service_obj = services[service_name]
+                if "volumes" in service_obj:
+                    for mount_string in service_obj["volumes"]:
+                        mount_split = mount_string.split(":")
+                        volume_source = mount_split[0]
+                        if is_host_path_mount(volume_source):
+                            sanitized_name = sanitize_host_path_to_volume_name(
+                                volume_source
+                            )
+                            if sanitized_name not in seen_host_path_volumes:
+                                seen_host_path_volumes.add(sanitized_name)
+                                # Create hostPath volume for mount inside kind node
+                                kind_mount_path = get_kind_host_path_mount_path(
+                                    sanitized_name
+                                )
+                                host_path_source = client.V1HostPathVolumeSource(
+                                    path=kind_mount_path, type="FileOrCreate"
+                                )
+                                volume = client.V1Volume(
+                                    name=sanitized_name, host_path=host_path_source
+                                )
+                                result.append(volume)
+                                if opts.o.debug:
+                                    print(f"Created hostPath volume: {sanitized_name}")
     return result
 
 
@@ -238,6 +507,8 @@ def _make_absolute_host_path(data_mount_path: Path, deployment_dir: Path) -> Pat
 def _generate_kind_mounts(parsed_pod_files, deployment_dir, deployment_context):
     volume_definitions = []
     volume_host_path_map = _get_host_paths_for_volumes(deployment_context)
+    seen_host_path_mounts = set()  # Track to avoid duplicate mounts
+
     # Note these paths are relative to the location of the pod files (at present)
     # So we need to fix up to make them correct and absolute because kind assumes
     # relative to the cwd.
@@ -252,28 +523,58 @@ def _generate_kind_mounts(parsed_pod_files, deployment_dir, deployment_context):
                     for mount_string in volumes:
                         # Looks like: test-data:/data
                         # or test-data:/data:ro or test-data:/data:rw
+                        # or ../config/file.sh:/opt/file.sh (host path mount)
                         if opts.o.debug:
                             print(f"mount_string: {mount_string}")
                         mount_split = mount_string.split(":")
                         volume_name = mount_split[0]
                         mount_path = mount_split[1]
-                        if opts.o.debug:
+
-                            print(f"volume_name: {volume_name}")
+                        if is_host_path_mount(volume_name):
-                            print(f"map: {volume_host_path_map}")
+                            # Host path mount - add extraMount for kind
-                            print(f"mount path: {mount_path}")
+                            sanitized_name = sanitize_host_path_to_volume_name(
-                        if volume_name not in deployment_context.spec.get_configmaps():
+                                volume_name
-                            if volume_host_path_map[volume_name]:
+                            )
-                                host_path = _make_absolute_host_path(
+                            if sanitized_name not in seen_host_path_mounts:
-                                    volume_host_path_map[volume_name],
+                                seen_host_path_mounts.add(sanitized_name)
-                                    deployment_dir,
+                                # Resolve path relative to compose directory
+                                host_path = resolve_host_path_for_kind(
+                                    volume_name, deployment_dir
                                 )
-                                container_path = get_kind_pv_bind_mount_path(
+                                container_path = get_kind_host_path_mount_path(
-                                    volume_name
+                                    sanitized_name
                                 )
                                 volume_definitions.append(
                                     f"  - hostPath: {host_path}\n"
                                     f"    containerPath: {container_path}\n"
                                 )
+                                if opts.o.debug:
+                                    print(f"Added host path mount: {host_path}")
+                        else:
+                            # Named volume
+                            if opts.o.debug:
+                                print(f"volume_name: {volume_name}")
+                                print(f"map: {volume_host_path_map}")
+                                print(f"mount path: {mount_path}")
+                            if (
+                                volume_name
+                                not in deployment_context.spec.get_configmaps()
+                            ):
+                                if (
+                                    volume_name in volume_host_path_map
+                                    and volume_host_path_map[volume_name]
+                                ):
+                                    host_path = _make_absolute_host_path(
+                                        volume_host_path_map[volume_name],
+                                        deployment_dir,
+                                    )
+                                    container_path = get_kind_pv_bind_mount_path(
+                                        volume_name
+                                    )
+                                    volume_definitions.append(
+                                        f"  - hostPath: {host_path}\n"
+                                        f"    containerPath: {container_path}\n"
+                                    )
     return (
         ""
         if len(volume_definitions) == 0

stack_orchestrator/deploy/spec.py

@@ -179,6 +179,9 @@ class Spec:
     def get_deployment_type(self):
         return self.obj.get(constants.deploy_to_key)
 
+    def get_acme_email(self):
+        return self.obj.get(constants.network_key, {}).get(constants.acme_email_key, "")
+
     def is_kubernetes_deployment(self):
         return self.get_deployment_type() in [
             constants.k8s_kind_deploy_type,

stack_orchestrator/deploy/webapp/deploy_webapp.py

@@ -94,7 +94,7 @@ def create_deployment(
     # Add the TLS and DNS spec
     _fixup_url_spec(spec_file_name, url)
     create_operation(
-        deploy_command_context, spec_file_name, deployment_dir, False, None, None
+        deploy_command_context, spec_file_name, deployment_dir, False, False, None, None
     )
     # Fix up the container tag inside the deployment compose file
     _fixup_container_tag(deployment_dir, image)

tests/database/run-test.sh

@@ -86,7 +86,7 @@ fi
 echo "deploy init test: passed"
 
 # Switch to a full path for the data dir so it gets provisioned as a host bind mounted volume and preserved beyond cluster lifetime
-sed -i "s|^\(\s*db-data:$\)$|\1 ${test_deployment_dir}/data/db-data|" $test_deployment_spec
+sed -i.bak "s|^\(\s*db-data:$\)$|\1 ${test_deployment_dir}/data/db-data|" $test_deployment_spec
 
 $TEST_TARGET_SO --stack ${stack} deploy create --spec-file $test_deployment_spec --deployment-dir $test_deployment_dir
 # Check the deployment dir exists

tests/deploy/run-deploy-test.sh

@@ -34,6 +34,7 @@ mkdir -p $CERC_REPO_BASE_DIR
 # with and without volume removal
 $TEST_TARGET_SO --stack test setup-repositories
 $TEST_TARGET_SO --stack test build-containers
+
 # Test deploy command execution
 $TEST_TARGET_SO --stack test deploy setup $CERC_REPO_BASE_DIR
 # Check that we now have the expected output directory
@@ -85,6 +86,7 @@ else
     exit 1
 fi
 $TEST_TARGET_SO --stack test deploy down --delete-volumes
+
 # Basic test of creating a deployment
 test_deployment_dir=$CERC_REPO_BASE_DIR/test-deployment-dir
 test_deployment_spec=$CERC_REPO_BASE_DIR/test-deployment-spec.yml
@@ -122,6 +124,101 @@ fi
 echo "dbfc7a4d-44a7-416d-b5f3-29842cc47650" > $test_deployment_dir/data/test-config/test_config
 
 echo "deploy create output file test: passed"
+
+# Test sync functionality: update deployment without destroying data
+# First, create a marker file in the data directory to verify it's preserved
+test_data_marker="$test_deployment_dir/data/test-data-bind/sync-test-marker.txt"
+echo "original-data-$(date +%s)" > "$test_data_marker"
+original_marker_content=$(<$test_data_marker)
+
+# Modify a config file in the deployment to differ from source (to test backup)
+test_config_file="$test_deployment_dir/config/test/settings.env"
+test_config_file_original_content=$(<$test_config_file)
+test_config_file_changed_content="ANSWER=69"
+echo "$test_config_file_changed_content" > "$test_config_file"
+
+# Check a config file that matches the source (to test no backup for unchanged files)
+test_unchanged_config="$test_deployment_dir/config/test/script.sh"
+
+# Modify spec file to simulate an update
+sed -i.bak 's/CERC_TEST_PARAM_3:/CERC_TEST_PARAM_3: FASTER/' $test_deployment_spec
+
+# Create/modify config.env to test it isn't overwritten during sync
+config_env_file="$test_deployment_dir/config.env"
+config_env_persistent_content="PERSISTENT_VALUE=should-not-be-overwritten-$(date +%s)"
+echo "$config_env_persistent_content" >> "$config_env_file"
+original_config_env_content=$(<$config_env_file)
+
+# Run sync to update deployment files without destroying data
+$TEST_TARGET_SO --stack test deploy create --spec-file $test_deployment_spec --deployment-dir $test_deployment_dir --update
+
+# Verify config.env was not overwritten
+synced_config_env_content=$(<$config_env_file)
+if [ "$synced_config_env_content" == "$original_config_env_content" ]; then
+    echo "deployment update test: config.env preserved - passed"
+else
+    echo "deployment update test: config.env was overwritten - FAILED"
+    echo "Expected: $original_config_env_content"
+    echo "Got: $synced_config_env_content"
+    exit 1
+fi
+
+# Verify the spec file was updated in deployment dir
+updated_deployed_spec=$(<$test_deployment_dir/spec.yml)
+if [[ "$updated_deployed_spec" == *"FASTER"* ]]; then
+    echo "deployment update test: spec file updated"
+else
+    echo "deployment update test: spec file not updated - FAILED"
+    exit 1
+fi
+
+# Verify changed config file was backed up
+test_config_backup="${test_config_file}.bak"
+if [ -f "$test_config_backup" ]; then
+    backup_content=$(<$test_config_backup)
+    if [ "$backup_content" == "$test_config_file_changed_content" ]; then
+        echo "deployment update test: changed config file backed up - passed"
+    else
+        echo "deployment update test: backup content incorrect - FAILED"
+        exit 1
+    fi
+else
+    echo "deployment update test: backup file not created for changed file - FAILED"
+    exit 1
+fi
+
+# Verify unchanged config file was NOT backed up
+test_unchanged_backup="$test_unchanged_config.bak"
+if [ -f "$test_unchanged_backup" ]; then
+    echo "deployment update test: backup created for unchanged file - FAILED"
+    exit 1
+else
+    echo "deployment update test: no backup for unchanged file - passed"
+fi
+
+# Verify the config file was updated from source
+updated_config_content=$(<$test_config_file)
+if [ "$updated_config_content" == "$test_config_file_original_content" ]; then
+    echo "deployment update test: config file updated from source - passed"
+else
+    echo "deployment update test: config file not updated correctly - FAILED"
+    exit 1
+fi
+
+# Verify the data marker file still exists with original content
+if [ ! -f "$test_data_marker" ]; then
+    echo "deployment update test: data file deleted - FAILED"
+    exit 1
+fi
+synced_marker_content=$(<$test_data_marker)
+if [ "$synced_marker_content" == "$original_marker_content" ]; then
+    echo "deployment update test: data preserved - passed"
+else
+    echo "deployment update test: data corrupted - FAILED"
+    exit 1
+fi
+echo "deployment update test: passed"
+
 # Try to start the deployment
 $TEST_TARGET_SO deployment --dir $test_deployment_dir start
 # Check logs command works

tests/external-stack/run-test.sh

@@ -125,6 +125,49 @@ fi
 echo "dbfc7a4d-44a7-416d-b5f3-29842cc47650" > $test_deployment_dir/data/test-config/test_config
 
 echo "deploy create output file test: passed"
+
+# Test sync functionality: update deployment without destroying data
+# First, create a marker file in the data directory to verify it's preserved
+test_data_marker="$test_deployment_dir/data/test-data/sync-test-marker.txt"
+mkdir -p "$test_deployment_dir/data/test-data"
+echo "external-stack-data-$(date +%s)" > "$test_data_marker"
+original_marker_content=$(<$test_data_marker)
+# Verify deployment file exists and preserve its cluster ID
+original_cluster_id=$(grep "cluster-id:" "$test_deployment_dir/deployment.yml" 2>/dev/null || echo "")
+# Modify spec file to simulate an update
+sed -i.bak 's/CERC_TEST_PARAM_1=PASSED/CERC_TEST_PARAM_1=UPDATED/' $test_deployment_spec
+# Run sync to update deployment files without destroying data
+$TEST_TARGET_SO_STACK deploy create --spec-file $test_deployment_spec --deployment-dir $test_deployment_dir --update
+# Verify the spec file was updated in deployment dir
+updated_deployed_spec=$(<$test_deployment_dir/spec.yml)
+if [[ "$updated_deployed_spec" == *"UPDATED"* ]]; then
+    echo "deploy sync test: spec file updated"
+else
+    echo "deploy sync test: spec file not updated - FAILED"
+    exit 1
+fi
+# Verify the data marker file still exists with original content
+if [ ! -f "$test_data_marker" ]; then
+    echo "deploy sync test: data file deleted - FAILED"
+    exit 1
+fi
+synced_marker_content=$(<$test_data_marker)
+if [ "$synced_marker_content" == "$original_marker_content" ]; then
+    echo "deploy sync test: data preserved - passed"
+else
+    echo "deploy sync test: data corrupted - FAILED"
+    exit 1
+fi
+# Verify cluster ID was preserved (not regenerated)
+new_cluster_id=$(grep "cluster-id:" "$test_deployment_dir/deployment.yml" 2>/dev/null || echo "")
+if [ -n "$original_cluster_id" ] && [ "$original_cluster_id" == "$new_cluster_id" ]; then
+    echo "deploy sync test: cluster ID preserved - passed"
+else
+    echo "deploy sync test: cluster ID not preserved - FAILED"
+    exit 1
+fi
+echo "deploy sync test: passed"
+
 # Try to start the deployment
 $TEST_TARGET_SO deployment --dir $test_deployment_dir start
 # Check logs command works