feat(k8s): ACME email fix, etcd persistence, volume paths #986

2026-02-03T22:29:02Z

AFDudley commented

2026-02-03 22:29:02 +00:00

Summary

Fix Caddy ACME email race condition by templating YAML directly
Add etcd + PKI extraMounts for certificate persistence across cluster recreation
Allow relative volume paths for k8s-kind deployments
Add idempotency check for ingress installation (skip if already running)

Key Changes

helpers.py: Added is_ingress_running() check, etcd/PKI extraMounts
deploy_k8s.py: Skip ingress install if already running
spec.py: ACME email getter using constants
cluster_info.py: Relative volume path support
docs/deployment_patterns.md: Volume persistence and registry auth docs

## Summary - Fix Caddy ACME email race condition by templating YAML directly - Add etcd + PKI extraMounts for certificate persistence across cluster recreation - Allow relative volume paths for k8s-kind deployments - Add idempotency check for ingress installation (skip if already running) ## Key Changes - `helpers.py`: Added `is_ingress_running()` check, etcd/PKI extraMounts - `deploy_k8s.py`: Skip ingress install if already running - `spec.py`: ACME email getter using constants - `cluster_info.py`: Relative volume path support - `docs/deployment_patterns.md`: Volume persistence and registry auth docs

AFDudley added 27 commits 2026-02-03 22:29:03 +00:00

docs(CLAUDE.md): add external stacks preferred guideline 4713107546

Document that external stack pattern should be used when creating new
stacks for any reason, with directory structure and usage examples.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

feat(deploy): add deployment restart command c197406cc7

Add `laconic-so deployment restart` command that:
- Pulls latest code from stack git repository
- Regenerates spec.yml from stack's commands.py
- Verifies DNS if hostname changed (with --force to skip)
- Syncs deployment directory preserving cluster ID and data
- Stops and restarts deployment with --skip-cluster-management

Also stores stack-source path in deployment.yml during create
for automatic stack location on restart.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Fix Caddy ingress ACME email and RBAC issues 8d3191e4fd

- Add acme_email_key constant for spec.yml parsing
- Add get_acme_email() method to Spec class
- Modify install_ingress_for_kind() to patch ConfigMap with email
- Pass acme-email from spec to ingress installation
- Add 'delete' verb to leases RBAC for certificate lock cleanup

The acme-email field in spec.yml was previously ignored, causing
Let's Encrypt to fail with "unable to parse email address".
The missing delete permission on leases caused lock cleanup failures.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Clear stale CNI resources from persisted etcd before cluster creation 675ee87544

When etcd is persisted (for certificate backup) and a cluster is
recreated, kind tries to install CNI (kindnet) fresh but the
persisted etcd already has those resources, causing 'AlreadyExists'
errors and cluster creation failure.

This fix:
- Detects etcd mount path from kind config
- Before cluster creation, clears stale CNI resources (kindnet, coredns)
- Preserves certificate and other important data

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Fix etcd cleanup to use docker for root-owned files 8948f5bfec

Use docker containers with volume mounts to handle all file
operations on root-owned etcd directories, avoiding the need
for sudo on the host.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Use whitelist approach for etcd cleanup 5b06cffe17

Instead of trying to delete specific stale resources (blacklist),
keep only the valuable data (caddy TLS certs) and delete everything
else. This is more robust as we don't need to maintain a list of
all possible stale resources.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Preserve original etcd backup until restore is verified 720e01fc75

Move original to .bak, move new into place, then delete bak.
If anything fails before the swap, original remains intact.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Keep timestamped backup of etcd forever e2d3c44321

Create member.backup-YYYYMMDD-HHMMSS before cleaning.
Each cluster recreation creates a new backup, preserving history.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Use docker for etcd existence check (root-owned dir) 3fbd854b8c

The etcd directory is root-owned, so shell test -f fails.
Use docker with volume mount to check file existence.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Fix restart command for GitOps deployments 14258500bc

- Remove init_operation() from restart - don't regenerate spec from
  commands.py defaults, use existing git-tracked spec.yml instead
- Add docs/deployment_patterns.md documenting GitOps workflow
- Add pre-commit rule to CLAUDE.md
- Fix line length issues in helpers.py

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Add --spec-file option to restart and auto-detect GitOps spec 22d64f1e97

- Add --spec-file option to specify spec location in repo
- Auto-detect deployment/spec.yml in repo as GitOps location
- Fall back to deployment dir if no repo spec found

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Fix repo root path calculation (4 parents from stack path) 4408725b08

Add cluster reuse for multi-stack k8s-kind deployments 2d3721efa4

When deploying a second stack to k8s-kind, automatically reuse an existing
kind cluster instead of trying to create a new one (which would fail due
to port 80/443 conflicts).

Changes:
- helpers.py: create_cluster() now checks for existing cluster first
- deploy_k8s.py: up() captures returned cluster name and updates self

This enables deploying multiple stacks (e.g., gorbagana-rpc + trashscan-explorer)
to the same kind cluster.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Add $generate:type:length$ token support for K8s secrets ca090d2cd5

- Add GENERATE_TOKEN_PATTERN to detect $generate:hex:N$ and $generate:base64:N$ tokens
- Add _generate_and_store_secrets() to create K8s Secrets from spec.yml config
- Modify _write_config_file() to separate secrets from regular config
- Add env_from with secretRef to container spec in cluster_info.py
- Secrets are injected directly into containers via K8s native mechanism

This enables declarative secret generation in spec.yml:
  config:
    SESSION_SECRET: $generate:hex:32$
    DB_PASSWORD: $generate:hex:16$

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Clarify create_cluster docstring: one cluster per host by design b057969ddd

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Add k8s-kind architecture documentation to CLAUDE.md d292e7c48d

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Split documentation: README for users, CLAUDE.md for agents 1128c95969

README.md: deployment types, external stacks, commands, spec.yml reference
CLAUDE.md: implementation details, code locations, codebase navigation

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Add setup-repositories to key files list a75138093b

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Fix deployment name extraction from path 3bc7832d8c

When stack: field in spec.yml contains a path (e.g., stack_orchestrator/data/stacks/name),
extract just the final name component for K8s secret naming. K8s resource names must
be valid RFC 1123 subdomains and cannot contain slashes.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Only load locally-built images into kind, auto-detect ingress d82b3fb881

- Check stack.yml containers: field to determine which images are local builds
- Only load local images via kind load; let k8s pull registry images directly
- Add is_ingress_running() to skip ingress installation if already running
- Fixes deployment failures when public registry images aren't in local Docker

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Add private registry authentication support 73ba13aaa5

Add ability to configure private container registry credentials in spec.yml
for deployments using images from registries like GHCR.

- Add get_image_registry_config() to spec.py for parsing image-registry config
- Add create_registry_secret() to create K8s docker-registry secrets
- Update cluster_info.py to use dynamic {deployment}-registry secret names
- Update deploy_k8s.py to create registry secret before deployment
- Document feature in deployment_patterns.md

The token-env pattern keeps credentials out of git - the spec references an
environment variable name, and the actual token is passed at runtime.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Rename image-registry to registry-credentials to avoid collision cb6fdb77a6

The existing 'image-registry' key is used for pushing images to a remote
registry (URL string). Rename the new auth config to 'registry-credentials'
to avoid collision.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Fix Caddy ACME email race condition by templating YAML 7cecf2caa6

Previously, install_ingress_for_kind() applied the YAML (which starts
the Caddy pod with email: ""), then patched the ConfigMap afterward.
The pod had already read the empty email and Caddy doesn't hot-reload.

Now template the email into the YAML before applying, so the pod starts
with the correct email from the beginning.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

docs: Add cluster and volume management section 581ceaea94

Document that:
- Volumes persist across cluster deletion by design
- Only use --delete-volumes when explicitly requested
- Multiple deployments share one kind cluster
- Use --skip-cluster-management to stop single deployment

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Allow relative volume paths for k8s-kind deployments ee59918082

For k8s-kind, relative paths (e.g., ./data/rpc-config) are resolved to
$DEPLOYMENT_DIR/path by _make_absolute_host_path() during kind config
generation. This provides Docker Host persistence that survives cluster
restarts.

Previously, validation threw an exception before paths could be resolved,
making it impossible to use relative paths for persistent storage.

Changes:
- deployment_create.py: Skip relative path check for k8s-kind
- cluster_info.py: Allow relative paths to reach PV generation
- docs/deployment_patterns.md: Document volume persistence patterns

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

feat(k8s): support acme-email config for Caddy ingress 5bc6c978ac

Adds support for configuring ACME email for Let's Encrypt certificates
in kind deployments. The email can be specified in the spec under
network.acme-email and will be used to configure the Caddy ingress
controller ConfigMap.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Add etcd + PKI extraMounts for offline data recovery

Lint Checks / Run linter (push) Failing after 13s

Details

Lint Checks / Run linter (pull_request) Failing after 16s

Details

Deploy Test / Run deploy test suite (pull_request) Successful in 2m18s

Details

K8s Deployment Control Test / Run deployment control suite on kind/k8s (pull_request) Failing after 2m43s

Details

K8s Deploy Test / Run deploy test suite on kind/k8s (pull_request) Successful in 3m31s

Details

Smoke Test / Run basic test suite (pull_request) Successful in 4m8s

Details

Webapp Test / Run webapp test suite (pull_request) Successful in 4m21s

Details

f70e87b848

Mount /var/lib/etcd and /etc/kubernetes/pki to host filesystem
so cluster state is preserved for offline recovery. Each deployment
gets its own backup directory keyed by deployment ID.

Directory structure:
  data/cluster-backups/{deployment_id}/etcd/
  data/cluster-backups/{deployment_id}/pki/

This enables extracting secrets from etcd backups using etcdctl
with the preserved PKI certificates.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

AFDudley merged commit 21d47908cc into main

2026-02-03 22:31:48 +00:00

AFDudley deleted branch fix-caddy-acme-email-rbac

2026-02-03 22:31:48 +00:00

AFDudley referenced this issue from a commit

2026-02-03 22:31:49 +00:00

Merge pull request 'feat(k8s): ACME email fix, etcd persistence, volume paths' (#986) from fix-caddy-acme-email-rbac into main

This repo is archived. You cannot comment on pull requests.

1 Participants

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: cerc-io/stack-orchestrator#986