fix(k8s): persist Caddy TLS certificates with PVC #981

Open

AFDudley wants to merge 3 commits from caddy-pvc-persistence into main

pull from: caddy-pvc-persistence

merge into: cerc-io:main

cerc-io:main

cerc-io:multi-port-service

cerc-io:helm-charts-with-caddy

cerc-io:afd-caddy-ingress

cerc-io:afd

cerc-io:feat/trashscan-explorer-stack

cerc-io:helm-charts-support

cerc-io:vaasl-deploy

cerc-io:roysc/deploy-create-extra-args

cerc-io:zach/atom-payments

cerc-io:zramsay-patch-1

cerc-io:roysc/deploy-create-pass-cluster

cerc-io:zach/update-url

cerc-io:telackey/defaultplatform

cerc-io:telackey/924

cerc-io:telackey/laconicdv1

cerc-io:zach/pin-cli-version

cerc-io:telackey/wagit

cerc-io:add-vega-stack

cerc-io:blast-stack

cerc-io:lotus-stack

cerc-io:roysc/fix-eth-stacks

cerc-io:telackey/na

cerc-io:telackey/fqdn

cerc-io:zach/snowdocs

cerc-io:zach/fixturenet-2d

cerc-io:telackey/wild

cerc-io:dboreham/mobymask-v3-demo-test

cerc-io:zach/fix-for-mars

cerc-io:ci-test

cerc-io:optimism-fix

cerc-io:telackey/envsubst

cerc-io:dboreham/laconicd-k8s

cerc-io:zach/birbit

cerc-io:osmosis

cerc-io:iskay/update-optimism

cerc-io:iskay/plugeth-test-update

cerc-io:iskay/fixturenet-payments-test

cerc-io:iskay/fixturenet-laconicd-test

cerc-io:iskay/fixturenet-eth-test

cerc-io:new-gitea-test

cerc-io:erc20-fix

cerc-io:update-uniswap

cerc-io:tel/1.20

cerc-io:telackey/systest

cerc-io:ng-deny-multiaddr

cerc-io:publish-test

cerc-io:telackey/datanet

Conversation 0 Commits 3 Files Changed 4 +45 -3

AFDudley commented 2026-01-25 05:30:33 +00:00

Owner

Copy Link

Caddy ingress was using emptyDir for /data storage, causing TLS
certificates to be lost on pod restarts or cluster recreations.
This led to Let's Encrypt rate limit issues from repeatedly
requesting new certificates.

Add a PersistentVolumeClaim for Caddy's data directory to persist
ACME certificates across redeployments.

Co-Authored-By: Claude Opus 4.5 noreply@anthropic.com

Caddy ingress was using emptyDir for /data storage, causing TLS certificates to be lost on pod restarts or cluster recreations. This led to Let's Encrypt rate limit issues from repeatedly requesting new certificates. Add a PersistentVolumeClaim for Caddy's data directory to persist ACME certificates across redeployments. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

AFDudley added 1 commit 2026-01-25 05:30:33 +00:00

fix(k8s): persist Caddy TLS certificates with PVC ... d5e1a6652c

Caddy ingress was using emptyDir for /data storage, causing TLS
certificates to be lost on pod restarts or cluster recreations.
This led to Let's Encrypt rate limit issues from repeatedly
requesting new certificates.

Add a PersistentVolumeClaim for Caddy's data directory to persist
ACME certificates across redeployments.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

AFDudley added 1 commit 2026-01-25 22:19:52 +00:00

feat(k8s): enable relative volume paths for kind deployments ... 79b7870a6a

Makes kind deployments use the same volume pattern as Docker Compose:
./data/{volume-name} relative to deployment directory.

Changes:
- Allow relative paths for kind (single host, like Docker Compose)
- Default kind volumes to ./data/ instead of provisioner-managed PVCs
- Update Caddy manifest to use hostPath /mnt/caddy-data
- Add caddy-data infrastructure volume support in kind mounts

This enables Caddy certificate persistence across cluster recreation
without requiring system-level directories like /opt/caddy-data.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

AFDudley added 1 commit 2026-01-25 22:28:07 +00:00

fix(deploy): merge volumes from stack init() instead of overwriting ... 6ff3da76ee

Previously, volumes defined in a stack's commands.py init() function
were being overwritten by volumes discovered from compose files.
This prevented stacks from adding infrastructure volumes like caddy-data
that aren't defined in the compose files.

Now volumes are merged, with init() volumes taking precedence over
compose-discovered defaults.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

AFDudley added 1 commit 2026-01-25 22:36:16 +00:00

feat(k8s): support acme-email config for Caddy ingress ... 1eed7987e2

Adds support for configuring ACME email for Let's Encrypt certificates
in kind deployments. The email can be specified in the spec under
network.acme-email and will be used to configure the Caddy ingress
controller ConfigMap.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

AFDudley added 1 commit 2026-01-25 22:40:26 +00:00

fix(k8s): allow relative paths for kind deployment PVs ... b04ce627f1

Kind deployments can use relative paths because the extraMounts config
resolves them to absolute paths on the host. The PV creation should not
skip relative paths for kind deployments since the PV will use the
/mnt/<volume> path inside the kind node.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

AFDudley added 1 commit 2026-01-25 22:44:37 +00:00

fix(deploy): create ConfigMaps for kind deployments ... 9f732c2226

Volumes with "config" in the name should be ConfigMaps for all k8s
deployment types, including k8s-kind. Previously only full k8s
deployments would create ConfigMaps for these volumes.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

AFDudley added 1 commit 2026-01-26 00:13:03 +00:00

Add etcd + PKI extraMounts for offline data recovery ... 3ff4ac6185

Mount /var/lib/etcd and /etc/kubernetes/pki to host filesystem
so cluster state is preserved for offline recovery. Each deployment
gets its own backup directory keyed by deployment ID.

Directory structure:
  data/cluster-backups/{deployment_id}/etcd/
  data/cluster-backups/{deployment_id}/pki/

This enables extracting secrets from etcd backups using etcdctl
with the preserved PKI certificates.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

AFDudley force-pushed caddy-pvc-persistence from 3ff4ac6185 to 1b9204da98 2026-01-27 03:04:00 +00:00 Compare

Some checks failed

Hide all checks

Lint Checks / Run linter (push) Successful in 4m35s

Details

Lint Checks / Run linter (pull_request) Successful in 7m46s

Details

Deploy Test / Run deploy test suite (pull_request) Successful in 13m51s

Details

K8s Deploy Test / Run deploy test suite on kind/k8s (pull_request) Failing after 18m29s

Details

K8s Deployment Control Test / Run deployment control suite on kind/k8s (pull_request) Successful in 21m22s

Details

Webapp Test / Run webapp test suite (pull_request) Successful in 24m48s

Details

Smoke Test / Run basic test suite (pull_request) Successful in 25m23s

Details

This pull request doesn't have enough approvals yet. 0 of 1 approvals granted.

You are not authorized to merge this pull request.

View command line instructions.

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin caddy-pvc-persistence:caddy-pvc-persistence

git checkout caddy-pvc-persistence

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

bug

Something isn't working

  

documentation

Improvements or additions to documentation

  

duplicate

This issue or pull request already exists

  

enhancement

New feature or request

  

feature

  

good first issue

Good for newcomers

  

help wanted

Extra attention is needed

  

in progress

  

invalid

This doesn't seem right

  

question

Further information is requested

  

wontfix

This will not be worked on

  

Copied from Github

An issue or PR manually copied from GitHub.

  

Kind/Breaking

Breaking change that won't be backward compatible

  

Kind/Bug

Something is not working

  

Kind/Documentation

Documentation changes

  

Kind/Enhancement

Improve existing functionality

  

Kind/Feature

New functionality

  

Kind/Security

This is security issue

  

Kind/Testing

Issue or pull request related to testing

  

Priority

Critical

The priority is critical

  

Priority

High

The priority is high

  

Priority

Low

The priority is low

  

Priority

Medium

The priority is medium

  

Reviewed

Confirmed

Issue has been confirmed

  

Reviewed

Duplicate

This issue or pull request already exists

  

Reviewed

Invalid

Invalid issue

  

Reviewed

Won't Fix

This issue won't be fixed

  

Status

Abandoned

Somebody has started to work on this but abandoned work

  

Status

Blocked

Something is blocking this issue or pull request

  

Status

Need More Info

Feedback is required to reproduce issue or to continue work

No Label

bug

documentation

duplicate

enhancement

feature

good first issue

help wanted

in progress

invalid

question

wontfix

Copied from Github

Kind/Breaking

Kind/Bug

Kind/Documentation

Kind/Enhancement

Kind/Feature

Kind/Security

Kind/Testing

Priority

Critical

Priority

High

Priority

Low

Priority

Medium

Reviewed

Confirmed

Reviewed

Duplicate

Reviewed

Invalid

Reviewed

Won't Fix

Status

Abandoned

Status

Blocked

Status

Need More Info

Milestone

Clear milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

No Assignees

1 Participants

Notifications

Subscribe

Due Date

The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: cerc-io/stack-orchestrator#981

No description provided.