forked from cerc-io/stack-orchestrator
feat: layer 4 invariants, mount checks, and deployment layer docs
- Rename biscayne-boot.yml → biscayne-prepare-agave.yml (layer 4) - Document deployment layers and layer 4 invariants in playbook header - Add zvol, ramdisk, rbind fstab management with stale entry cleanup - Add kind node XFS verification (reads cluster-id from deployment) - Add mount checks to health-check.yml (host mounts, kind visibility, propagation) - Fix health-check discovery tasks with tags: [always] and non-fatal pod lookup - Fix biscayne-redeploy.yml shell tasks missing executable: /bin/bash - Add ansible_python_interpreter to inventory - Update CLAUDE.md with deployment layers table and mount propagation notes Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
1 parent
b40883ef65
commit
14c0f63775
6 files changed
+342
-123
No files matched your search
@@ -1,5 +1,30 @@
|
||||
# Biscayne Agave Runbook
|
||||
|
||||
## Deployment Layers
|
||||
|
||||
Operations on biscayne follow a strict layering. Each layer assumes the layers
|
||||
below it are correct. Playbooks belong to exactly one layer.
|
||||
|
||||
| Layer | What | Playbooks |
|
||||
|-------|------|-----------|
|
||||
| 1. Base system | Docker, ZFS, packages | Out of scope (manual/PXE) |
|
||||
| 2. Prepare kind | `/srv/kind` exists (ZFS dataset) | None needed (ZFS handles it) |
|
||||
| 3. Install kind | `laconic-so deployment start` creates kind cluster, mounts `/srv/kind` → `/mnt` in kind node | `biscayne-redeploy.yml` (deploy tags) |
|
||||
| 4. Prepare agave | Host storage for agave: zvol, ramdisk, rbind into `/srv/kind/solana` | `biscayne-prepare-agave.yml` |
|
||||
| 5. Deploy agave | Deploy agave-stack into kind, snapshot download, scale up | `biscayne-redeploy.yml` (snapshot/verify tags), `biscayne-recover.yml` |
|
||||
|
||||
**Layer 4 invariants** (asserted by `biscayne-prepare-agave.yml`):
|
||||
- `/srv/solana` is XFS on a zvol — agave uses io_uring which deadlocks on ZFS
|
||||
- `/srv/solana/ramdisk` is XFS on `/dev/ram0` — accounts must be on ramdisk
|
||||
- `/srv/kind/solana` is an rbind of `/srv/solana` — makes the zvol visible to kind at `/mnt/solana`
|
||||
|
||||
These invariants are checked at runtime and persisted to fstab/systemd so they
|
||||
survive reboot. They are agave's requirements reaching into the boot sequence,
|
||||
not base system concerns.
|
||||
|
||||
**Cross-cutting**: `health-check.yml` (read-only diagnostics), `biscayne-stop.yml`
|
||||
(layer 5 — graceful shutdown), `fix-pv-mounts.yml` (layer 5 — PV repair).
|
||||
|
||||
## Cluster Operations
|
||||
|
||||
### Shutdown Order
|
||||
@@ -36,7 +61,7 @@ Correct shutdown sequence:
|
||||
The accounts directory must be on a ramdisk for performance. `/dev/ram0` loses its
|
||||
filesystem on reboot and must be reformatted before mounting.
|
||||
|
||||
**Boot ordering is handled by systemd units** (installed by `biscayne-boot.yml`):
|
||||
**Boot ordering is handled by systemd units** (installed by `biscayne-prepare-agave.yml`):
|
||||
- `format-ramdisk.service`: runs `mkfs.xfs -f /dev/ram0` before `local-fs.target`
|
||||
- fstab entry: mounts `/dev/ram0` at `/srv/solana/ramdisk` with
|
||||
`x-systemd.requires=format-ramdisk.service`
|
||||
@@ -46,11 +71,12 @@ filesystem on reboot and must be reformatted before mounting.
|
||||
These units run before docker, so the kind node's bind mounts always see the
|
||||
ramdisk. **No manual intervention is needed after reboot.**
|
||||
|
||||
**Mount propagation**: The kind node bind-mounts `/srv/kind` → `/mnt`. Because
|
||||
the ramdisk is mounted at `/srv/solana/ramdisk` and symlinked/overlaid through
|
||||
`/srv/kind/solana/ramdisk`, mount propagation makes it visible inside the kind
|
||||
node at `/mnt/solana/ramdisk` without restarting the kind node. **Do NOT restart
|
||||
the kind node just to pick up a ramdisk mount.**
|
||||
**Mount propagation**: The kind node bind-mounts `/srv/kind` → `/mnt` at container
|
||||
start. New mounts under `/srv/kind` on the host (like the rbind at
|
||||
`/srv/kind/solana`) do NOT propagate into the kind node because kind's default
|
||||
mount propagation is `None`. A kind node restart is required to pick up new host
|
||||
mounts. **TODO**: Fix laconic-so to set `propagation: HostToContainer` on the
|
||||
kind-mount-root extraMount, which would make host mounts propagate automatically.
|
||||
|
||||
### KUBECONFIG
|
||||
|
||||
|
||||
Reference in new issue
Block a user