Process bug fix: no pre-commit existed for this repo's Python code.
Added pyproject.toml with unified dependencies (ruff, mypy, ansible-lint),
.pre-commit-config.yaml with repo-based hooks (ruff) and local uv-run
hooks (mypy, ansible-lint).
Fixed 249 ruff errors (B023, B904, B006, B007, UP008, UP031, C408),
~13 mypy type errors, 11 ansible-lint violations, and ruff-format
across all Python files including stack-orchestrator subtree.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
kind load docker-image serializes the full image (docker save | ctr import),
taking 5-10 minutes per cluster recreate. Replace with a persistent local
registry (registry:2 on port 5001) that survives kind cluster deletes.
stack-orchestrator changes:
- helpers.py: replace load_images_into_kind() with ensure_local_registry(),
connect_registry_to_kind_network(), push_images_to_local_registry()
- helpers.py: add registry mirror to containerdConfigPatches so kind nodes
pull from localhost:5001 via the kind-registry container
- deploy_k8s.py: rewrite local container image refs to localhost:5001/...
so containerd pulls from the registry instead of local store
Ansible changes:
- biscayne-sync-tools.yml: ensure registry container before build, then
tag+push to local registry after build (build-container tag)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Three related fixes in the k8s deployer restart/up flow:
1. Clear stale claimRefs on Released PVs (_clear_released_pv_claim_refs):
After namespace deletion, PVs survive in Released state with claimRefs
pointing to deleted PVC UIDs. New PVCs can't bind until the stale
claimRef is removed. Now clears them before PVC creation.
2. Wait for namespace termination (_wait_for_namespace_deletion):
_ensure_namespace() now detects a terminating namespace and polls
until deletion completes (up to 120s) before creating the new one.
Replaces the racy 5s sleep in deployment restart.
3. Resilient PVC creation: wrap each PVC creation in error handling so
one failure doesn't prevent subsequent PVCs from being attempted.
All errors are collected and reported together.
Closes: bar-6cb, bar-31a, bar-fec
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Use \033[H\033[J (home + clear-to-end) instead of just \033[H to
prevent stale lines from previous frames persisting when output
shrinks between refreshes.
- Fix cursor restore on exit: was \033[?25l (hide) instead of
\033[?25h (show), leaving terminal with invisible cursor.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Also fix --include filter: container name uses slash (laconicnetwork/agave)
not dash (laconicnetwork-agave). The old filter silently skipped the build.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Two fixes for biscayne-restart.yml:
1. ansible_become_flags: "-E" on the restart task preserves SSH_AUTH_SOCK
through sudo so laconic-so can git pull the stack repo.
2. After restart, clear claimRef on any Released PVs. laconic-so restart
deletes the namespace (cascading to PVCs) then recreates, but the PVs
retain stale claimRefs that prevent new PVCs from binding.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Uses laconic-so deployment restart (GitOps) to pick up new container
images and config. Gracefully stops the validator first (scale to 0,
wait for pod termination, verify no agave processes). Preserves the
kind cluster, all data volumes, and cluster state.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Moving container scripts into agave-stack subtree (correct direction).
The source of truth will be agave-stack/ in this repo, pushed out to
LaconicNetwork/agave-stack via git subtree push.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
ip_echo_preflight.py: reimplements Solana ip_echo client protocol in
Python. Verifies UDP port reachability before snapshot download, called
from entrypoint.py. Prevents wasting hours on a snapshot only to
crash-loop on port reachability.
docs/postmortem-ashburn-relay-outbound.md: root cause analysis of the
firewalld nftables FORWARD chain blocking outbound relay traffic.
docs/ashburn-relay-checklist.md: 7-layer verification checklist for
relay path debugging.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Firewalld zones/policies for forwarding (Docker bridge → gre-ashburn),
iptables for Docker-specific rules (DNAT, DOCKER-USER, mangle, SNAT).
Both coexist at different netfilter priorities.
See docs/postmortem-ashburn-relay-outbound.md for root cause analysis.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Entrypoint changes:
- Always require full + incremental before starting (retry until found)
- Check incremental freshness against convergence threshold (500 slots)
- Gap monitor thread: if validator falls >5000 slots behind for 3
consecutive checks, graceful stop + restart with fresh incremental
- cmd_serve is now a loop: download → run → monitor → leapfrog → repeat
- --no-snapshot-fetch moved to common args (both RPC and validator modes)
- --maximum-full-snapshots-to-retain default 1 (validator deletes
downloaded full after generating its own)
- SNAPSHOT_MAX_AGE_SLOTS default 100000 (one full snapshot generation)
snapshot_download.py refactoring:
- Extract _discover_and_benchmark() and _rolling_incremental_download()
as shared helpers
- Restore download_incremental_for_slot() using shared helpers (downloads
only an incremental for an existing full snapshot)
- download_best_snapshot() uses shared helpers, downloads full then
incremental as separate operations
The leapfrog cycle: validator generates full snapshots at standard 100k
block height intervals (same slots as the rest of the network). When the
gap monitor triggers, the entrypoint loops back to maybe_download_snapshot
which finds the validator's local full, downloads a fresh network
incremental (generated every ~40s, converges within the ~11hr full
generation window), and restarts.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Overwrite lines in place instead of clear+redraw (no flicker)
- Pad lines to terminal width to clear stale characters
- Blank leftover rows when output shrinks between frames
- Hide cursor during watch mode
- Remove section comment bars
- Replace unicode checkmarks with +/x
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
laconic-so creates PV hostPath dirs as root. Grafana runs as UID 472
and crashes on startup because it can't write to /var/lib/grafana.
Fix ownership inside the kind node before scaling the deployment up.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- biscayne-migrate-storage.yml: stop docker to release bind mounts
before destroying zvol, no data copy (stale, fresh snapshot needed),
handle partially-migrated state, restart docker at end
- biscayne-upgrade-zfs.yml: use add-apt-repository CLI (module times
out), fix libzfs package name (libzfs4linux not 5), allow apt update
warnings from stale influxdata GPG key
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Instead of hardcoding the laconic cluster ID, namespace, deployment
name, and pod label, read cluster-id from deployment.yml on biscayne
and derive everything from it.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The recovery playbook now exits after scaling to 1. The container
entrypoint handles snapshot download (60+ min) and validator startup
autonomously. Removed all polling/verification steps that would
time out waiting.
Added scripts/check-status.py for monitoring download progress,
validator slot, gap to mainnet, catch-up rate, and ramdisk usage.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The container's entrypoint.py already handles snapshot freshness checks,
cleanup, download (with rolling incremental convergence), and validator
startup. Remove the host-side download and let the container do the work.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Without a bound, the loop runs forever if sources never serve an
incremental close enough to head (e.g. full snapshot base slot is
too old). After 30 minutes, proceed with the best incremental
available or none.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
After the full snapshot downloads, continuously re-probe all fast sources
for newer incrementals until the best available is within convergence_slots
(default 500) of head. Each iteration finds the highest-slot incremental
matching our full snapshot's base slot, downloads it (replacing any previous),
and checks the gap to mainnet head.
- Extract probe_incremental() from inline re-probe code
- Add convergence_slots param to download_best_snapshot() (default 500)
- Add --convergence-slots CLI arg
- Pass SNAPSHOT_CONVERGENCE_SLOTS env var from entrypoint.py
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The container entrypoint (entrypoint.py) handles snapshot download
internally via aria2c. Ansible no longer needs to scale-to-0, download,
scale-to-1 — it just deploys and lets the container manage startup.
- biscayne-redeploy.yml: remove snapshot download section, simplify to
teardown → wipe → deploy → verify
- biscayne-sync-tools.yml: new playbook to sync laconic-so and
agave-stack repos on biscayne, with separate branch controls
- snapshot_download.py: re-probe for fresh incremental after full
snapshot download completes (old incremental is stale by then)
- Switch laconic_so_branch to fix/kind-mount-propagation (has
hostNetwork translation code)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
scripts/agave-container/ is a git subtree of agave-stack's container-build
directory. Replaces fragile cross-repo symlink with proper subtree.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- playbooks/biscayne-iptables.yml: manages PREROUTING DNAT and DOCKER-USER
rules for both host IP (186.233.184.235) and relay loopback (137.239.194.65).
Idempotent, persists via netfilter-persistent.
- scripts/snapshot-download.py: replaced standalone copy with symlink to
agave-stack source of truth, eliminating duplication.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remounting tmpfs is instant (kernel frees pages), while rm -rf on 400GB+
of accounts files traverses every inode. Recover playbook keeps rm -rf
because the kind node's bind mount prevents umount while the container
is running.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The /dev/ram0 + XFS + format-ramdisk.service approach was unnecessary
complexity from a migration confusion — there was no actual tmpfs bug
with io_uring. tmpfs is simpler (no format-on-boot), resizable on the
fly, and what every other Solana operator uses.
Changes:
- prepare-agave: remove format-ramdisk.service and ramdisk-accounts.service,
use tmpfs fstab entry with size=1024G (was 600G /dev/ram0, too small)
- recover: remove ramdisk_device var (no longer needed)
- redeploy: wipe accounts by rm -rf instead of umount+mkfs
- snapshot-download.py: extract download_best_snapshot() public API for
use by the new container entrypoint.py (in agave-stack)
- CLAUDE.md: update ramdisk docs, fix /srv/solana → /srv/kind/solana paths
- health-check: fix ramdisk path references
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Refactors K8sDeployer.up() into three composable methods:
- _setup_cluster_and_namespace(): kind cluster, API, namespace, ingress
- _create_infrastructure(): PVs, PVCs, ConfigMaps, Services, NodePorts
- _create_deployment(): Deployment resource (pods)
`prepare` calls the first two only — creates all cluster infrastructure
without starting pods. This eliminates the scale-to-0 workaround where
operators had to run `deployment start` then immediately scale down.
Usage: laconic-so deployment --dir <dir> prepare
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Track stack-orchestrator work items with pebbles (append-only event log).
Epic so-076: Stack composition — deploy multiple stacks into one kind cluster
with independent lifecycle management per sub-stack.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Revert snapshot_dir to /srv/solana/snapshots — aria2c runs on the host
where this is the direct zvol mount (always available), unlike
/srv/kind/solana/snapshots which depends on the bind mount
- Add laconic_so_branch variable (default: main) and use it in both
git reset commands so the branch can be overridden via -e
- Move "Verify ramdisk visible inside kind node" from preflight to after
"Wait for deployment to exist" — the kind container may not exist
during preflight after teardown
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
K8s PUT (replace) operations require metadata.resourceVersion for
optimistic concurrency control. Services additionally have immutable
spec.clusterIP that must be preserved from the existing object.
On 409 conflict, all _ensure_* methods now read the existing resource
first and copy resourceVersion (and clusterIP for Services) into the
body before calling replace.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add laconic_so_repo variable (/home/rix/stack-orchestrator) and a
git pull task before deployment start — the editable install must be
current or stale code causes deploy failures
- Downgrade unified mount root check from fatal assertion to debug
warning — the mount style depends on which laconic-so version is
deployed, and individual PV mounts (/mnt/validator-*) work fine
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
All K8s resource creation in deploy_k8s.py now uses try-create, catch
ApiException(409), then replace — matching the pattern already used for
secrets in deployment_create.py. This allows `deployment start` to be
safely re-run without 409 Conflict errors.
Resources made idempotent:
- Deployment (create_namespaced_deployment → replace on 409)
- Service (create_namespaced_service → replace on 409)
- Ingress (create_namespaced_ingress → replace on 409)
- NodePort services (same as Service)
- ConfigMap (create_namespaced_config_map → replace on 409)
- PV/PVC: bare `except: pass` replaced with explicit ApiException
catch for 404
Extracted _ensure_deployment(), _ensure_service(), _ensure_ingress(),
and _ensure_config_map() helpers to keep cyclomatic complexity in check.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix snapshot_dir: /srv/solana/snapshots → /srv/kind/solana/snapshots
(kind node reads from the bind mount, not the zvol mount directly)
- Fix kind-internal paths: /mnt/solana/... → /mnt/validator-... to match
actual PV hostPath layout (individual mounts, not unified)
- Add 'scale-up' tag to "Scale validator to 1" task for partial recovery
(--tags snapshot,scale-up,verify resumes without re-running deploy)
- Make 'Start deployment' idempotent: failed_when: false + follow-up
check so existing deployment doesn't fail the play
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Root cause: Docker FORWARD chain policy DROP blocked all DNAT'd relay
traffic (UDP/TCP 8001, UDP 9000-9025) to the kind node. The DOCKER
chain only ACCEPTs specific TCP ports (6443, 443, 80). Added ACCEPT
rules in DOCKER-USER chain which runs before all Docker chains.
Changes:
- ashburn-relay-biscayne.yml: add DOCKER-USER ACCEPT rules (inbound
tag) and rollback cleanup
- ashburn-relay-setup.sh.j2: persist DOCKER-USER rules across reboot
- relay-inbound-udp-test.yml: controlled e2e test — listener in kind
netns, sender from kelce, assert arrival
- relay-link-test.yml: link-by-link tcpdump captures at each hop
- relay-test-udp-listen.py, relay-test-udp-send.py: test helpers
- relay-test-ip-echo.py: full ip_echo protocol test
- inventory/kelce.yml, inventory/panic.yml: test host inventories
- test-ashburn-relay.sh: add ip_echo UDP reachability test
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Destroying the kind cluster on stop/start is almost never the intent.
The cluster holds PVs, ConfigMaps, and networking state that are
expensive to recreate. Default to preserving the cluster; pass
--perform-cluster-management explicitly when a full teardown is needed.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The update command only patches environment variables and adds a
restart annotation. It does not update ports, volumes, configmaps,
or any other deployment spec. The old name was misleading — it
implied a full spec update, causing operators to expect changes
that never took effect.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Three Python scripts send real packets from the kind node through the
full relay path (biscayne → tunnel → mia-sw01 → was-sw01 → internet)
and verify responses come back via the inbound path. No indirect
counter-checking — a response proves both directions work.
- relay-test-udp.py: DNS query with sport 8001
- relay-test-tcp-sport.py: HTTP request with sport 8001
- relay-test-tcp-dport.py: TCP connect to entrypoint dport 8001 (ip_echo)
- test-ashburn-relay.sh: orchestrates from ansible controller via nsenter
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Inventories what the DZ agent controls (tunnels, ACLs, VRFs, BGP,
route-maps, loopbacks) so we don't accidentally modify objects that
the agent will silently overwrite. Includes a "safe to modify" section
listing our own relay infrastructure.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
mia-sw01: Replace PBR-based outbound routing with VRF isolation.
TCAM profile tunnel-interface-acl doesn't support PBR or traffic-policy
on tunnel interfaces. Tunnel100 now lives in VRF "relay" whose default
route sends decapsulated traffic to was-sw01 via backbone, avoiding
BCP38 drops on the ISP uplink for src 137.239.194.65.
biscayne: Add TCP dport mangle rule for ip_echo (port 8001). Without it,
outbound ip_echo probes use biscayne's real IP instead of the Ashburn
relay IP, causing entrypoints to probe the wrong address. Also fix
loopback IP idempotency (handle "already assigned" error).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Ashburn relay and shred relay lab configs for local end-to-end
testing with cEOS. No secrets — only public IPs and test scripts.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Normal playbooks should never hardcode hostnames — that's an inventory
concern. Changed all playbooks to hosts:all. The one exception is
ashburn-relay-check.yml which legitimately spans both inventories
(switches + biscayne) and uses explicit hostnames.
Also adds:
- ashburn-relay-check.yml: full-path relay diagnostics (switches + host)
- biscayne-start.yml: start kind container and scale validator to 1
- ashburn-relay-setup.sh.j2: boot persistence script for relay state
- Direct device mounts replacing rbind (ZFS shared propagation fix)
- systemd service replacing broken if-up.d/netfilter-persistent
- PV mount path corrections (/mnt/validator-* not /mnt/solana/*)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
_clean_etcd_keeping_certs() only preserved /registry/secrets/caddy-system,
deleting everything else including the kubernetes ClusterIP service in the
default namespace. When kind recreated the cluster with the cleaned etcd,
kube-apiserver saw existing data and skipped bootstrapping the service.
kindnet panicked on KUBERNETES_SERVICE_HOST missing, blocking all pod
networking.
Expand the whitelist to also preserve:
- /registry/services/specs/default/kubernetes
- /registry/services/endpoints/default/kubernetes
Loop over multiple prefixes instead of a single etcdctl get --prefix call.
See docs/bug-laconic-so-etcd-cleanup.md in biscayne-agave-runbook.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Without propagation, rbind submounts on the host (e.g., XFS zvol at
/srv/kind/solana) are invisible inside the kind node — it sees the
underlying filesystem (ZFS) instead. This causes agave's io_uring to
deadlock on ZFS transaction commits (D-state in dsl_dir_tempreserve_space).
HostToContainer propagation ensures host submounts propagate into the
kind node, so /mnt/solana correctly resolves to the XFS zvol.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move switches.yml to inventory-switches/ so ansible.cfg's
`inventory = inventory/` only loads biscayne. Switch playbooks
must pass `-i inventory-switches/` explicitly.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- FQCN for all modules (ansible.builtin.*)
- changed_when/failed_when on all command/shell tasks
- set -o pipefail on all shell tasks
- Add KUBECONFIG environment to health-check.yml
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Resolve container resources using layered priority:
1. spec.yml per-container override (resources.containers.<name>)
2. Compose file deploy.resources block
3. spec.yml global resources
4. DEFAULT_CONTAINER_RESOURCES fallback
This prevents monitoring sidecars from inheriting the validator's
resource requests (e.g., 256G memory). Each service gets appropriate
resources from its compose definition unless explicitly overridden.
Note: existing deployments with a global resources block in spec.yml
can remove it once compose files declare per-service defaults.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- ansible.cfg: enable SSH agent forwarding for git operations
- biscayne-redeploy.yml: add git pull, deploy create --update, and
clear stale PV claimRefs after namespace deletion
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Config is committed to running-config immediately (no 5-min timer).
Safety net is the checkpoint (rollback) and the fact that startup-config
is only written with -e commit=true. A reboot reverts uncommitted changes.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Root cause: the doublezero-agent on mia-sw01 manages Tunnel500's ACL
(SEC-USER-500-IN) and drops outbound gossip with src 137.239.194.65.
The agent overwrites any custom ACL entries.
Fix: create a separate GRE tunnel (Tunnel100) using mia-sw01's free
LAN IP (209.42.167.137) as tunnel source. This tunnel goes over the
ISP uplink, completely independent of the DZ overlay:
- mia-sw01: Tunnel100 src 209.42.167.137, dst 186.233.184.235
- biscayne: gre-ashburn src 186.233.184.235, dst 209.42.167.137
- Link addresses: 169.254.100.0/31
Playbook changes:
- ashburn-relay-mia-sw01: Tunnel100 + Loopback101 + SEC-VALIDATOR-100-IN
- ashburn-relay-biscayne: gre-ashburn tunnel + updated policy routing
- New template: ashburn-routing-ifup.sh.j2 for boot persistence
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Playbook fixes from testing:
- ashburn-relay-biscayne: insert DNAT rules at position 1 before
Docker's ADDRTYPE LOCAL rule (was being swallowed at position 3+)
- ashburn-relay-mia-sw01: add inbound route for 137.239.194.65 via
egress-vrf vrf1 (nexthop only, no interface — EOS silently drops
cross-VRF routes that specify a tunnel interface)
- ashburn-relay-was-sw01: replace PBR with static route, remove
Loopback101
Bug doc (bug-ashburn-tunnel-port-filtering.md): root cause is the
DoubleZero agent on mia-sw01 overwrites SEC-USER-500-IN ACL, dropping
outbound gossip with src 137.239.194.65. The DZ agent controls
Tunnel500's lifecycle. Fix requires a separate GRE tunnel using
mia-sw01's free LAN IP (209.42.167.137) to bypass DZ infrastructure.
Also adds all repo docs, scripts, inventory, and remaining playbooks.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Three playbooks for routing all validator traffic through 137.239.194.65:
- was-sw01: Loopback101 + PBR redirect on Et1/1 (already applied/committed)
Will be simplified to a static route in next iteration.
- mia-sw01: ACL permit for src 137.239.194.65 on Tunnel500 + default route
in vrf1 via egress-vrf default to was-sw01 backbone. No PBR needed —
per-tunnel ACLs already scope what enters vrf1.
- biscayne: DNAT inbound (137.239.194.65 → kind node), SNAT + policy
routing outbound (validator sport 8001,9000-9025 → doublezero0 GRE).
Inbound already applied.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Captured via ansible `show running-config` before applying
mia-sw01 outbound validator redirect changes.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
In docker-compose, services can reference each other by name (e.g., 'db:5432').
In Kubernetes, when multiple containers are in the same pod (sidecars), they
share the same network namespace and must use 'localhost' instead.
This fix adds translate_sidecar_service_names() which replaces docker-compose
service name references with 'localhost' in environment variable values for
containers that share the same pod.
Fixes issue where multi-container pods fail because one container tries to
connect to a sibling using the compose service name instead of localhost.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Each deployment now gets its own Kubernetes namespace (laconic-{deployment_id}).
This provides:
- Resource isolation between deployments on the same cluster
- Simplified cleanup: deleting the namespace cascades to all namespaced resources
- No orphaned resources possible when deployment IDs change
Changes:
- Set k8s_namespace based on deployment name in __init__
- Add _ensure_namespace() to create namespace before deploying resources
- Add _delete_namespace() for cleanup
- Simplify down() to just delete PVs (cluster-scoped) and the namespace
- Fix hardcoded "default" namespace in logs function
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Previously, down() generated resource names from the deployment config
and deleted those specific names. This failed to clean up orphaned
resources when deployment IDs changed (e.g., after force_redeploy).
Changes:
- Add 'app' label to all resources: Ingress, Service, NodePort, ConfigMap, PV
- Refactor down() to query K8s by label selector instead of generating names
- This ensures all resources for a deployment are cleaned up, even if
the deployment config has changed or been deleted
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Mount /var/lib/etcd and /etc/kubernetes/pki to host filesystem
so cluster state is preserved for offline recovery. Each deployment
gets its own backup directory keyed by deployment ID.
Directory structure:
data/cluster-backups/{deployment_id}/etcd/
data/cluster-backups/{deployment_id}/pki/
This enables extracting secrets from etcd backups using etcdctl
with the preserved PKI certificates.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Adds support for configuring ACME email for Let's Encrypt certificates
in kind deployments. The email can be specified in the spec under
network.acme-email and will be used to configure the Caddy ingress
controller ConfigMap.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
For k8s-kind, relative paths (e.g., ./data/rpc-config) are resolved to
$DEPLOYMENT_DIR/path by _make_absolute_host_path() during kind config
generation. This provides Docker Host persistence that survives cluster
restarts.
Previously, validation threw an exception before paths could be resolved,
making it impossible to use relative paths for persistent storage.
Changes:
- deployment_create.py: Skip relative path check for k8s-kind
- cluster_info.py: Allow relative paths to reach PV generation
- docs/deployment_patterns.md: Document volume persistence patterns
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Document that:
- Volumes persist across cluster deletion by design
- Only use --delete-volumes when explicitly requested
- Multiple deployments share one kind cluster
- Use --skip-cluster-management to stop single deployment
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Previously, install_ingress_for_kind() applied the YAML (which starts
the Caddy pod with email: ""), then patched the ConfigMap afterward.
The pod had already read the empty email and Caddy doesn't hot-reload.
Now template the email into the YAML before applying, so the pod starts
with the correct email from the beginning.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The existing 'image-registry' key is used for pushing images to a remote
registry (URL string). Rename the new auth config to 'registry-credentials'
to avoid collision.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add ability to configure private container registry credentials in spec.yml
for deployments using images from registries like GHCR.
- Add get_image_registry_config() to spec.py for parsing image-registry config
- Add create_registry_secret() to create K8s docker-registry secrets
- Update cluster_info.py to use dynamic {deployment}-registry secret names
- Update deploy_k8s.py to create registry secret before deployment
- Document feature in deployment_patterns.md
The token-env pattern keeps credentials out of git - the spec references an
environment variable name, and the actual token is passed at runtime.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Check stack.yml containers: field to determine which images are local builds
- Only load local images via kind load; let k8s pull registry images directly
- Add is_ingress_running() to skip ingress installation if already running
- Fixes deployment failures when public registry images aren't in local Docker
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
When stack: field in spec.yml contains a path (e.g., stack_orchestrator/data/stacks/name),
extract just the final name component for K8s secret naming. K8s resource names must
be valid RFC 1123 subdomains and cannot contain slashes.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add GENERATE_TOKEN_PATTERN to detect $generate:hex:N$ and $generate:base64:N$ tokens
- Add _generate_and_store_secrets() to create K8s Secrets from spec.yml config
- Modify _write_config_file() to separate secrets from regular config
- Add env_from with secretRef to container spec in cluster_info.py
- Secrets are injected directly into containers via K8s native mechanism
This enables declarative secret generation in spec.yml:
config:
SESSION_SECRET: $generate:hex:32$
DB_PASSWORD: $generate:hex:16$
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
When deploying a second stack to k8s-kind, automatically reuse an existing
kind cluster instead of trying to create a new one (which would fail due
to port 80/443 conflicts).
Changes:
- helpers.py: create_cluster() now checks for existing cluster first
- deploy_k8s.py: up() captures returned cluster name and updates self
This enables deploying multiple stacks (e.g., gorbagana-rpc + trashscan-explorer)
to the same kind cluster.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add --spec-file option to specify spec location in repo
- Auto-detect deployment/spec.yml in repo as GitOps location
- Fall back to deployment dir if no repo spec found
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The etcd directory is root-owned, so shell test -f fails.
Use docker with volume mount to check file existence.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Create member.backup-YYYYMMDD-HHMMSS before cleaning.
Each cluster recreation creates a new backup, preserving history.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Move original to .bak, move new into place, then delete bak.
If anything fails before the swap, original remains intact.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Instead of trying to delete specific stale resources (blacklist),
keep only the valuable data (caddy TLS certs) and delete everything
else. This is more robust as we don't need to maintain a list of
all possible stale resources.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Use docker containers with volume mounts to handle all file
operations on root-owned etcd directories, avoiding the need
for sudo on the host.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
When etcd is persisted (for certificate backup) and a cluster is
recreated, kind tries to install CNI (kindnet) fresh but the
persisted etcd already has those resources, causing 'AlreadyExists'
errors and cluster creation failure.
This fix:
- Detects etcd mount path from kind config
- Before cluster creation, clears stale CNI resources (kindnet, coredns)
- Preserves certificate and other important data
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add acme_email_key constant for spec.yml parsing
- Add get_acme_email() method to Spec class
- Modify install_ingress_for_kind() to patch ConfigMap with email
- Pass acme-email from spec to ingress installation
- Add 'delete' verb to leases RBAC for certificate lock cleanup
The acme-email field in spec.yml was previously ignored, causing
Let's Encrypt to fail with "unable to parse email address".
The missing delete permission on leases caused lock cleanup failures.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add `laconic-so deployment restart` command that:
- Pulls latest code from stack git repository
- Regenerates spec.yml from stack's commands.py
- Verifies DNS if hostname changed (with --force to skip)
- Syncs deployment directory preserving cluster ID and data
- Stops and restarts deployment with --skip-cluster-management
Also stores stack-source path in deployment.yml during create
for automatic stack location on restart.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Document that external stack pattern should be used when creating new
stacks for any reason, with directory structure and usage examples.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Previously, volumes defined in a stack's commands.py init() function
were being overwritten by volumes discovered from compose files.
This prevented stacks from adding infrastructure volumes like caddy-data
that aren't defined in the compose files.
Now volumes are merged, with init() volumes taking precedence over
compose-discovered defaults.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add support for Docker Compose host path mounts (like ../config/file:/path)
in k8s deployments. Previously these were silently skipped, causing k8s
deployments to fail when compose files used host path mounts.
Changes:
- Add helper functions for host path detection and name sanitization
- Generate kind extraMounts for host path mounts
- Create hostPath volumes in pod specs for host path mounts
- Create volumeMounts with sanitized names for host path mounts
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
To allow updating an existing deployment
- Check the deployment dir exists when updating
- Write to temp dir, then safely copy tree
- Don't overwrite data dir or config.env
Resolve conflicts:
- deployment_context.py: Keep single modify_yaml method from main
- fixturenet-optimism/commands.py: Use modify_yaml helper from main
- deployment_create.py: Keep helm-chart, network-dir, initial-peers options
- deploy_webapp.py: Update create_operation call signature
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
get_service() returns None when there are no http-proxy routes,
so we must check before calling create_namespaced_service().
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Change 'docker remove -f' to 'docker rm -f' - the 'remove' subcommand
doesn't exist in docker CLI.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Previously these workflows ran on PRs to any branch. Now:
- PRs to main: run all tests (full CI gate)
- Pushes to other branches: use existing path filtering
This reduces CI load on feature branch PRs while maintaining
full test coverage for PRs targeting main.
Affected workflows:
- test-k8s-deploy.yml
- test-k8s-deployment-control.yml
- test-webapp.yml
- test-deploy.yml
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Previously get_service() only exposed the first port from pod definition.
Now it collects all unique ports from http-proxy routes and exposes them
all in the Service spec.
This is needed for WebSocket support where RPC runs on one port (8899)
and WebSocket pubsub on another (8900) - both need to be accessible
through the ingress.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Update stack command for continuous deployment workflow
- Separate deployer from CLI
- Separate stacks from orchestrator repo
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The ingress annotation was still set to nginx class even though we're now
using Caddy as the ingress controller. Caddy won't pick up ingresses
annotated with the nginx class.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Caddy provides automatic HTTPS with Let's Encrypt, but needs port 443
mapped from the kind container to the host. Previously only port 80 was
mapped.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The helm-charts-with-caddy branch had the Caddy manifest file but was still
using nginx in the code. This change:
- Switch install_ingress_for_kind() to use ingress-caddy-kind-deploy.yaml
- Update wait_for_ingress_in_kind() to watch caddy-system namespace
- Use correct label selector for Caddy ingress controller pods
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The base_runtime_spec for containerd requires a complete OCI spec,
not just the rlimits section. The minimal spec was causing runc to
fail with "open /proc/self/fd: no such file or directory" because
essential mounts and namespaces were missing.
This commit uses kind's default cri-base.json as the base and adds
the rlimits configuration on top. The spec includes all necessary
mounts, namespaces, capabilities, and kind-specific hooks.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The previous approach of mounting cri-base.json into kind nodes failed
because we didn't tell containerd to use it via containerdConfigPatches.
RuntimeClass allows different stacks to have different rlimit profiles,
which is essential since kind only supports one cluster per host and
multiple stacks share the same cluster.
Changes:
- Add containerdConfigPatches to kind-config.yml to define runtime handlers
- Create RuntimeClass resources after cluster creation
- Add runtimeClassName to pod specs based on stack's security settings
- Rename cri-base.json to high-memlock-spec.json for clarity
- Add get_runtime_class() method to Spec that auto-derives from
unlimited-memlock setting
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add pyrightconfig.json for pyright 1.1.408 TOML parsing workaround
- Add NoReturn annotations to fatal() functions for proper type narrowing
- Add None checks and assertions after require=True get_record() calls
- Fix AttrDict class with __getattr__ for dynamic attribute access
- Add type annotations and casts for Kubernetes client objects
- Store compose config as DockerDeployer instance attributes
- Filter None values from dotenv and environment mappings
- Use hasattr/getattr patterns for optional container attributes
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Format code with black (line length 88)
- Fix E501 line length errors by breaking long strings and comments
- Fix F841 unused variable (removed unused 'quiet' variable)
- Configure pyright to disable common type issues in existing codebase
(reportGeneralTypeIssues, reportOptionalMemberAccess, etc.)
- All pre-commit hooks now pass
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add spec.yml option `security.unlimited-memlock` that configures
RLIMIT_MEMLOCK to unlimited for Kind cluster pods. This is needed
for workloads like Solana validators that require large amounts of
locked memory for memory-mapped files during snapshot decompression.
When enabled, generates a cri-base.json file with rlimits and mounts
it into the Kind node to override the default containerd runtime spec.
Also includes flake8 line-length fixes for affected files.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add black formatter (rev 23.12.1)
- Add pyright type checker (rev v1.1.345)
- Add yamllint with relaxed mode (rev v1.35.1)
- Update flake8 args: max-line-length=88, extend-ignore=E203,W503,E402
- Remove ansible-lint from dev dependencies (no ansible files in repo)
- Sync pyproject.toml flake8 config with pre-commit
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add Caddy ingress controller manifest for kind deployments
- Add k8s cluster list command for kind cluster management
- Add k8s_command import and registration in deploy.py
- Fix network section merge to preserve http-proxy settings
- Increase default container resources (4 CPUs, 8GB memory)
- Add UDP protocol support for K8s port definitions
- Add command/entrypoint support for K8s deployments
- Implement docker-compose variable expansion for K8s
- Set ConfigMap defaultMode to 0755 for executable scripts
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Create comprehensive documentation for deploying stacks using Docker
Compose, which is the default and recommended deployment mode.
The guide covers:
- Complete deployment workflows (deployment directory and quick deploy)
- Real-world examples (test stack and fixturenet-eth)
- Configuration options (ports, volumes, environment variables)
- Common operations and troubleshooting
- CLI commands reference
Mentions that Kubernetes deployment options exist but are out of scope
for this guide (covered in separate K8s documentation).
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
- AI-FRIENDLY-PLAN.md: Plan for making repo AI-friendly
- STACK-CREATION-GUIDE.md: Implementation details for create-stack command
- laconic-network-deployment.md: Laconic network deployment overview
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This is needed to allow custom deploy commands to handle arbitrary args.
* Adds a `DeploymentContext.modify_yaml` helper
* Removes `laconicd` from test stack to simplify it
Reviewed-on: cerc-io/stack-orchestrator#972
Reviewed-by: ashwin <ashwin@noreply.git.vdb.to>
Otherwise we sometimes see errors like:
```
cerc-webapp-deployer: File "/root/.shiv/laconic-so_0f937aa98c2748ef9af8585d6f441dbc01546ace0d6660cbb159d1e5040aeddf/site-packages/stack_orchestrator/deploy/webapp/deploy_webapp_from_registry.py", line 671, in command
cerc-webapp-deployer: shutil.rmtree(tempdir)
cerc-webapp-deployer: File "/usr/lib/python3.10/shutil.py", line 725, in rmtree
cerc-webapp-deployer: _rmtree_safe_fd(fd, path, onerror)
cerc-webapp-deployer: File "/usr/lib/python3.10/shutil.py", line 681, in _rmtree_safe_fd
cerc-webapp-deployer: onerror(os.unlink, fullname, sys.exc_info())
cerc-webapp-deployer: File "/usr/lib/python3.10/shutil.py", line 679, in _rmtree_safe_fd
cerc-webapp-deployer: os.unlink(entry.name, dir_fd=topfd)
cerc-webapp-deployer: FileNotFoundError: [Errno 2] No such file or directory: 'S.gpg-agent.extra'
```
Reviewed-on: cerc-io/stack-orchestrator#941
Co-authored-by: Thomas E Lackey <telackey@bozemanpass.com>
Co-committed-by: Thomas E Lackey <telackey@bozemanpass.com>
This adds two new commands: `publish-webapp-deployer` and `request-webapp-deployment`.
`publish-webapp-deployer` creates a `WebappDeployer` record, which provides information to requestors like the API URL, minimum required payment, payment address, and public key to use for encrypting config.
```
$ laconic-so publish-deployer-to-registry \
--laconic-config ~/.laconic/laconic.yml \
--api-url https://webapp-deployer-api.dev.vaasl.io \
--public-key-file webapp-deployer-api.dev.vaasl.io.pgp.pub \
--lrn lrn://laconic/deployers/webapp-deployer-api.dev.vaasl.io \
--min-required-payment 100000
```
`request-webapp-deployment` simplifies publishing a `WebappDeploymentRequest` and can also handle automatic payment, and encryption and upload of configuration.
```
$ laconic-so request-webapp-deployment \
--laconic-config ~/.laconic/laconic.yml \
--deployer lrn://laconic/deployers/webapp-deployer-api.dev.vaasl.io \
--app lrn://cerc-io/applications/webapp-hello-world@0.1.3 \
--env-file ~/yaml/hello.env \
--make-payment auto
```
Related changes are included for the deploy/undeploy commands for decrypting and using config, using the payment address from the WebappDeployer record, etc.
Reviewed-on: cerc-io/stack-orchestrator#938
Adds three new options for deployment/undeployment:
```
"--min-required-payment",
help="Requests must have a minimum payment to be processed",
"--payment-address",
help="The address to which payments should be made. Default is the current laconic account.",
"--all-requests",
help="Handle requests addressed to anyone (by default only requests to my payment address are examined).",
```
In this mode, requests should be designated for a particular address with the attribute `to` and include a `payment` attribute which is the tx hash for the payment.
The deployer will confirm the payment (to the right account, right amount, not used before, etc.) and then proceed with the deployment or undeployment.
Reviewed-on: cerc-io/stack-orchestrator#928
Reviewed-by: David Boreham <dboreham@noreply.git.vdb.to>
Co-authored-by: Thomas E Lackey <telackey@bozemanpass.com>
Co-committed-by: Thomas E Lackey <telackey@bozemanpass.com>
Related to cerc-io/webapp-deployment-status-api#10
There are two issues in that. One is that the output probably changed recently, whether in the client or server, where no matching record is found by ID (Note this is specific to `laconic record get --id <v>` and does not seem to apply to the similar command to retrieve a record by name, `laconic name resolve <n>`).
Rather than returning `[]` it is now returning `[ null ]`. This cause us to think there *was* an application record found, and we attempt to treat the `null` entry like an Application object. That's fixed by filtering out null responses, which is a good precaution for the deployer, though I think it makes sense to ask whether this new behavior by the client/server is correct. Seems suspicious.
The other issue is that all the defensive checks we had in place to deal with broken/bad AppDeploymentRequests were around the _build_. This error was coming much earlier, merely when parsing and examining the request to see if it needed to be handled at all.
I have now added similar defensive error handling around that portion of the code.
Reviewed-on: cerc-io/stack-orchestrator#922
Reviewed-by: zramsay <zramsay@noreply.git.vdb.to>
Co-authored-by: Thomas E Lackey <telackey@bozemanpass.com>
Co-committed-by: Thomas E Lackey <telackey@bozemanpass.com>
The str type check doesn't work if the port is a ruamel.yaml.scalarstring.SingleQuotedScalarString or ruamel.yaml.scalarstring.DoubleQuotedScalarString
Reviewed-on: cerc-io/stack-orchestrator#919
Co-authored-by: Thomas E Lackey <telackey@bozemanpass.com>
Co-committed-by: Thomas E Lackey <telackey@bozemanpass.com>
This emulates the K8S ConfigMap behavior on Docker by using a regular volume.
If a directory exists under `config/` which matches a named volume, the contents will be copied to the volume on `create` (provided the destination volume is empty). That is, rather than a ConfigMap, it is essentially a "config volume".
For example, with a compose file like:
```
version: '3.7'
services:
caddy:
image: cerc/caddy-ethcache:local
restart: always
volumes:
- caddyconfig:/etc/caddy:ro
volumes:
caddyconfig:
```
And a directory:
```
❯ ls stack-orchestrator/config/caddyconfig/
Caddyfile
```
After `laconic-so deploy create --spec-file caddy.yml --deployment-dir /srv/caddy` there will be:
```
❯ ls /srv/caddy/data/caddyconfig
Caddyfile
```
Mounted at `/etc/caddy`
Reviewed-on: cerc-io/stack-orchestrator#914
Reviewed-by: David Boreham <dboreham@noreply.git.vdb.to>
Co-authored-by: Thomas E Lackey <telackey@bozemanpass.com>
Co-committed-by: Thomas E Lackey <telackey@bozemanpass.com>
NodePort example:
```
network:
ports:
caddy:
- 1234
- 32020:2020
```
Replicas example:
```
replicas: 2
```
This also adds an optimization for k8s where if a directory matching the name of a configmap exists in beneath config/ in the stack, its contents will be copied into the corresponding configmap.
For example:
```
# Config files in the stack
❯ ls stack-orchestrator/config/caddyconfig
Caddyfile Caddyfile.one-req-per-upstream-example
# ConfigMap in the spec
❯ cat foo.yml | grep config
...
configmaps:
caddyconfig: ./configmaps/caddyconfig
# Create the deployment
❯ laconic-so --stack ~/cerc/caddy-ethcache/stack-orchestrator/stacks/caddy-ethcache deploy create --spec-file foo.yml
# The files from beneath config/<config_map_name> have been copied to the ConfigMap directory from the spec.
❯ ls deployment-001/configmaps/caddyconfig
Caddyfile Caddyfile.one-req-per-upstream-example
```
Reviewed-on: cerc-io/stack-orchestrator#913
Reviewed-by: David Boreham <dboreham@noreply.git.vdb.to>
Co-authored-by: Thomas E Lackey <telackey@bozemanpass.com>
Co-committed-by: Thomas E Lackey <telackey@bozemanpass.com>
This is working off pull request "[Add support for pnpm as a webapp build tool. #767](https://git.vdb.to/cerc-io/stack-orchestrator/pulls/767/files)" that adds `pnpm` package manager support for `nextjs` & `webapps`.
`bun` default build output directory (defined as `CERC_BUILD_OUTPUT_DIR`) is `dist` which should already be handled with `pnpm` support in the previously mentioned [pull request](https://git.vdb.to/cerc-io/stack-orchestrator/pulls/767/files)
Installing `bun` using `npm` following our previous `pnpm` installation
```zsh
npm install -g bun
```
We'll be using `bun` as a package manager that works with `Node.js` projects as defined in bun's [docs](https://bun.sh/docs/cli/install)
> The bun CLI contains a Node.js-compatible package manager designed to be a dramatically faster replacement for npm, yarn, and pnpm. It's a standalone tool that will work in pre-existing Node.js projects; if your project has a package.json, bun install can help you speed up your workflow.
To test `next.js` apps using `node.js` and compatibility with all four packager managers -- `npm`, `yarn`, `pnpm`, and `bun` -- use the branches of snowball's [nextjs-package-manager-example-app](https://git.vdb.to/snowball/nextjs-package-manager-example-app) repo: `nextjs-package-manager/npm`, `nextjs-package-manager/yarn`, `nextjs-package-manager/pnpm`, `nextjs-package-manager/bun`.
Co-authored-by: Vivian Phung <dev+github@vivianphung.com>
Co-authored-by: David Boreham <dboreham@noreply.git.vdb.to>
Reviewed-on: https://git.vdb.to/cerc-io/stack-orchestrator/pulls/800
Reviewed-by: David Boreham <dboreham@noreply.git.vdb.to>
Co-authored-by: VPhung24 <vphung24@noreply.git.vdb.to>
Co-committed-by: VPhung24 <vphung24@noreply.git.vdb.to>
Fixes
- stack path resolution for `build`
- external stack path resolution for deployments
- "extra" config detection
- `deployment ports` command
- `version` command in dist or source install (without build_tag.txt)
- `setup-repos`, so it won't die when an existing repo is not at a branch or exact tag
Used in cerc-io/fixturenet-eth-stacks#14
Reviewed-on: cerc-io/stack-orchestrator#851
Reviewed-by: David Boreham <dboreham@noreply.git.vdb.to>
This add a new option `--fqdn-policy` to the `deploy-webapp-from-registry`.
The default policy, `prohibit` means that `ApplicationDeploymentRequests` which specify a FQDN will be rejected. The `allow` policy will cause them to be processed. The `preexisting` policy will only process them if an existing `DnsRecord` exists in the registry with the correct ownership.
The latter would be useful in conjunction with a pre-checking scheme in the UI (eg, that the DNS entry is properly configured, the domain is under the control of the requestor, etc.) Only after all the checks were successful would the `DnsRecord` be created, allowing for `ApplicationDeploymentRequests` to use it.
Reviewed-on: cerc-io/stack-orchestrator#802
Reviewed-by: David Boreham <dboreham@noreply.git.vdb.to>
Co-authored-by: Thomas E Lackey <telackey@bozemanpass.com>
Co-committed-by: Thomas E Lackey <telackey@bozemanpass.com>
Updates fixturenet-plugeth stack for the Deneb fork based on Geth v1.13.x:
- updates genesis generator tool, and simplifies the config: the default from `ethereum-genesis-generator` can be used for a from-genesis Merged chain.
Reviewed-on: cerc-io/stack-orchestrator#789
Reviewed-by: jonathanface <jonathanface@noreply.git.vdb.to>
Reviewed-by: Thomas E Lackey <telackey@noreply.git.vdb.to>
Rather than always requesting a certificate, attempt to re-use an existing certificate if it already exists in the k8s cluster. This includes matching to a wildcard certificate.
Reviewed-on: cerc-io/stack-orchestrator#779
Co-authored-by: Thomas E Lackey <telackey@bozemanpass.com>
Co-committed-by: Thomas E Lackey <telackey@bozemanpass.com>
Implementation of a command to fetch pre-built images from a remote registry, complementing the --push-images option already present on build-containers.
The two subcommands used together allow a stack to be deployed without needing to built its images, provided they have been already built and pushed to the specified container image registry.
This implementation simply picks the newest image with the right name and platform (matches against the platform Python is running on, so watch out for scenarios where Python is an x86 binary on M1 macs).
Reviewed-on: cerc-io/stack-orchestrator#768
Co-authored-by: David Boreham <david@bozemanpass.com>
Co-committed-by: David Boreham <david@bozemanpass.com>
webapps are meant to be build-once/deploy-many, but we were rebuilding them for every request. This changes that, so that we rebuild only for every unique ApplicationRecord.
When we push the image, we now tag it according to its ApplicationRecord.
We don't want to use that tag directly in the compose file for the deployment, however, as the deployment needs to be able to adjust to new builds w/o re-writing the file all the time. Instead, we use a per-deployment unique tag (same as before), we just update what image it references as needed.
Reviewed-on: cerc-io/stack-orchestrator#764
This creates a new environment variable, CERC_SINGLE_PAGE_APP, which controls whether a catchall redirection back to / is applied.
If the value is not explicitly set, we try to detect if the page looks like a single-page app.
Reviewed-on: cerc-io/stack-orchestrator#763
Co-authored-by: Thomas E Lackey <telackey@bozemanpass.com>
Co-committed-by: Thomas E Lackey <telackey@bozemanpass.com>
Hopefully the last one for a bit.
This only output the cmdline if log_file is present (ie, not to stdout). It also fixes a bug where the log_file was not passed in one line.
Reviewed-on: cerc-io/stack-orchestrator#757
Co-authored-by: Thomas E Lackey <telackey@bozemanpass.com>
Co-committed-by: Thomas E Lackey <telackey@bozemanpass.com>
Minor fixes to envsubst for webapps. Somewhat specially treated is `LACONIC_HOSTED_CONFIG_homepage` which can be used to replace the homepage in package.json. With react, this gets an extra `/` though, which we need to remove.
Reviewed-on: cerc-io/stack-orchestrator#746
Co-authored-by: Thomas E Lackey <telackey@bozemanpass.com>
Co-committed-by: Thomas E Lackey <telackey@bozemanpass.com>
In kind, when we bind-mount a host directory it is first mounted into the kind container at /mnt, then into the pod at the desired location.
We accidentally picked this up for full-blown k8s, and were creating volumes at /mnt. This changes the behavior for both kind and regular k8s so that bind mounts are only allowed if a fully-qualified path is specified. If no path is specified at all, a default storageClass is assumed to be present, and the volume managed by a provisioner.
Eg, for kind, the default provisioner is: https://github.com/rancher/local-path-provisioner
```
stack: test
deploy-to: k8s-kind
config:
test-variable-1: test-value-1
network:
ports:
test:
- '80'
volumes:
# this will be bind-mounted to a host-path
test-data-bind: /srv/data
# this will be managed by the k8s node
test-data-auto:
configmaps:
test-config: ./configmap/test-config
```
Reviewed-on: cerc-io/stack-orchestrator#741
Co-authored-by: Thomas E Lackey <telackey@bozemanpass.com>
Co-committed-by: Thomas E Lackey <telackey@bozemanpass.com>
```
--include-tags TEXT Only include requests with matching tags
(comma-separated).
--exclude-tags TEXT Exclude requests with matching tags (comma-
separated).
```
Reviewed-on: cerc-io/stack-orchestrator#734
Co-authored-by: Thomas E Lackey <telackey@bozemanpass.com>
Co-committed-by: Thomas E Lackey <telackey@bozemanpass.com>
* Add Postgres exporter and it's dashboard
* Use templating for watcher dashboard
* Add subgraph related panels to watcher dashboard
* Remove individual watcher dashboards and update instructions
* Deploy osmosis on Urbit fake ship
* Remove Urbit configuration from existing osmosis stack
* Add a separate stack for Osmosis front end on Urbit
* Run script for renaming build files with bash
* Add environment variables required in urbit osmosis build
* Fix BASEPATH in compose file
* Remove ipfs-glob-host from network config in osmosis readme
* Use laconic branch for osmosis frontend
---------
Co-authored-by: Nabarun <nabarun@deepstacksoft.com>
* Setup Prometheus and Grafana for monitoring stack
* Add dashboard for azimuth watchers
* Add a dashboard for sushiswap watcher
* Persist prometheus server data
* Additional metrics in watcher dashboards
* Update dashboards and add for merkl sushiswap watcher
* Add dashboards for remaining azimuth watchers
* Separate out preconfigured watcher dashboards and add instructions
* Keep the empty dashboards dir
* Refactor to make Urbit app deployment script generic
* Rename urbit pod and update instructions
* Add a flag to allow skipping app installation on Urbit
* Make remote Urbit app deployment scripts generic
* Move remote deployment scripts to urbit fixturenet
* Update and use existing kubo pod for Urbit glob hosting
* Separate out uniswap gql proxy in a stack
* Use proxy server from watcher-ts
* Add a flag to enable/disable the proxy server
* Update env configuratoin for uniswap urbit app stack
* Update stack file for uniswap urbit app stack
* Fix env variables in instructions
* Use IPFS for hosting glob files for Urbit
* Add env configuration for IPFS endpoints to instructions
* Make ship pier dir configurable in remote deployment script
* Update remote deployment script to accept glob hash arg
* Create uniswap-frontend stack
* Add stack for building uniswap frontend app
* Add a container for Urbit fake ship
* Update with deployment command
* Add a service for uniswap app deployment to urbit
* Use a script to start urbit ship to handle restarts
* Rename stack name to uniswap-urbit-app
* Rename build.sh to build-app.sh and check if build already exists
* Rename stack directory name
* Update uniswap build restart on failure
* Perform uniswap app deployment in the urbit container
* Add steps to create glob for the app
* Tail /dev/null after deployment
* Add steps to install the app to desk
* Host glob files for uniswap
* Update repo branch
* Update readme with command to get urbit password
* Update readme
* Update readme to open urbit web UI
* Expose the port on glob hosting container
* Avoid exposing urbit http port
* Add scripts for installing uniswap on remote urbit instance
* Configure GQL proxy for uniswap app
* Use laconic branch for app repo
* Rename urbit pod for uniswap app deployment
---------
Co-authored-by: Prathamesh Musale <prathamesh.musale0@gmail.com>
* Update stack to run azimuth job runner
* Run azimuth watcher in active mode
* Update stack to run job-runners for all watchers
* Update ports in job-runner health checks
* Map metrics ports to host
* Configure historical block processing batch size for Azimuth watcher
* Use deployment command for azimuth stack
---------
Co-authored-by: neeraj <neeraj.rtly@gmail.com>
Co-authored-by: Prathamesh Musale <prathamesh.musale0@gmail.com>
This fixes three issues:
1. #644 (build output)
2. #646 (error on startup)
3. automatic env quote handling (related to 2)
For the build output we now have:
```
#################################################################
Built host container for /home/telackey/tmp/iglootools-home with tag:
cerc/iglootools-home:local
To test locally run:
docker run -p 3000:3000 cerc/iglootools-home:local
```
For the startup error, it was hung waiting for the "success" message from the next generate output (itself a workaround for a nextjs bug fixed by this PR we submitted: https://github.com/vercel/next.js/pull/58276).
I added a timeout which will cause it to wait up to a maximum _n_ seconds before issuing:
```
ERROR: 'npm run cerc_generate' exceeded CERC_MAX_GENERATE_TIME.
```
On the quoting itself, I plan on adding a new run-webapp command, but I realized I had a decent spot to do effect the quote replacement on-the-fly after all when I am already escaping the values for insertion/replacement into JS.
The "dequoting" can be disabled with `CERC_RETAIN_ENV_QUOTES=true`.
* Upgrade merkl-sushiswap-v3-watcher-ts release
* Increase blockDelayInMilliSecs for merkl-sushiswap-v3 watcher
* Upgrade sushiswap-v3-watcher-ts release
* Add sushiswap-v3 watcher to stack list
* Avoid mapping ports that are not required to be exposed
* Add a sushiswap-v3 watcher stack
* Add services for watcher db and server
* Add service for watcher job-runner
* Use 0.0.0.0 for watcher server config
---------
Co-authored-by: Nabarun <nabarun@deepstacksoft.com>
* Separate ponder indexer and ponder watcher and add second ponder indexer
* Handle review changes
* Update config to point ponder watcher to indexer 2 to indexer 1
* Update Ponder demo
* Use deployed ERC20 contract in second Ponder indexer
* Add order by timestamp in Ponder watcher app entities query
* Upgrade go-nitro version to v0.1.2-ts-port-0.1.9
* Decrease Ponder start block to process contract transfer event at deployment
---------
Co-authored-by: Shreerang Kale <shreerangkale@gmail.com>
* Use durable store for in-process Nitro node
* Update setup for external go-nitro node
* Add a separate service for ipld-eth-server with remote Nitro node
* Update repo branches / versions
* Wait for external Nitro node endpoint and update instructions
* Update repo branches
* Add support to pass ratesFile in config
* Change branch for ponder to laconic-esm
* Fix repo name for ponder
---------
Co-authored-by: Shreerang Kale <shreerangkale@gmail.com>
* Remove reverse payment proxy service from payment stack
* Remove run-reverse-payment-proxy.sh
* Remove reverse payment proxy port from readme
---------
Co-authored-by: Shreerang Kale <shreerangkale@gmail.com>
* Use ponder in watcher mode and indexer mode separately in payments stack
* Refactor config file and configure env variables for watcher mode
* Update demo.md for payments stack
* Handle review changes
* Setup config to pay for watcher to indexer GQL queries
* Fix config in stack for making payments in watcher ponder app
* Update demo for payment from watcher to indexer mode Ponder apps
* Use laconic-esm brannch for ponder
---------
Co-authored-by: Shreerang Kale <shreerangkale@gmail.com>
* First part of deployments for external repos
* Generate deployment dir
* Create empty config file
* Copy script files into deployment
* Run scripts in deployment
* Refactor
* Integrate external plugins
* Remove debug output
* Add a fixturenet-payments stack
* Export the WebSocket port in fixturenet-eth-geth service
* Add container to run a go-nitro node
* Add container to deploy Nitro contracts
* Read contract addresses from a volume when running the Nitro node
* Add a service for Nitro reverse payment proxy
* Expose payment proxy endpoint to be accessible from host
* Map nitro node messaging and payment proxy ports to host
* Use container to deploy Nitro contracts in mobymask-v3 stack
* Use a common contract deployment script from mobymask-v3 stack
* Add MobyMask contract deployment and watcher services
* Fixes for contract deployment and watcher scripts
* Add a container and service for mobymask-snap
* Add MobyMask app service
* Add container and service for a ponder app
* Fix ponder setup and update instructions
* Handle review comments
* Use enablepaidrpcmethods flag in reverse payment proxy server
* Update go-nitro branch
* Fixes for mobymask-v3 stack
---------
Co-authored-by: Nabarun <nabarun@deepstacksoft.com>
* Add a mobymask-v3 stack
* Fix Nitro deployment script and add watcher container
* Setup Nitro config
* Run build after setting Nitro addresses
* Setup consensus config
* Add a container for web-app
* Use node 18 for the web-app
* Persist Nitro node data to a volume
* Add clean up steps
* Update query rates
* Add steps to force rebuild and persist peers_ids volume
* Update mobymask-v2 stack with pubsub option
* Update watcher-ts version
---------
Co-authored-by: Nabarun <nabarun@deepstacksoft.com>
* Update mobymask v1 watcher with new contract
* Update mobymask v1 stack with deployment commands
* Use release tag for mobymask-watcher-ts repo
* Upgrade MobyMask version in v2 stack to use latest contract
* Update sushiswap-subgraph stack to point to filecoin endpoint
* Deploy blocks subgraph in sushiswap-subgraph stack
* Update subgraph config
* Remove duplicate nativePricePool
* Enable debug logs in graph-node and update instructions
* Two additional miner nodes in fixturenet-lotus
* Revert experimental change for additional miner nodes
* Set ETHEREUM_REORG_THRESHOLD env for graph-node to catch up to head
* Take ETH RPC endpoint for graph-node from env
* Set default values for RPC endpoint in graph-node
* Rename fixturenet-graph-node pod to graph-node
* Clean up sushiswap-subgraph stack
* Use deployment command in sushiswap-subgraph stack
* Add a separate fixturenet-sushiswap-subgraph stack
* Fix pods in fixturenet-sushiswap-subgraph
* Fix fixturenet subgraph deployment commands and instructions
---------
Co-authored-by: Nabarun <nabarun@deepstacksoft.com>
* Use ip utility to get the required miner node multiaddr
* Persist lotus node data to support restarts
* Add clean up steps to instructions
* Fix lotus-seed sector-dir arg
* Add a sushiswap stack with contract deployments
* Add watcher services
* Add a service for the info app
* Add instructions to run smoke tests
* Use sushi-info-watcher in demo mode
* Turn off block prefetching
* Fix sushiswap demo instructions
* Use release version and add healthcheck in Lotus stack
* Wait for Lotus node to start before sushiswap watchers
---------
Co-authored-by: Nabarun <nabarun@deepstacksoft.com>
* move health check inside startup script
* remove pre-built genesis
* move health check inside startup script
* remove pre-built genesis
* Use hardcoded paths for Lotus node data directories
* Persist proof parameters
* Write out miner node's multiaddr with docker network IP
* Enable Lotus ETH RPC API and bind to all available interfaces
* Fund a known account
---------
Co-authored-by: iskay <ikay@lakeheadu.ca>
Co-authored-by: Ian Kay <ian@knowable.vc>
* forward more vars for debugging
forward CERC_GETH_VERBOSITY
forward CERC_STATEDIFF_DB_LOG_STATEMENTS
forward CERC_REMOTE_DEBUG
* fix env var
* remote flag can be set from env
* random nits
* geth - visibility of migration status
* forward CERC_RUN_STATEDIFF to geth container
* fix ipld-eth-server vars
* fix fixturenet-eth-loaded stack
* fixturenet geth genesis - include mergeNetsplitBlock
* forward CERC_STATEDIFF_DB_GOOSE_MIN_VER to env file
* add TAG_SUFFIX arg to lighthouse build
intended to avoid sporadic failures when running lcli on github CI runners, likely related to non-portable builds
* Add a stack for Gelato watcher
* Add option to create and use a state snapshot
* Add commands to create and import a state checkpoint
* Rename ipld-eth-server endpoint env variables
* Fix default env variable
Former-commit-id: 8b4b5deba8
* Setup gateway-server with watchers
* Add js script to merge toml config files
* Remove individual watcher configs
* Add all azimuth watchers in stack
* Fix toml-js install
* Use env variables for ipld-eth-server endpoints
* Checkout to version tag in azimuth-watcher-ts repo
Former-commit-id: 5a94aed7f7
* add fixturenet-gaia stack
* add fixturenet-pocket
* integrate with eth fixturenet
* separate out fixturenet-gaia
* use pocket-deployments Dockerfile
---------
Co-authored-by: iskay <ian@knowable.vc>
Co-authored-by: Ian <ikay@lakeheadu.ca>
Former-commit-id: b23b5ae3bf
* Remove unnecessary check on watcher endpoint
* Add instructions to run MobyMask app with a watcher on network
* Move watcher on network docs to a separate folder
* Add nginx config for watcher endpoint
* Add expected output logs
* Add sample nginx config for hosting the app
* Update instructions
Former-commit-id: 018950858b
* Use the latest stable optimism release
* Remove unnecessary repos from repo-list
* Add op-proposer service to fixturenet-optimism stack
* Add jq and bash to op-proposer image
* Update instructions
* Update op-batcher and op-geth commands
Former-commit-id: 988be0ef9a
* Refactor L2 enpoint check to contract deployment script
* Add instructions to join to an existing watcher network
* Include mobymask-v2-watcher-ts in repositories setup
* Add a clean up section and expected outputs
* Add a troubleshooting section
* Use lxdao frontend
* Update instructions for updated UI
Former-commit-id: f78176a27f
* Add container to stack for lxdao mobymask-app
* Remove shm_size
* Use cerc-io scoped alias for lxdao app package
* Change alias to @cerc-io/mobymask-ui-lxdao
Former-commit-id: 46b36c3cb6
* Use standalone mobymask-v2-watcher-ts to run peer test
* Add watcher-ts image for running peer tests
* Run separate containers for peer ids generation and tests
* Wait for watcher to be up before starting peer-test-app
* Resolve peer-test-app compose file and remove setup-repositories for web-apps
Former-commit-id: c4002dcc5c
* Use mounted volumes for data in geth nodes
* Use mounted volumes for data in lighthouse nodes
* Avoid resetting genesis time in a lighthouse node on restart
* Mount parent datadir for lighthouse nodes
* Trap signals on shutdown and clean up in lighthouse nodes
* Allow stalled sync in lighthouse beacon nodes
* Gracefully shutdown geth nodes
* Add clean up instructions
* Gracefully shutdown lighthouse boot node
Former-commit-id: 3130af1615
* Build MobyMask web-app at container build step
* Fix web-app start script to use env variables in config
* Replace variables in built web-app files
* Use published mobymask-ui package from gitea
* Use published react-peer/test-app from gitea
* Remove local gitea publish TODO
Former-commit-id: cf79f0de0a
* Fix contract deployment script in fixturenet-optimism stack
* Configure relay node's announce domain from env
* Configure relay peers list for the relay node from env
* Create and use peer ids from a mounted volume
* Fix command to create watcher config
* Fix mobymask-app deployment script
Former-commit-id: 882f0b16aa
* Add an option to pass env file to deploy command
* Use env variable mapping in fixturenet-optimism stack
* Use default values from checked in env files
* Use env variable mapping in mobymask-v2 stack
* Update instructions
* Add extra hosts in app compose files and update instructions
* Add CERC prefix to env variables in fixturenet-optimism stack
* Add CERC prefix to env variables in mobymask-v2 stack
Former-commit-id: 6b62247ef7
{"type":"create","timestamp":"2026-03-08T06:56:07.080584539Z","issue_id":"so-076","payload":{"description":"Currently laconic-so maps one stack to one deployment to one pod. All containers\nin a stack's compose files become containers in a single k8s pod. This means:\n\n- Can't upgrade doublezero without restarting agave-validator\n- Can't restart monitoring without disrupting the validator\n- Can't independently scale or lifecycle-manage components\n\nThe fix is stack composition. A meta-stack (e.g. biscayne-stack) composes\nsub-stacks (agave, doublezero, agave-monitoring), each becoming its own\nk8s Deployment with independent lifecycle.","priority":"2","title":"Stack composition: deploy multiple stacks into one kind cluster","type":"epic"}}
{"type":"create","timestamp":"2026-03-08T06:56:07.551986919Z","issue_id":"so-ab0","payload":{"description":"Add laconic-so deployment prepare that creates cluster infrastructure without pods. Already implemented, needs review.","priority":"2","title":"deployment prepare command","type":"task"}}
{"type":"create","timestamp":"2026-03-08T06:56:07.884418759Z","issue_id":"so-04f","payload":{"description":"deployment stop on ANY deployment deletes the shared kind cluster. Should only delete its own namespace.","priority":"2","title":"deployment stop should not destroy shared cluster","type":"bug"}}
{"type":"create","timestamp":"2026-03-08T06:56:08.253520249Z","issue_id":"so-370","payload":{"description":"Allow stack.yml to reference sub-stacks. Each sub-stack becomes its own k8s Deployment sharing namespace and PVs.","priority":"2","title":"Add stacks: field to stack.yml for composition","type":"task"}}
{"type":"create","timestamp":"2026-03-08T06:56:08.646764337Z","issue_id":"so-f7c","payload":{"description":"Create three independent stacks from the monolithic agave-stack. Each gets its own compose file and independent lifecycle.","priority":"2","title":"Split agave-stack into agave + doublezero + monitoring","type":"task"}}
Make the stack-orchestrator repository easier for AI tools (Claude Code, Cursor, Copilot) to understand and use for generating stacks, including adding a `create-stack` command.
---
## Part 1: Documentation & Context Files
### 1.1 Add CLAUDE.md
Create a root-level context file for AI assistants.
**File:**`CLAUDE.md`
Contents:
- Project overview (what stack-orchestrator does)
- Stack creation workflow (step-by-step)
- File naming conventions
- Required vs optional fields in stack.yml
- Common patterns and anti-patterns
- Links to example stacks (simple, medium, complex)
This file provides guidance to Claude Code when working with the stack-orchestrator project.
## Some rules to follow
NEVER speculate about the cause of something
NEVER assume your hypotheses are true without evidence
ALWAYS clearly state when something is a hypothesis
ALWAYS use evidence from the systems your interacting with to support your claims and hypotheses
ALWAYS run `pre-commit run --all-files` before committing changes
## Key Principles
### Development Guidelines
- **Single responsibility** - Each component has one clear purpose
- **Fail fast** - Let errors propagate, don't hide failures
- **DRY/KISS** - Minimize duplication and complexity
## Development Philosophy: Conversational Literate Programming
### Approach
This project follows principles inspired by literate programming, where development happens through explanatory conversation rather than code-first implementation.
### Core Principles
- **Documentation-First**: All changes begin with discussion of intent and reasoning
- **Narrative-Driven**: Complex systems are explained through conversational exploration
- **Justification Required**: Every coding task must have a corresponding TODO.md item explaining the "why"
- **Iterative Understanding**: Architecture and implementation evolve through dialogue
### Working Method
1.**Explore and Understand**: Read existing code to understand current state
2.**Discuss Architecture**: Workshop complex design decisions through conversation
3.**Document Intent**: Update TODO.md with clear justification before coding
4.**Explain Changes**: Each modification includes reasoning and context
5.**Maintain Narrative**: Conversations serve as living documentation of design evolution
### Implementation Guidelines
- Treat conversations as primary documentation
- Explain architectural decisions before implementing
- Use TODO.md as the "literate document" that justifies all work
- Maintain clear narrative threads across sessions
- Workshop complex ideas before coding
This approach treats the human-AI collaboration as a form of **conversational literate programming** where understanding emerges through dialogue before code implementation.
## External Stacks Preferred
When creating new stacks for any reason, **use the external stack pattern** rather than adding stacks directly to this repository.
- **When something times out that doesn't mean it needs a longer timeout it means something that was expected never happened, not that we need to wait longer for it.**
- **NEVER change a timeout because you believe something truncated, you don't understand timeouts, don't edit them unless told to explicitly by user.**
@@ -6,7 +6,7 @@ Stack Orchestrator allows building and deployment of a Laconic Stack on a single
## Install
**To get started quickly** on a fresh Ubuntu instance (e.g, Digital Ocean); [try this script](./scripts/quick-install-ubuntu.sh). **WARNING:** always review scripts prior to running them so that you know what is happening on your machine.
**To get started quickly** on a fresh Ubuntu instance (e.g, Digital Ocean); [try this script](./scripts/quick-install-linux.sh). **WARNING:** always review scripts prior to running them so that you know what is happening on your machine.
For any other installation, follow along below and **adapt these instructions based on the specifics of your system.**
@@ -16,6 +16,7 @@ Ensure that the following are already installed:
- [Python3](https://wiki.python.org/moin/BeginnersGuide/Download): `python3 --version` >= `3.8.10` (the Python3 shipped in Ubuntu 20+ is good to go)
Note: if installing docker-compose via package manager on Linux (as opposed to Docker Desktop), you must [install the plugin](https://docs.docker.com/compose/install/linux/#install-the-plugin-manually), e.g. :
Next decide on a directory where you would like to put the stack-orchestrator program. Typically this would be
Next decide on a directory where you would like to put the stack-orchestrator program. Typically this would be
a "user" binary directory such as `~/bin` or perhaps `/usr/local/laconic` or possibly just the current working directory.
Now, having selected that directory, download the latest release from [this page](https://github.com/cerc-io/stack-orchestrator/tags) into it (we're using `~/bin` below for concreteness but edit to suit if you selected a different directory). Also be sure that the destination directory exists and is writable:
Now, having selected that directory, download the latest release from [this page](https://git.vdb.to/cerc-io/stack-orchestrator/tags) into it (we're using `~/bin` below for concreteness but edit to suit if you selected a different directory). Also be sure that the destination directory exists and is writable:
If Stack Orchestrator was installed using the process described above, it is able to subsequently self-update to the current latest version by running:
```bash
laconic-so update
```
## Usage
Three sub-commands: `setup-repositories`, `build-containers` and `deploy-system` are generally run in order. The following is a slim example for standing up the `erc20-watcher`. Go further with the [erc20 watcher demo](/app/data/stacks/erc20) and other pieces of the stack, within the [`stacks` directory](/app/data/stacks).
The various [stacks](/stack_orchestrator/data/stacks) each contain instructions for running different stacks based on your use case. For example:
We need an "update stack" command in stack orchestrator and cleaner documentation regarding how to do continuous deployment with and without payments.
**Context**: Currently, `deploy init` generates a spec file and `deploy create` creates a deployment directory. The `deployment update` command (added by Thomas Lackey) only syncs env vars and restarts - it doesn't regenerate configurations. There's a gap in the workflow for updating stack configurations after initial deployment.
## Bugs
### `deploy create` doesn't auto-generate volume mappings for new pods
When a new pod is added to `stack.yml` (e.g. `monitoring`), `deploy create`
does not generate default host path mappings in spec.yml for the new pod's
volumes. The deployment then fails at scheduling because the PVCs don't exist.
**Expected**: `deploy create` enumerates all volumes from all compose files
in the stack and generates default host paths for any that aren't already
mapped in the spec.yml `volumes:` section.
**Actual**: Only volumes already in spec.yml get PVs. New volumes are silently
missing, causing `FailedScheduling: persistentvolumeclaim not found`.
**Workaround**: Manually add volume entries to spec.yml and create host dirs.
The deployer logic should be decoupled from the CLI tool to allow independent development and reuse.
### Separate Stacks from Stack Orchestrator Repo
Stacks should live in their own repositories, not bundled with the orchestrator tool. This allows stacks to evolve independently and be maintained by different teams.
Instructions for deploying a local Laconic blockchain "fixturenet" for development and testing purposes using laconic-stack-orchestrator.
## 1. Install Laconic Stack Orchestrator
Installation is covered in detail [here](https://github.com/cerc-io/stack-orchestrator#user-mode) but if you're on Linux and already have docker installed it should be as simple as:
Instructions to setup and deploy an end-to-end L1+L2 stack with [fixturenet-eth](../fixturenet-eth/) (L1) and [Optimism](https://stack.optimism.io) (L2)
We support running just the L2 part of stack, given an external L1 endpoint. Follow [l2-only](./l2-only.md) for the same.
This should create the required docker images in the local image registry:
*`cerc/go-ethereum`
*`cerc/lighthouse`
*`cerc/fixturenet-eth-geth`
*`cerc/fixturenet-eth-lighthouse`
*`cerc/foundry`
*`cerc/optimism-contracts`
*`cerc/optimism-l2geth`
*`cerc/optimism-op-batcher`
*`cerc/optimism-op-node`
## Deploy
Deploy the stack:
```bash
laconic-so --stack fixturenet-optimism deploy up
```
The `fixturenet-optimism-contracts` service may take a while (`~15 mins`) to complete running as it:
1. waits for the 'Merge' to happen on L1
2. waits for a finalized block to exist on L1 (so that it can be taken as a starting block for roll ups)
3. deploys the L1 contracts
To list down and monitor the running containers:
```bash
laconic-so --stack fixturenet-optimism deploy ps
# With status
docker ps
# Check logs for a container
docker logs -f <CONTAINER_ID>
```
## Clean up
Stop all services running in the background:
```bash
laconic-so --stack fixturenet-optimism deploy down
```
Clear volumes created by this stack:
```bash
# List all relevant volumes
docker volume ls -q --filter "name=.*fixturenet_geth_accounts|.*l1_deployment|.*l2_accounts|.*l2_config|.*l2_geth_data"
# Remove all the listed volumes
docker volume rm $(docker volume ls -q --filter "name=.*fixturenet_geth_accounts|.*l1_deployment|.*l2_accounts|.*l2_config|.*l2_geth_data")
```
## Troubleshooting
* If `op-geth` service aborts or is restarted, the following error might occur in the `op-node` service:
```bash
WARN [02-16|21:22:02.868] Derivation process temporary error attempts=14 err="stage 0 failed resetting: temp: failed to find the L2 Heads to start from: failed to fetch L2 block by hash 0x0000000000000000000000000000000000000000000000000000000000000000: failed to determine block-hash of hash 0x0000000000000000000000000000000000000000000000000000000000000000, could not get payload: not found"
```
* This means that the data directory that `op-geth` is using is corrupted and needs to be reinitialized; the containers `op-geth`, `op-node` and `op-batcher` need to be started afresh:
This should create the required docker images in the local image registry
## Deploy
### Configuration
* In [mobymask-params.env](../../config/watcher-mobymask-v2/mobymask-params.env) file set `DEPLOYED_CONTRACT` to existing deployed mobymask contract address
* Setting `DEPLOYED_CONTRACT` will skip contract deployment when running stack
*`ENABLE_PEER_L2_TXS` is used to enable/disable sending txs to L2 chain from watcher peer.
* Update the [optimism-params.env](../../config/watcher-mobymask-v2/optimism-params.env) file with Optimism endpoints and other params for the Optimism running separately
*`PRIVATE_KEY_PEER` is used by watcher peer to send txs to L2 chain
* NOTE:
* Stack Orchestrator needs to be run in [`dev`](/docs/CONTRIBUTING.md#install-developer-mode) mode to be able to edit the env file
* If Optimism is running on the host machine, use `host.docker.internal` as the hostname to access the host port
### Deploy the stack
```bash
laconic-so --stack mobymask-v2 deploy --include watcher-mobymask-v2 up
This should create the required docker images in the local image registry
## Deploy
### Configuration
* Update the [mobymask-params.env](../../config/watcher-mobymask-v2/mobymask-params.env) file with watcher endpoints and other params required by the web-apps
*`WATCHER_HOST` and `WATCHER_PORT` is used to check if watcher is up before building and deploying mobymask-app
*`APP_WATCHER_URL` is used by mobymask-app to make GQL queries
*`DEPLOYED_CONTRACT` and `CHAIN_ID` is used by mobymask-app in app config when creating messgaes for L2 txs
*`RELAY_NODES` is used by the web-apps to connect to the relay nodes (run in watcher)
* NOTE:
* Stack Orchestrator needs to be run in [`dev`](/docs/CONTRIBUTING.md#install-developer-mode) mode to be able to edit the env file
* If watcher is running on the host machine, use `host.docker.internal` as the hostname to access the host port
### Deploy the stack
For running mobymask-app
```bash
laconic-so --stack mobymask-v2 deploy --include mobymask-app up
```
For running peer-test-app
```bash
laconic-so --stack mobymask-v2 deploy --include peer-test-app up
```
To list down and monitor the running containers:
```bash
docker ps
# Check logs for a container
docker logs -f <CONTAINER_ID>
```
## Clean up
Stop all services running in the background:
For mobymask-app
```bash
laconic-so --stack mobymask-v2 deploy --include mobymask-app down
```
For peer-test-app
```bash
laconic-so --stack mobymask-v2 deploy --include peer-test-app down
```
Clear volumes created by this stack:
```bash
# List all relevant volumes
docker volume ls -q --filter "name=.*mobymask_deployment"
# Remove all the listed volumes
docker volume rm $(docker volume ls -q --filter "name=.*mobymask_deployment")
The MobyMask watcher is a Laconic Network component that provides efficient access to MobyMask contract data from Ethereum, along with evidence allowing users to verify the correctness of that data. The watcher source code is available in [this repository](https://github.com/cerc-io/watcher-ts/tree/main/packages/mobymask-watcher) and a developer-oriented Docker Compose setup for the watcher can be found [here](https://github.com/cerc-io/mobymask-watcher). The watcher can be deployed automatically using the Laconic Stack Orchestrator tool as detailed below:
## Deploy the MobyMask Watcher
The instructions below show how to deploy a MobyMask watcher using laconic-stack-orchestrator (the installation of which is covered [here](https://github.com/cerc-io/stack-orchestrator#user-mode)).
This deployment expects that ipld-eth-server's endpoints are available on the local machine at http://ipld-eth-server.example.com:8083/graphql and http://ipld-eth-server.example.com:8082. More advanced configurations are supported by modifying the watcher's [config file](../../config/watcher-mobymask/mobymask-watcher.toml).
$ laconic-so deploy-system --include watcher-mobymask up
```
Correct operation should be verified by following the instructions [here](https://github.com/cerc-io/mobymask-watcher/tree/main/mainnet-watcher-only#run), checking GraphQL queries return valid results in the watcher's [playground](http://127.0.0.1:3001/graphql).
## Clean up
Stop all the services running in background:
```bash
$ laconic-so deploy-system --include watcher-mobymask down
The Package Registry Stack supports a build environment that requires a package registry (initially for NPM packages only).
Setup instructions can be found [here](../build-support/README.md).
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.