multi-port-service #980

Merged
AFDudley merged 26 commits from multi-port-service into main 2026-01-24 23:05:14 +00:00

26 Commits

Author SHA1 Message Date
A. F. Dudley
d07a3afd27 Merge origin/main into multi-port-service
All checks were successful
Lint Checks / Run linter (push) Successful in 24m22s
Lint Checks / Run linter (pull_request) Successful in 23m2s
Deploy Test / Run deploy test suite (pull_request) Successful in 25m37s
K8s Deploy Test / Run deploy test suite on kind/k8s (pull_request) Successful in 28m31s
K8s Deployment Control Test / Run deployment control suite on kind/k8s (pull_request) Successful in 27m46s
Webapp Test / Run webapp test suite (pull_request) Successful in 27m34s
Smoke Test / Run basic test suite (pull_request) Successful in 28m59s
Resolve conflicts:
- deployment_context.py: Keep single modify_yaml method from main
- fixturenet-optimism/commands.py: Use modify_yaml helper from main
- deployment_create.py: Keep helm-chart, network-dir, initial-peers options
- deploy_webapp.py: Update create_operation call signature

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-24 16:48:11 -05:00
A. F. Dudley
a5b373da26 Check for None before creating k8s service
All checks were successful
Lint Checks / Run linter (push) Successful in 4m1s
K8s Deploy Test / Run deploy test suite on kind/k8s (push) Successful in 10m28s
K8s Deployment Control Test / Run deployment control suite on kind/k8s (push) Successful in 13m7s
Lint Checks / Run linter (pull_request) Successful in 14m12s
Deploy Test / Run deploy test suite (pull_request) Successful in 19m51s
K8s Deploy Test / Run deploy test suite on kind/k8s (pull_request) Successful in 24m55s
K8s Deployment Control Test / Run deployment control suite on kind/k8s (pull_request) Successful in 24m33s
Webapp Test / Run webapp test suite (pull_request) Successful in 20m56s
Smoke Test / Run basic test suite (pull_request) Successful in 22m48s
get_service() returns None when there are no http-proxy routes,
so we must check before calling create_namespaced_service().

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-24 16:39:11 -05:00
A. F. Dudley
99db75da19 Fix invalid docker command in webapp-test
Change 'docker remove -f' to 'docker rm -f' - the 'remove' subcommand
doesn't exist in docker CLI.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-24 16:39:00 -05:00
A. F. Dudley
d4e935484f Limit test workflow PR triggers to main branch only
Previously these workflows ran on PRs to any branch. Now:
- PRs to main: run all tests (full CI gate)
- Pushes to other branches: use existing path filtering

This reduces CI load on feature branch PRs while maintaining
full test coverage for PRs targeting main.

Affected workflows:
- test-k8s-deploy.yml
- test-k8s-deployment-control.yml
- test-webapp.yml
- test-deploy.yml

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-24 15:44:54 -05:00
A. F. Dudley
4f01054781 Expose all ports from http-proxy routes in k8s Service
Some checks failed
Lint Checks / Run linter (push) Successful in 6m2s
Lint Checks / Run linter (pull_request) Successful in 5m1s
Deploy Test / Run deploy test suite (pull_request) Successful in 10m57s
K8s Deploy Test / Run deploy test suite on kind/k8s (pull_request) Failing after 13m32s
K8s Deployment Control Test / Run deployment control suite on kind/k8s (pull_request) Failing after 17m24s
Webapp Test / Run webapp test suite (pull_request) Failing after 21m3s
Smoke Test / Run basic test suite (pull_request) Successful in 21m54s
Previously get_service() only exposed the first port from pod definition.
Now it collects all unique ports from http-proxy routes and exposes them
all in the Service spec.

This is needed for WebSocket support where RPC runs on one port (8899)
and WebSocket pubsub on another (8900) - both need to be accessible
through the ingress.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-24 15:14:48 -05:00
A. F. Dudley
811bbd9db4 Add TODO.md with planned features and refactoring
All checks were successful
Lint Checks / Run linter (push) Successful in 4m1s
- Update stack command for continuous deployment workflow
- Separate deployer from CLI
- Separate stacks from orchestrator repo

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-24 10:43:12 -05:00
A. F. Dudley
8d9682eb47 Use caddy ingress class instead of nginx in cluster_info.py
All checks were successful
Lint Checks / Run linter (push) Successful in 14s
The ingress annotation was still set to nginx class even though we're now
using Caddy as the ingress controller. Caddy won't pick up ingresses
annotated with the nginx class.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-22 03:41:35 -05:00
A. F. Dudley
638435873c Add port 443 mapping for kind clusters with Caddy ingress
All checks were successful
Lint Checks / Run linter (push) Successful in 14s
Caddy provides automatic HTTPS with Let's Encrypt, but needs port 443
mapped from the kind container to the host. Previously only port 80 was
mapped.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-22 03:35:03 -05:00
A. F. Dudley
97a85359ff Fix helpers.py to use Caddy ingress instead of nginx
All checks were successful
Lint Checks / Run linter (push) Successful in 14s
The helm-charts-with-caddy branch had the Caddy manifest file but was still
using nginx in the code. This change:

- Switch install_ingress_for_kind() to use ingress-caddy-kind-deploy.yaml
- Update wait_for_ingress_in_kind() to watch caddy-system namespace
- Use correct label selector for Caddy ingress controller pods

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-22 03:22:07 -05:00
A. F. Dudley
ffa00767d4 Add extra_args support to deploy create command
All checks were successful
Lint Checks / Run linter (push) Successful in 13s
- Add @click.argument for generic args passthrough to stack commands
- Keep explicit --network-dir and --initial-peers options
- Add DeploymentContext.get_compose_file() helper
- Add DeploymentContext.modify_yaml() helper for stack commands
- Update init() to use absolute paths

This allows stack-specific create commands to receive arbitrary
arguments via: laconic-so deploy create ... -- --custom-arg value

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-22 03:06:45 -05:00
A. F. Dudley
86462c940f Fix high-memlock spec to include complete OCI runtime config
All checks were successful
Lint Checks / Run linter (push) Successful in 14s
The base_runtime_spec for containerd requires a complete OCI spec,
not just the rlimits section. The minimal spec was causing runc to
fail with "open /proc/self/fd: no such file or directory" because
essential mounts and namespaces were missing.

This commit uses kind's default cri-base.json as the base and adds
the rlimits configuration on top. The spec includes all necessary
mounts, namespaces, capabilities, and kind-specific hooks.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-22 02:12:11 -05:00
A. F. Dudley
87db167d7f Add RuntimeClass support for unlimited RLIMIT_MEMLOCK
All checks were successful
Lint Checks / Run linter (push) Successful in 14s
The previous approach of mounting cri-base.json into kind nodes failed
because we didn't tell containerd to use it via containerdConfigPatches.

RuntimeClass allows different stacks to have different rlimit profiles,
which is essential since kind only supports one cluster per host and
multiple stacks share the same cluster.

Changes:
- Add containerdConfigPatches to kind-config.yml to define runtime handlers
- Create RuntimeClass resources after cluster creation
- Add runtimeClassName to pod specs based on stack's security settings
- Rename cri-base.json to high-memlock-spec.json for clarity
- Add get_runtime_class() method to Spec that auto-derives from
  unlimited-memlock setting

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-22 01:58:38 -05:00
A. F. Dudley
dd856af2d3 Fix pyright type errors across codebase
- Add pyrightconfig.json for pyright 1.1.408 TOML parsing workaround
- Add NoReturn annotations to fatal() functions for proper type narrowing
- Add None checks and assertions after require=True get_record() calls
- Fix AttrDict class with __getattr__ for dynamic attribute access
- Add type annotations and casts for Kubernetes client objects
- Store compose config as DockerDeployer instance attributes
- Filter None values from dotenv and environment mappings
- Use hasattr/getattr patterns for optional container attributes

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-22 01:10:36 -05:00
A. F. Dudley
cd3d908d0d Apply pre-commit linting fixes
- Format code with black (line length 88)
- Fix E501 line length errors by breaking long strings and comments
- Fix F841 unused variable (removed unused 'quiet' variable)
- Configure pyright to disable common type issues in existing codebase
  (reportGeneralTypeIssues, reportOptionalMemberAccess, etc.)
- All pre-commit hooks now pass

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-21 20:58:31 -05:00
A. F. Dudley
03f9acf869 Add unlimited-memlock support for Kind clusters
All checks were successful
Lint Checks / Run linter (push) Successful in 14s
Add spec.yml option `security.unlimited-memlock` that configures
RLIMIT_MEMLOCK to unlimited for Kind cluster pods. This is needed
for workloads like Solana validators that require large amounts of
locked memory for memory-mapped files during snapshot decompression.

When enabled, generates a cri-base.json file with rlimits and mounts
it into the Kind node to override the default containerd runtime spec.

Also includes flake8 line-length fixes for affected files.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-21 20:20:19 -05:00
A. F. Dudley
ba1aad9fa6 Add black, pyright, yamllint to pre-commit hooks
- Add black formatter (rev 23.12.1)
- Add pyright type checker (rev v1.1.345)
- Add yamllint with relaxed mode (rev v1.35.1)
- Update flake8 args: max-line-length=88, extend-ignore=E203,W503,E402
- Remove ansible-lint from dev dependencies (no ansible files in repo)
- Sync pyproject.toml flake8 config with pre-commit

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-21 20:04:15 -05:00
A. F. Dudley
dc36a6564a Fix misleading error message in load_images_into_kind
All checks were successful
Lint Checks / Run linter (push) Successful in 14s
2026-01-21 19:32:53 -05:00
A. F. Dudley
c5c3fc1618 Retrigger test-container-registry CI
Some checks failed
Lint Checks / Run linter (push) Successful in 14s
Container Registry Test / Run contaier registry hosting test on kind/k8s (push) Failing after 1m36s
2026-01-21 19:28:29 -05:00
A. F. Dudley
2e384b7179 Trigger test-container-registry CI
Some checks failed
Lint Checks / Run linter (push) Successful in 14s
Container Registry Test / Run contaier registry hosting test on kind/k8s (push) Failing after 2m33s
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-21 19:12:05 -05:00
A. F. Dudley
b708836aa9 Add flake8 to pre-commit hooks
All checks were successful
Lint Checks / Run linter (push) Successful in 14s
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-21 19:05:12 -05:00
A. F. Dudley
d8da9b6515 Add missing get_kind_cluster function to helpers.py
All checks were successful
Lint Checks / Run linter (push) Successful in 13s
Fixes ImportError in k8s_command.py that was causing CI failure.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-21 19:04:46 -05:00
A. F. Dudley
5a1399f2b2 Apply pre-commit linting fixes
Some checks failed
Lint Checks / Run linter (push) Successful in 14s
K8s Deployment Control Test / Run deployment control suite on kind/k8s (push) Failing after 31s
Database Test / Run database hosting test on kind/k8s (push) Failing after 31s
Container Registry Test / Run contaier registry hosting test on kind/k8s (push) Failing after 36s
Fix trailing whitespace and end-of-file issues across codebase.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-20 23:16:44 -05:00
A. F. Dudley
89db6e1e92 Add Caddy ingress and k8s cluster management features
- Add Caddy ingress controller manifest for kind deployments
- Add k8s cluster list command for kind cluster management
- Add k8s_command import and registration in deploy.py
- Fix network section merge to preserve http-proxy settings
- Increase default container resources (4 CPUs, 8GB memory)
- Add UDP protocol support for K8s port definitions
- Add command/entrypoint support for K8s deployments
- Implement docker-compose variable expansion for K8s
- Set ConfigMap defaultMode to 0755 for executable scripts

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-20 23:14:22 -05:00
A. F. Dudley
9bd59f29d9 Add CLAUDE.md, pre-commit config, and pyproject.toml
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-20 22:40:59 -05:00
8afae1904b Add support for running jobs from a stack (#975)
All checks were successful
Lint Checks / Run linter (push) Successful in 30s
Part of https://plan.wireit.in/deepstack/browse/VUL-265/

Reviewed-on: #975
Co-authored-by: Prathamesh Musale <prathamesh.musale0@gmail.com>
Co-committed-by: Prathamesh Musale <prathamesh.musale0@gmail.com>
2025-12-04 06:13:28 +00:00
7acabb0743 Add support for generating Helm charts when creating a deployment (#974)
All checks were successful
Lint Checks / Run linter (push) Successful in 29s
Part of https://plan.wireit.in/deepstack/browse/VUL-265/

- Added a flag `--helm-chart` to `deploy create` command
- Uses Kompose CLI wrapper to generate a helm chart from compose files in a stack
- To be handled in a follow on PR(s):
  - Templatize generated charts and generate a `values.yml` file with defaults

Reviewed-on: #974
Co-authored-by: Prathamesh Musale <prathamesh.musale0@gmail.com>
Co-committed-by: Prathamesh Musale <prathamesh.musale0@gmail.com>
2025-11-27 06:43:07 +00:00