multi-port-service #980

Merged

AFDudley merged 26 commits from multi-port-service into main 2026-01-24 23:05:14 +00:00

Conversation 0 Commits 26 Files Changed 136

+4334 -1193

AFDudley commented 2026-01-24 20:18:55 +00:00

Owner

Copy Link

Copy Source

No description provided.

AFDudley added 22 commits 2026-01-24 20:18:56 +00:00

Add support for generating Helm charts when creating a deployment (#974) ... 7acabb0743

Part of https://plan.wireit.in/deepstack/browse/VUL-265/

- Added a flag `--helm-chart` to `deploy create` command
- Uses Kompose CLI wrapper to generate a helm chart from compose files in a stack
- To be handled in a follow on PR(s):
  - Templatize generated charts and generate a `values.yml` file with defaults

Reviewed-on: #974
Co-authored-by: Prathamesh Musale <prathamesh.musale0@gmail.com>
Co-committed-by: Prathamesh Musale <prathamesh.musale0@gmail.com>

Add support for running jobs from a stack (#975) ... 8afae1904b

Part of https://plan.wireit.in/deepstack/browse/VUL-265/

Reviewed-on: #975
Co-authored-by: Prathamesh Musale <prathamesh.musale0@gmail.com>
Co-committed-by: Prathamesh Musale <prathamesh.musale0@gmail.com>

Add CLAUDE.md, pre-commit config, and pyproject.toml ... 9bd59f29d9

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Add Caddy ingress and k8s cluster management features ... 89db6e1e92

- Add Caddy ingress controller manifest for kind deployments
- Add k8s cluster list command for kind cluster management
- Add k8s_command import and registration in deploy.py
- Fix network section merge to preserve http-proxy settings
- Increase default container resources (4 CPUs, 8GB memory)
- Add UDP protocol support for K8s port definitions
- Add command/entrypoint support for K8s deployments
- Implement docker-compose variable expansion for K8s
- Set ConfigMap defaultMode to 0755 for executable scripts

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Apply pre-commit linting fixes ... 5a1399f2b2

Fix trailing whitespace and end-of-file issues across codebase.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Add missing get_kind_cluster function to helpers.py ... d8da9b6515

Fixes ImportError in k8s_command.py that was causing CI failure.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Add flake8 to pre-commit hooks ... b708836aa9

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Trigger test-container-registry CI ... 2e384b7179

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Retrigger test-container-registry CI c5c3fc1618

Fix misleading error message in load_images_into_kind dc36a6564a

Add black, pyright, yamllint to pre-commit hooks ... ba1aad9fa6

- Add black formatter (rev 23.12.1)
- Add pyright type checker (rev v1.1.345)
- Add yamllint with relaxed mode (rev v1.35.1)
- Update flake8 args: max-line-length=88, extend-ignore=E203,W503,E402
- Remove ansible-lint from dev dependencies (no ansible files in repo)
- Sync pyproject.toml flake8 config with pre-commit

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Add unlimited-memlock support for Kind clusters ... 03f9acf869

Add spec.yml option `security.unlimited-memlock` that configures
RLIMIT_MEMLOCK to unlimited for Kind cluster pods. This is needed
for workloads like Solana validators that require large amounts of
locked memory for memory-mapped files during snapshot decompression.

When enabled, generates a cri-base.json file with rlimits and mounts
it into the Kind node to override the default containerd runtime spec.

Also includes flake8 line-length fixes for affected files.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Apply pre-commit linting fixes ... cd3d908d0d

- Format code with black (line length 88)
- Fix E501 line length errors by breaking long strings and comments
- Fix F841 unused variable (removed unused 'quiet' variable)
- Configure pyright to disable common type issues in existing codebase
  (reportGeneralTypeIssues, reportOptionalMemberAccess, etc.)
- All pre-commit hooks now pass

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Fix pyright type errors across codebase ... dd856af2d3

- Add pyrightconfig.json for pyright 1.1.408 TOML parsing workaround
- Add NoReturn annotations to fatal() functions for proper type narrowing
- Add None checks and assertions after require=True get_record() calls
- Fix AttrDict class with __getattr__ for dynamic attribute access
- Add type annotations and casts for Kubernetes client objects
- Store compose config as DockerDeployer instance attributes
- Filter None values from dotenv and environment mappings
- Use hasattr/getattr patterns for optional container attributes

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Add RuntimeClass support for unlimited RLIMIT_MEMLOCK ... 87db167d7f

The previous approach of mounting cri-base.json into kind nodes failed
because we didn't tell containerd to use it via containerdConfigPatches.

RuntimeClass allows different stacks to have different rlimit profiles,
which is essential since kind only supports one cluster per host and
multiple stacks share the same cluster.

Changes:
- Add containerdConfigPatches to kind-config.yml to define runtime handlers
- Create RuntimeClass resources after cluster creation
- Add runtimeClassName to pod specs based on stack's security settings
- Rename cri-base.json to high-memlock-spec.json for clarity
- Add get_runtime_class() method to Spec that auto-derives from
  unlimited-memlock setting

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Fix high-memlock spec to include complete OCI runtime config ... 86462c940f

The base_runtime_spec for containerd requires a complete OCI spec,
not just the rlimits section. The minimal spec was causing runc to
fail with "open /proc/self/fd: no such file or directory" because
essential mounts and namespaces were missing.

This commit uses kind's default cri-base.json as the base and adds
the rlimits configuration on top. The spec includes all necessary
mounts, namespaces, capabilities, and kind-specific hooks.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Add extra_args support to deploy create command ... ffa00767d4

- Add @click.argument for generic args passthrough to stack commands
- Keep explicit --network-dir and --initial-peers options
- Add DeploymentContext.get_compose_file() helper
- Add DeploymentContext.modify_yaml() helper for stack commands
- Update init() to use absolute paths

This allows stack-specific create commands to receive arbitrary
arguments via: laconic-so deploy create ... -- --custom-arg value

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Fix helpers.py to use Caddy ingress instead of nginx ... 97a85359ff

The helm-charts-with-caddy branch had the Caddy manifest file but was still
using nginx in the code. This change:

- Switch install_ingress_for_kind() to use ingress-caddy-kind-deploy.yaml
- Update wait_for_ingress_in_kind() to watch caddy-system namespace
- Use correct label selector for Caddy ingress controller pods

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Add port 443 mapping for kind clusters with Caddy ingress ... 638435873c

Caddy provides automatic HTTPS with Let's Encrypt, but needs port 443
mapped from the kind container to the host. Previously only port 80 was
mapped.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Use caddy ingress class instead of nginx in cluster_info.py ... 8d9682eb47

The ingress annotation was still set to nginx class even though we're now
using Caddy as the ingress controller. Caddy won't pick up ingresses
annotated with the nginx class.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Add TODO.md with planned features and refactoring ... 811bbd9db4

- Update stack command for continuous deployment workflow
- Separate deployer from CLI
- Separate stacks from orchestrator repo

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Expose all ports from http-proxy routes in k8s Service ... 4f01054781

Previously get_service() only exposed the first port from pod definition.
Now it collects all unique ports from http-proxy routes and exposes them
all in the Service spec.

This is needed for WebSocket support where RPC runs on one port (8899)
and WebSocket pubsub on another (8900) - both need to be accessible
through the ingress.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

AFDudley added 3 commits 2026-01-24 21:39:40 +00:00

Limit test workflow PR triggers to main branch only ... d4e935484f

Previously these workflows ran on PRs to any branch. Now:
- PRs to main: run all tests (full CI gate)
- Pushes to other branches: use existing path filtering

This reduces CI load on feature branch PRs while maintaining
full test coverage for PRs targeting main.

Affected workflows:
- test-k8s-deploy.yml
- test-k8s-deployment-control.yml
- test-webapp.yml
- test-deploy.yml

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Fix invalid docker command in webapp-test ... 99db75da19

Change 'docker remove -f' to 'docker rm -f' - the 'remove' subcommand
doesn't exist in docker CLI.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Check for None before creating k8s service ... a5b373da26

get_service() returns None when there are no http-proxy routes,
so we must check before calling create_namespaced_service().

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

AFDudley added 1 commit 2026-01-24 21:48:53 +00:00

Merge origin/main into multi-port-service ... d07a3afd27

Resolve conflicts:
- deployment_context.py: Keep single modify_yaml method from main
- fixturenet-optimism/commands.py: Use modify_yaml helper from main
- deployment_create.py: Keep helm-chart, network-dir, initial-peers options
- deploy_webapp.py: Update create_operation call signature

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

AFDudley self-assigned this 2026-01-24 22:59:02 +00:00

AFDudley merged commit 55b76b9b57 into main 2026-01-24 23:05:14 +00:00

AFDudley referenced this issue from a commit 2026-01-24 23:05:15 +00:00

Merge pull request 'multi-port-service' (#980) from multi-port-service into main

This repo is archived. You cannot comment on pull requests.

Reviewers

No Reviewers

Labels

Clear labels

K8s

bug

Something isn't working

documentation

Improvements or additions to documentation

duplicate

This issue or pull request already exists

enhancement

New feature or request

feature

good first issue

Good for newcomers

help wanted

Extra attention is needed

in progress

invalid

This doesn't seem right

question

Further information is requested

wontfix

This will not be worked on

Copied from Github

An issue or PR manually copied from GitHub.

Kind/Breaking

Breaking change that won't be backward compatible

Kind/Bug

Something is not working

Kind/Documentation

Documentation changes

Kind/Enhancement

Improve existing functionality

Kind/Feature

New functionality

Kind/Security

This is security issue

Kind/Testing

Issue or pull request related to testing

Priority

Critical

The priority is critical

Priority

High

The priority is high

Priority

Low

The priority is low

Priority

Medium

The priority is medium

Reviewed

Confirmed

Issue has been confirmed

Reviewed

Duplicate

This issue or pull request already exists

Reviewed

Invalid

Invalid issue

Reviewed

Won't Fix

This issue won't be fixed

Status

Abandoned

Somebody has started to work on this but abandoned work

Status

Blocked

Something is blocking this issue or pull request

Status

Need More Info

Feedback is required to reproduce issue or to continue work

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

AFDudley Alex ashwin cerccicd ericlewis i-norden (Ian Norden) prathamesh (Prathamesh Musale) rachmaninovquar rjk-laconic roysc (Roy Crihfield) srw zramsay

No Assignees AFDudley

1 Participants

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: cerc-io/stack-orchestrator#980