Skip to content

feat(helm): size cluster sandbox capacity from Helm values - #5756

Open
FrostGod wants to merge 11 commits into
mainfrom
feat/5748-configurable-supervisor-pool-limits/FrostGod
Open

FrostGod wants to merge 11 commits into
mainfrom
feat/5748-configurable-supervisor-pool-limits/FrostGod

Conversation

@FrostGod

@FrostGod FrostGod commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Make shared supervisor pool capacity configurable from Helm, so operators can size a cluster by gateways, supervisors, and sandboxes per supervisor.

Related Issue

Closes #5748

Changes

  • New Helm values under supervisor.pool:
    • maxInstancesPerMember: sandboxes per supervisor, 1–1024, default 128
    • maxMembersPerGateway: supervisors per gateway, 1 or more, default 32
  • Total capacity is live gateway replicas × supervisors per gateway × sandboxes per supervisor, never more than 131072.
  • New sandboxes go to the least-loaded supervisor and never past its limit.
  • At capacity, sandbox create fails right away with FAILED_PRECONDITION: sandbox capacity reached; add supervisors or gateway replicas, or raise the supervisor pool limits. The gateway log says which limit was hit.
  • Message buffers are sized from the limits.
  • Chart defaults:
    • supervisor.replicas: 1 (unchanged)
    • supervisor autoscaling off, maxReplicas: 4
    • supervisor resources unset; the values file recommends requests 250m / 768Mi and a 1.5Gi memory limit
  • Rendering fails when supervisor replicas (or autoscaling maxReplicas) exceed gateway replicas × maxMembersPerGateway.
  • Added a values-scale-10k.yaml profile: 2 gateways × up to 40 supervisors each × 128 sandboxes = 10,240.
  • Docs updated: gateway configuration, the HA guide, and the cluster and dev-environment skills.

Release Notes

Helm users can set shared supervisor capacity with supervisor.pool.maxInstancesPerMember and supervisor.pool.maxMembersPerGateway. Creates beyond capacity fail with "sandbox capacity reached".

Testing

  • Rust unit tests for openshell-supervisor-control, the shared supervisor host, server config, and placement; clippy clean
  • helm unittest (503 tests), mise run helm:lint, test-gateway-config-parser.sh, test-supervisor-pool-scaling.sh
  • Local k3d cluster with 10–20 sandboxes across several limit settings: even spread (7/6/7 on 3 supervisors), creates refused at the limit
  • E2E tests added/updated (if applicable)

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)

@copy-pr-bot

copy-pr-bot Bot commented Nov 19, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@github-actions github-actions Bot added the state:accepted A maintainer decided OpenShell should pursue this issue label Nov 19, 2026
@github-actions

Copy link
Copy Markdown

@FrostGod
FrostGod requested a review from drew November 19, 2026 06:51
@FrostGod

Copy link
Copy Markdown
Contributor Author

/ok to test c17f546

Comment thread deploy/helm/openshell/values.yaml Resolved
@FrostGod

Copy link
Copy Markdown
Contributor Author

/ok to test abd2964

@FrostGod
FrostGod requested a review from TaylorMutch November 19, 2026 21:58
Comment thread deploy/helm/openshell/values.yaml Resolved
@TaylorMutch

Copy link
Copy Markdown
Contributor

Local review: two findings. Both still apply at abd2964a59e669b8c5a8249236a79c5f44afe24a.

  1. Include the pool-wide route directory in the supervisor decode limit. pool_limits.rs:113–115 sizes incoming gateway messages from the per-supervisor cap, but SupervisorHostController::snapshot includes the entire pool's route directory in every member's snapshot. With max_instances_per_member = 16, the decode limit is 4,210,688 bytes. Encoding 100,000 normal route entries (rt- plus 32 hex characters and runtime address 10.0.0.1:5500) with the PR's actual protobuf schema produces a 5,400,002-byte AssignmentSnapshot, before adding any assignments or launch material. This is within the new configurable pool ceiling, but supervisors would reject the snapshots and repeatedly reconnect before the pool reached its configured capacity. Account for the pool-wide directory in the bound, or send it separately; add a serialized-size test combining a small member cap with a large pool.

  2. Extract or download the example values file before passing it to -f. The new example in docs/kubernetes/high-availability.mdx uses helm install openshell <chart> ... -f values-scale-10k.yaml. Helm packages that file, but -f reads a local path; installing a published chart does not extract the file for the caller. Without a separately obtained copy, the command fails with open values-scale-10k.yaml: no such file or directory. Show helm pull --untar and the extracted path, or provide a download step/link.

Validation on the originally reviewed commit, c114e789b933a832a1c6ea3d0b9f5064543b3edf: Helm lint, rendering the 10k profile, and verifying the packaged values file passed. The scaling script stopped on a leading-newline difference after local yq serialization. Rust and cluster tests were not run. The two affected sections were rechecked at the current head above; the full review and validation were not rerun against that update.

@FrostGod

Copy link
Copy Markdown
Contributor Author

Route list counted in the receive limit: "Fixed in b989be8: the supervisor receive limit now adds room for the whole cluster route list at 131072 sandboxes (128 bytes each), with a test that a supervisor limited to 1 sandbox can still accept that list."

-f values-scale-10k.yaml on a published chart: "Fixed in 131fb46: the guide now runs helm pull --untar first and passes -f helm-chart/values-scale-10k.yaml from the extracted chart."

@FrostGod

Copy link
Copy Markdown
Contributor Author

/ok to test b989be8

@FrostGod
FrostGod enabled auto-merge November 20, 2026 19:40
@FrostGod
FrostGod disabled auto-merge November 20, 2026 20:20
@FrostGod
FrostGod enabled auto-merge November 20, 2026 20:20
@FrostGod

FrostGod commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor Author

I have read the DCO document and I hereby sign the DCO

@FrostGod
FrostGod disabled auto-merge November 20, 2026 20:21
@FrostGod
FrostGod enabled auto-merge November 20, 2026 20:21
…imits

Add [openshell.gateway.supervisor_pool] max_instances_per_member (16-1024,
default 128) and max_members_per_gateway (1-64, default 32). The pool cap is
live gateway replicas * members * instances, never above 131072. Placement
picks the least-loaded member with room and never charges a member past its
cap. Supervisors take the same cap through --pool-max-instances and size
their gateway stream buffers from it.

Refs #5748

Signed-off-by: divesh <dgude@nvidia.com>
Render supervisor.pool.maxInstancesPerMember and maxMembersPerGateway into the
gateway config and supervisor env, validate their ranges, and fail the render
when supervisor replicas or HPA maxReplicas exceed gateway replicas *
maxMembersPerGateway. Default to two supervisors with configurable resources
(250m/768Mi requests, 1.5Gi memory limit) and add values-scale-100k.yaml.

Closes #5748

Signed-off-by: divesh <dgude@nvidia.com>
Lower max_instances_per_member and supervisor.pool.maxInstancesPerMember to
1..=1024. Gateway message and status-report bounds keep the floor a cap of 16
gave, since snapshots also carry the pool-wide route directory. Document how
to add gateway replicas for more supervisors.

Refs #5748

Signed-off-by: divesh <dgude@nvidia.com>
…ool limits plumbing

Only sandboxes per supervisor stays bounded (1-1024). Members per gateway
accepts any positive value; the 131072 pool ceiling still applies.
Placement refusals now say "sandbox capacity reached" and name the
limit that was hit.

Refs #5748

Signed-off-by: divesh <dgude@nvidia.com>
…imits

Refs #5748

Signed-off-by: divesh <dgude@nvidia.com>
Two gateways with up to 40 supervisors each, at 128 sandboxes per
supervisor, hold 10,240 sandboxes.

Refs #5748

Signed-off-by: divesh <dgude@nvidia.com>
…tion

Refs #5748

Signed-off-by: divesh <dgude@nvidia.com>
Installing the chart no longer reserves supervisor CPU and memory. The
values file, the HA guide, and the 10k profile carry the recommended
requests and limits instead.

Refs #5748

Signed-off-by: divesh <dgude@nvidia.com>
…file

Refs #5748

Signed-off-by: divesh <dgude@nvidia.com>
Refs #5748

Signed-off-by: divesh <dgude@nvidia.com>
…tory

Every snapshot lists where each instance in the pool runs, so a member
with a small cap rejected snapshots of a large pool. The decode bound now
adds room for the route directory of the largest pool.

Refs #5748

Signed-off-by: divesh <dgude@nvidia.com>
@FrostGod
FrostGod force-pushed the feat/5748-configurable-supervisor-pool-limits/FrostGod branch from b989be8 to 0077bb7 Compare November 20, 2026 20:32

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

state:accepted A maintainer decided OpenShell should pursue this issue

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(helm): size cluster sandbox capacity from Helm values

2 participants