You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
perf(supervisor): stop sending the cluster-wide route list in every supervisor update #5783
I size OpenShell Kubernetes clusters with a shared supervisor pool (#5748). I need supervisor updates to stay small as the cluster grows, so large clusters don't hit message limits or flood the gateway.
Problem Statement
Every update the gateway sends a shared supervisor (AssignmentSnapshot) includes route_directory: where every sandbox in the whole cluster runs. Supervisors use it only to redirect a sandbox that reconnects to the wrong supervisor through the Service.
The update size grows with the total number of sandboxes, not with the supervisor's own sandboxes. That's about 54 bytes per sandbox, so 5.4 MB at 100k sandboxes.
The gateway re-sends the full list to every supervisor on every create and delete, so traffic grows with sandboxes × supervisors.
Impact / Why This Matters
Supervisors with small per-supervisor limits reject updates (they exceed the receive limit) and keep reconnecting before the cluster is full. Reported in #5756.
Total capacity is held at 131072 sandboxes mainly to bound this list.
Gateway CPU, memory, and network use grow quickly with cluster size.
Proposed Design
A supervisor's update carries only its own sandboxes.
A sandbox that reaches the wrong supervisor is still redirected to its owner.
Update size depends only on sandboxes per supervisor, not on cluster size.
The exact mechanism is open for discussion; see Alternatives Considered.
Acceptance Criteria
Supervisor update size does not grow with sandboxes owned by other supervisors.
A misrouted sandbox reconnect still reaches its owner, including after a failover.
A test combines a small per-supervisor limit with a large cluster.
The 131072 total limit can be raised or removed.
Alternatives Considered
Ask the gateway on a miss: a supervisor asks the gateway for an unknown route's owner, then redirects. The gateway already stores the owners.
Connect straight to the owner: the gateway gives each sandbox its owner's address, so most connections skip the Service. This still needs a fallback after a failover.
Compute the owner from a hash of the route: no list needed, but it conflicts with least-loaded placement, per-supervisor limits, and failover moving sandboxes.
Send only what changed in the list: smaller updates, but every supervisor still holds the full list.
Raise the receive limit (short-term): fixes the rejections for small per-supervisor limits, but not the growth.
Activity