You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
bug(compute): a deleted sandbox stays in the store for ~5 min when the driver's gone event hits a CAS conflict #5769
Under load (tens of sandboxes created and deleted concurrently, Docker compute driver), a sandbox whose DeleteSandbox succeeded (HTTP 200, ~0.4 s) and whose container is gone is still returned by GetSandbox for about five minutes. A client that waits for the sandbox to be gone (polling GetSandbox until NotFound) times out.
Seen with v0.1.8 (f3d9765); the code below is unchanged on main (6c3d773).
Why
When the compute driver reports the container gone, the watch loop applies the event with a single compare-and-swap (update_sandbox_record, crates/openshell-server/src/compute/mod.rs, around line 5733). On a conflict it returns "concurrent modification detected during sandbox reconciliation", which watch_loop logs as Failed to apply compute driver event and drops. The comment says "On conflict the next watch event will naturally retry", but after a deletion no further event comes for that sandbox.
The record is then removed only by the periodic reconcile (RECONCILE_INTERVAL, 1 min), and only once the sandbox has been missing from the driver's snapshot for ORPHAN_GRACE_PERIOD (5 min, line 558): "Removing sandbox from store after it disappeared from the compute driver snapshot".
DeleteSandbox itself updates the same record, so a deletion and the driver's event for it conflict often. #5227 (reconciliation advancing resource_version on unchanged sandboxes) likely makes conflicts more frequent.
Evidence
In one run: 48 Failed to apply compute driver event ... concurrent modification warnings in the gateway log; three deleted sandboxes traced, removed 302 s, 318 s and 350 s after a successful DeleteSandbox.
Suggested fix
Retry the CAS for a terminal (deleted/gone) driver event after re-reading the record, as other paths already do ("concurrent modification detected after {attempt} attempts"); or, on a conflict for a sandbox already in the Deleting phase whose resource is gone, remove it from the store directly.
No, not Kubernetes. It was the Docker compute driver: a single-host gateway (v0.1.8) on macOS, with Docker provided by OrbStack, and tens of sandboxes created and deleted concurrently by a test suite.
The code path itself is driver-independent, though: the dropped event is in the gateway's watch loop (watch_loop → apply_watch_event → update_sandbox_record's single CAS in crates/openshell-server/src/compute/mod.rs), and the 5-minute removal is the shared ORPHAN_GRACE_PERIOD reconcile, so I'd expect any driver whose "gone" event can race a DeleteSandbox update to show it.
Activity
FrostGod commentedon Nov 19, 2026
@jbadeau is the issue with k8s deployment of openshell ?
jbadeau commentedon Nov 19, 2026
No, not Kubernetes. It was the Docker compute driver: a single-host gateway (v0.1.8) on macOS, with Docker provided by OrbStack, and tens of sandboxes created and deleted concurrently by a test suite.
The code path itself is driver-independent, though: the dropped event is in the gateway's watch loop (
watch_loop→apply_watch_event→update_sandbox_record's single CAS incrates/openshell-server/src/compute/mod.rs), and the 5-minute removal is the sharedORPHAN_GRACE_PERIODreconcile, so I'd expect any driver whose "gone" event can race aDeleteSandboxupdate to show it.jbadeau commentedon Nov 19, 2026
Cool project BTW, you guys saved me allot of time!