Skip to content

bug(compute): a deleted sandbox stays in the store for ~5 min when the driver's gone event hits a CAS conflict #5769

Open

Description

@jbadeau

What happens

Under load (tens of sandboxes created and deleted concurrently, Docker compute driver), a sandbox whose DeleteSandbox succeeded (HTTP 200, ~0.4 s) and whose container is gone is still returned by GetSandbox for about five minutes. A client that waits for the sandbox to be gone (polling GetSandbox until NotFound) times out.

Seen with v0.1.8 (f3d9765); the code below is unchanged on main (6c3d773).

Why

When the compute driver reports the container gone, the watch loop applies the event with a single compare-and-swap (update_sandbox_record, crates/openshell-server/src/compute/mod.rs, around line 5733). On a conflict it returns "concurrent modification detected during sandbox reconciliation", which watch_loop logs as Failed to apply compute driver event and drops. The comment says "On conflict the next watch event will naturally retry", but after a deletion no further event comes for that sandbox.

The record is then removed only by the periodic reconcile (RECONCILE_INTERVAL, 1 min), and only once the sandbox has been missing from the driver's snapshot for ORPHAN_GRACE_PERIOD (5 min, line 558): "Removing sandbox from store after it disappeared from the compute driver snapshot".

DeleteSandbox itself updates the same record, so a deletion and the driver's event for it conflict often. #5227 (reconciliation advancing resource_version on unchanged sandboxes) likely makes conflicts more frequent.

Evidence

In one run: 48 Failed to apply compute driver event ... concurrent modification warnings in the gateway log; three deleted sandboxes traced, removed 302 s, 318 s and 350 s after a successful DeleteSandbox.

Suggested fix

Retry the CAS for a terminal (deleted/gone) driver event after re-reading the record, as other paths already do ("concurrent modification detected after {attempt} attempts"); or, on a conflict for a sandbox already in the Deleting phase whose resource is gone, remove it from the store directly.

Activity

  1. FrostGod commented on Nov 19, 2026

    @FrostGod
    Contributor

    @jbadeau is the issue with k8s deployment of openshell ?

  2. jbadeau commented on Nov 19, 2026

    @jbadeau
    Author

    No, not Kubernetes. It was the Docker compute driver: a single-host gateway (v0.1.8) on macOS, with Docker provided by OrbStack, and tens of sandboxes created and deleted concurrently by a test suite.

    The code path itself is driver-independent, though: the dropped event is in the gateway's watch loop (watch_loop → apply_watch_event → update_sandbox_record's single CAS in crates/openshell-server/src/compute/mod.rs), and the 5-minute removal is the shared ORPHAN_GRACE_PERIOD reconcile, so I'd expect any driver whose "gone" event can race a DeleteSandbox update to show it.

  3. jbadeau commented on Nov 19, 2026

    @jbadeau
    Author

    Cool project BTW, you guys saved me allot of time!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    state:triage-neededOpened without agent diagnostics and needs triage

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions