Skip to content

bug: k3s-ha delete/recreate flake exposes ambiguous sandbox deletion results #5844

Open

Description

@matthewgrossman

User Story

I use OpenShell for prerelease qualification and saw the SSH host identity conformance test fail on k3s-ha after deleting and recreating a sandbox. I need deletion completion to reflect the original sandbox's absence so qualification can reliably exercise same-name recreation.

Problem Statement

In v0.1.10-pre.3 attempt 1, stop/start and SSH succeeded. Delete of test-ixd1xhjz-hk returned success at 11:20:54.383593Z and the SDK's wait for the old sandbox ID returned. Same-name creation started at 11:20:54.415222Z but Kubernetes still held default--test-ixd1xhjz-hk and rejected it with AlreadyExists. Attempt 2 passed at the same SHA.

PR #5843 addresses one Kubernetes driver path that can cause this result. A deterministic regression reproduces a rejected DELETE being reported as absence; the original Kubernetes DELETE response was not captured in CI, so that diagnosis is inferred.

This issue tracks the wider contract gap: DeleteSandboxResponse.deleted=false says the request did not delete a resource, but the gateway treats it as proof the original sandbox is absent and releases its name without checking. Audit the false paths across compute drivers, including Podman and VM; the same bug has not been established in those drivers. Compare an identity-scoped gateway lookup on false with tightening the driver contract or introducing explicit outcomes for confirmed absence and accepted asynchronous deletion.

Impact / Why This Matters

Release qualification can fail intermittently even though startup and SSH work. OpenShell callers that wait for deletion can still see AlreadyExists on recreate. Rerunning qualification can pass but does not make the deletion signal reliable.

Acceptance Criteria

  • An ambiguous driver deletion result cannot make the gateway report the original sandbox as gone or release its name.
  • Confirmed absence is tied to the original sandbox ID. A same-name replacement is not queried or deleted as if it were the original.
  • Audit the relevant false return paths and document the chosen gateway/driver contract for asynchronous deletion and confirmed absence.
  • Add deterministic coverage for both confirmed absence and an ambiguous result where the original still exists.

Reproduction Steps

  1. Run the OpenShell ssh_host_identity_stable_across_stop_start_and_new_on_recreate conformance case against a k3s-ha OpenShell gateway.
  2. The case creates a sandbox, verifies its SSH host identity across stop/start, deletes it, and uses SDK wait_deleted with the original sandbox ID.
  3. It immediately creates a sandbox under the same name. In the failing run, create returned AlreadyExists even though the wait had completed. The race is intermittent.

Environment

  • OpenShell v0.1.10-pre.3, SHA 93761e1578aecf7bb953dc7e651fd0c85a850f25
  • OpenShell release qualification: ubuntu-k3s / k3s-ha, Kubernetes compute driver
  • Failing job · Conformance artifact

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions