Skip to content

Scheduler: prefer a live candidate over a completed one in try_subscribe - #2885

Closed
b7r6 wants to merge 1 commit into
TraceMachina:mainfrom
b7r6:fork/upstream-fix-sched-try-subscribe-live
Closed

b7r6 wants to merge 1 commit into
TraceMachina:mainfrom
b7r6:fork/upstream-fix-sched-try-subscribe-live

Conversation

@b7r6

@b7r6 b7r6 commented Oct 3, 2026

Copy link
Copy Markdown
Contributor

What and why

try_subscribe in store_awaited_action_db.rs stopped at the first exact unique-qualifier match, even when that match was a completed record. The unique-qualifier index retains completed records alongside any live successor, and the backend does not order results by state, so a retained completed record could be returned ahead of a joinable live one — forcing an unnecessary recreate and a brand-new operation instead of joining the live successor (duplicate scheduling of the same action). This change scans every exact match and prefers a joinable (neither finished nor abandoned) candidate, keeping the first exact match only as a fallback. Note: this is the selection half of the issue; a fully atomic qualifier-to-live-op claim (two concurrent same-qualifier submissions) needs a compare-and-set store primitive that does not exist today and is out of scope here.

How was this verified?

Found via our fleet's scheduler consistency chaos suite. New regression coverage in nativelink-scheduler/tests/store_awaited_action_db_test.rs (~170 lines) constructs a retained completed record ahead of a live successor for the same qualifier and asserts the live one is selected — fails on current main, passes with the fix. cargo check -p nativelink-scheduler --tests passes on this branch.

Risk

Low. A 27-line change to candidate selection in one function; when only one candidate exists, behavior is unchanged, and the first-match fallback preserves today's behavior when no joinable candidate is found.

AI assistance

Yes. The fix was developed on our fleet-hardened fork with AI assistance (chaos-test triage and patch authoring), and an agent curated and rebased it onto current main for this PR. A human reviewed the diff before opening.

@vercel

vercel Bot commented Oct 3, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
nativelink Ready Ready Preview Oct 3, 2026 9:41pm UTC
nativelink-aidm Ready Ready Preview Oct 3, 2026 9:41pm UTC

Request Review

`try_subscribe` stopped at the first exact unique-qualifier match, even
when that match was a completed record. The unique-qualifier index retains
completed records alongside any live successor, and the backend does not
order results by state, so a retained completed record could be returned
ahead of a joinable live one — forcing an unnecessary recreate and a new
operation instead of joining the live successor.

Scan every exact match and prefer a joinable (neither finished nor
abandoned) candidate, keeping the first exact match only as a fallback.

This is the selection half of finding N7. The atomic qualifier->live-op
claim (two concurrent same-qualifier submissions can still both create an
operation) needs a store primitive that does not exist today (a
compare-and-set on the shared qualifier key); that is tracked separately.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
(cherry picked from commit c8a7b7b)

This branch was successfully deployed

2 active deployments
Preview – nativelink — c4f37faf Deployed Oct 3, 2026 by vercel[bot]
Preview – nativelink-aidm — c4f37faf Deployed Oct 3, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants