Repository navigation
fix(scheduler): read the memory store's listing whole, never under the lock inside the stream - #3013
Merged
MarcusSorealheis merged 1 commit intoOct 22, 2026
Conversation
…e lock inside the stream Since 4edf822 (TraceMachina#3001) the filter pass reads listed rows 64 at a time with `buffered`, which keeps asking the listing for the next row while it hands rows downstream. The memory store's listing took the store's lock inside the stream (a `ChunkedStream` whose chunk fetch awaits `inner.lock()`), so under load the next chunk fetch was left parked inside the stream, waiting for the lock, while the matcher wrote for a row it had been handed, on the same task, and waited for the same lock. Once the parked wait had starved past async-lock's fairness threshold the lock was reserved for it, nothing polled it, and the scheduler stopped for good: every state update blocked, actions the workers had finished and uploaded "timed out with no update from their worker", and nothing dispatched again. On railgun a `bazel build //...` of this repository through a three-worker fleet wedged within 15 s on TraceMachina#3001 and on main, every time; v1.7.7, TraceMachina#2954 and v1.8.0 run it clean. The listing is now read whole under one hold of the lock, taken in the `async fn` before the stream exists, and the stream only hands out what was read. The chunk fetch already read the whole range in one go, so the snapshot is what the stream yielded before; the lock just moves out of it. Rows are still handed on one at a time as they are read, which the matcher's mid-pass placement relies on, and `buffered` keeps its concurrency for the Redis store, whose reads are round trips. The test pins what the fix relies on: a row added after the listing was requested is not in it. The lazy listing read it at the first poll. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
palfrey
approved these changes
Oct 22, 2026
b7r6
added a commit
to b7r6/nativelink
that referenced
this pull request
Oct 22, 2026
The concurrency chapter had six failure families, all of them about state you can see: a counter, a record, a timestamp, a map entry. It had nothing about the liveness contract of a future, which is the category TraceMachina#3001 fell into — a lock acquisition that starts and then stops being polled. No cycle, no guard held across an await, no lint that fires; the lock ends up free with every task waiting on it. Adds: - Family 7 in the concurrency chapter: the four-layer anatomy of TraceMachina#3001/TraceMachina#3013, the ways a started acquisition gets parked, how to recognise the wedge in the field, and why no unit test reaches it. Notes that this tree has layer 1 already — the lock lives inside the listing stream at memory_awaited_action_db.rs:990 and :1025 — and is held safe only by the current consumer's laziness. One `buffered` re-creates the bug. - Chapter 8, "Locks, and Who Owns the State": the census (69 async acquisitions across nine production files, seven lock flavours from four crates), what each flavour's fairness costs, the undocumented three-lock order the matcher crosses on every action, the absent `await_holding_lock` lint, the Redis listing as the next instance, and the level-1 discipline that removes the class. - Appendix H, "Scheduler State Ownership": the single-owner refactor and the openraft mapping, with the HA question that has to be decided before level 3 means anything. Also rewrites the concurrency chapter's tooling section, which claimed a scheduler DST fuzzer that is not in this tree, to describe the harness that is (current-thread `nativelink_test` plus `MockClock`) and to state the two limits that matter: hand-arranged interleavings only, and a current-thread runtime cannot reach family 7 at all. Every count and line citation is reproducible against this tree; the commands are in the chapter. Where the source gist described upstream main at v1.8.0+7, the numbers here were re-measured here, and two diverge: both central scheduler locks are `async_lock::Mutex` (not one tokio and one async-lock), and nativelink-worker has 9 acquisitions, not 11. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
Upstream
mainsince #3001 (4edf8224ea, 2026-10-22) wedges the scheduler within ~15 s of real remote-execution load. Found on the railgun soak fleet withbazel build //...of this repository through three workers; it reproduced on every run of #3001 and ofmain, and never on v1.7.7, #2954 or v1.8.0.Mechanism. #3001 reads listed rows 64 at a time with
buffered, which keeps asking the listing for the next row while it hands rows downstream. The memory store's listing took the store's lock inside the stream (aChunkedStreamwhose chunk fetch awaitsinner.lock()). Under load the next chunk fetch was left parked inside the stream, waiting for the lock, while the matcher wrote for a row it had been handed, on the same task, and waited for the same lock. Once the parked wait had starved pastasync-lock's 0.5 ms fairness threshold the lock was reserved for it; nothing polled it; every state update blocked. Symptoms: twoinner_update_operation STARTlines with noSUCCESS(one from the matcher, one from the worker API), the state DB silent, then "timed out after 10 seconds with no update from its worker" for actions the workers had already finished and uploaded, and nothing dispatched again. CAS and AC on the same process keep serving.Fix. The listing is read whole under one hold of the lock, taken in the
async fnbefore the stream exists; the stream then only hands out what was read. The chunk fetch already read the whole range in one go, so the snapshot is exactly what the stream yielded before; only the lock moves out of it. Rows are still handed on one at a time (the matcher's mid-pass placement relies on that; a chunked pipeline brokeroom_opening_mid_pass_keeps_the_listing_order_across_shapesandlost_parked_assignment_skips_its_shape_until_the_next_pass), andbufferedkeeps its concurrency for the Redis store, whose reads are round trips.How was this verified?
Bisect on the fleet, pure upstream binaries, same configs and workload each time:
843115085c), v1.8.021ae17884e4edf8224eamainb01417643emain+ this fixbazel test //nativelink-scheduler/...: 27/27 on the fix.The new test
the_listing_is_read_whole_when_it_is_requestedfails on the old listing (4 rows, expected 3) and passes on the fix. A deterministic unit test of the deadlock itself isn't possible: in a single-threaded runtime the memory store's lock is never contended at the moment the prefetch polls it, and a multi-threaded stress test is confounded by executor behaviour. The fleet is the reproduction, and the fixed binary completes the same build every time.Risk
The memory listing now allocates a
Vecof subscriber handles (awatch::Receiverclone each) for the whole range at request time instead of lazily. Rows added after the listing was requested are not in it; the matcher runs passes continuously and the lazy listing's window for them was the first poll anyway.AI assistance
Investigated and written with Claude Code on the railgun soak fleet; b7r6 reviewed the mechanism and the fix.
🤖 Generated with Claude Code