Skip to content

fix(scheduler): read the memory store's listing whole, never under the lock inside the stream - #3013

Merged
MarcusSorealheis merged 1 commit into
TraceMachina:mainfrom
b7r6:fix/memory-listing-read-whole
Oct 22, 2026
Merged

MarcusSorealheis merged 1 commit into
TraceMachina:mainfrom
b7r6:fix/memory-listing-read-whole

Conversation

@b7r6

@b7r6 b7r6 commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

What and why

Upstream main since #3001 (4edf8224ea, 2026-10-22) wedges the scheduler within ~15 s of real remote-execution load. Found on the railgun soak fleet with bazel build //... of this repository through three workers; it reproduced on every run of #3001 and of main, and never on v1.7.7, #2954 or v1.8.0.

Mechanism. #3001 reads listed rows 64 at a time with buffered, which keeps asking the listing for the next row while it hands rows downstream. The memory store's listing took the store's lock inside the stream (a ChunkedStream whose chunk fetch awaits inner.lock()). Under load the next chunk fetch was left parked inside the stream, waiting for the lock, while the matcher wrote for a row it had been handed, on the same task, and waited for the same lock. Once the parked wait had starved past async-lock's 0.5 ms fairness threshold the lock was reserved for it; nothing polled it; every state update blocked. Symptoms: two inner_update_operation START lines with no SUCCESS (one from the matcher, one from the worker API), the state DB silent, then "timed out after 10 seconds with no update from its worker" for actions the workers had already finished and uploaded, and nothing dispatched again. CAS and AC on the same process keep serving.

Fix. The listing is read whole under one hold of the lock, taken in the async fn before the stream exists; the stream then only hands out what was read. The chunk fetch already read the whole range in one go, so the snapshot is exactly what the stream yielded before; only the lock moves out of it. Rows are still handed on one at a time (the matcher's mid-pass placement relies on that; a chunked pipeline broke room_opening_mid_pass_keeps_the_listing_order_across_shapes and lost_parked_assignment_skips_its_shape_until_the_next_pass), and buffered keeps its concurrency for the Redis store, whose reads are round trips.

How was this verified?

Risk

The memory listing now allocates a Vec of subscriber handles (a watch::Receiver clone each) for the whole range at request time instead of lazily. Rows added after the listing was requested are not in it; the matcher runs passes continuously and the lazy listing's window for them was the first poll anyway.

AI assistance

Investigated and written with Claude Code on the railgun soak fleet; b7r6 reviewed the mechanism and the fix.

🤖 Generated with Claude Code

…e lock inside the stream

Since 4edf822 (TraceMachina#3001) the filter pass reads listed rows 64 at a time
with `buffered`, which keeps asking the listing for the next row while it
hands rows downstream. The memory store's listing took the store's lock
inside the stream (a `ChunkedStream` whose chunk fetch awaits
`inner.lock()`), so under load the next chunk fetch was left parked inside
the stream, waiting for the lock, while the matcher wrote for a row it had
been handed, on the same task, and waited for the same lock. Once the
parked wait had starved past async-lock's fairness threshold the lock was
reserved for it, nothing polled it, and the scheduler stopped for good:
every state update blocked, actions the workers had finished and uploaded
"timed out with no update from their worker", and nothing dispatched
again. On railgun a `bazel build //...` of this repository through a
three-worker fleet wedged within 15 s on TraceMachina#3001 and on main, every time;
v1.7.7, TraceMachina#2954 and v1.8.0 run it clean.

The listing is now read whole under one hold of the lock, taken in the
`async fn` before the stream exists, and the stream only hands out what
was read. The chunk fetch already read the whole range in one go, so the
snapshot is what the stream yielded before; the lock just moves out of it.
Rows are still handed on one at a time as they are read, which the
matcher's mid-pass placement relies on, and `buffered` keeps its
concurrency for the Redis store, whose reads are round trips.

The test pins what the fix relies on: a row added after the listing was
requested is not in it. The lazy listing read it at the first poll.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@vercel

vercel Bot commented Oct 22, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
nativelink Ready Ready Preview Oct 22, 2026 1:35pm UTC
nativelink-aidm Ready Ready Preview Oct 22, 2026 1:35pm UTC

Request Review

@MarcusSorealheis
MarcusSorealheis merged commit 13939b5 into TraceMachina:main Oct 22, 2026
30 checks passed
b7r6 added a commit to b7r6/nativelink that referenced this pull request Oct 22, 2026
The concurrency chapter had six failure families, all of them about
state you can see: a counter, a record, a timestamp, a map entry. It
had nothing about the liveness contract of a future, which is the
category TraceMachina#3001 fell into — a lock acquisition that starts and then
stops being polled. No cycle, no guard held across an await, no lint
that fires; the lock ends up free with every task waiting on it.

Adds:

- Family 7 in the concurrency chapter: the four-layer anatomy of
  TraceMachina#3001/TraceMachina#3013, the ways a started acquisition gets parked, how to
  recognise the wedge in the field, and why no unit test reaches it.
  Notes that this tree has layer 1 already — the lock lives inside the
  listing stream at memory_awaited_action_db.rs:990 and :1025 — and is
  held safe only by the current consumer's laziness. One `buffered`
  re-creates the bug.
- Chapter 8, "Locks, and Who Owns the State": the census (69 async
  acquisitions across nine production files, seven lock flavours from
  four crates), what each flavour's fairness costs, the undocumented
  three-lock order the matcher crosses on every action, the absent
  `await_holding_lock` lint, the Redis listing as the next instance,
  and the level-1 discipline that removes the class.
- Appendix H, "Scheduler State Ownership": the single-owner refactor
  and the openraft mapping, with the HA question that has to be
  decided before level 3 means anything.

Also rewrites the concurrency chapter's tooling section, which claimed
a scheduler DST fuzzer that is not in this tree, to describe the
harness that is (current-thread `nativelink_test` plus `MockClock`)
and to state the two limits that matter: hand-arranged interleavings
only, and a current-thread runtime cannot reach family 7 at all.

Every count and line citation is reproducible against this tree; the
commands are in the chapter. Where the source gist described upstream
main at v1.8.0+7, the numbers here were re-measured here, and two
diverge: both central scheduler locks are `async_lock::Mutex` (not one
tokio and one async-lock), and nativelink-worker has 9 acquisitions,
not 11.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

This branch was successfully deployed

2 active deployments
Preview – nativelink — 2a23827b Deployed Oct 22, 2026 by vercel[bot]
Preview – nativelink-aidm — 2a23827b Deployed Oct 22, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants