Skip to content

Read a filter pass's listed records concurrently so a deep queue on Redis lists inside the provisioner's timeout - #3001

Merged
amankrx merged 2 commits into
TraceMachina:mainfrom
amankrx:fix/admin-demand-pipelined-keepalive
Oct 22, 2026
Merged

amankrx merged 2 commits into
TraceMachina:mainfrom
amankrx:fix/admin-demand-pipelined-keepalive

Conversation

@amankrx

@amankrx amankrx commented Oct 22, 2026 •

Copy link
Copy Markdown
Member

What and why

A filter pass reads the record behind every listed row one at a time, and on the Redis backend each read is a round trip for the row's client keepalive. A queue of 7,000 took several seconds to list, on every matching pass and on every admin demand listing, which the worker provisioner gives 5 s before it falls back to reading Redis itself. The per-stage listing now reads its rows 64 at a time with an order-preserving buffered, so both callers see the store's order as before; the predicate behind the reads, which can write, still runs one row at a time. The admin listing only has two comments corrected.

How was this verified?

A new state manager test puts a slow store under the listing, longest read first, and counts reads in flight: 100 queued actions list highest priority first with 64 reads in flight at the peak, in 0.14 s. Against the previous serial listing it fails with a peak of 1 after 5.25 s, and against buffer_unordered it fails on the order. The scheduler tests, clippy and a check of the binary pass on 1.98.1 and 1.97.1.

Risk

A listing that fails part way may have up to 63 further record reads in flight; those are reads only. The all-stages listing, which collects and sorts, keeps its serial read.

AI assistance

An agent drafted the change and I reviewed every line.

@vercel

vercel Bot commented Oct 22, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
nativelink Ready Ready Preview Oct 22, 2026 9:46am UTC
nativelink-aidm Ready Ready Preview Oct 22, 2026 9:46am UTC

Request Review

@MarcusSorealheis MarcusSorealheis left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for digging into the slow demand listing. I don't think this change makes it faster on Redis, though, and the new test passes without the fix. Details are inline. The main point is here because it's on lines outside this diff.

The per-record round trip is in the shared listing, so the fix belongs there (code observation). filter_operations runs try_filter_map(borrow_listed) one record at a time (nativelink-scheduler/src/simple_scheduler_state_manager.rs:1629-1631). On Redis, borrow() is where the client keepalive read happens, and its result is cached in the subscriber's listed_with_keepalive OnceCell (store_awaited_action_db.rs:236-249) before the record is yielded. The matcher's get_queued_operations pays the same serial cost. Two options that would fix both callers:

  • run borrow_listed as an order-preserving .map(...).buffered(N) before the predicate, or
  • fetch the keepalive keys for each FT.AGGREGATE page with one MGET in get_range_of_actions.

Either would make the admin-side concurrency unnecessary.

What I ran

  • cargo test -p nativelink-scheduler --test admin_test on 0d2db546: 6/6 pass. Also 6/6 with .buffer_unordered(...) swapped in, and 6/6 with the pre-PR admin.rs restored.
  • nativelink-scheduler builds with the 'a removed from read_demand_entry.

Not checked: no timing against a real or mock Redis (the latency points come from reading the code), no clippy or Bazel run, and no full nativelink build.

Comment thread nativelink-scheduler/src/admin.rs Resolved
Comment thread nativelink-scheduler/src/admin.rs Resolved
Comment thread nativelink-scheduler/src/admin.rs Resolved
Comment thread nativelink-scheduler/src/admin.rs Resolved
Comment thread nativelink-scheduler/src/admin.rs Resolved
Comment thread nativelink-scheduler/src/admin.rs Resolved
… on Redis lists inside the provisioner's timeout
@amankrx amankrx changed the title Read the admin demand listing's records concurrently so a deep queue answers inside the provisioner's timeout Read a filter pass's listed records concurrently so a deep queue on Redis lists inside the provisioner's timeout Oct 22, 2026
@amankrx
amankrx force-pushed the fix/admin-demand-pipelined-keepalive branch from 0d2db54 to 97c78a9 Compare October 22, 2026 09:09

@MarcusSorealheis MarcusSorealheis left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@amankrx
amankrx merged commit 4edf822 into TraceMachina:main Oct 22, 2026
29 of 34 checks passed
MarcusSorealheis pushed a commit that referenced this pull request Oct 22, 2026
…e lock inside the stream (#3013)

Since 4edf822 (#3001) the filter pass reads listed rows 64 at a time
with `buffered`, which keeps asking the listing for the next row while it
hands rows downstream. The memory store's listing took the store's lock
inside the stream (a `ChunkedStream` whose chunk fetch awaits
`inner.lock()`), so under load the next chunk fetch was left parked inside
the stream, waiting for the lock, while the matcher wrote for a row it had
been handed, on the same task, and waited for the same lock. Once the
parked wait had starved past async-lock's fairness threshold the lock was
reserved for it, nothing polled it, and the scheduler stopped for good:
every state update blocked, actions the workers had finished and uploaded
"timed out with no update from their worker", and nothing dispatched
again. On railgun a `bazel build //...` of this repository through a
three-worker fleet wedged within 15 s on #3001 and on main, every time;
v1.7.7, #2954 and v1.8.0 run it clean.

The listing is now read whole under one hold of the lock, taken in the
`async fn` before the stream exists, and the stream only hands out what
was read. The chunk fetch already read the whole range in one go, so the
snapshot is what the stream yielded before; the lock just moves out of it.
Rows are still handed on one at a time as they are read, which the
matcher's mid-pass placement relies on, and `buffered` keeps its
concurrency for the Redis store, whose reads are round trips.

The test pins what the fix relies on: a row added after the listing was
requested is not in it. The lazy listing read it at the first poll.

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
b7r6 added a commit to b7r6/nativelink that referenced this pull request Oct 22, 2026
The concurrency chapter had six failure families, all of them about
state you can see: a counter, a record, a timestamp, a map entry. It
had nothing about the liveness contract of a future, which is the
category TraceMachina#3001 fell into — a lock acquisition that starts and then
stops being polled. No cycle, no guard held across an await, no lint
that fires; the lock ends up free with every task waiting on it.

Adds:

- Family 7 in the concurrency chapter: the four-layer anatomy of
  TraceMachina#3001/TraceMachina#3013, the ways a started acquisition gets parked, how to
  recognise the wedge in the field, and why no unit test reaches it.
  Notes that this tree has layer 1 already — the lock lives inside the
  listing stream at memory_awaited_action_db.rs:990 and :1025 — and is
  held safe only by the current consumer's laziness. One `buffered`
  re-creates the bug.
- Chapter 8, "Locks, and Who Owns the State": the census (69 async
  acquisitions across nine production files, seven lock flavours from
  four crates), what each flavour's fairness costs, the undocumented
  three-lock order the matcher crosses on every action, the absent
  `await_holding_lock` lint, the Redis listing as the next instance,
  and the level-1 discipline that removes the class.
- Appendix H, "Scheduler State Ownership": the single-owner refactor
  and the openraft mapping, with the HA question that has to be
  decided before level 3 means anything.

Also rewrites the concurrency chapter's tooling section, which claimed
a scheduler DST fuzzer that is not in this tree, to describe the
harness that is (current-thread `nativelink_test` plus `MockClock`)
and to state the two limits that matter: hand-arranged interleavings
only, and a current-thread runtime cannot reach family 7 at all.

Every count and line citation is reproducible against this tree; the
commands are in the chapter. Where the source gist described upstream
main at v1.8.0+7, the numbers here were re-measured here, and two
diverge: both central scheduler locks are `async_lock::Mutex` (not one
tokio and one async-lock), and nativelink-worker has 9 acquisitions,
not 11.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

This branch was successfully deployed

2 active deployments
Preview – nativelink — 5372daef Deployed Oct 22, 2026 by vercel[bot]
Preview – nativelink-aidm — 5372daef Deployed Oct 22, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants