Repository navigation
Read a filter pass's listed records concurrently so a deep queue on Redis lists inside the provisioner's timeout - #3001
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
49e3270 to
9f856fc
Compare
MarcusSorealheis
left a comment
There was a problem hiding this comment.
Thanks for digging into the slow demand listing. I don't think this change makes it faster on Redis, though, and the new test passes without the fix. Details are inline. The main point is here because it's on lines outside this diff.
The per-record round trip is in the shared listing, so the fix belongs there (code observation). filter_operations runs try_filter_map(borrow_listed) one record at a time (nativelink-scheduler/src/simple_scheduler_state_manager.rs:1629-1631). On Redis, borrow() is where the client keepalive read happens, and its result is cached in the subscriber's listed_with_keepalive OnceCell (store_awaited_action_db.rs:236-249) before the record is yielded. The matcher's get_queued_operations pays the same serial cost. Two options that would fix both callers:
- run
borrow_listedas an order-preserving.map(...).buffered(N)before the predicate, or - fetch the keepalive keys for each FT.AGGREGATE page with one MGET in
get_range_of_actions.
Either would make the admin-side concurrency unnecessary.
What I ran
cargo test -p nativelink-scheduler --test admin_teston0d2db546: 6/6 pass. Also 6/6 with.buffer_unordered(...)swapped in, and 6/6 with the pre-PRadmin.rsrestored.nativelink-schedulerbuilds with the'aremoved fromread_demand_entry.
Not checked: no timing against a real or mock Redis (the latency points come from reading the code), no clippy or Bazel run, and no full nativelink build.
… on Redis lists inside the provisioner's timeout
0d2db54 to
97c78a9
Compare
…e lock inside the stream (#3013) Since 4edf822 (#3001) the filter pass reads listed rows 64 at a time with `buffered`, which keeps asking the listing for the next row while it hands rows downstream. The memory store's listing took the store's lock inside the stream (a `ChunkedStream` whose chunk fetch awaits `inner.lock()`), so under load the next chunk fetch was left parked inside the stream, waiting for the lock, while the matcher wrote for a row it had been handed, on the same task, and waited for the same lock. Once the parked wait had starved past async-lock's fairness threshold the lock was reserved for it, nothing polled it, and the scheduler stopped for good: every state update blocked, actions the workers had finished and uploaded "timed out with no update from their worker", and nothing dispatched again. On railgun a `bazel build //...` of this repository through a three-worker fleet wedged within 15 s on #3001 and on main, every time; v1.7.7, #2954 and v1.8.0 run it clean. The listing is now read whole under one hold of the lock, taken in the `async fn` before the stream exists, and the stream only hands out what was read. The chunk fetch already read the whole range in one go, so the snapshot is what the stream yielded before; the lock just moves out of it. Rows are still handed on one at a time as they are read, which the matcher's mid-pass placement relies on, and `buffered` keeps its concurrency for the Redis store, whose reads are round trips. The test pins what the fix relies on: a row added after the listing was requested is not in it. The lazy listing read it at the first poll. Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
The concurrency chapter had six failure families, all of them about state you can see: a counter, a record, a timestamp, a map entry. It had nothing about the liveness contract of a future, which is the category TraceMachina#3001 fell into — a lock acquisition that starts and then stops being polled. No cycle, no guard held across an await, no lint that fires; the lock ends up free with every task waiting on it. Adds: - Family 7 in the concurrency chapter: the four-layer anatomy of TraceMachina#3001/TraceMachina#3013, the ways a started acquisition gets parked, how to recognise the wedge in the field, and why no unit test reaches it. Notes that this tree has layer 1 already — the lock lives inside the listing stream at memory_awaited_action_db.rs:990 and :1025 — and is held safe only by the current consumer's laziness. One `buffered` re-creates the bug. - Chapter 8, "Locks, and Who Owns the State": the census (69 async acquisitions across nine production files, seven lock flavours from four crates), what each flavour's fairness costs, the undocumented three-lock order the matcher crosses on every action, the absent `await_holding_lock` lint, the Redis listing as the next instance, and the level-1 discipline that removes the class. - Appendix H, "Scheduler State Ownership": the single-owner refactor and the openraft mapping, with the HA question that has to be decided before level 3 means anything. Also rewrites the concurrency chapter's tooling section, which claimed a scheduler DST fuzzer that is not in this tree, to describe the harness that is (current-thread `nativelink_test` plus `MockClock`) and to state the two limits that matter: hand-arranged interleavings only, and a current-thread runtime cannot reach family 7 at all. Every count and line citation is reproducible against this tree; the commands are in the chapter. Where the source gist described upstream main at v1.8.0+7, the numbers here were re-measured here, and two diverge: both central scheduler locks are `async_lock::Mutex` (not one tokio and one async-lock), and nativelink-worker has 9 acquisitions, not 11. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
What and why
A filter pass reads the record behind every listed row one at a time, and on the Redis backend each read is a round trip for the row's client keepalive. A queue of 7,000 took several seconds to list, on every matching pass and on every admin demand listing, which the worker provisioner gives 5 s before it falls back to reading Redis itself. The per-stage listing now reads its rows 64 at a time with an order-preserving
buffered, so both callers see the store's order as before; the predicate behind the reads, which can write, still runs one row at a time. The admin listing only has two comments corrected.How was this verified?
A new state manager test puts a slow store under the listing, longest read first, and counts reads in flight: 100 queued actions list highest priority first with 64 reads in flight at the peak, in 0.14 s. Against the previous serial listing it fails with a peak of 1 after 5.25 s, and against
buffer_unorderedit fails on the order. The scheduler tests, clippy and a check of the binary pass on 1.98.1 and 1.97.1.Risk
A listing that fails part way may have up to 63 further record reads in flight; those are reads only. The all-stages listing, which collects and sorts, keeps its serial read.
AI assistance
An agent drafted the change and I reviewed every line.