Skip to content

Reduce scheduler contention after lost assignments - #2954

Merged
MarcusSorealheis merged 3 commits into
TraceMachina:mainfrom
MarcusSorealheis:codex/tra-1475-matcher-contention
Oct 18, 2026
Merged

MarcusSorealheis merged 3 commits into
TraceMachina:mainfrom
MarcusSorealheis:codex/tra-1475-matcher-contention

Conversation

@MarcusSorealheis

@MarcusSorealheis MarcusSorealheis commented Oct 16, 2026 •

Copy link
Copy Markdown
Member

What and why

When schedulers race to assign the same queued action, the loser currently keeps competing for every following action with the same platform properties. After a lost assignment, this change skips that property shape for the remainder of the pass, including parked actions revisited when capacity opens. Other shapes remain eligible, and the next pass considers all shapes again.

Diagnostic logging now samples up to five oldest queued actions from the matcher's existing listing, removing the additional queued and executing database listings. Matching diagnostics remain disabled by default. The platform-properties documentation describes the new behavior.

The executing sweep, running-operation reports, and default logging change already landed in #2923, #2925, and #2924. This preserves their reviewed decisions: lost results consume a retry, and worker keepalives do not extend the execution ceiling. Reliable result resends require an acknowledgment protocol and are outside this change.

How was this verified?

  • cargo test -p nativelink-scheduler --locked: 246 passed, 1 ignored on macOS.
  • Both new tests fail against the original matcher and pass with this change. They force a peer to win an assignment, check that only the matching property shape waits, and confirm it runs on the next pass. The second case loses the assignment while replaying parked actions after capacity opens.
  • The logging-enabled test gates database listings so an extra diagnostic queue query cannot complete the matching pass.
  • cargo clippy -p nativelink-scheduler --test simple_scheduler_test --locked -- -D warnings passed.
  • Repository pre-commit hooks passed for the changed files. Docs anchor, snippet, navigation, and corpus generation checks passed after generating the changelog prerequisite.

Bazel and live-cluster validation were not run.

Risk

A contested property shape may wait for the next matching pass while local capacity is available. If fallback matching is disabled, that pass needs a task or worker notification. No retry budget, worker reservation size, or persistent state changes. Diagnostic consumers should account for the new Oldest actions in the matching pass message, which samples queued actions rather than logging a separate executing list.

AI assistance

Automated assistance was used to draft the changes and run the listed checks.


This change is Reviewable

@vercel

vercel Bot commented Oct 16, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
nativelink Ready Ready Preview Oct 17, 2026 11:11pm UTC
nativelink-aidm Ready Ready Preview Oct 17, 2026 11:11pm UTC

Request Review

@MarcusSorealheis
MarcusSorealheis merged commit 8431150 into TraceMachina:main Oct 18, 2026
41 checks passed
MarcusSorealheis pushed a commit that referenced this pull request Oct 22, 2026
…e lock inside the stream (#3013)

Since 4edf822 (#3001) the filter pass reads listed rows 64 at a time
with `buffered`, which keeps asking the listing for the next row while it
hands rows downstream. The memory store's listing took the store's lock
inside the stream (a `ChunkedStream` whose chunk fetch awaits
`inner.lock()`), so under load the next chunk fetch was left parked inside
the stream, waiting for the lock, while the matcher wrote for a row it had
been handed, on the same task, and waited for the same lock. Once the
parked wait had starved past async-lock's fairness threshold the lock was
reserved for it, nothing polled it, and the scheduler stopped for good:
every state update blocked, actions the workers had finished and uploaded
"timed out with no update from their worker", and nothing dispatched
again. On railgun a `bazel build //...` of this repository through a
three-worker fleet wedged within 15 s on #3001 and on main, every time;
v1.7.7, #2954 and v1.8.0 run it clean.

The listing is now read whole under one hold of the lock, taken in the
`async fn` before the stream exists, and the stream only hands out what
was read. The chunk fetch already read the whole range in one go, so the
snapshot is what the stream yielded before; the lock just moves out of it.
Rows are still handed on one at a time as they are read, which the
matcher's mid-pass placement relies on, and `buffered` keeps its
concurrency for the Redis store, whose reads are round trips.

The test pins what the fix relies on: a row added after the listing was
requested is not in it. The lazy listing read it at the first poll.

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>

This branch was successfully deployed

2 active deployments
Preview – nativelink — 23d87ef9 Deployed Oct 17, 2026 by vercel[bot]
Preview – nativelink-aidm — 23d87ef9 Deployed Oct 17, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants