Repository navigation
Conversation
The fuzz input decodes into a schedule of events — client submissions, worker connect/disconnect, keepalives, dispatch acks, completions, drains, evictions, and virtual-clock advances — replayed against a real SimpleScheduler on a mock clock inside a single-threaded runtime. Coverage guidance then searches the interleaving space that hand-written tests cannot: the scheduler's timeout and requeue logic races wall-clock heuristics against worker liveness, and that space is where its production incidents live. Findings surface two ways: any scheduler panic, and two explicit invariants — an operation StartAction'd on a second worker while the first assignment is live (double execution), and a drain epilogue proving that with one healthy worker attached and every configured ceiling elapsed, every operation reaches a terminal stage. Anything still Queued or Executing after that is a wedge, which is exactly the bug class of the dead-peer hangs and the Action.timeout pre-emption cascade (TraceMachina#2826). Deterministic by construction — mock clock, current-thread runtime, no I/O — so every crashing input replays exactly. A stable-toolchain smoke test (FUZZ_SMOKE_CASES, default 200) runs the same simulator without nightly; the libfuzzer target registers with the existing ClusterFuzzLite setup for coverage-guided runs in CI. 5000 random schedules pass at the time of writing; during bring-up the wedge detector caught and shrank three deliberate protocol violations by the harness itself to minimal repros, which is the evidence it detects what it claims to. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Three harness corrections from the first live campaign, each found by chasing a wedge to root cause: - Pin mock_instant to the workspace's 0.5.3: the 0.6 declaration gave the harness and the scheduler two independent thread-local clocks, so scheduler-side time never advanced and every time-based path was untested dead weight. - Spawn a polling task per client listener: operation-timeout recovery is client-driven by design (ClientActionStateResult wraps the matching-engine subscriber whose changed() runs the timeout check), so an undriven listener models a client that never polls and masks every timeout path. - Drain rewrite: single loop, reserved drain-worker id namespace, yield bursts so mock-clock spin-sleeps observe advances, and a zombie-adoption phase that re-attaches a fresh polling client to every operation via cache-key merge before asserting terminality — the liveness contract is "heals once any client polls". With the harness honest, schedule case 117183 (seed 5EED_CAFE) now deterministically reproduces a real scheduler bug: same-worker-id re-registration silently replaces the Worker object (workers.put), orphaning its running_action_infos; eviction then requeues nothing and the operation is re-dispatched while the original worker still runs it — the harness's DOUBLE EXECUTION invariant fires. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> (cherry picked from commit 589fe50)
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
cas_config must always build; scheduler_race pulls workspace crates whose rust-version pin can outrun the oss-fuzz image's pinned nightly, so it degrades gracefully (with a loud log line) until the image catches up rather than failing the whole fuzz build. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This was referenced Sep 30, 2026
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The scheduler simulation target pulls in nativelink-scheduler/util/proto, which pin a newer rustc than the oss-fuzz base image's nightly. Adding them to the shared fuzz manifest broke the pre-existing cas_config target too (whole-manifest MSRV resolution). Make those deps optional behind a scheduler_race feature so cas_config builds on the old image exactly as before; scheduler_race builds with --features locally and in the tolerant ClusterFuzzLite branch. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
b7r6
marked this pull request as draft
September 30, 2026 03:57
…e the generator Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ing — Astra review H1/H2/H3 H3 (input decoding): Vec::<Op>::arbitrary_take_rest reads a continuation byte per element, so a random even byte ended the schedule — expected event count was ~1 and the campaign tested almost-empty schedules. Decode now reads a target length from the first byte and pulls Ops in a loop (cap 256) until the buffer is exhausted; typical inputs yield tens of events. Guarded by decoded_schedules_are_rich. H1 (clocks): the harness added a 100k-second offset to worker timestamps while MockClock started at zero, so the worker registry and the operation-timeout checker disagreed on liveness at the same instant. MockClock is now seeded with the base epoch and every reader shares it; advance_clocks() moves the mock clock and tokio's paused clock in lockstep (the abandoned-action sweep sleeps on tokio time) and self-checks reader agreement on every advance. Guarded by clock_readers_agree_across_deadline. H2 (replay determinism): OperationId::default() minted random UUIDs and SortedAwaitedAction breaks ties by operation id, so identical inputs dispatched in different orders; the drain also iterated a randomly-seeded HashMap. New nativelink-util 'mock_operation_id' feature provides a thread-local counter override for OperationId::default(); infos is now a BTreeMap; run_trace() returns a normalized trace. Guarded by replay_is_deterministic. Invariants recalibrated for the now-rich schedules (both prior formulations only ever saw ~1-event inputs): double-execution is checked as eventual REVOKE-KILL, since timeout requeue revokes asynchronously and the kill lands on a later pass; NO-GHOST gets a grace window of the healing ceilings, and superseded same-id connections (workers.put replacement closes the old channel) are marked dead instead of fabricating conflicts. CLIENT_ACTION_TIMEOUT_S drops to 120s so the drain's 768 virtual seconds actually cross every configured ceiling — the abandoned-executing backstop only fires at ~2x that ceiling. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
b7r6
force-pushed
the
pr/scheduler-race-fuzzer
branch
from
September 30, 2026 05:04
5e716de to
d37ec26
Compare
|
Deployment failed for project nativelink-aidm with the following error: |
|
Deployment failed for project nativelink with the following error: |
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
b7r6
force-pushed
the
pr/scheduler-race-fuzzer
branch
from
September 30, 2026 05:05
d37ec26 to
f1dabfa
Compare
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Contributor
Author
|
Withdrawing this from the queue to respect the team's review bandwidth. It's a verified fix (fails-without regression, retained on our fork) — rather than keep a flood of drafts open, we'll re-propose one clean, reviewed PR at a time. Happy to prioritize any of these on request. |
b7r6
added a commit
to b7r6/nativelink
that referenced
this pull request
Oct 7, 2026
…under a live id ApiWorkerSchedulerImpl::add_worker used LruCache::put, which silently replaced a live entry when a worker re-registered under an existing id. The replaced Worker was dropped together with its running_action_infos: nothing was requeued, the old channel never heard Disconnect, and the gauges were only partially adjusted. Downstream, the replaced worker's operations stayed Executing assigned to the id; when the new incarnation was later evicted its empty list requeued nothing, and the eventual timeout requeued the operation to another worker while the original process might still be running it -- double execution. Now ApiWorkerScheduler::add_worker treats a live entry under the same id exactly like a disconnect before admitting the new incarnation: it goes through immediate_evict_worker, which drains running_action_infos via UpdateOperationType::UpdateWithDisconnect, sends Disconnect on the old channel, and balances the connect/disconnect gauges. A requeue failure is logged and does not block the reconnecting worker. Deterministically reproduced by the DST fuzzer schedule in PR TraceMachina#2831; the new regression test (worker re-registers mid-execution, asserts Disconnect on the old channel, re-dispatch of the same operation to the new channel, and completion) fails on unfixed main. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
b7r6
added a commit
to b7r6/nativelink
that referenced
this pull request
Oct 7, 2026
…under a live id ApiWorkerSchedulerImpl::add_worker used LruCache::put, which silently replaced a live entry when a worker re-registered under an existing id. The replaced Worker was dropped together with its running_action_infos: nothing was requeued, the old channel never heard Disconnect, and the gauges were only partially adjusted. Downstream, the replaced worker's operations stayed Executing assigned to the id; when the new incarnation was later evicted its empty list requeued nothing, and the eventual timeout requeued the operation to another worker while the original process might still be running it -- double execution. Now ApiWorkerScheduler::add_worker treats a live entry under the same id exactly like a disconnect before admitting the new incarnation: it goes through immediate_evict_worker, which drains running_action_infos via UpdateOperationType::UpdateWithDisconnect, sends Disconnect on the old channel, and balances the connect/disconnect gauges. A requeue failure is logged and does not block the reconnecting worker. Deterministically reproduced by the DST fuzzer schedule in PR TraceMachina#2831; the new regression test (worker re-registers mid-execution, asserts Disconnect on the old channel, re-dispatch of the same operation to the new channel, and completion) fails on unfixed main. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
b7r6
added a commit
to b7r6/nativelink
that referenced
this pull request
Oct 7, 2026
…under a live id ApiWorkerSchedulerImpl::add_worker used LruCache::put, which silently replaced a live entry when a worker re-registered under an existing id. The replaced Worker was dropped together with its running_action_infos: nothing was requeued, the old channel never heard Disconnect, and the gauges were only partially adjusted. Downstream, the replaced worker's operations stayed Executing assigned to the id; when the new incarnation was later evicted its empty list requeued nothing, and the eventual timeout requeued the operation to another worker while the original process might still be running it -- double execution. Now ApiWorkerScheduler::add_worker treats a live entry under the same id exactly like a disconnect before admitting the new incarnation: it goes through immediate_evict_worker, which drains running_action_infos via UpdateOperationType::UpdateWithDisconnect, sends Disconnect on the old channel, and balances the connect/disconnect gauges. A requeue failure is logged and does not block the reconnecting worker. Deterministically reproduced by the DST fuzzer schedule in PR TraceMachina#2831; the new regression test (worker re-registers mid-execution, asserts Disconnect on the old channel, re-dispatch of the same operation to the new channel, and completion) fails on unfixed main. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
b7r6
added a commit
to b7r6/nativelink
that referenced
this pull request
Oct 8, 2026
…under a live id ApiWorkerSchedulerImpl::add_worker used LruCache::put, which silently replaced a live entry when a worker re-registered under an existing id. The replaced Worker was dropped together with its running_action_infos: nothing was requeued, the old channel never heard Disconnect, and the gauges were only partially adjusted. Downstream, the replaced worker's operations stayed Executing assigned to the id; when the new incarnation was later evicted its empty list requeued nothing, and the eventual timeout requeued the operation to another worker while the original process might still be running it -- double execution. Now ApiWorkerScheduler::add_worker treats a live entry under the same id exactly like a disconnect before admitting the new incarnation: it goes through immediate_evict_worker, which drains running_action_infos via UpdateOperationType::UpdateWithDisconnect, sends Disconnect on the old channel, and balances the connect/disconnect gauges. A requeue failure is logged and does not block the reconnecting worker. Deterministically reproduced by the DST fuzzer schedule in PR TraceMachina#2831; the new regression test (worker re-registers mid-execution, asserts Disconnect on the old channel, re-dispatch of the same operation to the new channel, and completion) fails on unfixed main. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
b7r6
added a commit
to b7r6/nativelink
that referenced
this pull request
Oct 14, 2026
…under a live id ApiWorkerSchedulerImpl::add_worker used LruCache::put, which silently replaced a live entry when a worker re-registered under an existing id. The replaced Worker was dropped together with its running_action_infos: nothing was requeued, the old channel never heard Disconnect, and the gauges were only partially adjusted. Downstream, the replaced worker's operations stayed Executing assigned to the id; when the new incarnation was later evicted its empty list requeued nothing, and the eventual timeout requeued the operation to another worker while the original process might still be running it -- double execution. Now ApiWorkerScheduler::add_worker treats a live entry under the same id exactly like a disconnect before admitting the new incarnation: it goes through immediate_evict_worker, which drains running_action_infos via UpdateOperationType::UpdateWithDisconnect, sends Disconnect on the old channel, and balances the connect/disconnect gauges. A requeue failure is logged and does not block the reconnecting worker. Deterministically reproduced by the DST fuzzer schedule in PR TraceMachina#2831; the new regression test (worker re-registers mid-execution, asserts Disconnect on the old channel, re-dispatch of the same operation to the new channel, and completion) fails on unfixed main. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
b7r6
added a commit
to b7r6/nativelink
that referenced
this pull request
Oct 15, 2026
…under a live id ApiWorkerSchedulerImpl::add_worker used LruCache::put, which silently replaced a live entry when a worker re-registered under an existing id. The replaced Worker was dropped together with its running_action_infos: nothing was requeued, the old channel never heard Disconnect, and the gauges were only partially adjusted. Downstream, the replaced worker's operations stayed Executing assigned to the id; when the new incarnation was later evicted its empty list requeued nothing, and the eventual timeout requeued the operation to another worker while the original process might still be running it -- double execution. Now ApiWorkerScheduler::add_worker treats a live entry under the same id exactly like a disconnect before admitting the new incarnation: it goes through immediate_evict_worker, which drains running_action_infos via UpdateOperationType::UpdateWithDisconnect, sends Disconnect on the old channel, and balances the connect/disconnect gauges. A requeue failure is logged and does not block the reconnecting worker. Deterministically reproduced by the DST fuzzer schedule in PR TraceMachina#2831; the new regression test (worker re-registers mid-execution, asserts Disconnect on the old channel, re-dispatch of the same operation to the new channel, and completion) fails on unfixed main. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
b7r6
added a commit
to b7r6/nativelink
that referenced
this pull request
Oct 21, 2026
…under a live id ApiWorkerSchedulerImpl::add_worker used LruCache::put, which silently replaced a live entry when a worker re-registered under an existing id. The replaced Worker was dropped together with its running_action_infos: nothing was requeued, the old channel never heard Disconnect, and the gauges were only partially adjusted. Downstream, the replaced worker's operations stayed Executing assigned to the id; when the new incarnation was later evicted its empty list requeued nothing, and the eventual timeout requeued the operation to another worker while the original process might still be running it -- double execution. Now ApiWorkerScheduler::add_worker treats a live entry under the same id exactly like a disconnect before admitting the new incarnation: it goes through immediate_evict_worker, which drains running_action_infos via UpdateOperationType::UpdateWithDisconnect, sends Disconnect on the old channel, and balances the connect/disconnect gauges. A requeue failure is logged and does not block the reconnecting worker. Deterministically reproduced by the DST fuzzer schedule in PR TraceMachina#2831; the new regression test (worker re-registers mid-execution, asserts Disconnect on the old channel, re-dispatch of the same operation to the new channel, and completion) fails on unfixed main. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
b7r6
added a commit
to b7r6/nativelink
that referenced
this pull request
Oct 22, 2026
Five places told contributors to run a `scheduler_race` deterministic- simulation fuzzer before touching the scheduler. It does not exist here. `nativelink-test/fuzz` declares exactly one `[[bin]]`, `cas_config`, and depends on `nativelink-config` alone — so it could not host a scheduler target as it stands. The book was written against a tree carrying PR TraceMachina#2831; this branch is based on fork main, which is not. Corrects the fuzzing section to describe the one target that exists, says plainly that no scheduler fuzzer is in the tree and that the prototype lives on the fork, and replaces "run the fuzzer" in the contribution checklist with the check that actually applies: the interleaving tests under nativelink-scheduler/tests/ and a regression that fails without the fix. A checkable instruction that cannot be carried out is worse than no instruction, because the reader spends their afternoon before concluding the book is wrong. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
A deterministic-simulation fuzz target for
SimpleScheduler. The fuzz input decodes into a schedule of events — client submissions, worker connect/disconnect, keepalives, dispatch acks/declines, completions, retryable failures, drains, evictions, virtual-clock advances — replayed against a real scheduler (memory backend) on a mock clock in a single-threaded runtime. Coverage guidance then searches the interleaving space that hand-written tests cannot reach; the scheduler's timeout and requeue logic races wall-clock heuristics against worker liveness, and that space is where its production incidents have lived (#2826 being the recent example).Findings surface two ways: any scheduler panic, and two explicit invariants — an operation
StartActioned on a second worker while a live worker still runs it (double execution), and a drain epilogue asserting that with a healthy worker attached, actively-polled clients, and every configured ceiling elapsed, every operation reaches a terminal stage. Anything still Queued or Executing after that is a wedge.Deterministic by construction — mock clock, current-thread runtime, no I/O — so every crashing input replays exactly. A stable-toolchain smoke mode (
FUZZ_SMOKE_CASES, default 200) runs the same simulator under plaincargo test; the libfuzzer target registers with the existing ClusterFuzzLite setup for coverage-guided CI.How was this verified?
Harness honesty was earned the hard way, and the second commit documents it: the first campaign's wedges were traced to three harness modeling errors (a duplicate
mock_instantversion giving the harness and scheduler two independent thread-local clocks; undriven client listeners, when operation-timeout recovery is client-driven by design via the subscriber thatClientActionStateResultwraps; and drain bookkeeping), each shrunk to a minimal schedule and fixed. With the harness honest, 5000+ random schedules pass on a correct scheduler — and schedule case 117183 (seed0x5EED_CAFE_F00D_D00D) deterministically reproduces a real bug: same-worker-id re-registration silently replaces theWorkerobject (workers.putinapi_worker_scheduler), orphaning itsrunning_action_infos; eviction then requeues nothing and the operation is re-dispatched while the original worker still runs it — the double-execution invariant fires every run. A fix PR with a minimized regression test follows separately; this PR is the instrument.Risk
None to runtime — test/fuzz infrastructure only. The fuzz crate gains dev-only dependencies (
arbitrary,tokio,mock_instantpinned to the workspace's 0.5.3) and one new ClusterFuzzLite target.AI assistance
Claude (Fable) designed and wrote the harness, ran the campaigns, and root-caused both the harness's own modeling errors and the scheduler bug it found. Human-directed throughout.
This change is