Repository navigation
Conversation
b7r6
force-pushed
the
pr/scheduler-race-fuzzer
branch
3 times, most recently
from
October 8, 2026 03:07
c12ad01 to
04daec0
Compare
b7r6
force-pushed
the
pr/scheduler-race-fuzzer
branch
from
October 14, 2026 17:40
04daec0 to
663f756
Compare
b7r6
force-pushed
the
pr/scheduler-race-fuzzer
branch
from
October 15, 2026 11:19
663f756 to
4f51802
Compare
Completes the no-per-PR-OCI-builds intent: image-off-pr moved image.yaml off PRs and TraceMachina#2877 moved the nix integration job off PRs, but main.yaml's integration-tests (docker-compose image build + run_integration_tests.sh) fell between them and still ran on every PR. Defer to nightly + merge-to-main, matching TraceMachina#2880's pattern. Fork-local until folded into the upstream OCI/nightly set.
The fuzz input decodes into a schedule of events — client submissions, worker connect/disconnect, keepalives, dispatch acks, completions, drains, evictions, and virtual-clock advances — replayed against a real SimpleScheduler on a mock clock inside a single-threaded runtime. Coverage guidance then searches the interleaving space that hand-written tests cannot: the scheduler's timeout and requeue logic races wall-clock heuristics against worker liveness, and that space is where its production incidents live. Findings surface two ways: any scheduler panic, and two explicit invariants — an operation StartAction'd on a second worker while the first assignment is live (double execution), and a drain epilogue proving that with one healthy worker attached and every configured ceiling elapsed, every operation reaches a terminal stage. Anything still Queued or Executing after that is a wedge, which is exactly the bug class of the dead-peer hangs and the Action.timeout pre-emption cascade (TraceMachina#2826). Deterministic by construction — mock clock, current-thread runtime, no I/O — so every crashing input replays exactly. A stable-toolchain smoke test (FUZZ_SMOKE_CASES, default 200) runs the same simulator without nightly; the libfuzzer target registers with the existing ClusterFuzzLite setup for coverage-guided runs in CI. 5000 random schedules pass at the time of writing; during bring-up the wedge detector caught and shrank three deliberate protocol violations by the harness itself to minimal repros, which is the evidence it detects what it claims to. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Three harness corrections from the first live campaign, each found by chasing a wedge to root cause: - Pin mock_instant to the workspace's 0.5.3: the 0.6 declaration gave the harness and the scheduler two independent thread-local clocks, so scheduler-side time never advanced and every time-based path was untested dead weight. - Spawn a polling task per client listener: operation-timeout recovery is client-driven by design (ClientActionStateResult wraps the matching-engine subscriber whose changed() runs the timeout check), so an undriven listener models a client that never polls and masks every timeout path. - Drain rewrite: single loop, reserved drain-worker id namespace, yield bursts so mock-clock spin-sleeps observe advances, and a zombie-adoption phase that re-attaches a fresh polling client to every operation via cache-key merge before asserting terminality — the liveness contract is "heals once any client polls". With the harness honest, schedule case 117183 (seed 5EED_CAFE) now deterministically reproduces a real scheduler bug: same-worker-id re-registration silently replaces the Worker object (workers.put), orphaning its running_action_infos; eviction then requeues nothing and the operation is re-dispatched while the original worker still runs it — the harness's DOUBLE EXECUTION invariant fires. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> (cherry picked from commit 589fe50)
cas_config must always build; scheduler_race pulls workspace crates whose rust-version pin can outrun the oss-fuzz image's pinned nightly, so it degrades gracefully (with a loud log line) until the image catches up rather than failing the whole fuzz build. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The scheduler simulation target pulls in nativelink-scheduler/util/proto, which pin a newer rustc than the oss-fuzz base image's nightly. Adding them to the shared fuzz manifest broke the pre-existing cas_config target too (whole-manifest MSRV resolution). Make those deps optional behind a scheduler_race feature so cas_config builds on the old image exactly as before; scheduler_race builds with --features locally and in the tolerant ClusterFuzzLite branch. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…e the generator Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ing — Astra review H1/H2/H3 H3 (input decoding): Vec::<Op>::arbitrary_take_rest reads a continuation byte per element, so a random even byte ended the schedule — expected event count was ~1 and the campaign tested almost-empty schedules. Decode now reads a target length from the first byte and pulls Ops in a loop (cap 256) until the buffer is exhausted; typical inputs yield tens of events. Guarded by decoded_schedules_are_rich. H1 (clocks): the harness added a 100k-second offset to worker timestamps while MockClock started at zero, so the worker registry and the operation-timeout checker disagreed on liveness at the same instant. MockClock is now seeded with the base epoch and every reader shares it; advance_clocks() moves the mock clock and tokio's paused clock in lockstep (the abandoned-action sweep sleeps on tokio time) and self-checks reader agreement on every advance. Guarded by clock_readers_agree_across_deadline. H2 (replay determinism): OperationId::default() minted random UUIDs and SortedAwaitedAction breaks ties by operation id, so identical inputs dispatched in different orders; the drain also iterated a randomly-seeded HashMap. New nativelink-util 'mock_operation_id' feature provides a thread-local counter override for OperationId::default(); infos is now a BTreeMap; run_trace() returns a normalized trace. Guarded by replay_is_deterministic. Invariants recalibrated for the now-rich schedules (both prior formulations only ever saw ~1-event inputs): double-execution is checked as eventual REVOKE-KILL, since timeout requeue revokes asynchronously and the kill lands on a later pass; NO-GHOST gets a grace window of the healing ceilings, and superseded same-id connections (workers.put replacement closes the old channel) are marked dead instead of fabricating conflicts. CLIENT_ACTION_TIMEOUT_S drops to 120s so the drain's 768 virtual seconds actually cross every configured ceiling — the abandoned-executing backstop only fires at ~2x that ceiling. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
cargo-machete (the repo-wide pre-commit hook) flags it: the harness only uses tracing-subscriber. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
b7r6
force-pushed
the
pr/scheduler-race-fuzzer
branch
2 times, most recently
from
October 21, 2026 17:37
ade934d to
b2b7072
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
Lane A stability fix from the findings catalog (see
nativelink-findings-2026-10-05.csvon the
meta/findings-catalog-2026-10-05branch), rebased onto OSS main@bf4b3aa855. This fork PR exists to validate the change on CI that is
representative of upstream PR lanes (public read-only cache, same endpoints)
before any upstream submission.
How was this verified?
The branch carries its own regression tests (fails-without where the fix
class allows it — see the branch's test commits). This PR's purpose is the
full representative CI run; green here rehearses the exact checks an
upstream PR would face.
Risk
Fork-internal validation PR; nothing lands anywhere by merging this. The
underlying fix's risk assessment happens at upstream-submission time.
AI assistance
Agent-assisted pipeline (Claude) for rebase and PR mechanics under my
direction; the fix itself carries verification in its tests.