Skip to content

fuzz: taplo format fuzz/Cargo.toml (2-space, sorted arrays) - #64

Open
b7r6 wants to merge 13 commits into
mainfrom
pr/scheduler-race-fuzzer
Open

b7r6 wants to merge 13 commits into
mainfrom
pr/scheduler-race-fuzzer

Conversation

@b7r6

@b7r6 b7r6 commented Oct 7, 2026 •

Copy link
Copy Markdown
Owner

What and why

Lane A stability fix from the findings catalog (see nativelink-findings-2026-10-05.csv
on the meta/findings-catalog-2026-10-05 branch), rebased onto OSS main
@bf4b3aa855. This fork PR exists to validate the change on CI that is
representative of upstream PR lanes (public read-only cache, same endpoints)
before any upstream submission.

How was this verified?

The branch carries its own regression tests (fails-without where the fix
class allows it — see the branch's test commits). This PR's purpose is the
full representative CI run; green here rehearses the exact checks an
upstream PR would face.

Risk

Fork-internal validation PR; nothing lands anywhere by merging this. The
underlying fix's risk assessment happens at upstream-submission time.

AI assistance

Agent-assisted pipeline (Claude) for rebase and PR mechanics under my
direction; the fix itself carries verification in its tests.

@b7r6
b7r6 force-pushed the pr/scheduler-race-fuzzer branch 3 times, most recently from c12ad01 to 04daec0 Compare October 8, 2026 03:07
@b7r6
b7r6 force-pushed the pr/scheduler-race-fuzzer branch from 04daec0 to 663f756 Compare October 14, 2026 17:40
@b7r6
b7r6 force-pushed the pr/scheduler-race-fuzzer branch from 663f756 to 4f51802 Compare October 15, 2026 11:19
Completes the no-per-PR-OCI-builds intent: image-off-pr moved image.yaml off
PRs and TraceMachina#2877 moved the nix integration job off PRs, but main.yaml's
integration-tests (docker-compose image build + run_integration_tests.sh)
fell between them and still ran on every PR. Defer to nightly + merge-to-main,
matching TraceMachina#2880's pattern. Fork-local until folded into the upstream OCI/nightly
set.
b7r6 and others added 12 commits October 21, 2026 17:19
The fuzz input decodes into a schedule of events — client submissions,
worker connect/disconnect, keepalives, dispatch acks, completions,
drains, evictions, and virtual-clock advances — replayed against a real
SimpleScheduler on a mock clock inside a single-threaded runtime.
Coverage guidance then searches the interleaving space that
hand-written tests cannot: the scheduler's timeout and requeue logic
races wall-clock heuristics against worker liveness, and that space is
where its production incidents live.

Findings surface two ways: any scheduler panic, and two explicit
invariants — an operation StartAction'd on a second worker while the
first assignment is live (double execution), and a drain epilogue
proving that with one healthy worker attached and every configured
ceiling elapsed, every operation reaches a terminal stage. Anything
still Queued or Executing after that is a wedge, which is exactly the
bug class of the dead-peer hangs and the Action.timeout pre-emption
cascade (TraceMachina#2826).

Deterministic by construction — mock clock, current-thread runtime, no
I/O — so every crashing input replays exactly. A stable-toolchain smoke
test (FUZZ_SMOKE_CASES, default 200) runs the same simulator without
nightly; the libfuzzer target registers with the existing
ClusterFuzzLite setup for coverage-guided runs in CI. 5000 random
schedules pass at the time of writing; during bring-up the wedge
detector caught and shrank three deliberate protocol violations by the
harness itself to minimal repros, which is the evidence it detects what
it claims to.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Three harness corrections from the first live campaign, each found by
chasing a wedge to root cause:

- Pin mock_instant to the workspace's 0.5.3: the 0.6 declaration gave
  the harness and the scheduler two independent thread-local clocks, so
  scheduler-side time never advanced and every time-based path was
  untested dead weight.
- Spawn a polling task per client listener: operation-timeout recovery
  is client-driven by design (ClientActionStateResult wraps the
  matching-engine subscriber whose changed() runs the timeout check),
  so an undriven listener models a client that never polls and masks
  every timeout path.
- Drain rewrite: single loop, reserved drain-worker id namespace,
  yield bursts so mock-clock spin-sleeps observe advances, and a
  zombie-adoption phase that re-attaches a fresh polling client to
  every operation via cache-key merge before asserting terminality —
  the liveness contract is "heals once any client polls".

With the harness honest, schedule case 117183 (seed 5EED_CAFE) now
deterministically reproduces a real scheduler bug: same-worker-id
re-registration silently replaces the Worker object (workers.put),
orphaning its running_action_infos; eviction then requeues nothing and
the operation is re-dispatched while the original worker still runs it
— the harness's DOUBLE EXECUTION invariant fires.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit 589fe50)
cas_config must always build; scheduler_race pulls workspace crates
whose rust-version pin can outrun the oss-fuzz image's pinned nightly,
so it degrades gracefully (with a loud log line) until the image
catches up rather than failing the whole fuzz build.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The scheduler simulation target pulls in nativelink-scheduler/util/proto,
which pin a newer rustc than the oss-fuzz base image's nightly. Adding
them to the shared fuzz manifest broke the pre-existing cas_config
target too (whole-manifest MSRV resolution). Make those deps optional
behind a scheduler_race feature so cas_config builds on the old image
exactly as before; scheduler_race builds with --features locally and in
the tolerant ClusterFuzzLite branch.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…e the generator

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ing — Astra review H1/H2/H3

H3 (input decoding): Vec::<Op>::arbitrary_take_rest reads a continuation
byte per element, so a random even byte ended the schedule — expected
event count was ~1 and the campaign tested almost-empty schedules. Decode
now reads a target length from the first byte and pulls Ops in a loop
(cap 256) until the buffer is exhausted; typical inputs yield tens of
events. Guarded by decoded_schedules_are_rich.

H1 (clocks): the harness added a 100k-second offset to worker timestamps
while MockClock started at zero, so the worker registry and the
operation-timeout checker disagreed on liveness at the same instant.
MockClock is now seeded with the base epoch and every reader shares it;
advance_clocks() moves the mock clock and tokio's paused clock in
lockstep (the abandoned-action sweep sleeps on tokio time) and
self-checks reader agreement on every advance. Guarded by
clock_readers_agree_across_deadline.

H2 (replay determinism): OperationId::default() minted random UUIDs and
SortedAwaitedAction breaks ties by operation id, so identical inputs
dispatched in different orders; the drain also iterated a
randomly-seeded HashMap. New nativelink-util 'mock_operation_id' feature
provides a thread-local counter override for OperationId::default();
infos is now a BTreeMap; run_trace() returns a normalized trace. Guarded
by replay_is_deterministic.

Invariants recalibrated for the now-rich schedules (both prior
formulations only ever saw ~1-event inputs): double-execution is checked
as eventual REVOKE-KILL, since timeout requeue revokes asynchronously
and the kill lands on a later pass; NO-GHOST gets a grace window of the
healing ceilings, and superseded same-id connections (workers.put
replacement closes the old channel) are marked dead instead of
fabricating conflicts. CLIENT_ACTION_TIMEOUT_S drops to 120s so the
drain's 768 virtual seconds actually cross every configured ceiling —
the abandoned-executing backstop only fires at ~2x that ceiling.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
cargo-machete (the repo-wide pre-commit hook) flags it: the harness only
uses tracing-subscriber.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant