Skip to content

Deterministic simulation fuzzing for SimpleScheduler - #2831

Closed
b7r6 wants to merge 11 commits into
TraceMachina:mainfrom
b7r6:pr/scheduler-race-fuzzer
Closed

b7r6 wants to merge 11 commits into
TraceMachina:mainfrom
b7r6:pr/scheduler-race-fuzzer

Conversation

@b7r6

@b7r6 b7r6 commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

What and why

A deterministic-simulation fuzz target for SimpleScheduler. The fuzz input decodes into a schedule of events — client submissions, worker connect/disconnect, keepalives, dispatch acks/declines, completions, retryable failures, drains, evictions, virtual-clock advances — replayed against a real scheduler (memory backend) on a mock clock in a single-threaded runtime. Coverage guidance then searches the interleaving space that hand-written tests cannot reach; the scheduler's timeout and requeue logic races wall-clock heuristics against worker liveness, and that space is where its production incidents have lived (#2826 being the recent example).

Findings surface two ways: any scheduler panic, and two explicit invariants — an operation StartActioned on a second worker while a live worker still runs it (double execution), and a drain epilogue asserting that with a healthy worker attached, actively-polled clients, and every configured ceiling elapsed, every operation reaches a terminal stage. Anything still Queued or Executing after that is a wedge.

Deterministic by construction — mock clock, current-thread runtime, no I/O — so every crashing input replays exactly. A stable-toolchain smoke mode (FUZZ_SMOKE_CASES, default 200) runs the same simulator under plain cargo test; the libfuzzer target registers with the existing ClusterFuzzLite setup for coverage-guided CI.

How was this verified?

Harness honesty was earned the hard way, and the second commit documents it: the first campaign's wedges were traced to three harness modeling errors (a duplicate mock_instant version giving the harness and scheduler two independent thread-local clocks; undriven client listeners, when operation-timeout recovery is client-driven by design via the subscriber that ClientActionStateResult wraps; and drain bookkeeping), each shrunk to a minimal schedule and fixed. With the harness honest, 5000+ random schedules pass on a correct scheduler — and schedule case 117183 (seed 0x5EED_CAFE_F00D_D00D) deterministically reproduces a real bug: same-worker-id re-registration silently replaces the Worker object (workers.put in api_worker_scheduler), orphaning its running_action_infos; eviction then requeues nothing and the operation is re-dispatched while the original worker still runs it — the double-execution invariant fires every run. A fix PR with a minimized regression test follows separately; this PR is the instrument.

Risk

None to runtime — test/fuzz infrastructure only. The fuzz crate gains dev-only dependencies (arbitrary, tokio, mock_instant pinned to the workspace's 0.5.3) and one new ClusterFuzzLite target.

AI assistance

Claude (Fable) designed and wrote the harness, ran the campaigns, and root-caused both the harness's own modeling errors and the scheduler bug it found. Human-directed throughout.


This change is Reviewable

b7r6 and others added 2 commits September 29, 2026 20:09
The fuzz input decodes into a schedule of events — client submissions,
worker connect/disconnect, keepalives, dispatch acks, completions,
drains, evictions, and virtual-clock advances — replayed against a real
SimpleScheduler on a mock clock inside a single-threaded runtime.
Coverage guidance then searches the interleaving space that
hand-written tests cannot: the scheduler's timeout and requeue logic
races wall-clock heuristics against worker liveness, and that space is
where its production incidents live.

Findings surface two ways: any scheduler panic, and two explicit
invariants — an operation StartAction'd on a second worker while the
first assignment is live (double execution), and a drain epilogue
proving that with one healthy worker attached and every configured
ceiling elapsed, every operation reaches a terminal stage. Anything
still Queued or Executing after that is a wedge, which is exactly the
bug class of the dead-peer hangs and the Action.timeout pre-emption
cascade (TraceMachina#2826).

Deterministic by construction — mock clock, current-thread runtime, no
I/O — so every crashing input replays exactly. A stable-toolchain smoke
test (FUZZ_SMOKE_CASES, default 200) runs the same simulator without
nightly; the libfuzzer target registers with the existing
ClusterFuzzLite setup for coverage-guided runs in CI. 5000 random
schedules pass at the time of writing; during bring-up the wedge
detector caught and shrank three deliberate protocol violations by the
harness itself to minimal repros, which is the evidence it detects what
it claims to.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Three harness corrections from the first live campaign, each found by
chasing a wedge to root cause:

- Pin mock_instant to the workspace's 0.5.3: the 0.6 declaration gave
  the harness and the scheduler two independent thread-local clocks, so
  scheduler-side time never advanced and every time-based path was
  untested dead weight.
- Spawn a polling task per client listener: operation-timeout recovery
  is client-driven by design (ClientActionStateResult wraps the
  matching-engine subscriber whose changed() runs the timeout check),
  so an undriven listener models a client that never polls and masks
  every timeout path.
- Drain rewrite: single loop, reserved drain-worker id namespace,
  yield bursts so mock-clock spin-sleeps observe advances, and a
  zombie-adoption phase that re-attaches a fresh polling client to
  every operation via cache-key merge before asserting terminality —
  the liveness contract is "heals once any client polls".

With the harness honest, schedule case 117183 (seed 5EED_CAFE) now
deterministically reproduces a real scheduler bug: same-worker-id
re-registration silently replaces the Worker object (workers.put),
orphaning its running_action_infos; eviction then requeues nothing and
the operation is re-dispatched while the original worker still runs it
— the harness's DOUBLE EXECUTION invariant fires.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit 589fe50)
@vercel

vercel Bot commented Sep 30, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
nativelink Ready Ready Preview Sep 30, 2026 9:18pm UTC
nativelink-aidm Ready Ready Preview Sep 30, 2026 9:18pm UTC

Request Review

cas_config must always build; scheduler_race pulls workspace crates
whose rust-version pin can outrun the oss-fuzz image's pinned nightly,
so it degrades gracefully (with a loud log line) until the image
catches up rather than failing the whole fuzz build.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The scheduler simulation target pulls in nativelink-scheduler/util/proto,
which pin a newer rustc than the oss-fuzz base image's nightly. Adding
them to the shared fuzz manifest broke the pre-existing cas_config
target too (whole-manifest MSRV resolution). Make those deps optional
behind a scheduler_race feature so cas_config builds on the old image
exactly as before; scheduler_race builds with --features locally and in
the tolerant ClusterFuzzLite branch.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
b7r6 and others added 2 commits September 30, 2026 00:23
…e the generator

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ing — Astra review H1/H2/H3

H3 (input decoding): Vec::<Op>::arbitrary_take_rest reads a continuation
byte per element, so a random even byte ended the schedule — expected
event count was ~1 and the campaign tested almost-empty schedules. Decode
now reads a target length from the first byte and pulls Ops in a loop
(cap 256) until the buffer is exhausted; typical inputs yield tens of
events. Guarded by decoded_schedules_are_rich.

H1 (clocks): the harness added a 100k-second offset to worker timestamps
while MockClock started at zero, so the worker registry and the
operation-timeout checker disagreed on liveness at the same instant.
MockClock is now seeded with the base epoch and every reader shares it;
advance_clocks() moves the mock clock and tokio's paused clock in
lockstep (the abandoned-action sweep sleeps on tokio time) and
self-checks reader agreement on every advance. Guarded by
clock_readers_agree_across_deadline.

H2 (replay determinism): OperationId::default() minted random UUIDs and
SortedAwaitedAction breaks ties by operation id, so identical inputs
dispatched in different orders; the drain also iterated a
randomly-seeded HashMap. New nativelink-util 'mock_operation_id' feature
provides a thread-local counter override for OperationId::default();
infos is now a BTreeMap; run_trace() returns a normalized trace. Guarded
by replay_is_deterministic.

Invariants recalibrated for the now-rich schedules (both prior
formulations only ever saw ~1-event inputs): double-execution is checked
as eventual REVOKE-KILL, since timeout requeue revokes asynchronously
and the kill lands on a later pass; NO-GHOST gets a grace window of the
healing ceilings, and superseded same-id connections (workers.put
replacement closes the old channel) are marked dead instead of
fabricating conflicts. CLIENT_ACTION_TIMEOUT_S drops to 120s so the
drain's 768 virtual seconds actually cross every configured ceiling —
the abandoned-executing backstop only fires at ~2x that ceiling.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@b7r6
b7r6 force-pushed the pr/scheduler-race-fuzzer branch from 5e716de to d37ec26 Compare September 30, 2026 05:04
@vercel

vercel Bot commented Sep 30, 2026

Copy link
Copy Markdown

Deployment failed for project nativelink-aidm with the following error:

The provided GitHub repository does not contain the requested branch or commit reference. Please ensure the repository is not empty.

@vercel

vercel Bot commented Sep 30, 2026

Copy link
Copy Markdown

Deployment failed for project nativelink with the following error:

The provided GitHub repository does not contain the requested branch or commit reference. Please ensure the repository is not empty.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@b7r6

b7r6 commented Oct 3, 2026

Copy link
Copy Markdown
Contributor Author

Withdrawing this from the queue to respect the team's review bandwidth. It's a verified fix (fails-without regression, retained on our fork) — rather than keep a flood of drafts open, we'll re-propose one clean, reviewed PR at a time. Happy to prioritize any of these on request.

@b7r6 b7r6 closed this Oct 3, 2026
b7r6 added a commit to b7r6/nativelink that referenced this pull request Oct 7, 2026
…under a live id

ApiWorkerSchedulerImpl::add_worker used LruCache::put, which silently
replaced a live entry when a worker re-registered under an existing id.
The replaced Worker was dropped together with its running_action_infos:
nothing was requeued, the old channel never heard Disconnect, and the
gauges were only partially adjusted. Downstream, the replaced worker's
operations stayed Executing assigned to the id; when the new incarnation
was later evicted its empty list requeued nothing, and the eventual
timeout requeued the operation to another worker while the original
process might still be running it -- double execution.

Now ApiWorkerScheduler::add_worker treats a live entry under the same id
exactly like a disconnect before admitting the new incarnation: it goes
through immediate_evict_worker, which drains running_action_infos via
UpdateOperationType::UpdateWithDisconnect, sends Disconnect on the old
channel, and balances the connect/disconnect gauges. A requeue failure
is logged and does not block the reconnecting worker.

Deterministically reproduced by the DST fuzzer schedule in PR TraceMachina#2831;
the new regression test (worker re-registers mid-execution, asserts
Disconnect on the old channel, re-dispatch of the same operation to the
new channel, and completion) fails on unfixed main.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
b7r6 added a commit to b7r6/nativelink that referenced this pull request Oct 7, 2026
…under a live id

ApiWorkerSchedulerImpl::add_worker used LruCache::put, which silently
replaced a live entry when a worker re-registered under an existing id.
The replaced Worker was dropped together with its running_action_infos:
nothing was requeued, the old channel never heard Disconnect, and the
gauges were only partially adjusted. Downstream, the replaced worker's
operations stayed Executing assigned to the id; when the new incarnation
was later evicted its empty list requeued nothing, and the eventual
timeout requeued the operation to another worker while the original
process might still be running it -- double execution.

Now ApiWorkerScheduler::add_worker treats a live entry under the same id
exactly like a disconnect before admitting the new incarnation: it goes
through immediate_evict_worker, which drains running_action_infos via
UpdateOperationType::UpdateWithDisconnect, sends Disconnect on the old
channel, and balances the connect/disconnect gauges. A requeue failure
is logged and does not block the reconnecting worker.

Deterministically reproduced by the DST fuzzer schedule in PR TraceMachina#2831;
the new regression test (worker re-registers mid-execution, asserts
Disconnect on the old channel, re-dispatch of the same operation to the
new channel, and completion) fails on unfixed main.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
b7r6 added a commit to b7r6/nativelink that referenced this pull request Oct 7, 2026
…under a live id

ApiWorkerSchedulerImpl::add_worker used LruCache::put, which silently
replaced a live entry when a worker re-registered under an existing id.
The replaced Worker was dropped together with its running_action_infos:
nothing was requeued, the old channel never heard Disconnect, and the
gauges were only partially adjusted. Downstream, the replaced worker's
operations stayed Executing assigned to the id; when the new incarnation
was later evicted its empty list requeued nothing, and the eventual
timeout requeued the operation to another worker while the original
process might still be running it -- double execution.

Now ApiWorkerScheduler::add_worker treats a live entry under the same id
exactly like a disconnect before admitting the new incarnation: it goes
through immediate_evict_worker, which drains running_action_infos via
UpdateOperationType::UpdateWithDisconnect, sends Disconnect on the old
channel, and balances the connect/disconnect gauges. A requeue failure
is logged and does not block the reconnecting worker.

Deterministically reproduced by the DST fuzzer schedule in PR TraceMachina#2831;
the new regression test (worker re-registers mid-execution, asserts
Disconnect on the old channel, re-dispatch of the same operation to the
new channel, and completion) fails on unfixed main.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
b7r6 added a commit to b7r6/nativelink that referenced this pull request Oct 8, 2026
…under a live id

ApiWorkerSchedulerImpl::add_worker used LruCache::put, which silently
replaced a live entry when a worker re-registered under an existing id.
The replaced Worker was dropped together with its running_action_infos:
nothing was requeued, the old channel never heard Disconnect, and the
gauges were only partially adjusted. Downstream, the replaced worker's
operations stayed Executing assigned to the id; when the new incarnation
was later evicted its empty list requeued nothing, and the eventual
timeout requeued the operation to another worker while the original
process might still be running it -- double execution.

Now ApiWorkerScheduler::add_worker treats a live entry under the same id
exactly like a disconnect before admitting the new incarnation: it goes
through immediate_evict_worker, which drains running_action_infos via
UpdateOperationType::UpdateWithDisconnect, sends Disconnect on the old
channel, and balances the connect/disconnect gauges. A requeue failure
is logged and does not block the reconnecting worker.

Deterministically reproduced by the DST fuzzer schedule in PR TraceMachina#2831;
the new regression test (worker re-registers mid-execution, asserts
Disconnect on the old channel, re-dispatch of the same operation to the
new channel, and completion) fails on unfixed main.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
b7r6 added a commit to b7r6/nativelink that referenced this pull request Oct 14, 2026
…under a live id

ApiWorkerSchedulerImpl::add_worker used LruCache::put, which silently
replaced a live entry when a worker re-registered under an existing id.
The replaced Worker was dropped together with its running_action_infos:
nothing was requeued, the old channel never heard Disconnect, and the
gauges were only partially adjusted. Downstream, the replaced worker's
operations stayed Executing assigned to the id; when the new incarnation
was later evicted its empty list requeued nothing, and the eventual
timeout requeued the operation to another worker while the original
process might still be running it -- double execution.

Now ApiWorkerScheduler::add_worker treats a live entry under the same id
exactly like a disconnect before admitting the new incarnation: it goes
through immediate_evict_worker, which drains running_action_infos via
UpdateOperationType::UpdateWithDisconnect, sends Disconnect on the old
channel, and balances the connect/disconnect gauges. A requeue failure
is logged and does not block the reconnecting worker.

Deterministically reproduced by the DST fuzzer schedule in PR TraceMachina#2831;
the new regression test (worker re-registers mid-execution, asserts
Disconnect on the old channel, re-dispatch of the same operation to the
new channel, and completion) fails on unfixed main.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
b7r6 added a commit to b7r6/nativelink that referenced this pull request Oct 15, 2026
…under a live id

ApiWorkerSchedulerImpl::add_worker used LruCache::put, which silently
replaced a live entry when a worker re-registered under an existing id.
The replaced Worker was dropped together with its running_action_infos:
nothing was requeued, the old channel never heard Disconnect, and the
gauges were only partially adjusted. Downstream, the replaced worker's
operations stayed Executing assigned to the id; when the new incarnation
was later evicted its empty list requeued nothing, and the eventual
timeout requeued the operation to another worker while the original
process might still be running it -- double execution.

Now ApiWorkerScheduler::add_worker treats a live entry under the same id
exactly like a disconnect before admitting the new incarnation: it goes
through immediate_evict_worker, which drains running_action_infos via
UpdateOperationType::UpdateWithDisconnect, sends Disconnect on the old
channel, and balances the connect/disconnect gauges. A requeue failure
is logged and does not block the reconnecting worker.

Deterministically reproduced by the DST fuzzer schedule in PR TraceMachina#2831;
the new regression test (worker re-registers mid-execution, asserts
Disconnect on the old channel, re-dispatch of the same operation to the
new channel, and completion) fails on unfixed main.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
b7r6 added a commit to b7r6/nativelink that referenced this pull request Oct 21, 2026
…under a live id

ApiWorkerSchedulerImpl::add_worker used LruCache::put, which silently
replaced a live entry when a worker re-registered under an existing id.
The replaced Worker was dropped together with its running_action_infos:
nothing was requeued, the old channel never heard Disconnect, and the
gauges were only partially adjusted. Downstream, the replaced worker's
operations stayed Executing assigned to the id; when the new incarnation
was later evicted its empty list requeued nothing, and the eventual
timeout requeued the operation to another worker while the original
process might still be running it -- double execution.

Now ApiWorkerScheduler::add_worker treats a live entry under the same id
exactly like a disconnect before admitting the new incarnation: it goes
through immediate_evict_worker, which drains running_action_infos via
UpdateOperationType::UpdateWithDisconnect, sends Disconnect on the old
channel, and balances the connect/disconnect gauges. A requeue failure
is logged and does not block the reconnecting worker.

Deterministically reproduced by the DST fuzzer schedule in PR TraceMachina#2831;
the new regression test (worker re-registers mid-execution, asserts
Disconnect on the old channel, re-dispatch of the same operation to the
new channel, and completion) fails on unfixed main.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
b7r6 added a commit to b7r6/nativelink that referenced this pull request Oct 22, 2026
Five places told contributors to run a `scheduler_race` deterministic-
simulation fuzzer before touching the scheduler. It does not exist
here. `nativelink-test/fuzz` declares exactly one `[[bin]]`,
`cas_config`, and depends on `nativelink-config` alone — so it could
not host a scheduler target as it stands. The book was written against
a tree carrying PR TraceMachina#2831; this branch is based on fork main, which is
not.

Corrects the fuzzing section to describe the one target that exists,
says plainly that no scheduler fuzzer is in the tree and that the
prototype lives on the fork, and replaces "run the fuzzer" in the
contribution checklist with the check that actually applies: the
interleaving tests under nativelink-scheduler/tests/ and a regression
that fails without the fix.

A checkable instruction that cannot be carried out is worse than no
instruction, because the reader spends their afternoon before
concluding the book is wrong.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

This branch was successfully deployed

2 active deployments
Preview – nativelink — 314e9d7d Deployed Sep 30, 2026 by vercel[bot]
Preview – nativelink-aidm — 314e9d7d Deployed Sep 30, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant