Skip to content

fix(scheduler): re-validate worker liveness under the eviction lock in remove_timedout_workers - #40

Open
b7r6 wants to merge 2 commits into
mainfrom
fix/eviction-rechecks-heartbeat
Open

b7r6 wants to merge 2 commits into
mainfrom
fix/eviction-rechecks-heartbeat

Conversation

@b7r6

@b7r6 b7r6 commented Oct 7, 2026 •

Copy link
Copy Markdown
Owner

What and why

Lane A stability fix from the findings catalog (see nativelink-findings-2026-10-05.csv
on the meta/findings-catalog-2026-10-05 branch), rebased onto OSS main
@bf4b3aa855. This fork PR exists to validate the change on CI that is
representative of upstream PR lanes (public read-only cache, same endpoints)
before any upstream submission.

How was this verified?

The branch carries its own regression tests (fails-without where the fix
class allows it — see the branch's test commits). This PR's purpose is the
full representative CI run; green here rehearses the exact checks an
upstream PR would face.

Risk

Fork-internal validation PR; nothing lands anywhere by merging this. The
underlying fix's risk assessment happens at upstream-submission time.

AI assistance

Agent-assisted pipeline (Claude) for rebase and PR mechanics under my
direction; the fix itself carries verification in its tests.

@github-actions

This comment has been minimized.

@b7r6
b7r6 force-pushed the fix/eviction-rechecks-heartbeat branch 3 times, most recently from ce608bf to f7483a1 Compare October 8, 2026 17:51
@b7r6
b7r6 force-pushed the fix/eviction-rechecks-heartbeat branch from f7483a1 to d154a5e Compare October 14, 2026 17:40
@b7r6
b7r6 force-pushed the fix/eviction-rechecks-heartbeat branch from d154a5e to 113b729 Compare October 15, 2026 11:19
b7r6 and others added 2 commits October 21, 2026 17:14
Completes the no-per-PR-OCI-builds intent: image-off-pr moved image.yaml off
PRs and TraceMachina#2877 moved the nix integration job off PRs, but main.yaml's
integration-tests (docker-compose image build + run_integration_tests.sh)
fell between them and still ran on every PR. Defer to nightly + merge-to-main,
matching TraceMachina#2880's pattern. Fork-local until folded into the upstream OCI/nightly
set.
…n remove_timedout_workers

remove_timedout_workers had a TOCTOU window: it snapshots each worker's
last_update_timestamp under the inner lock, releases the lock to consult
the worker registry (an await boundary every sweep crosses), then
re-acquires the lock and calls immediate_evict_worker using the stale
snapshot. A keepalive processed in that window refreshes
last_update_timestamp under the very same mutex, yet the sweep never
re-consults it, so a demonstrably-alive worker is evicted and all of its
running actions are requeued.

Fix: after re-acquiring the inner lock, re-read the worker's
last_update_timestamp and skip the eviction if it has moved past the
timeout threshold. Because keepalives update the timestamp under the
same mutex, the staleness decision and the eviction are now atomic with
respect to keepalive processing. No new locking is added to any dispatch
path. Kill-overdue evictions deliberately do not re-check: keepalives
from a worker that refuses to acknowledge a kill are exactly the
wedged-worker symptom that path exists to handle.

Tests: two deterministic repros. One parks the sweep at the pre-existing
await boundary via a cfg(test)-only Notify rendezvous and delivers a
full worker_keep_alive_received; the other forces the interleaving with
genuine lock contention (holding the registry write lock, then the inner
lock) via a cfg(test)-only WorkerRegistry::write_lock_for_test accessor.
Both fail without the fix and pass with it. The write_lock_for_test seam
is kept: it exercises the window through real lock contention rather
than the injected gate, so the two tests do not share a failure mode.

Provenance: adversarial review fleet.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant