Skip to content

Report running operations on the worker keepalive and requeue what the worker no longer runs - #2925

Merged
amankrx merged 4 commits into
TraceMachina:mainfrom
amankrx:fix/worker-keepalive-liveness
Oct 12, 2026
Merged

amankrx merged 4 commits into
TraceMachina:mainfrom
amankrx:fix/worker-keepalive-liveness

Conversation

@amankrx

@amankrx amankrx commented Oct 12, 2026 •

Copy link
Copy Markdown
Member

What and why

The scheduler knows what it dispatched to a worker and what the worker reported done; it never hears what the worker is running. A result whose store write failed, or a dispatch the worker took and lost, leaves the operation in the worker's running set on the scheduler, holding a slot, with the record Executing, until a ceiling measured from assignment fires, which with the default ceiling is never.

A worker's keepalive now carries the operations it is running, from the dispatch until the result is published (KeepAliveRequest.running_operations, absent from older workers). The scheduler compares that with what it holds for the worker. An acknowledged operation the worker leaves out of two reports in a row (one may have crossed the dispatch) is released and requeued through the error path, so the loss spends one of max_job_retries and a result the store keeps refusing ends in a failure the client sees. Operations not yet acknowledged are the acknowledgement sweep's; ones told to die are on their way off. The executing ceiling is untouched and still measures from assignment.

How was this verified?

A new scheduler test over a recording state manager: an acknowledged operation reported three times is left alone, one report leaving it out keeps the slot, a report naming it again restarts the count, and the second consecutive omission releases the slot and requeues with Unavailable; an unacknowledged operation and one being killed are never counted. The scheduler, service and worker crates pass all tests and clippy with -D warnings; the generated proto code is regenerated.

Risk

One field on every keepalive, a few dozen operation ids at most. A worker older than this change sends no report and nothing changes for it. A lost result now costs a retry where before it cost the slot for good.

AI assistance

An agent drafted the change and I reviewed every line.

@vercel

vercel Bot commented Oct 12, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
nativelink Ready Ready Preview Oct 12, 2026 7:00pm UTC
nativelink-aidm Ready Ready Preview Oct 12, 2026 7:00pm UTC

Request Review

@amankrx amankrx changed the title Refresh running operations' worker-update timestamp from the worker keepalive Report running operations on the worker keepalive and requeue what the worker no longer runs Oct 12, 2026

This branch was successfully deployed

2 active deployments
Preview – nativelink — 621db016 Deployed Oct 12, 2026 by vercel[bot]
Preview – nativelink-aidm — 621db016 Deployed Oct 12, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants