Repository navigation
Report running operations on the worker keepalive and requeue what the worker no longer runs - #2925
Merged
amankrx merged 4 commits intoOct 12, 2026
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
corcillo
reviewed
Oct 12, 2026
corcillo
reviewed
Oct 12, 2026
corcillo
reviewed
Oct 12, 2026
…e worker no longer runs
corcillo
approved these changes
Oct 12, 2026
This was referenced Oct 13, 2026
This was referenced Oct 16, 2026
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
The scheduler knows what it dispatched to a worker and what the worker reported done; it never hears what the worker is running. A result whose store write failed, or a dispatch the worker took and lost, leaves the operation in the worker's running set on the scheduler, holding a slot, with the record
Executing, until a ceiling measured from assignment fires, which with the default ceiling is never.A worker's keepalive now carries the operations it is running, from the dispatch until the result is published (
KeepAliveRequest.running_operations, absent from older workers). The scheduler compares that with what it holds for the worker. An acknowledged operation the worker leaves out of two reports in a row (one may have crossed the dispatch) is released and requeued through the error path, so the loss spends one ofmax_job_retriesand a result the store keeps refusing ends in a failure the client sees. Operations not yet acknowledged are the acknowledgement sweep's; ones told to die are on their way off. The executing ceiling is untouched and still measures from assignment.How was this verified?
A new scheduler test over a recording state manager: an acknowledged operation reported three times is left alone, one report leaving it out keeps the slot, a report naming it again restarts the count, and the second consecutive omission releases the slot and requeues with
Unavailable; an unacknowledged operation and one being killed are never counted. The scheduler, service and worker crates pass all tests and clippy with-D warnings; the generated proto code is regenerated.Risk
One field on every keepalive, a few dozen operation ids at most. A worker older than this change sends no report and nothing changes for it. A lost result now costs a retry where before it cost the slot for good.
AI assistance
An agent drafted the change and I reviewed every line.