Repository navigation
GoingAway with a drain flag, reconnect backoff with jitter, and a shutdown guard that waits for the drain - #2807
Merged
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
… work, and reconnect backoff with jitter (cherry picked from commit 3b2b6d033335da13654742ab7eaffd01d9dd60ca)
…o drain (cherry picked from commit dbb4ce0e930313e1794c4906a8b9edcada028282)
amankrx
force-pushed
the
pr/going-away-drain
branch
from
September 28, 2026 21:11
f6added to
d52258d
Compare
MarcusSorealheis
approved these changes
Sep 28, 2026
This was referenced Oct 31, 2026
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
A worker shutting down finished its actions first and announced
GoingAwaylast, so until then the scheduler kept offering it work it refused.GoingAwaynow carries a drain flag: withdrain_on_shutdown: truethe worker tells the scheduler at once that it is draining, gets nothing new, finishes what it has and exits, and the scheduler removes it when the stream closes. Reconnects back off with jitter instead of hammering the scheduler in lockstep. The original shutdown guard is counted, so a SIGTERM waits for the worker to drain instead of racing it.How was this verified?
worker_api_server_testcovers aGoingAwaywith the drain flag pausing dispatch to that worker;shutdown_guard_testcovers the guard count holding the shutdown until the worker's clone is dropped. On a test cluster a worker stopped with the flag on took no new action after its announcement and finished its ten running ones; without the flag it kept receiving and refusing work until it exited.Risk
One optional proto field on
GoingAwayRequest, ignored by an older scheduler.drain_on_shutdownis off by default; ship the scheduler side before turning it on anywhere, since an older scheduler would treat the earlyGoingAwayas a plain leave and requeue the running actions. The backoff only changes reconnect timing.AI assistance
An agent drafted the change and I reviewed every line.
This change is