Skip to content

GoingAway with a drain flag, reconnect backoff with jitter, and a shutdown guard that waits for the drain - #2807

Merged
amankrx merged 3 commits into
TraceMachina:mainfrom
amankrx:pr/going-away-drain
Sep 28, 2026
Merged

amankrx merged 3 commits into
TraceMachina:mainfrom
amankrx:pr/going-away-drain

Conversation

@amankrx

@amankrx amankrx commented Sep 28, 2026 •

Copy link
Copy Markdown
Member

What and why

A worker shutting down finished its actions first and announced GoingAway last, so until then the scheduler kept offering it work it refused. GoingAway now carries a drain flag: with drain_on_shutdown: true the worker tells the scheduler at once that it is draining, gets nothing new, finishes what it has and exits, and the scheduler removes it when the stream closes. Reconnects back off with jitter instead of hammering the scheduler in lockstep. The original shutdown guard is counted, so a SIGTERM waits for the worker to drain instead of racing it.

How was this verified?

worker_api_server_test covers a GoingAway with the drain flag pausing dispatch to that worker; shutdown_guard_test covers the guard count holding the shutdown until the worker's clone is dropped. On a test cluster a worker stopped with the flag on took no new action after its announcement and finished its ten running ones; without the flag it kept receiving and refusing work until it exited.

Risk

One optional proto field on GoingAwayRequest, ignored by an older scheduler. drain_on_shutdown is off by default; ship the scheduler side before turning it on anywhere, since an older scheduler would treat the early GoingAway as a plain leave and requeue the running actions. The backoff only changes reconnect timing.

AI assistance

An agent drafted the change and I reviewed every line.


This change is Reviewable

@vercel

vercel Bot commented Sep 28, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
nativelink Ready Ready Preview Sep 28, 2026 10:32pm UTC
nativelink-aidm Ready Ready Preview Sep 28, 2026 10:32pm UTC

Request Review

… work, and reconnect backoff with jitter

(cherry picked from commit 3b2b6d033335da13654742ab7eaffd01d9dd60ca)
…o drain

(cherry picked from commit dbb4ce0e930313e1794c4906a8b9edcada028282)
@amankrx
amankrx merged commit 2bfa020 into TraceMachina:main Sep 28, 2026
47 checks passed

This branch was successfully deployed

2 active deployments
Preview – nativelink — 2fa5a05a Deployed Sep 28, 2026 by vercel[bot]
Preview – nativelink-aidm — 2fa5a05a Deployed Sep 28, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants