Skip to content

Worker reaps the zombies actions leave behind, and an action's stragglers no longer hold its output open - #2823

Merged
amankrx merged 5 commits into
TraceMachina:mainfrom
amankrx:pr/worker-reaper-and-drain
Sep 29, 2026
Merged

amankrx merged 5 commits into
TraceMachina:mainfrom
amankrx:pr/worker-reaper-and-drain

Conversation

@amankrx

@amankrx amankrx commented Sep 29, 2026 •

Copy link
Copy Markdown
Member

What and why

A worker was found with hundreds of zombie processes, and actions on it that had exited but never finished. Both come from what an action leaves behind. The worker waits only on the process it spawned; a background job or a daemonized helper that process forked is reparented when it exits, usually to the worker as the container's PID 1, which never waited on it, so it stayed a zombie for the life of the pod. And such a process holds the action's stdout pipe open, so the worker's read to end-of-file, and with it the action, waited until the action's timeout. The worker now makes itself the subreaper of its actions' orphans and reaps their zombies every five seconds. Every child the worker spawns itself (an action, a precondition script, a persistent worker, the Buck2 helper) is registered while its handle lives and is never touched, whatever its state; on top of that a zombie is reaped only on the second sweep it appears in, and a reaped pid is not carried forward. The scan reads one stat line per process and runs off the runtime. When the action's process exits, whatever is left in its process group is killed, as the sandboxes Bazel runs locally do, so the pipes close; the drain is bounded at five seconds only for a killed action or one that left stragglers, and what was read by then is kept; an action that exited cleanly with nothing left behind is read to the end, so a reader slowed by a busy spill never loses output to a clock.

How was this verified?

reaper_test: a shell backgrounds a sleep and exits, the sleep becomes the worker's zombie, the first sweep notes it and the second reaps it (Linux only, like the code); the status parser only reports zombies. running_actions_manager_test: an action that exits leaving a sleep on its stdout finishes at once with what it printed, on Linux by the group kill and elsewhere by the drain deadline, rather than after its timeout.

Risk

A process an action deliberately leaves running past its exit is now killed. That was never valid under remote execution, since the outputs have to be complete at exit, and the local sandboxes end them the same way. Under use_namespaces the stub already reaps inside the action's namespace; this adds nothing there. The subreaper flag is a Linux prctl; elsewhere both are no-ops.

AI assistance

An agent drafted the change and I reviewed every line.


This change is Reviewable

@vercel

vercel Bot commented Sep 29, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
nativelink Ready Ready Preview Sep 29, 2026 11:08pm UTC
nativelink-aidm Ready Ready Preview Sep 29, 2026 11:08pm UTC

Request Review

Comment thread nativelink-worker/src/reaper.rs Resolved
Comment thread nativelink-worker/src/reaper.rs Resolved
Comment thread nativelink-worker/src/reaper.rs Resolved
corcillo
corcillo previously approved these changes Sep 29, 2026

@corcillo corcillo left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LG after comments

…reaped pids; drain bounded only for killed or straggling actions; biased read
@amankrx
amankrx merged commit 4fe0a27 into TraceMachina:main Sep 29, 2026
45 checks passed

This branch was successfully deployed

2 active deployments
Preview – nativelink — 1f26cafa Deployed Sep 29, 2026 by vercel[bot]
Preview – nativelink-aidm — 1f26cafa Deployed Sep 29, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants