Repository navigation
Worker reaps the zombies actions leave behind, and an action's stragglers no longer hold its output open - #2823
Merged
Conversation
…lers no longer hold its output open
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
… the zombie sets by lookup
corcillo
reviewed
Sep 29, 2026
corcillo
reviewed
Sep 29, 2026
corcillo
reviewed
Sep 29, 2026
corcillo
reviewed
Sep 29, 2026
corcillo
reviewed
Sep 29, 2026
…reaped pids; drain bounded only for killed or straggling actions; biased read
corcillo
approved these changes
Sep 29, 2026
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
A worker was found with hundreds of zombie processes, and actions on it that had exited but never finished. Both come from what an action leaves behind. The worker waits only on the process it spawned; a background job or a daemonized helper that process forked is reparented when it exits, usually to the worker as the container's PID 1, which never waited on it, so it stayed a zombie for the life of the pod. And such a process holds the action's stdout pipe open, so the worker's read to end-of-file, and with it the action, waited until the action's timeout. The worker now makes itself the subreaper of its actions' orphans and reaps their zombies every five seconds. Every child the worker spawns itself (an action, a precondition script, a persistent worker, the Buck2 helper) is registered while its handle lives and is never touched, whatever its state; on top of that a zombie is reaped only on the second sweep it appears in, and a reaped pid is not carried forward. The scan reads one
statline per process and runs off the runtime. When the action's process exits, whatever is left in its process group is killed, as the sandboxes Bazel runs locally do, so the pipes close; the drain is bounded at five seconds only for a killed action or one that left stragglers, and what was read by then is kept; an action that exited cleanly with nothing left behind is read to the end, so a reader slowed by a busy spill never loses output to a clock.How was this verified?
reaper_test: a shell backgrounds a sleep and exits, the sleep becomes the worker's zombie, the first sweep notes it and the second reaps it (Linux only, like the code); the status parser only reports zombies.running_actions_manager_test: an action that exits leaving a sleep on its stdout finishes at once with what it printed, on Linux by the group kill and elsewhere by the drain deadline, rather than after its timeout.Risk
A process an action deliberately leaves running past its exit is now killed. That was never valid under remote execution, since the outputs have to be complete at exit, and the local sandboxes end them the same way. Under
use_namespacesthe stub already reaps inside the action's namespace; this adds nothing there. The subreaper flag is a Linuxprctl; elsewhere both are no-ops.AI assistance
An agent drafted the change and I reviewed every line.
This change is