Skip to content

Worker: a disk soft limit, and peak_disk_kb from the files an action writes - #2825

Merged
amankrx merged 9 commits into
TraceMachina:mainfrom
amankrx:pr/disk-soft-limit
Sep 30, 2026
Merged

amankrx merged 9 commits into
TraceMachina:mainfrom
amankrx:pr/disk-soft-limit

Conversation

@amankrx

@amankrx amankrx commented Sep 29, 2026 •

Copy link
Copy Markdown
Member

What and why

The disk guard refuses an action whose reservation does not fit before its inputs are fetched, but nothing watched what the action then wrote: one that filled the work volume took every action on the worker down with it, and the sizing loop had no disk figure to learn from. disk: soft is the guard plus a kill. Every ten seconds the worker adds up the files under the action's own directory that the action itself wrote (those modified since its command started; inputs were materialized before that, and a hard link count would lie once the store evicts a blob), and an action past its disk_kb reservation by disk_headroom_percent (default 20) is killed and fails with FAILED_PRECONDITION naming the numbers, its outcome KILLED_DISK. With either disk mode the usage report carries peak_disk_kb, what the action wrote. The sampler that measures memory now carries both ceilings, and the first one breached ends the action. The walk runs as its own task on the blocking pool and is polled from the sampler, so memory sampling never waits on it; guard pays one walk at completion and no periodic one. peak_disk_kb is reported and exported for now; the scheduler's sizing consumer comes with the size ladder.

How was this verified?

running_actions_manager_test: an action that writes 30 MiB against a 1 MiB reservation is killed at the ten-second sample, fails naming the disk reservation, and reports KILLED_DISK with the bytes it wrote (Linux only, like the sampler); the directory walk counts an action's own files and not a hard-linked input, a symlink or the directories themselves. The enforcement config test covers disk: soft alone.

Risk

Off unless disk: soft is configured; guard gains one walk of the action directory at completion for the measurement, none behaves as before. The walk runs on the blocking pool every ten seconds over the action directory only. An action that writes large temporary files and removes them again within ten seconds is never seen over the limit; one that keeps them past it is.

AI assistance

An agent drafted the change and I reviewed every line.


This change is Reviewable

@vercel

vercel Bot commented Sep 29, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
nativelink Ready Ready Preview Sep 30, 2026 12:22am UTC
nativelink-aidm Ready Ready Preview Sep 30, 2026 12:22am UTC

Request Review

…ect the sample to catch the write in progress
corcillo
corcillo previously approved these changes Sep 29, 2026

@corcillo corcillo left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LG after comments

…a polled task, periodic walks only for the soft limit, the disk figure counts as sampled
corcillo
corcillo previously approved these changes Sep 29, 2026
@amankrx
amankrx merged commit 7bd3c54 into TraceMachina:main Sep 30, 2026
47 checks passed

This branch was successfully deployed

2 active deployments
Preview – nativelink — f3b12b97 Deployed Sep 30, 2026 by vercel[bot]
Preview – nativelink-aidm — f3b12b97 Deployed Sep 30, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants