Skip to content

Restore per-action input fetches as the worker default and suggest a shared pool under load - #2984

Merged
MarcusSorealheis merged 3 commits into
TraceMachina:mainfrom
MarcusSorealheis:bugfix/worker-input-downloads-default
Oct 21, 2026
Merged

MarcusSorealheis merged 3 commits into
TraceMachina:mainfrom
MarcusSorealheis:bugfix/worker-input-downloads-default

Conversation

@MarcusSorealheis

@MarcusSorealheis MarcusSorealheis commented Oct 20, 2026 •

Copy link
Copy Markdown
Member

What and why

This makes the shared input fetch pool from #2952 opt-in again, before any release ships it on by default.

  • v1.7.7: each action fetched its inputs on its own.
  • Bound a worker's input fetches with one pool across the actions it stages #2952 (on main, not yet released): made 64 the default limit across every action a worker stages. A worker staging several cold actions at once shares those 64 fetches between them instead of each action having its own. That this slows staging follows from the limit; it has not been measured.
  • This change: max_concurrent_input_downloads of 0 (the default) shares no pool, and any positive value is the size of the pool every action shares. The standalone download_to_directory loses its per-call pool too, matching v1.7.7.

A worker without the pool now says when it might need one. It logs one warning when either of these happens:

  • an input fetch fails with a transient store error other than DataLoss;
  • a fetch runs past max_download_timeout.

DataLoss is still retried, but it means a truncated or corrupt stored copy rather than an overloaded store, so it gives no advice. The warning names max_concurrent_input_downloads, suggests 64, and explains that several cold actions fetching independently can overload the CAS and the object store behind it. It is logged at most once every 10 minutes, so a storm of failed fetches produces one line of advice rather than one per failure:

Input fetches are failing or stalling while max_concurrent_input_downloads is unset: every action this worker stages fetches its inputs on its own, and several cold actions at once can overload the CAS and the object store behind it. Consider setting max_concurrent_input_downloads to 64 so they share one pool of fetches (logged at most every 10 minutes)

This is meant for v1.8.0: please merge it before v1.8.0 is tagged, so no release changes the default. The release PR, #2979, currently describes the 64 default; it will merge main, regenerate its changelog and drop that note once this lands.

How was this verified?

Commands run.

  • cargo test -p nativelink-worker: all suites pass, including all 77 tests in running_actions_manager_test.
  • cargo test -p nativelink-config: passes.
  • cargo clippy -p nativelink-worker -p nativelink-config --all-targets: no warnings from the code. The only warning is the existing unused workspace.cargo-features-manager key in Cargo.toml.

New and changed tests.

  • input_download_pool_is_unbounded_unless_configured (renamed from input_download_pool_defaults_to_sixty_four): unset means unbounded, and 7 gives a pool of 7.
  • prepare_action_suggests_a_shared_input_pool_once: two Unavailable input reads through a default-configured manager produce exactly one warning.
  • prepare_action_with_data_loss_suggests_nothing: two DataLoss reads through a default-configured manager are retried, and the action is staged without a warning.
  • prepare_action_with_a_shared_input_pool_suggests_nothing: a worker configured with 64 logs none.
  • prepare_action_times_out_on_a_hung_fetch: now also checks that a stall produces the warning.
  • Unit tests cover the 10-minute rate limit and the pool-size mapping, including clamping an oversized value instead of panicking.

Checked against the previous source.

Docs. The config reference was regenerated with gen:config-reference, and only its main page is committed. The generator also refreshes the per-release pages for v1.7.7, which is a separate post-release step. On main.mdx only the max_concurrent_input_downloads row changed: the default reads "0 (no shared pool)", and the description ends with "0 shares no pool; each action fetches on its own." The table shows only a field's first doc paragraph, so that sentence moved into it.

Not run. Pre-commit passed on every changed file. Bazel was not run locally, and nothing was tested against a deployment.

Risk

  • No default bound on input fetches. As in v1.7.7, a worker places no shared limit on the input fetches it sends to the CAS and the object store unless max_concurrent_input_downloads is set. max_concurrent_reads (Bound object store reads with an adaptive permit pool, Azure first #2941) is off by default too. Bound a worker's input fetches with one pool across the actions it stages #2952 made the pool the default after throttled-store incidents, where fresh workers' cold actions together held hundreds of reads open against one account. Deployments like that should set max_concurrent_input_downloads: 64, and the warning points there when fetches start failing or stalling.
  • Default change on main. No release ships the 64 default. A build from main between Bound a worker's input fetches with one pool across the actions it stages #2952 and this change has it; such a worker gets no shared limit after updating.
  • Warning volume. The warning adds at most one line every 10 minutes per worker, and only fires when fetches are already failing or stalling.
  • Hot path. The input fetch hot path is unchanged, apart from taking a permit from a pool that never runs out.

AI assistance

Claude Code wrote the change, its tests and this description at Marcus Eagan's request, after reading #2952 and the input fetch retry path. After review, it made the advice skip DataLoss, moved the meaning of 0 into the first doc paragraph, and added the DataLoss test. It also ran the tests above, including the checks that the new tests fail without the change. Marcus Eagan chose to keep the opt-in default and target v1.8.0. A maintainer should review the warning's wording and its triggers.

🤖 Generated with Claude Code

https://claude.ai/code/session_01HBxQtV9S561AzBo7aPCYFj

…shared pool under load

The shared input fetch pool made 64 the default limit across every action
a worker stages. Before it, each action fetched its inputs on its own, so a
worker staging several cold actions at once now stages them more slowly
than it did. Keep the pool, but make it opt-in again:
max_concurrent_input_downloads of 0, now the default, shares no pool, and
any other value is the size of the pool every action shares.

A worker without the pool says when it might need one. When an input fetch
fails with a transient store error, or the fetch runs past
max_download_timeout, it logs a warning naming
max_concurrent_input_downloads and suggesting 64, at most once every 10
minutes so a storm of failed fetches yields one line of advice.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HBxQtV9S561AzBo7aPCYFj
@vercel

vercel Bot commented Oct 20, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
nativelink Ready Ready Preview Oct 21, 2026 1:26pm UTC
nativelink-aidm Ready Ready Preview Oct 21, 2026 1:26pm UTC

Request Review

);
let max_downloads = args.execution_configuration.max_concurrent_input_downloads;
let input_downloads = input_download_pool(max_downloads);
let input_pool_hint = (max_downloads == 0).then(InputPoolHint::default);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On the direction: with this merged, 1.8.1 has no default limit on how many input fetches a worker sends to the object store. max_concurrent_reads (#2941) and max_concurrent_requests (#2985) are both off by default too. #2952 made the pool the default because of throttled-store incidents: fresh worker pools whose cold actions together held hundreds of reads open against one account. Do we have a measurement of how much slower staging got with the shared 64? If the slowdown is the concern, a larger shared default (say 256) might keep most of the protection without the regression.

Whichever way we go, I'd rather decide before v1.8.0 is tagged than change the default in 1.8.0 and change it back in 1.8.1. If this is the direction, merging it into 1.8.0 means the default never changes at all.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, there is no measurement of the slowdown; the description now says so. We are taking the second option: keep the v1.7.7 default (no shared pool) and merge this before v1.8.0 is tagged, so no release changes the default. #2979 will wait for this, then merge main and regenerate its changelog. The Risk section now covers the throttled-store case and points those deployments at max_concurrent_input_downloads: 64. The worker warning suggests the same setting once fetches start failing or stalling.

}
};
self.running_actions_manager
.suggest_input_download_pool(&self.operation_id, &err);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is_transient_fetch_error also counts DataLoss as transient, so a truncated or corrupt blob triggers advice about store overload. That points operators at the wrong cause, especially now that #2974 and #2993 turn bad stored copies into DataLoss and NotFound. Could the hint skip DataLoss?

if err.code != Code::DataLoss {
self.running_actions_manager
.suggest_input_download_pool(&self.operation_id, &err);
}

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 6bfc9a9. The check is in suggest_input_download_pool rather than at this call site, so it holds for any caller. prepare_action_with_data_loss_suggests_nothing covers it: two DataLoss reads are retried and the action stages with no advice. The test fails with the check removed.

Comment thread nativelink-config/src/cas_server.rs Resolved
DataLoss is retried as transient, but it means a truncated or corrupt
stored copy rather than an overloaded store, so it no longer suggests
max_concurrent_input_downloads. The field's first doc paragraph, the
one the config reference shows, now says that 0 shares no pool, and the
reference default reads "0 (no shared pool)".

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@MarcusSorealheis
MarcusSorealheis merged commit 2a3ad61 into TraceMachina:main Oct 21, 2026
34 checks passed
MarcusSorealheis added a commit to MarcusSorealheis/nativelink that referenced this pull request Oct 21, 2026
Regenerate the v1.8.0 section after merging main, and pin TraceMachina#2984 to Bug
Fixes by subject, since the generic rules file it under Miscellaneous.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

This branch was successfully deployed

2 active deployments
Preview – nativelink — 6bfc9a90 Deployed Oct 21, 2026 by vercel[bot]
Preview – nativelink-aidm — 6bfc9a90 Deployed Oct 21, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants