Repository navigation
Restore per-action input fetches as the worker default and suggest a shared pool under load - #2984
Conversation
…shared pool under load The shared input fetch pool made 64 the default limit across every action a worker stages. Before it, each action fetched its inputs on its own, so a worker staging several cold actions at once now stages them more slowly than it did. Keep the pool, but make it opt-in again: max_concurrent_input_downloads of 0, now the default, shares no pool, and any other value is the size of the pool every action shares. A worker without the pool says when it might need one. When an input fetch fails with a transient store error, or the fetch runs past max_download_timeout, it logs a warning naming max_concurrent_input_downloads and suggesting 64, at most once every 10 minutes so a storm of failed fetches yields one line of advice. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HBxQtV9S561AzBo7aPCYFj
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
| ); | ||
| let max_downloads = args.execution_configuration.max_concurrent_input_downloads; | ||
| let input_downloads = input_download_pool(max_downloads); | ||
| let input_pool_hint = (max_downloads == 0).then(InputPoolHint::default); |
There was a problem hiding this comment.
On the direction: with this merged, 1.8.1 has no default limit on how many input fetches a worker sends to the object store. max_concurrent_reads (#2941) and max_concurrent_requests (#2985) are both off by default too. #2952 made the pool the default because of throttled-store incidents: fresh worker pools whose cold actions together held hundreds of reads open against one account. Do we have a measurement of how much slower staging got with the shared 64? If the slowdown is the concern, a larger shared default (say 256) might keep most of the protection without the regression.
Whichever way we go, I'd rather decide before v1.8.0 is tagged than change the default in 1.8.0 and change it back in 1.8.1. If this is the direction, merging it into 1.8.0 means the default never changes at all.
There was a problem hiding this comment.
No, there is no measurement of the slowdown; the description now says so. We are taking the second option: keep the v1.7.7 default (no shared pool) and merge this before v1.8.0 is tagged, so no release changes the default. #2979 will wait for this, then merge main and regenerate its changelog. The Risk section now covers the throttled-store case and points those deployments at max_concurrent_input_downloads: 64. The worker warning suggests the same setting once fetches start failing or stalling.
| } | ||
| }; | ||
| self.running_actions_manager | ||
| .suggest_input_download_pool(&self.operation_id, &err); |
There was a problem hiding this comment.
is_transient_fetch_error also counts DataLoss as transient, so a truncated or corrupt blob triggers advice about store overload. That points operators at the wrong cause, especially now that #2974 and #2993 turn bad stored copies into DataLoss and NotFound. Could the hint skip DataLoss?
if err.code != Code::DataLoss {
self.running_actions_manager
.suggest_input_download_pool(&self.operation_id, &err);
}
There was a problem hiding this comment.
Done in 6bfc9a9. The check is in suggest_input_download_pool rather than at this call site, so it holds for any caller. prepare_action_with_data_loss_suggests_nothing covers it: two DataLoss reads are retried and the action stages with no advice. The test fails with the check removed.
DataLoss is retried as transient, but it means a truncated or corrupt stored copy rather than an overloaded store, so it no longer suggests max_concurrent_input_downloads. The field's first doc paragraph, the one the config reference shows, now says that 0 shares no pool, and the reference default reads "0 (no shared pool)". Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Regenerate the v1.8.0 section after merging main, and pin TraceMachina#2984 to Bug Fixes by subject, since the generic rules file it under Miscellaneous. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
What and why
This makes the shared input fetch pool from #2952 opt-in again, before any release ships it on by default.
main, not yet released): made 64 the default limit across every action a worker stages. A worker staging several cold actions at once shares those 64 fetches between them instead of each action having its own. That this slows staging follows from the limit; it has not been measured.max_concurrent_input_downloadsof 0 (the default) shares no pool, and any positive value is the size of the pool every action shares. The standalonedownload_to_directoryloses its per-call pool too, matching v1.7.7.A worker without the pool now says when it might need one. It logs one warning when either of these happens:
DataLoss;max_download_timeout.DataLossis still retried, but it means a truncated or corrupt stored copy rather than an overloaded store, so it gives no advice. The warning namesmax_concurrent_input_downloads, suggests 64, and explains that several cold actions fetching independently can overload the CAS and the object store behind it. It is logged at most once every 10 minutes, so a storm of failed fetches produces one line of advice rather than one per failure:This is meant for v1.8.0: please merge it before v1.8.0 is tagged, so no release changes the default. The release PR, #2979, currently describes the 64 default; it will merge
main, regenerate its changelog and drop that note once this lands.How was this verified?
Commands run.
cargo test -p nativelink-worker: all suites pass, including all 77 tests inrunning_actions_manager_test.cargo test -p nativelink-config: passes.cargo clippy -p nativelink-worker -p nativelink-config --all-targets: no warnings from the code. The only warning is the existing unusedworkspace.cargo-features-managerkey inCargo.toml.New and changed tests.
input_download_pool_is_unbounded_unless_configured(renamed frominput_download_pool_defaults_to_sixty_four): unset means unbounded, and 7 gives a pool of 7.prepare_action_suggests_a_shared_input_pool_once: twoUnavailableinput reads through a default-configured manager produce exactly one warning.prepare_action_with_data_loss_suggests_nothing: twoDataLossreads through a default-configured manager are retried, and the action is staged without a warning.prepare_action_with_a_shared_input_pool_suggests_nothing: a worker configured with 64 logs none.prepare_action_times_out_on_a_hung_fetch: now also checks that a stall produces the warning.Checked against the previous source.
prepare_action_with_data_loss_suggests_nothingfails with theDataLosscheck removed ("1 pool suggestions for corrupt reads"), and passes with it.download_to_directory_bounds_fetches_by_the_poolandconcurrent_downloads_share_one_pool.Docs. The config reference was regenerated with
gen:config-reference, and only itsmainpage is committed. The generator also refreshes the per-release pages for v1.7.7, which is a separate post-release step. Onmain.mdxonly themax_concurrent_input_downloadsrow changed: the default reads "0 (no shared pool)", and the description ends with "0 shares no pool; each action fetches on its own." The table shows only a field's first doc paragraph, so that sentence moved into it.Not run. Pre-commit passed on every changed file. Bazel was not run locally, and nothing was tested against a deployment.
Risk
max_concurrent_input_downloadsis set.max_concurrent_reads(Bound object store reads with an adaptive permit pool, Azure first #2941) is off by default too. Bound a worker's input fetches with one pool across the actions it stages #2952 made the pool the default after throttled-store incidents, where fresh workers' cold actions together held hundreds of reads open against one account. Deployments like that should setmax_concurrent_input_downloads: 64, and the warning points there when fetches start failing or stalling.main. No release ships the 64 default. A build frommainbetween Bound a worker's input fetches with one pool across the actions it stages #2952 and this change has it; such a worker gets no shared limit after updating.AI assistance
Claude Code wrote the change, its tests and this description at Marcus Eagan's request, after reading #2952 and the input fetch retry path. After review, it made the advice skip
DataLoss, moved the meaning of 0 into the first doc paragraph, and added theDataLosstest. It also ran the tests above, including the checks that the new tests fail without the change. Marcus Eagan chose to keep the opt-in default and target v1.8.0. A maintainer should review the warning's wording and its triggers.🤖 Generated with Claude Code
https://claude.ai/code/session_01HBxQtV9S561AzBo7aPCYFj