Repository navigation
Expose a max upload size for stores (#2077) - #3097
Merged
palfrey merged 2 commits intoNov 2, 2026
Merged
Conversation
Add StoreDriver::max_upload_size(), defaulting to u64::MAX. S3 (and R2/OCI), Azure and GCS report their provider limits. FastSlowStore announces the slow store's limit instead of u64::MAX when it populates the fast store without a has() check, so an S3 fast tier no longer rejects the fill with FailedPrecondition. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
Author
|
@MarcusSorealheis this implements #2077 as discussed. CLA is signed; could you approve the workflows when you have a moment? |
This was referenced Nov 4, 2026
This was referenced Nov 11, 2026
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
FastSlowStorepassesUploadSizeInfo::MaxSize(u64::MAX)to the fast store when it fills it from an S3, GCS or Azure slow store, because those stores skip thehas()that would give the exact size. A fast store that sizes its upload from that bound mishandles it: an S3 fast store refusesu64::MAXwithFailedPrecondition(it exceedsMAX_UPLOAD_SIZE), so the fill always fails.This adds
StoreDriver::max_upload_size(), defaulting tou64::MAX(no known limit). S3 (so also R2 and OCI) and Azure return the limits their stores already hard-code; GCS gets its documented 5 TiB limit.FastSlowStorenow announces the slow store's limit, since nothing larger could have been stored there.The digest's size is deliberately not used as an exact size: an AC value's length differs from its key's (see
lazy_slow_read_fills_a_compression_fast_tier_with_an_ac_entry).Fixes #2077
How was this verified?
Two new tests:
fast_slow_store_test::lazy_slow_read_announces_the_slow_store_limit_to_the_fast_store: a lazy slow store advertising a limit, and a fast store that records the size it is announced. Fails with the oldu64::MAX, passes with the change.s3_store_test::update_accepts_its_advertised_max_upload_size: an S3 store refusesMaxSize(u64::MAX)withFailedPrecondition, and acceptsMaxSize(store.max_upload_size()), completing a multipart upload.Ran
cargo test -p nativelink-store --test fast_slow_store_test --test s3_store_teston Windows: 39 and 20 tests pass. I did not run Bazel or clippy locally.Risk
Low. Only the
LazyExistenceOnSyncfill path changes, and only the bound it announces.verify,compression, ...) don't forward the limit and reportu64::MAX. This doesn't affect the fill path, which only applies when the slow store resolves to S3/GCS/Azure itself.AI assistance
Claude Code helped investigate the issue, draft the change and tests, and run them. I reviewed the diff and can answer for every line.
🤖 Generated with Claude Code