Repository navigation
Reject incomplete CAS uploads before publishing blobs - #2974
Merged
MarcusSorealheis merged 7 commits intoOct 20, 2026
Merged
MarcusSorealheis merged 7 commits into
MarcusSorealheis merged 7 commits into
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
palfrey
requested changes
Oct 20, 2026
palfrey
approved these changes
Oct 20, 2026
This was referenced Oct 20, 2026
b7r6
added a commit
to b7r6/nativelink
that referenced
this pull request
Oct 21, 2026
…ssionStore::has() a1e6efb (TraceMachina#2960) made CompressionStore::has() report a digest key's declared size instead of the compressed stream's length, and left string keys reporting the stored length as a follow-up. This is that follow-up. A FastSlowStore declares the slow tier's has() size to its fast tier, and since 086289a (TraceMachina#2974) the populate itself rejects a fill whose length differs from that size. A string key read through a compression slow tier therefore still fails every populate on main: Expected 514 upload bytes, received 88064 Slow store ended before the cache fill was complete A string key has no size of its own, so the size comes from the footer. Its last three fields (uncompressed_data_sz u64, block_size u32, version u8) have fixed widths and always end the stream, so one ranged get_part of 13 bytes at the inner store's reported length finds it without reading the blocks; the version and block size are checked. If the object went away since the inner has(), the key reports missing. If the footer is not at the reported length, the inner store reported something other than the stream's exact length (FilesystemStore reports its block-rounded footprint), and locating the footer would mean reading the whole stream, so has() logs a warning and reports the stored length as before; reporting None instead would call a present object missing. String keys rarely reach a compression store, since CAS traffic is digest keyed, so the extra read stays off the FindMissingBlobs path. Tests: - has_reports_uncompressed_size_of_string_key: the footer path, plus an empty object (0) and a missing key. - populate_from_compression_slow_store_into_filesystem_fast_store: end to end for a digest key and a string key. On main the string key fails as above. - has_reports_uncompressed_size_of_digest: the digest path TraceMachina#2960 fixed, over a two-block payload and has_many with a zero and a missing digest. - has_preserves_missing_digests_and_stored_string_key_sizes (from TraceMachina#2960) pinned the stored length for a string key; it now expects the blob's length and is renamed to match. - has_mixed_digests_preserves_result_order and has_nonzero_digests_delegates_single_batch used string keys against a recording inner store that reports sizes it cannot serve. They now use digest keys. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
b7r6
added a commit
to b7r6/nativelink
that referenced
this pull request
Oct 21, 2026
…stSlowStore's fast tier On main the refusal comes from 086289a (TraceMachina#2974), not TraceMachina#2881, which was reverted (bd8c8d9): the populate itself now checks the bytes it received against this size and fails with DataLoss. A fast_slow store whose slow tier is a FilesystemStore at the default 4 KiB block size fails every populate of a blob that is not a block multiple: Expected 4096 upload bytes, received 17 Slow store ended before the cache fill was complete That includes the dedup index_store (memory over filesystem) in nativelink-config/examples/filesystem_cas.json5. The rest of this message predates the rebase. FastSlowStore::populate_and_maybe_stream took the size it declares to the fast tier, as ExactSize, from the slow tier's has(). Since 264b7eb (TraceMachina#2881) a filesystem fast tier refuses a fill whose length differs from it, so any slow tier that reports a physical size fails every populate. The compression store was one (fixed in the previous commit). Another is FilesystemStore, whose has() reports its block-rounded footprint (4096 bytes for a 17-byte blob at the default block size), so a fast_slow store with filesystem tiers on both sides, as in deployment-examples/docker-compose/test-multi-worker-simple.json5, fails the same way. So do dedup over a filesystem content store, which sums those footprints, and GrpcStore in AC mode, which reports u64::MAX. Declaring ExactSize(digest.size_bytes()) for every digest key would fix those for a CAS, but a digest's size is not always the size of what it names. An action cache keys an ActionResult by the action's digest, a dedup index store keys a blob's index by the blob's digest, and the chunking layout store does the same. A fast_slow AC with a filesystem fast tier over Redis, Mongo or memory works today because has() reports the entry's size; declaring the digest's size would make the filesystem tier refuse every such entry, and the memory-over-filesystem dedup index store in filesystem_cas.json5 would skip caching the index of every blob larger than the memory tier's max_bytes. So for a digest key the populate now checks one answer against the other. When has() agrees with the digest's size, that size is declared exact, as before. When they disagree, this store cannot tell which is wrong, so the larger is declared as MaxSize. A filesystem or memory fast tier accepts any length under MaxSize, and a compression fast tier gets a bound that covers the blob as long as either answer does. A missing blob still fails the has() check with the same NotFound and MissingDigest context. The LazyExistenceOnSync branch, which skips has(), keeps MaxSize(u64::MAX); AC_MAIN_STORE in gcs_backend.json5 is a filesystem tier over GCS, so the digest's size cannot go there either. String keys keep ExactSize(has()), as there is nothing to check it against. The cost is that when the two disagree, the fast tier loses the ExactSize guard, and MemoryStore's skip of writes over max_bytes, which only trusts ExactSize, does not apply. For a CAS that happens only behind a slow tier that misreports, which after the previous commit leaves FilesystemStore's rounding; once its has() reports the stored length, it stops. update_with_whole_file and the other paths take their sizes from the caller's UploadSizeInfo, the file's metadata or update()'s return value; populate was the only place that derived an ExactSize from has(). Tests: - populate_does_not_trust_slow_store_has_size_for_digest: a slow store whose has() reports 1 or 4096 for a 3000-byte blob. The read returns the blob and the filesystem fast tier holds all 3000 bytes. It fails on 7149987, and with only the previous commit, with "refusing to store a partial blob". - populate_keeps_entry_whose_size_is_not_its_digests: an AC-shaped entry, 3000 bytes under a digest that says 100, behind an exact memory slow tier. It passes on 7149987 and fails if the populate declares ExactSize(digest.size_bytes()); it guards against that simplification. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ashishsonamm
added a commit
to ashishsonamm/nativelink
that referenced
this pull request
Oct 22, 2026
A filesystem store reported the block-rounded size an entry occupies on disk from has(), not the length of its content. FastSlowStore takes the slow store's has() result as the exact length of the copy when it fills the fast store, and since TraceMachina#2974 it rejects a fill whose length differs. With the default 4 KiB block size every blob that is not a whole number of blocks failed that fill as DATA_LOSS: Expected 4096 upload bytes, received 206 : Slow store ended before the cache fill was complete whenever it had to come from a filesystem slow tier: any CAS layout with a filesystem store behind a fast tier, the Helm chart's filesystem provider among them, once the fast tier had evicted the blob or been emptied by a restart. The rounding came from TraceMachina#661, which made FileEntryImpl::len() return size_on_disk() so eviction charges a file the whole blocks it takes. EvictingMap::sizes_for_keys reported that same len() as the entry's size, so has() inherited the rounding. LenEntry gains content_len(), the length a read of the entry returns, defaulting to len(). sizes_for_keys reports content_len(), and FileEntryImpl returns its data size from it while len() keeps returning size_on_disk(), so eviction accounting is unchanged. Other entries have no separate on-disk size and behave as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ashishsonamm
added a commit
to ashishsonamm/nativelink
that referenced
this pull request
Oct 22, 2026
A filesystem store reported the block-rounded size an entry occupies on disk from has(), not the length of its content. FastSlowStore takes the slow store's has() result as the exact length of the copy when it fills the fast store, and since TraceMachina#2974 it rejects a fill whose length differs. With the default 4 KiB block size every blob that is not a whole number of blocks failed that fill as DATA_LOSS: Expected 4096 upload bytes, received 206 : Slow store ended before the cache fill was complete whenever it had to come from a filesystem slow tier: any CAS layout with a filesystem store behind a fast tier, the Helm chart's filesystem provider among them, once the fast tier had evicted the blob or been emptied by a restart. The rounding came from TraceMachina#661, which made FileEntryImpl::len() return size_on_disk() so eviction charges a file the whole blocks it takes. EvictingMap::sizes_for_keys reported that same len() as the entry's size, so has() inherited the rounding. LenEntry gains content_len(), the length a read of the entry returns, defaulting to len(). sizes_for_keys reports content_len(), and FileEntryImpl returns its data size from it while len() keeps returning size_on_disk(), so eviction accounting is unchanged. Other entries have no separate on-disk size and behave as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ashishsonamm
added a commit
to ashishsonamm/nativelink
that referenced
this pull request
Oct 22, 2026
A filesystem store reported the block-rounded size an entry occupies on disk from has(), not the length of its content. FastSlowStore takes the slow store's has() result as the exact length of the copy when it fills the fast store, and since TraceMachina#2974 it rejects a fill whose length differs. With the default 4 KiB block size every blob that is not a whole number of blocks failed that fill as DATA_LOSS: Expected 4096 upload bytes, received 206 : Slow store ended before the cache fill was complete whenever it had to come from a filesystem slow tier: any CAS layout with a filesystem store behind a fast tier, the Helm chart's filesystem provider among them, once the fast tier had evicted the blob or been emptied by a restart. The rounding came from TraceMachina#661, which made FileEntryImpl::len() return size_on_disk() so eviction charges a file the whole blocks it takes. EvictingMap::sizes_for_keys reported that same len() as the entry's size, so has() inherited the rounding. LenEntry gains content_len(), the length a read of the entry returns, defaulting to len(). sizes_for_keys reports content_len(), and FileEntryImpl returns its data size from it while len() keeps returning size_on_disk(), so eviction accounting is unchanged. Other entries have no separate on-disk size and behave as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ashishsonamm
added a commit
to ashishsonamm/nativelink
that referenced
this pull request
Oct 22, 2026
A filesystem store reported the block-rounded size an entry occupies on disk from has(), not the length of its content. FastSlowStore takes the slow store's has() result as the exact length of the copy when it fills the fast store, and since TraceMachina#2974 it rejects a fill whose length differs. With the default 4 KiB block size every blob that is not a whole number of blocks failed that fill as DATA_LOSS: Expected 4096 upload bytes, received 206 : Slow store ended before the cache fill was complete whenever it had to come from a filesystem slow tier: any CAS layout with a filesystem store behind a fast tier, the Helm chart's filesystem provider among them, once the fast tier had evicted the blob or been emptied by a restart. The rounding came from TraceMachina#661, which made FileEntryImpl::len() return size_on_disk() so eviction charges a file the whole blocks it takes. EvictingMap::sizes_for_keys reported that same len() as the entry's size, so has() inherited the rounding. LenEntry gains content_len(), the length a read of the entry returns, defaulting to len(). sizes_for_keys reports content_len(), and FileEntryImpl returns its data size from it while len() keeps returning size_on_disk(), so eviction accounting is unchanged. Other entries have no separate on-disk size and behave as before.
b7r6
added a commit
to b7r6/nativelink
that referenced
this pull request
Oct 22, 2026
…stem slow tier Since 086289a (TraceMachina#2974) FastSlowStore's populate checks the bytes it received from the slow tier against the size it declared to the fast tier, and it takes that size from the slow tier's has(). A FilesystemStore reports its block-rounded footprint there (4096 for a 17-byte blob at the default block size), so every read that has to fill the fast tier from a filesystem slow tier fails with DataLoss: Expected 4096 upload bytes, received 17 Slow store ended before the cache fill was complete unless the blob is a block multiple. That is the dedup index store in nativelink-config/examples/filesystem_cas.json5 and the action cache in kubernetes/nativelink/nativelink-config.json5, which shipped broken in v1.8.0. This commit adds only tests, so CI goes red on it; the fix follows. - populate_from_filesystem_slow_tier_at_default_block_size: the shipped shape, with a real FilesystemStore slow tier. - populate_does_not_trust_slow_store_has_size_for_digest: a slow tier whose has() reports less (a compressed stream) or more (a block-rounded footprint) than the blob. - populate_keeps_entry_whose_size_is_not_its_digests: passes today and guards the fix against the obvious simplification of declaring the digest's size, which an AC or dedup index entry does not have. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
A clean EOF could publish an empty or truncated object while the upload declared a nonzero size. Local reproductions confirmed this in ByteStream without a VerifyStore and in Azure's small-upload path; this does not establish the origin of any production corruption.
Validate transfer sizes before committing uploads across Azure, S3 (including R2/OCI), ONTAP, GCS, Redis, MongoDB, memory and filesystem, including whole-file uploads. Apply the same boundary to compression, deduplication, gRPC forwarding and cache fills with a known source size. Preserve Action Cache and encoded-value semantics by checking the declared transfer size rather than the storage key's digest size.
Skip unnecessary uploads of the canonical empty CAS blob. The REAPI specification defines empty blobs as always available without upload; an empty payload under a nonempty digest is invalid. The source comments link this rule too.
Related: #2968. Preventing incomplete writes and evicting corrupt cached copies do not repair an existing corrupt Azure object.
How was this verified?
Before the fix, the new ByteStream and Azure regressions failed: zero/short payloads claiming 78 bytes were accepted. After the fix, tests reject those uploads without publishing them. Additional regressions cover disconnected producers, oversized input, preserved existing values, whole-file uploads, multipart commit/abort behavior, compressed and uncompressed gRPC, and memory/filesystem cache fills.
Ran the selected Cargo suites with
--offline --locked --profile=smol: 437 tests passed across 24 suites, including focused reruns after the final changes. One pre-existing MongoDB scheduler test is explicitly ignored. All 11 focused Bazel suites passed; the server build, Clippy/rustfmt for every changed library and test target, pre-commit checks, and docs anchor/snippet checks also passed. The Azure mock suite is now registered in Bazel so CI exercises it. Cloud requests use mocks, MongoDB uses a disposable local instance, and no production deployment was tested.After merging
mainto resolve the Azure multipart conflict, all 34 Azure tests and 67 Redis tests passed under both Cargo and Bazel. The new regression verifies that short and oversized multipart uploads discard staged blocks without replacing an existing committed copy. The merged store library and both test targets passed Clippy/rustfmt checks, and pre-commit passed.Review follow-up: direct streaming MemoryStore regressions now cover invalid exact and maximum sizes, preservation of existing values, and validation before callbacks in the over-budget drain path. The two failure regressions fail when the size checks are removed. A positive case preserves transfer-size semantics for AC/encoded values, and an Azure regression demonstrates the independent empty-digest and transfer-size constraints for both supported empty digests. All 49 tests across the MemoryStore and Azure Bazel suites passed with Clippy/rustfmt checks.
Risk
Small object uploads must reach EOF before their request starts. S3 now buffers these payloads, bounded by its existing small-upload threshold; this can increase first-byte latency and memory use for concurrent uploads. Large multipart uploads still stream parts and validate before final commit. Failed uploads can leave temporary keys, staged parts or unreferenced deduplication chunks.
No configuration defaults or wire formats change. These size checks do not detect same-length corruption; digest verification remains necessary. Existing durable objects are not deleted or repaired.
AI assistance
Codex drafted the implementation, regression tests, documentation and this description. The changes are provided for human review.
This change is