Skip to content

Reject incomplete CAS uploads before publishing blobs - #2974

Merged
MarcusSorealheis merged 7 commits into
TraceMachina:mainfrom
MarcusSorealheis:bugfix/reject-incomplete-cas-uploads
Oct 20, 2026
Merged

MarcusSorealheis merged 7 commits into
TraceMachina:mainfrom
MarcusSorealheis:bugfix/reject-incomplete-cas-uploads

Conversation

@MarcusSorealheis

@MarcusSorealheis MarcusSorealheis commented Oct 20, 2026 •

Copy link
Copy Markdown
Member

What and why

A clean EOF could publish an empty or truncated object while the upload declared a nonzero size. Local reproductions confirmed this in ByteStream without a VerifyStore and in Azure's small-upload path; this does not establish the origin of any production corruption.

Validate transfer sizes before committing uploads across Azure, S3 (including R2/OCI), ONTAP, GCS, Redis, MongoDB, memory and filesystem, including whole-file uploads. Apply the same boundary to compression, deduplication, gRPC forwarding and cache fills with a known source size. Preserve Action Cache and encoded-value semantics by checking the declared transfer size rather than the storage key's digest size.

Skip unnecessary uploads of the canonical empty CAS blob. The REAPI specification defines empty blobs as always available without upload; an empty payload under a nonempty digest is invalid. The source comments link this rule too.

Related: #2968. Preventing incomplete writes and evicting corrupt cached copies do not repair an existing corrupt Azure object.

How was this verified?

Before the fix, the new ByteStream and Azure regressions failed: zero/short payloads claiming 78 bytes were accepted. After the fix, tests reject those uploads without publishing them. Additional regressions cover disconnected producers, oversized input, preserved existing values, whole-file uploads, multipart commit/abort behavior, compressed and uncompressed gRPC, and memory/filesystem cache fills.

Ran the selected Cargo suites with --offline --locked --profile=smol: 437 tests passed across 24 suites, including focused reruns after the final changes. One pre-existing MongoDB scheduler test is explicitly ignored. All 11 focused Bazel suites passed; the server build, Clippy/rustfmt for every changed library and test target, pre-commit checks, and docs anchor/snippet checks also passed. The Azure mock suite is now registered in Bazel so CI exercises it. Cloud requests use mocks, MongoDB uses a disposable local instance, and no production deployment was tested.

After merging main to resolve the Azure multipart conflict, all 34 Azure tests and 67 Redis tests passed under both Cargo and Bazel. The new regression verifies that short and oversized multipart uploads discard staged blocks without replacing an existing committed copy. The merged store library and both test targets passed Clippy/rustfmt checks, and pre-commit passed.

Review follow-up: direct streaming MemoryStore regressions now cover invalid exact and maximum sizes, preservation of existing values, and validation before callbacks in the over-budget drain path. The two failure regressions fail when the size checks are removed. A positive case preserves transfer-size semantics for AC/encoded values, and an Azure regression demonstrates the independent empty-digest and transfer-size constraints for both supported empty digests. All 49 tests across the MemoryStore and Azure Bazel suites passed with Clippy/rustfmt checks.

Risk

Small object uploads must reach EOF before their request starts. S3 now buffers these payloads, bounded by its existing small-upload threshold; this can increase first-byte latency and memory use for concurrent uploads. Large multipart uploads still stream parts and validate before final commit. Failed uploads can leave temporary keys, staged parts or unreferenced deduplication chunks.

No configuration defaults or wire formats change. These size checks do not detect same-length corruption; digest verification remains necessary. Existing durable objects are not deleted or repaired.

AI assistance

Codex drafted the implementation, regression tests, documentation and this description. The changes are provided for human review.


This change is Reviewable

@vercel

vercel Bot commented Oct 20, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
nativelink Ready Ready Preview Oct 20, 2026 10:13am UTC
nativelink-aidm Ready Ready Preview Oct 20, 2026 10:13am UTC

Request Review

Comment thread nativelink-store/src/memory_store.rs Resolved
@MarcusSorealheis
MarcusSorealheis merged commit 086289a into TraceMachina:main Oct 20, 2026
31 checks passed
b7r6 added a commit to b7r6/nativelink that referenced this pull request Oct 21, 2026
…ssionStore::has()

a1e6efb (TraceMachina#2960) made CompressionStore::has() report a digest key's
declared size instead of the compressed stream's length, and left string
keys reporting the stored length as a follow-up. This is that follow-up.

A FastSlowStore declares the slow tier's has() size to its fast tier, and
since 086289a (TraceMachina#2974) the populate itself rejects a fill whose length
differs from that size. A string key read through a compression slow tier
therefore still fails every populate on main:

  Expected 514 upload bytes, received 88064
  Slow store ended before the cache fill was complete

A string key has no size of its own, so the size comes from the footer.
Its last three fields (uncompressed_data_sz u64, block_size u32, version
u8) have fixed widths and always end the stream, so one ranged get_part of
13 bytes at the inner store's reported length finds it without reading the
blocks; the version and block size are checked. If the object went away
since the inner has(), the key reports missing. If the footer is not at the
reported length, the inner store reported something other than the
stream's exact length (FilesystemStore reports its block-rounded
footprint), and locating the footer would mean reading the whole stream,
so has() logs a warning and reports the stored length as before; reporting
None instead would call a present object missing. String keys rarely reach
a compression store, since CAS traffic is digest keyed, so the extra read
stays off the FindMissingBlobs path.

Tests:

- has_reports_uncompressed_size_of_string_key: the footer path, plus an
  empty object (0) and a missing key.
- populate_from_compression_slow_store_into_filesystem_fast_store: end to
  end for a digest key and a string key. On main the string key fails as
  above.
- has_reports_uncompressed_size_of_digest: the digest path TraceMachina#2960 fixed, over
  a two-block payload and has_many with a zero and a missing digest.
- has_preserves_missing_digests_and_stored_string_key_sizes (from TraceMachina#2960)
  pinned the stored length for a string key; it now expects the blob's
  length and is renamed to match.
- has_mixed_digests_preserves_result_order and
  has_nonzero_digests_delegates_single_batch used string keys against a
  recording inner store that reports sizes it cannot serve. They now use
  digest keys.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
b7r6 added a commit to b7r6/nativelink that referenced this pull request Oct 21, 2026
…stSlowStore's fast tier

On main the refusal comes from 086289a (TraceMachina#2974), not TraceMachina#2881, which was
reverted (bd8c8d9): the populate itself now checks the bytes it
received against this size and fails with DataLoss. A fast_slow store
whose slow tier is a FilesystemStore at the default 4 KiB block size
fails every populate of a blob that is not a block multiple:

  Expected 4096 upload bytes, received 17
  Slow store ended before the cache fill was complete

That includes the dedup index_store (memory over filesystem) in
nativelink-config/examples/filesystem_cas.json5. The rest of this message
predates the rebase.

FastSlowStore::populate_and_maybe_stream took the size it declares to the
fast tier, as ExactSize, from the slow tier's has(). Since 264b7eb
(TraceMachina#2881) a filesystem fast tier refuses a fill whose length differs from
it, so any slow tier that reports a physical size fails every populate.
The compression store was one (fixed in the previous commit). Another is
FilesystemStore, whose has() reports its block-rounded footprint (4096
bytes for a 17-byte blob at the default block size), so a fast_slow store
with filesystem tiers on both sides, as in
deployment-examples/docker-compose/test-multi-worker-simple.json5, fails
the same way. So do dedup over a filesystem content store, which sums
those footprints, and GrpcStore in AC mode, which reports u64::MAX.

Declaring ExactSize(digest.size_bytes()) for every digest key would fix
those for a CAS, but a digest's size is not always the size of what it
names. An action cache keys an ActionResult by the action's digest, a
dedup index store keys a blob's index by the blob's digest, and the
chunking layout store does the same. A fast_slow AC with a filesystem
fast tier over Redis, Mongo or memory works today because has() reports
the entry's size; declaring the digest's size would make the filesystem
tier refuse every such entry, and the memory-over-filesystem dedup index
store in filesystem_cas.json5 would skip caching the index of every blob
larger than the memory tier's max_bytes.

So for a digest key the populate now checks one answer against the
other. When has() agrees with the digest's size, that size is declared
exact, as before. When they disagree, this store cannot tell which is
wrong, so the larger is declared as MaxSize. A filesystem or memory fast
tier accepts any length under MaxSize, and a compression fast tier gets a
bound that covers the blob as long as either answer does. A missing blob
still fails the has() check with the same NotFound and MissingDigest
context. The LazyExistenceOnSync branch, which skips has(), keeps
MaxSize(u64::MAX); AC_MAIN_STORE in gcs_backend.json5 is a filesystem
tier over GCS, so the digest's size cannot go there either. String keys
keep ExactSize(has()), as there is nothing to check it against.

The cost is that when the two disagree, the fast tier loses the ExactSize
guard, and MemoryStore's skip of writes over max_bytes, which only trusts
ExactSize, does not apply. For a CAS that happens only behind a slow tier
that misreports, which after the previous commit leaves FilesystemStore's
rounding; once its has() reports the stored length, it stops.

update_with_whole_file and the other paths take their sizes from the
caller's UploadSizeInfo, the file's metadata or update()'s return value;
populate was the only place that derived an ExactSize from has().

Tests:

- populate_does_not_trust_slow_store_has_size_for_digest: a slow store
  whose has() reports 1 or 4096 for a 3000-byte blob. The read returns
  the blob and the filesystem fast tier holds all 3000 bytes. It fails on
  7149987, and with only the previous commit, with "refusing to store
  a partial blob".
- populate_keeps_entry_whose_size_is_not_its_digests: an AC-shaped entry,
  3000 bytes under a digest that says 100, behind an exact memory slow
  tier. It passes on 7149987 and fails if the populate declares
  ExactSize(digest.size_bytes()); it guards against that simplification.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ashishsonamm added a commit to ashishsonamm/nativelink that referenced this pull request Oct 22, 2026
A filesystem store reported the block-rounded size an entry occupies
on disk from has(), not the length of its content. FastSlowStore takes
the slow store's has() result as the exact length of the copy when it
fills the fast store, and since TraceMachina#2974 it rejects a fill whose length
differs. With the default 4 KiB block size every blob that is not a
whole number of blocks failed that fill as

  DATA_LOSS: Expected 4096 upload bytes, received 206 : Slow store
  ended before the cache fill was complete

whenever it had to come from a filesystem slow tier: any CAS layout
with a filesystem store behind a fast tier, the Helm chart's
filesystem provider among them, once the fast tier had evicted the
blob or been emptied by a restart.

The rounding came from TraceMachina#661, which made FileEntryImpl::len() return
size_on_disk() so eviction charges a file the whole blocks it takes.
EvictingMap::sizes_for_keys reported that same len() as the entry's
size, so has() inherited the rounding.

LenEntry gains content_len(), the length a read of the entry returns,
defaulting to len(). sizes_for_keys reports content_len(), and
FileEntryImpl returns its data size from it while len() keeps
returning size_on_disk(), so eviction accounting is unchanged. Other
entries have no separate on-disk size and behave as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ashishsonamm added a commit to ashishsonamm/nativelink that referenced this pull request Oct 22, 2026
A filesystem store reported the block-rounded size an entry occupies
on disk from has(), not the length of its content. FastSlowStore takes
the slow store's has() result as the exact length of the copy when it
fills the fast store, and since TraceMachina#2974 it rejects a fill whose length
differs. With the default 4 KiB block size every blob that is not a
whole number of blocks failed that fill as

  DATA_LOSS: Expected 4096 upload bytes, received 206 : Slow store
  ended before the cache fill was complete

whenever it had to come from a filesystem slow tier: any CAS layout
with a filesystem store behind a fast tier, the Helm chart's
filesystem provider among them, once the fast tier had evicted the
blob or been emptied by a restart.

The rounding came from TraceMachina#661, which made FileEntryImpl::len() return
size_on_disk() so eviction charges a file the whole blocks it takes.
EvictingMap::sizes_for_keys reported that same len() as the entry's
size, so has() inherited the rounding.

LenEntry gains content_len(), the length a read of the entry returns,
defaulting to len(). sizes_for_keys reports content_len(), and
FileEntryImpl returns its data size from it while len() keeps
returning size_on_disk(), so eviction accounting is unchanged. Other
entries have no separate on-disk size and behave as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ashishsonamm added a commit to ashishsonamm/nativelink that referenced this pull request Oct 22, 2026
A filesystem store reported the block-rounded size an entry occupies
on disk from has(), not the length of its content. FastSlowStore takes
the slow store's has() result as the exact length of the copy when it
fills the fast store, and since TraceMachina#2974 it rejects a fill whose length
differs. With the default 4 KiB block size every blob that is not a
whole number of blocks failed that fill as

  DATA_LOSS: Expected 4096 upload bytes, received 206 : Slow store
  ended before the cache fill was complete

whenever it had to come from a filesystem slow tier: any CAS layout
with a filesystem store behind a fast tier, the Helm chart's
filesystem provider among them, once the fast tier had evicted the
blob or been emptied by a restart.

The rounding came from TraceMachina#661, which made FileEntryImpl::len() return
size_on_disk() so eviction charges a file the whole blocks it takes.
EvictingMap::sizes_for_keys reported that same len() as the entry's
size, so has() inherited the rounding.

LenEntry gains content_len(), the length a read of the entry returns,
defaulting to len(). sizes_for_keys reports content_len(), and
FileEntryImpl returns its data size from it while len() keeps
returning size_on_disk(), so eviction accounting is unchanged. Other
entries have no separate on-disk size and behave as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ashishsonamm added a commit to ashishsonamm/nativelink that referenced this pull request Oct 22, 2026
A filesystem store reported the block-rounded size an entry occupies
on disk from has(), not the length of its content. FastSlowStore takes
the slow store's has() result as the exact length of the copy when it
fills the fast store, and since TraceMachina#2974 it rejects a fill whose length
differs. With the default 4 KiB block size every blob that is not a
whole number of blocks failed that fill as

  DATA_LOSS: Expected 4096 upload bytes, received 206 : Slow store
  ended before the cache fill was complete

whenever it had to come from a filesystem slow tier: any CAS layout
with a filesystem store behind a fast tier, the Helm chart's
filesystem provider among them, once the fast tier had evicted the
blob or been emptied by a restart.

The rounding came from TraceMachina#661, which made FileEntryImpl::len() return
size_on_disk() so eviction charges a file the whole blocks it takes.
EvictingMap::sizes_for_keys reported that same len() as the entry's
size, so has() inherited the rounding.

LenEntry gains content_len(), the length a read of the entry returns,
defaulting to len(). sizes_for_keys reports content_len(), and
FileEntryImpl returns its data size from it while len() keeps
returning size_on_disk(), so eviction accounting is unchanged. Other
entries have no separate on-disk size and behave as before.
b7r6 added a commit to b7r6/nativelink that referenced this pull request Oct 22, 2026
…stem slow tier

Since 086289a (TraceMachina#2974) FastSlowStore's populate checks the bytes it
received from the slow tier against the size it declared to the fast
tier, and it takes that size from the slow tier's has(). A
FilesystemStore reports its block-rounded footprint there (4096 for a
17-byte blob at the default block size), so every read that has to fill
the fast tier from a filesystem slow tier fails with

  DataLoss: Expected 4096 upload bytes, received 17
  Slow store ended before the cache fill was complete

unless the blob is a block multiple. That is the dedup index store in
nativelink-config/examples/filesystem_cas.json5 and the action cache in
kubernetes/nativelink/nativelink-config.json5, which shipped broken in
v1.8.0.

This commit adds only tests, so CI goes red on it; the fix follows.

- populate_from_filesystem_slow_tier_at_default_block_size: the shipped
  shape, with a real FilesystemStore slow tier.
- populate_does_not_trust_slow_store_has_size_for_digest: a slow tier
  whose has() reports less (a compressed stream) or more (a block-rounded
  footprint) than the blob.
- populate_keeps_entry_whose_size_is_not_its_digests: passes today and
  guards the fix against the obvious simplification of declaring the
  digest's size, which an AC or dedup index entry does not have.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

This branch was successfully deployed

2 active deployments
Preview – nativelink — e4948681 Deployed Oct 20, 2026 by vercel[bot]
Preview – nativelink-aidm — e4948681 Deployed Oct 20, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants