Skip to content

Report the digest size from the compression store has, not the compressed one - #2960

Merged
amankrx merged 7 commits into
TraceMachina:mainfrom
amankrx:fix/compression-has-size
Oct 20, 2026
Merged

amankrx merged 7 commits into
TraceMachina:mainfrom
amankrx:fix/compression-has-size

Conversation

@amankrx

@amankrx amankrx commented Oct 19, 2026 •

Copy link
Copy Markdown
Member

What and why

CompressionStore reported the backend's framed, compressed length from has_with_results. FastSlow uses that answer as the transfer size when filling its fast tier, so decompressed reads could fail the filesystem's size check. Present digest keys now report the size declared by the digest; missing keys stay missing and string keys keep the backend's stored size.

The merge with current main preserves #2974's upload-size validation and its incomplete-compression regression. Both tests that conflicted at the end of compression_store_test.rs are retained.

How was this verified?

The original regression checks that a compressible 4 KiB blob reports 4096 bytes through CompressionStore while its backend reports the smaller framed size.

A new integration regression uses a real FastSlow store with a temporary filesystem fast tier and a compressed memory-backed slow tier. It exercises both a compressible blob and a tiny blob whose framing makes it larger, verifies the returned contents, and confirms that the filesystem cache is populated with the uncompressed length. With digest-size reporting temporarily disabled, this test failed with Expected 79 upload bytes, received 4096; restoring it made the test pass.

A mixed-key regression verifies that missing digests remain absent, canonical empty digests remain available, present digests report logical length, and string keys retain the stored size. The retained incomplete-upload regression confirms that empty and short payloads declaring 78 bytes are rejected without publishing a compressed blob.

On macOS, all 108 tests passed across the compression (17), FastSlow (35), and filesystem (56) Bazel suites, with Clippy/rustfmt checks for the affected library and tests. Pre-commit and documentation anchor, snippet, navigation, and corpus checks passed. The original author reported deployment verification under Redis failover and after a CAS restart; the merge verification used local stores and did not test a deployment.

Risk

The reported size changes only for present digest keys. It comes from the digest rather than by decoding the stored object, so this does not verify its contents or repair existing corruption. String keys still report the compressed size and can still need separate handling when used through FastSlow. Compressed frames, upload validation, configuration defaults, and wire formats are unchanged.

AI assistance

The original author disclosed that an agent drafted the change and that they reviewed every line. Codex resolved the merge with main, added the cache-fill and mixed-key regressions, updated documentation, and ran the local verification described above.

@vercel

vercel Bot commented Oct 19, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
nativelink Ready Ready Preview Oct 20, 2026 10:47am UTC
nativelink-aidm Ready Ready Preview Oct 20, 2026 10:47am UTC

Request Review

palfrey
palfrey previously approved these changes Oct 19, 2026
corcillo
corcillo previously approved these changes Oct 19, 2026

@MarcusSorealheis MarcusSorealheis left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@amankrx
amankrx merged commit a1e6efb into TraceMachina:main Oct 20, 2026
33 checks passed
b7r6 added a commit to b7r6/nativelink that referenced this pull request Oct 21, 2026
…ssionStore::has()

a1e6efb (TraceMachina#2960) made CompressionStore::has() report a digest key's
declared size instead of the compressed stream's length, and left string
keys reporting the stored length as a follow-up. This is that follow-up.

A FastSlowStore declares the slow tier's has() size to its fast tier, and
since 086289a (TraceMachina#2974) the populate itself rejects a fill whose length
differs from that size. A string key read through a compression slow tier
therefore still fails every populate on main:

  Expected 514 upload bytes, received 88064
  Slow store ended before the cache fill was complete

A string key has no size of its own, so the size comes from the footer.
Its last three fields (uncompressed_data_sz u64, block_size u32, version
u8) have fixed widths and always end the stream, so one ranged get_part of
13 bytes at the inner store's reported length finds it without reading the
blocks; the version and block size are checked. If the object went away
since the inner has(), the key reports missing. If the footer is not at the
reported length, the inner store reported something other than the
stream's exact length (FilesystemStore reports its block-rounded
footprint), and locating the footer would mean reading the whole stream,
so has() logs a warning and reports the stored length as before; reporting
None instead would call a present object missing. String keys rarely reach
a compression store, since CAS traffic is digest keyed, so the extra read
stays off the FindMissingBlobs path.

Tests:

- has_reports_uncompressed_size_of_string_key: the footer path, plus an
  empty object (0) and a missing key.
- populate_from_compression_slow_store_into_filesystem_fast_store: end to
  end for a digest key and a string key. On main the string key fails as
  above.
- has_reports_uncompressed_size_of_digest: the digest path TraceMachina#2960 fixed, over
  a two-block payload and has_many with a zero and a missing digest.
- has_preserves_missing_digests_and_stored_string_key_sizes (from TraceMachina#2960)
  pinned the stored length for a string key; it now expects the blob's
  length and is renamed to match.
- has_mixed_digests_preserves_result_order and
  has_nonzero_digests_delegates_single_batch used string keys against a
  recording inner store that reports sizes it cannot serve. They now use
  digest keys.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch was successfully deployed

2 active deployments
Preview – nativelink — ab99ab4e Deployed Oct 20, 2026 by vercel[bot]
Preview – nativelink-aidm — ab99ab4e Deployed Oct 20, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants