Repository navigation
Report the digest size from the compression store has, not the compressed one - #2960
Merged
Merged
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
palfrey
previously approved these changes
Oct 19, 2026
corcillo
reviewed
Oct 19, 2026
corcillo
previously approved these changes
Oct 19, 2026
modernmedici
approved these changes
Oct 20, 2026
This was referenced Oct 20, 2026
b7r6
added a commit
to b7r6/nativelink
that referenced
this pull request
Oct 21, 2026
…ssionStore::has() a1e6efb (TraceMachina#2960) made CompressionStore::has() report a digest key's declared size instead of the compressed stream's length, and left string keys reporting the stored length as a follow-up. This is that follow-up. A FastSlowStore declares the slow tier's has() size to its fast tier, and since 086289a (TraceMachina#2974) the populate itself rejects a fill whose length differs from that size. A string key read through a compression slow tier therefore still fails every populate on main: Expected 514 upload bytes, received 88064 Slow store ended before the cache fill was complete A string key has no size of its own, so the size comes from the footer. Its last three fields (uncompressed_data_sz u64, block_size u32, version u8) have fixed widths and always end the stream, so one ranged get_part of 13 bytes at the inner store's reported length finds it without reading the blocks; the version and block size are checked. If the object went away since the inner has(), the key reports missing. If the footer is not at the reported length, the inner store reported something other than the stream's exact length (FilesystemStore reports its block-rounded footprint), and locating the footer would mean reading the whole stream, so has() logs a warning and reports the stored length as before; reporting None instead would call a present object missing. String keys rarely reach a compression store, since CAS traffic is digest keyed, so the extra read stays off the FindMissingBlobs path. Tests: - has_reports_uncompressed_size_of_string_key: the footer path, plus an empty object (0) and a missing key. - populate_from_compression_slow_store_into_filesystem_fast_store: end to end for a digest key and a string key. On main the string key fails as above. - has_reports_uncompressed_size_of_digest: the digest path TraceMachina#2960 fixed, over a two-block payload and has_many with a zero and a missing digest. - has_preserves_missing_digests_and_stored_string_key_sizes (from TraceMachina#2960) pinned the stored length for a string key; it now expects the blob's length and is renamed to match. - has_mixed_digests_preserves_result_order and has_nonzero_digests_delegates_single_batch used string keys against a recording inner store that reports sizes it cannot serve. They now use digest keys. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 22, 2026
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
CompressionStore reported the backend's framed, compressed length from
has_with_results. FastSlow uses that answer as the transfer size when filling its fast tier, so decompressed reads could fail the filesystem's size check. Present digest keys now report the size declared by the digest; missing keys stay missing and string keys keep the backend's stored size.The merge with current
mainpreserves #2974's upload-size validation and its incomplete-compression regression. Both tests that conflicted at the end ofcompression_store_test.rsare retained.How was this verified?
The original regression checks that a compressible 4 KiB blob reports 4096 bytes through CompressionStore while its backend reports the smaller framed size.
A new integration regression uses a real FastSlow store with a temporary filesystem fast tier and a compressed memory-backed slow tier. It exercises both a compressible blob and a tiny blob whose framing makes it larger, verifies the returned contents, and confirms that the filesystem cache is populated with the uncompressed length. With digest-size reporting temporarily disabled, this test failed with
Expected 79 upload bytes, received 4096; restoring it made the test pass.A mixed-key regression verifies that missing digests remain absent, canonical empty digests remain available, present digests report logical length, and string keys retain the stored size. The retained incomplete-upload regression confirms that empty and short payloads declaring 78 bytes are rejected without publishing a compressed blob.
On macOS, all 108 tests passed across the compression (17), FastSlow (35), and filesystem (56) Bazel suites, with Clippy/rustfmt checks for the affected library and tests. Pre-commit and documentation anchor, snippet, navigation, and corpus checks passed. The original author reported deployment verification under Redis failover and after a CAS restart; the merge verification used local stores and did not test a deployment.
Risk
The reported size changes only for present digest keys. It comes from the digest rather than by decoding the stored object, so this does not verify its contents or repair existing corruption. String keys still report the compressed size and can still need separate handling when used through FastSlow. Compressed frames, upload validation, configuration defaults, and wire formats are unchanged.
AI assistance
The original author disclosed that an agent drafted the change and that they reviewed every line. Codex resolved the merge with main, added the cache-fill and mixed-key regressions, updated documentation, and ran the local verification described above.