Repository navigation
Conversation
…stem slow tier Since 086289a (TraceMachina#2974) FastSlowStore's populate checks the bytes it received from the slow tier against the size it declared to the fast tier, and it takes that size from the slow tier's has(). A FilesystemStore reports its block-rounded footprint there (4096 for a 17-byte blob at the default block size), so every read that has to fill the fast tier from a filesystem slow tier fails with DataLoss: Expected 4096 upload bytes, received 17 Slow store ended before the cache fill was complete unless the blob is a block multiple. That is the dedup index store in nativelink-config/examples/filesystem_cas.json5 and the action cache in kubernetes/nativelink/nativelink-config.json5, which shipped broken in v1.8.0. This commit adds only tests, so CI goes red on it; the fix follows. - populate_from_filesystem_slow_tier_at_default_block_size: the shipped shape, with a real FilesystemStore slow tier. - populate_does_not_trust_slow_store_has_size_for_digest: a slow tier whose has() reports less (a compressed stream) or more (a block-rounded footprint) than the blob. - populate_keeps_entry_whose_size_is_not_its_digests: passes today and guards the fix against the obvious simplification of declaring the digest's size, which an AC or dedup index entry does not have. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…g the fast tier FastSlowStore's populate took the size it declares to the fast tier, as ExactSize, from the slow tier's has(). Since 086289a (TraceMachina#2974) the populate also checks the bytes it received against that size, so any slow tier whose has() is not the blob's byte count fails every fill: FilesystemStore reports its block-rounded footprint (4096 for a 17-byte blob at the default block size), dedup sums those footprints, and GrpcStore in AC mode reports u64::MAX. Memory over filesystem, as in the filesystem_cas.json5 dedup index store and the Kubernetes example's action cache, fails every read that misses the fast tier. Declaring ExactSize(digest.size_bytes()) for every digest key would fix a CAS but break the stores where a digest's size is not the size of what it names: an action cache keys an ActionResult by the action's digest, and a dedup index store keys a blob's index by the blob's digest. A filesystem fast tier would then refuse every such entry. So for a digest key the populate now checks one answer against the other. When has() agrees with the digest's size, that size is declared exact. When they disagree, neither is trusted and the populate declares MaxSize of the larger, so the fast tier accepts the bytes it actually gets and the received-vs-declared check (which only applies to ExactSize) cannot misfire. A string key, which has no digest, keeps has()'s answer but as a MaxSize for the same reason. This is the follow-up TraceMachina#2953 reverted TraceMachina#2881 to wait for; TraceMachina#2974 relanded the check without it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This was referenced Oct 22, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
Stacked on #71 (tests only, red on main); this one takes it green. Supersedes #72, whose head branch was lost in a rebase onto the new fork main.
FastSlowStore's populate took the size it declares to the fast tier, asExactSize, from the slow tier'shas(). Since upstream TraceMachina#2974 the populate also checks the bytes it received against that size, so any slow tier whosehas()is not the blob's byte count fails every fill:FilesystemStorereports its block-rounded footprint, dedup sums those footprints,GrpcStorein AC mode reportsu64::MAX. Memory over filesystem (thefilesystem_cas.json5dedup index store, the Kubernetes example's action cache) fails every read that misses the fast tier. This shipped in v1.8.0.Declaring
ExactSize(digest.size_bytes())for every digest key would fix a CAS but break the stores where a digest's size is not the size of what it names: an action cache keys anActionResultby the action's digest, a dedup index store keys a blob's index by the blob's digest. A filesystem fast tier would then refuse every such entry.So for a digest key the populate now checks one answer against the other. When
has()agrees with the digest's size, that size is declared exact. When they disagree, neither is trusted and the populate declaresMaxSizeof the larger, so the fast tier accepts the bytes it actually gets and the received-vs-declared check (which only applies toExactSize) cannot misfire. A string key, which has no digest, keepshas()'s answer but as aMaxSizefor the same reason.This is the follow-up TraceMachina#2953 reverted TraceMachina#2881 to wait for; TraceMachina#2974 relanded the check without it.
How was this verified?
bazel testonfast_slow_store_test,filesystem_store_testanddedup_store_test: all pass, including the three tests from the base PR (38/38 infast_slow_store_test).Risk
Only the size hint the populate passes to the fast tier changes, and only when the slow tier's
has()and the digest disagree. A fast tier that trustedExactSizeto pre-size its write now getsMaxSizein that case.MaxSizeentries skip the received-vs-declared check, which is the point: that check assumes a size the store cannot know here.AI assistance
Written with Claude Code from b7r6's 10-15 branch
fix/compression-has-size, rebased onto v1.8.0+1 and split from its compression-store half (superseded for digest keys by TraceMachina#2960). b7r6 reviewed the change.🤖 Generated with Claude Code