Skip to content

fix(fast_slow_store): don't declare a disputed size exact when filling the fast tier - #72

Closed
b7r6 wants to merge 3 commits into
mainfrom
fix/fastslow-populate-block-rounded
Closed

b7r6 wants to merge 3 commits into
mainfrom
fix/fastslow-populate-block-rounded

Conversation

@b7r6

@b7r6 b7r6 commented Oct 22, 2026

Copy link
Copy Markdown
Owner

What and why

Stacked on the test-only PR (base of this PR), which goes red on main; this one takes it green.

FastSlowStore's populate took the size it declares to the fast tier, as ExactSize, from the slow tier's has(). Since upstream TraceMachina#2974 the populate also checks the bytes it received against that size, so any slow tier whose has() is not the blob's byte count fails every fill: FilesystemStore reports its block-rounded footprint, dedup sums those footprints, GrpcStore in AC mode reports u64::MAX. Memory over filesystem (the filesystem_cas.json5 dedup index store, the Kubernetes example's action cache) fails every read that misses the fast tier. This shipped in v1.8.0.

Declaring ExactSize(digest.size_bytes()) for every digest key would fix a CAS but break the stores where a digest's size is not the size of what it names: an action cache keys an ActionResult by the action's digest, a dedup index store keys a blob's index by the blob's digest. A filesystem fast tier would then refuse every such entry.

So for a digest key the populate now checks one answer against the other. When has() agrees with the digest's size, that size is declared exact. When they disagree, neither is trusted and the populate declares MaxSize of the larger, so the fast tier accepts the bytes it actually gets and the received-vs-declared check (which only applies to ExactSize) cannot misfire. A string key, which has no digest, keeps has()'s answer but as a MaxSize for the same reason.

This is the follow-up TraceMachina#2953 reverted TraceMachina#2881 to wait for; TraceMachina#2974 relanded the check without it.

How was this verified?

bazel test on fast_slow_store_test, filesystem_store_test and dedup_store_test: all pass, including the three tests from the base PR (38/38 in fast_slow_store_test).

Risk

Only the size hint the populate passes to the fast tier changes, and only when the slow tier's has() and the digest disagree. A fast tier that trusted ExactSize to pre-size its write now gets MaxSize in that case. MaxSize entries skip the received-vs-declared check, which is the point: that check assumes a size the store cannot know here.

AI assistance

Written with Claude Code from b7r6's 10-15 branch fix/compression-has-size, rebased onto v1.8.0+1 and split from its compression-store half (superseded for digest keys by TraceMachina#2960). b7r6 reviewed the change.

🤖 Generated with Claude Code

b7r6 and others added 3 commits October 21, 2026 17:14
Completes the no-per-PR-OCI-builds intent: image-off-pr moved image.yaml off
PRs and TraceMachina#2877 moved the nix integration job off PRs, but main.yaml's
integration-tests (docker-compose image build + run_integration_tests.sh)
fell between them and still ran on every PR. Defer to nightly + merge-to-main,
matching TraceMachina#2880's pattern. Fork-local until folded into the upstream OCI/nightly
set.
…stem slow tier

Since 086289a (TraceMachina#2974) FastSlowStore's populate checks the bytes it
received from the slow tier against the size it declared to the fast
tier, and it takes that size from the slow tier's has(). A
FilesystemStore reports its block-rounded footprint there (4096 for a
17-byte blob at the default block size), so every read that has to fill
the fast tier from a filesystem slow tier fails with

  DataLoss: Expected 4096 upload bytes, received 17
  Slow store ended before the cache fill was complete

unless the blob is a block multiple. That is the dedup index store in
nativelink-config/examples/filesystem_cas.json5 and the action cache in
kubernetes/nativelink/nativelink-config.json5, which shipped broken in
v1.8.0.

This commit adds only tests, so CI goes red on it; the fix follows.

- populate_from_filesystem_slow_tier_at_default_block_size: the shipped
  shape, with a real FilesystemStore slow tier.
- populate_does_not_trust_slow_store_has_size_for_digest: a slow tier
  whose has() reports less (a compressed stream) or more (a block-rounded
  footprint) than the blob.
- populate_keeps_entry_whose_size_is_not_its_digests: passes today and
  guards the fix against the obvious simplification of declaring the
  digest's size, which an AC or dedup index entry does not have.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…g the fast tier

FastSlowStore's populate took the size it declares to the fast tier, as
ExactSize, from the slow tier's has(). Since 086289a (TraceMachina#2974) the
populate also checks the bytes it received against that size, so any
slow tier whose has() is not the blob's byte count fails every fill:
FilesystemStore reports its block-rounded footprint (4096 for a 17-byte
blob at the default block size), dedup sums those footprints, and
GrpcStore in AC mode reports u64::MAX. Memory over filesystem, as in
the filesystem_cas.json5 dedup index store and the Kubernetes example's
action cache, fails every read that misses the fast tier.

Declaring ExactSize(digest.size_bytes()) for every digest key would fix
a CAS but break the stores where a digest's size is not the size of what
it names: an action cache keys an ActionResult by the action's digest,
and a dedup index store keys a blob's index by the blob's digest. A
filesystem fast tier would then refuse every such entry.

So for a digest key the populate now checks one answer against the
other. When has() agrees with the digest's size, that size is declared
exact. When they disagree, neither is trusted and the populate declares
MaxSize of the larger, so the fast tier accepts the bytes it actually
gets and the received-vs-declared check (which only applies to
ExactSize) cannot misfire. A string key, which has no digest, keeps
has()'s answer but as a MaxSize for the same reason.

This is the follow-up TraceMachina#2953 reverted TraceMachina#2881 to wait for; TraceMachina#2974 relanded
the check without it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@b7r6
b7r6 changed the base branch from repro/fastslow-populate-block-rounded to main October 22, 2026 11:36
@b7r6 b7r6 closed this Oct 22, 2026
@b7r6 b7r6 reopened this Oct 22, 2026
@b7r6 b7r6 closed this Oct 22, 2026
@b7r6
b7r6 deleted the fix/fastslow-populate-block-rounded branch October 22, 2026 13:33
@b7r6

b7r6 commented Oct 22, 2026

Copy link
Copy Markdown
Owner Author

Superseded by the PR above after the fork's main moved to upstream HEAD; same commit, rebased.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant