Skip to content

test(fast_slow_store): show the v1.8.0 populate failure from a filesystem slow tier (CI red on purpose) - #71

Open
b7r6 wants to merge 1 commit into
mainfrom
repro/fastslow-populate-block-rounded
Open

b7r6 wants to merge 1 commit into
mainfrom
repro/fastslow-populate-block-rounded

Conversation

@b7r6

@b7r6 b7r6 commented Oct 22, 2026 •

Copy link
Copy Markdown
Owner

What and why

This PR is tests only and is meant to go red. The fix is stacked on it: see the PR based on this branch.

Since upstream TraceMachina#2974 (086289a) FastSlowStore's populate checks the bytes it received from the slow tier against the size it declared to the fast tier, and it takes that size from the slow tier's has(). A FilesystemStore reports its block-rounded footprint there (4096 for a 17-byte blob at the default block size), so every read that has to fill the fast tier from a filesystem slow tier fails with

DataLoss: Expected 4096 upload bytes, received 17
Slow store ended before the cache fill was complete

unless the blob is a block multiple. That is the dedup index store in nativelink-config/examples/filesystem_cas.json5 and the action cache in kubernetes/nativelink/nativelink-config.json5, and it shipped in v1.8.0.

History: TraceMachina#2881 added the exact-size refusal on 10-15, TraceMachina#2953 reverted it the next day because "it needs a follow-up change before it can go into a release", TraceMachina#2974 relanded the check on 10-20, and TraceMachina#2960 covered the compression-store half. The FastSlow half (the follow-up) never landed.

Tests

  • populate_from_filesystem_slow_tier_at_default_block_size: the shipped shape, a real FilesystemStore slow tier at the default block size behind a memory fast tier. Failed on v1.8.0; upstream Report content length from filesystem store has() TraceMachina/nativelink#3006 (filesystem has() reports the content length) fixes this shape and is in the fork's main now, so this one passes here. Kept as the regression test for the shipped bug.
  • populate_does_not_trust_slow_store_has_size_for_digest: a slow tier whose has() reports less (a compressed stream) or more (a block-rounded footprint) than the blob. Fails on main.
  • populate_keeps_entry_whose_size_is_not_its_digests: passes today; it guards the fix against the obvious simplification of declaring the digest's size, which an AC or dedup index entry does not have.

How was this verified?

bazel test //nativelink-store:integration_tests/fast_slow_store_test_test on fork main (upstream b01417643e + CI commit): 38 passed, 1 failed (populate_does_not_trust_slow_store_has_size_for_digest), with the DataLoss message above. On v1.8.0 the real-store test failed too.

Risk

None to the code: tests only. The risk is in reading the red CI as anything other than the point of this PR; the fix stacked on it takes it green.

AI assistance

Written with Claude Code from a branch b7r6 wrote on 10-15 as the TraceMachina#2881 follow-up; b7r6 reviewed the tests and the history.

🤖 Generated with Claude Code

@github-actions

This comment has been minimized.

@b7r6
b7r6 force-pushed the repro/fastslow-populate-block-rounded branch from 0953a63 to 3bd8778 Compare October 22, 2026 13:33
…stem slow tier

Since 086289a (TraceMachina#2974) FastSlowStore's populate checks the bytes it
received from the slow tier against the size it declared to the fast
tier, and it takes that size from the slow tier's has(). A
FilesystemStore reports its block-rounded footprint there (4096 for a
17-byte blob at the default block size), so every read that has to fill
the fast tier from a filesystem slow tier fails with

  DataLoss: Expected 4096 upload bytes, received 17
  Slow store ended before the cache fill was complete

unless the blob is a block multiple. That is the dedup index store in
nativelink-config/examples/filesystem_cas.json5 and the action cache in
kubernetes/nativelink/nativelink-config.json5, which shipped broken in
v1.8.0.

This commit adds only tests, so CI goes red on it; the fix follows.

- populate_from_filesystem_slow_tier_at_default_block_size: the shipped
  shape, with a real FilesystemStore slow tier.
- populate_does_not_trust_slow_store_has_size_for_digest: a slow tier
  whose has() reports less (a compressed stream) or more (a block-rounded
  footprint) than the blob.
- populate_keeps_entry_whose_size_is_not_its_digests: passes today and
  guards the fix against the obvious simplification of declaring the
  digest's size, which an AC or dedup index entry does not have.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant