Skip to content

Forward cache repair through CompressionStore to the encoded copy - #2988

Open
MarcusSorealheis wants to merge 3 commits into
TraceMachina:mainfrom
MarcusSorealheis:bugfix/compression-store-remove
Open

MarcusSorealheis wants to merge 3 commits into
TraceMachina:mainfrom
MarcusSorealheis:bugfix/compression-store-remove

Conversation

@MarcusSorealheis

@MarcusSorealheis MarcusSorealheis commented Oct 20, 2026 •

Copy link
Copy Markdown
Member

What and why

Fixes #2971.

CompressionStore had no remove() of its own, so a repair request from verify stopped there. A valid compressed copy that decodes to fewer bytes than its digest names failed every read until the cache happened to evict it, even with a correct copy in the slow tier.

remove() now passes the request to the inner store, under the same key the compressed copy is stored under, and returns that store's answer and error. The inner store still decides whether to drop its copy. A tier that holds the only copy is protected the same way an unwrapped memory or filesystem tier already is: put a cache_metrics wrapper with allow_remove: false around it. true means the compressed copy was dropped, never that a durable copy was repaired.

An undecodable copy fails the read with its decoding error and is not removed. That is not a size check failure, so verify asks for no repair.

This is meant for 1.8.1, not 1.8.0. Please merge after v1.8.0 is tagged.

How was this verified?

New tests in nativelink-store/tests/compression_store_test.rs use the composition verify(size) -> fast_slow(fast: compression -> X, slow: memory protected by cache_metrics allow_remove false):

  • repair_drops_a_truncated_encoded_copy_from_memory and ..._from_filesystem: a valid compressed copy that decodes to 4 bytes is stored under a 10-byte digest. The read fails with DataLoss, the fast copy is dropped and the protected slow copy is kept. The next read returns the correct 10 bytes and refills the fast tier.
  • remove_forwards_to_the_tier_holding_the_encoded_copy:
    • with allow_remove: false, the value is kept, remove returns false, and the value is still readable through compression;
    • with allow_remove: true, the value is dropped and remove returns true;
    • a missing entry returns false.
  • remove_propagates_the_backend_error: an inner store that refuses removal makes remove fail with its Unavailable, and the inner store saw the original key.
  • an_undecodable_encoded_copy_is_an_error_not_a_repair: a corrupted compressed run fails the read with Decompression error, not DataLoss. Nothing is removed, and the slow copy is intact.

Against the source without the fix, the first four tests fail. The last passes either way; it guards the decode-error behavior.

Also run:

  • cargo test --no-fail-fast -p nativelink-store: 436 passed, 0 failed, 3 ignored.
  • cargo clippy -p nativelink-store --all-targets: no warnings in the changed files.
  • Pre-commit on the changed files.

Bazel was not run locally. Nothing was tested against a deployment.

Durable objects are safe on this path:

So a repair passing through compression to a Redis or Azure backend deletes nothing there. This follows from those stores' code; there is no Azure mock test for it here.

Risk

A repair request asked of a compression store now reaches its wrapped tier. A memory or filesystem tier under compression that holds the only copy is now evicted on a failed read, unless it is wrapped in cache_metrics with allow_remove: false. The same tier without compression is already evicted this way. The compose-stores page documents this. The compression format and the read and write paths are unchanged.

AI assistance

Claude Code wrote the change, its tests and the doc updates, ran the checks above and drafted this description, at Marcus Eagan's request.

🤖 Generated with Claude Code

https://claude.ai/code/session_01HBxQtV9S561AzBo7aPCYFj

CompressionStore inherited the trait's no-op remove(), so a repair
request from verify stopped at the wrapper: a valid compressed container
that expands to fewer bytes than its digest names failed every read
until the cache happened to evict it, even with a correct copy in the
slow tier.

remove() now forwards to the inner store under the same key the encoded
copy is stored under, returning the inner store's answer and error.
Whether that tier may drop its copy stays its own decision: a tier that
holds the only copy is protected by a cache_metrics wrapper with
allow_remove false, as an unwrapped memory or filesystem tier already
is. true means the encoded copy was dropped, never that a durable copy
was repaired. An encoded copy that cannot be decoded fails the read with
the decoding error, which is not a size verdict, so it is not removed.

Fixes TraceMachina#2971

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HBxQtV9S561AzBo7aPCYFj
@vercel

vercel Bot commented Oct 20, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
nativelink Ready Ready Preview Oct 26, 2026 2:07pm UTC
nativelink-aidm Ready Ready Preview Oct 26, 2026 2:07pm UTC

Request Review

@corcillo

Copy link
Copy Markdown
Collaborator

We will delay merging this until next release v1.8.3

This branch was successfully deployed

2 active deployments
Preview – nativelink — 99af1723 Deployed Oct 26, 2026 by vercel[bot]
Preview – nativelink-aidm — 99af1723 Deployed Oct 26, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Propagate cache repair through CompressionStore without deleting durable copies

2 participants