Repository navigation
Forward cache repair through CompressionStore to the encoded copy - #2988
Open
MarcusSorealheis wants to merge 3 commits into
Open
MarcusSorealheis wants to merge 3 commits into
MarcusSorealheis wants to merge 3 commits into
Conversation
CompressionStore inherited the trait's no-op remove(), so a repair request from verify stopped at the wrapper: a valid compressed container that expands to fewer bytes than its digest names failed every read until the cache happened to evict it, even with a correct copy in the slow tier. remove() now forwards to the inner store under the same key the encoded copy is stored under, returning the inner store's answer and error. Whether that tier may drop its copy stays its own decision: a tier that holds the only copy is protected by a cache_metrics wrapper with allow_remove false, as an unwrapped memory or filesystem tier already is. true means the encoded copy was dropped, never that a durable copy was repaired. An encoded copy that cannot be decoded fails the read with the decoding error, which is not a size verdict, so it is not removed. Fixes TraceMachina#2971 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HBxQtV9S561AzBo7aPCYFj
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
MarcusSorealheis
marked this pull request as draft
October 21, 2026 03:00
MarcusSorealheis
marked this pull request as ready for review
October 21, 2026 15:05
Collaborator
|
We will delay merging this until next release v1.8.3 |
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
Fixes #2971.
CompressionStorehad noremove()of its own, so a repair request fromverifystopped there. A valid compressed copy that decodes to fewer bytes than its digest names failed every read until the cache happened to evict it, even with a correct copy in the slow tier.remove()now passes the request to the inner store, under the same key the compressed copy is stored under, and returns that store's answer and error. The inner store still decides whether to drop its copy. A tier that holds the only copy is protected the same way an unwrapped memory or filesystem tier already is: put acache_metricswrapper withallow_remove: falsearound it.truemeans the compressed copy was dropped, never that a durable copy was repaired.An undecodable copy fails the read with its decoding error and is not removed. That is not a size check failure, so
verifyasks for no repair.This is meant for 1.8.1, not 1.8.0. Please merge after v1.8.0 is tagged.
How was this verified?
New tests in
nativelink-store/tests/compression_store_test.rsuse the compositionverify(size) -> fast_slow(fast: compression -> X, slow: memory protected by cache_metrics allow_remove false):repair_drops_a_truncated_encoded_copy_from_memoryand..._from_filesystem: a valid compressed copy that decodes to 4 bytes is stored under a 10-byte digest. The read fails withDataLoss, the fast copy is dropped and the protected slow copy is kept. The next read returns the correct 10 bytes and refills the fast tier.remove_forwards_to_the_tier_holding_the_encoded_copy:allow_remove: false, the value is kept,removereturnsfalse, and the value is still readable through compression;allow_remove: true, the value is dropped andremovereturnstrue;false.remove_propagates_the_backend_error: an inner store that refuses removal makesremovefail with itsUnavailable, and the inner store saw the original key.an_undecodable_encoded_copy_is_an_error_not_a_repair: a corrupted compressed run fails the read withDecompression error, notDataLoss. Nothing is removed, and the slow copy is intact.Against the source without the fix, the first four tests fail. The last passes either way; it guards the decode-error behavior.
Also run:
cargo test --no-fail-fast -p nativelink-store: 436 passed, 0 failed, 3 ignored.cargo clippy -p nativelink-store --all-targets: no warnings in the changed files.Bazel was not run locally. Nothing was tested against a deployment.
Durable objects are safe on this path:
remove().mainkeeps every object. With Repair blobs stored at the wrong size, through shards down to Azure #2986 it deletes only an empty object under a non-empty digest, and a compressed frame is never empty.So a repair passing through compression to a Redis or Azure backend deletes nothing there. This follows from those stores' code; there is no Azure mock test for it here.
Risk
A repair request asked of a
compressionstore now reaches its wrapped tier. A memory or filesystem tier under compression that holds the only copy is now evicted on a failed read, unless it is wrapped incache_metricswithallow_remove: false. The same tier without compression is already evicted this way. Thecompose-storespage documents this. The compression format and the read and write paths are unchanged.AI assistance
Claude Code wrote the change, its tests and the doc updates, ran the checks above and drafted this description, at Marcus Eagan's request.
🤖 Generated with Claude Code
https://claude.ai/code/session_01HBxQtV9S561AzBo7aPCYFj