Repository navigation
Repair blobs stored at the wrong size, through shards down to Azure - #2986
Closed
MarcusSorealheis wants to merge 2 commits into
Closed
MarcusSorealheis wants to merge 2 commits into
MarcusSorealheis wants to merge 2 commits into
Conversation
An upload that lost its body on a retry could leave an empty Azure blob under a digest that names content. The Azure store reported it present, so clients never uploaded it again, and every read of it failed size verification. The removal verification asks for never reached Azure: ShardStore did not forward remove(), and AzureBlobStore had none. - VerifyStore::has with verify_size reports a copy whose stored size disagrees with its digest as missing, so the client uploads the blob again and the upload replaces the bad copy in every tier. - ShardStore::remove forwards to the one store the key is sharded to. - AzureBlobStore::remove deletes only an empty object under a non-empty digest, conditional on the ETag it read. Any object holding bytes is kept: a store above may have written it shorter than its digest. Fixes TraceMachina#2970 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HBxQtV9S561AzBo7aPCYFj
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This was referenced Oct 20, 2026
MarcusSorealheis
marked this pull request as draft
October 21, 2026 03:00
MarcusSorealheis
marked this pull request as ready for review
October 21, 2026 15:05
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
Fixes #2970.
A CAS upload to Azure that lost its body on a retry could leave an empty blob under a digest that names content. #2966 and #2974 stop new ones from being written, but the blobs already written stay broken. The Azure store reports them present, so clients never upload them again, and every read of one fails size verification. The repair that verification triggers never reached Azure:
ShardStoredid not forwardremove(), andAzureBlobStorehad none. The enterprise chart always puts Azure under a shard on the CAS path (cloudShardStore, 10 shards by default).This change does three things:
VerifyStore::haswithverify_sizereports a copy whose stored size disagrees with its digest as missing. The client then uploads the blob again, and the upload overwrites the bad copy in every tier, Azure included. The same check runs where a worker decides whether to upload an output and wherecompleteness_checkingvalidates an Action Cache entry.ShardStore::removeforwards to the one store the key is sharded to, using the same routing asupdateandget_part. It returns that store's result or error.AzureBlobStore::removedeletes only an empty object stored under a non-empty digest. It reads the blob's properties and sends a delete conditional on the ETag (If-Match), so an upload that replaced the object in the meantime survives. Any object holding bytes is kept andfalsereturned, because a store above may have written it shorter than the digest on purpose, such as a compressed frame. No encoding stores content in zero bytes, so an empty object is never anyone's valid copy.After a failed read, the next read of the blob is a
NotFound, which clients answer with an upload, instead of the sameDataLossevery time.Intended for 1.8.1, not 1.8.0. Please merge after v1.8.0 is tagged.
Related: #2971, #2972 and #2973 cover the other wrappers repair stops at, in their own PRs. This one also replaces two
unimplemented!()bodies in averify_store_testmock with real behavior: the mock never keeps an upload, so it reports keys missing and reads asNotFound.How was this verified?
Code observations.
verify → fast_slow → fast_slow(filesystem | shard → 10× azure), checked withhelm template --set cas.cloudProvider=azure, with and without Redis.ShardStoreused the trait's defaultremove(), which returnsfalsewithout asking any shard.verifyreports the size a read delivers:STRLEN), and S3/GCS/ONTAP/Azure (content length) report what they store.grpcandcompression(since Report the digest size from the compression store has, not the compressed one #2960) report the digest size.dedupsums its chunks.New tests. Each was first run against the unfixed sources and seen to fail.
azure_blob_store_test::verify_over_a_shard_repairs_an_empty_azure_blobreproduces the deployed composition,verify(size) → shard → azure, against a mock transport.hasreports the empty blob missing, and a read fails withDataLossand sends oneDELETEcarrying the ETag it read. Unfixed,hasreturnedSome(0)and no delete was sent.azure_blob_store_test::remove_*(5 tests):If-Match;azure_blob_store_test::a_failed_remove_through_a_shard_reports_the_error: an Azure error reaches the caller of the shard'sremove, taggedIn ShardStore::remove().shard_store_test::remove_reaches_the_store_the_key_is_sharded_to: for 32 digest keys and 8 string keys across 3 shards, each removal empties only the owning shard, a second removal returnsfalse, and every key not yet removed stays readable.shard_store_test::a_truncated_copy_in_a_sharded_cache_is_evicted_and_refilledcoversverify(size) → fast_slow(fast: shard → 2× memory, slow: cache_metrics(allow_remove: false) → memory). A truncated cached copy fails withDataLossand is evicted from its shard, the protected durable copy is kept, and the next read succeeds and refills the shard.verify_store_test::verify_size_true_reports_a_wrong_size_copy_missing: empty and short copies are reported missing, a correct copy is reported with its size, and a re-upload replaces the empty copy.verify_store_test::verify_size_false_reports_the_stored_sizekeeps the old behavior without size verification.Commands run in this worktree.
cargo test --no-fail-fast -p nativelink-store -p nativelink-service: 45 test binaries, 577 passed, 0 failed. 3 are ignored, as onmain.cargo clippy -p nativelink-store -p nativelink-util --all-targets: no warnings in changed files.Not verified.
Risk
verify_size. A copy of the wrong size is now reported missing instead of present, which causes a re-upload. That is the intended effect. If a store belowverifyreported a size other than what a read delivers, its blobs would be re-uploaded on every check. The stores listed above don't. One that did would show up as a risingsize_verification_failurescount and the new warning.shardnow reaches its stores, so a memory or filesystem tier under a shard is evicted on a failed read. That already happened without a shard. Wrap such a tier incache_metricswithallow_remove: falseif it holds the only copy. The sharding docs now say so.removetake a request permit.AI assistance
Claude Code (Claude Opus 5.5) traced the repair path through the store composition, rendered the chart, wrote the change, tests and doc updates, ran the checks above and drafted this description. Marcus Eagan directed the scope.
🤖 Generated with Claude Code
https://claude.ai/code/session_01HBxQtV9S561AzBo7aPCYFj