Skip to content

Repair blobs stored at the wrong size, through shards down to Azure - #2986

Closed
MarcusSorealheis wants to merge 2 commits into
TraceMachina:mainfrom
MarcusSorealheis:bugfix/repair-wrong-size-blobs
Closed

MarcusSorealheis wants to merge 2 commits into
TraceMachina:mainfrom
MarcusSorealheis:bugfix/repair-wrong-size-blobs

Conversation

@MarcusSorealheis

@MarcusSorealheis MarcusSorealheis commented Oct 20, 2026 •

Copy link
Copy Markdown
Member

What and why

Fixes #2970.

A CAS upload to Azure that lost its body on a retry could leave an empty blob under a digest that names content. #2966 and #2974 stop new ones from being written, but the blobs already written stay broken. The Azure store reports them present, so clients never upload them again, and every read of one fails size verification. The repair that verification triggers never reached Azure: ShardStore did not forward remove(), and AzureBlobStore had none. The enterprise chart always puts Azure under a shard on the CAS path (cloudShardStore, 10 shards by default).

This change does three things:

  • VerifyStore::has with verify_size reports a copy whose stored size disagrees with its digest as missing. The client then uploads the blob again, and the upload overwrites the bad copy in every tier, Azure included. The same check runs where a worker decides whether to upload an output and where completeness_checking validates an Action Cache entry.
  • ShardStore::remove forwards to the one store the key is sharded to, using the same routing as update and get_part. It returns that store's result or error.
  • AzureBlobStore::remove deletes only an empty object stored under a non-empty digest. It reads the blob's properties and sends a delete conditional on the ETag (If-Match), so an upload that replaced the object in the meantime survives. Any object holding bytes is kept and false returned, because a store above may have written it shorter than the digest on purpose, such as a compressed frame. No encoding stores content in zero bytes, so an empty object is never anyone's valid copy.

After a failed read, the next read of the blob is a NotFound, which clients answer with an upload, instead of the same DataLoss every time.

Intended for 1.8.1, not 1.8.0. Please merge after v1.8.0 is tagged.

Related: #2971, #2972 and #2973 cover the other wrappers repair stops at, in their own PRs. This one also replaces two unimplemented!() bodies in a verify_store_test mock with real behavior: the mock never keeps an upload, so it reports keys missing and reads as NotFound.

How was this verified?

Code observations.

  • The enterprise chart renders the Azure CAS tier as verify → fast_slow → fast_slow(filesystem | shard → 10× azure), checked with helm template --set cas.cloudProvider=azure, with and without Redis.
  • Before this change, ShardStore used the trait's default remove(), which returns false without asking any shard.
  • Every store composed under verify reports the size a read delivers:

New tests. Each was first run against the unfixed sources and seen to fail.

  • azure_blob_store_test::verify_over_a_shard_repairs_an_empty_azure_blob reproduces the deployed composition, verify(size) → shard → azure, against a mock transport. has reports the empty blob missing, and a read fails with DataLoss and sends one DELETE carrying the ETag it read. Unfixed, has returned Some(0) and no delete was sent.
  • azure_blob_store_test::remove_* (5 tests):
    • an empty blob is deleted with If-Match;
    • objects of 1, 50, 100 and 200 bytes are kept;
    • a blob replaced since it was read (412) is kept;
    • an empty blob is kept when no ETag is returned;
    • a missing blob, the empty digest and string keys send no delete (the last two send no request at all).
  • azure_blob_store_test::a_failed_remove_through_a_shard_reports_the_error: an Azure error reaches the caller of the shard's remove, tagged In ShardStore::remove().
  • shard_store_test::remove_reaches_the_store_the_key_is_sharded_to: for 32 digest keys and 8 string keys across 3 shards, each removal empties only the owning shard, a second removal returns false, and every key not yet removed stays readable.
  • shard_store_test::a_truncated_copy_in_a_sharded_cache_is_evicted_and_refilled covers verify(size) → fast_slow(fast: shard → 2× memory, slow: cache_metrics(allow_remove: false) → memory). A truncated cached copy fails with DataLoss and is evicted from its shard, the protected durable copy is kept, and the next read succeeds and refills the shard.
  • verify_store_test::verify_size_true_reports_a_wrong_size_copy_missing: empty and short copies are reported missing, a correct copy is reported with its size, and a re-upload replaces the empty copy.
  • verify_store_test::verify_size_false_reports_the_stored_size keeps the old behavior without size verification.

Commands run in this worktree.

  • cargo test --no-fail-fast -p nativelink-store -p nativelink-service: 45 test binaries, 577 passed, 0 failed. 3 are ignored, as on main.
  • cargo clippy -p nativelink-store -p nativelink-util --all-targets: no warnings in changed files.
  • Pre-commit on every changed file passed, including rustfmt, typos and vale.

Not verified.

  • Bazel was not run locally.
  • There was no run against a real Azure account or a deployment.
  • Blobs already in a container are repaired only when a client checks for them or reads them. A blob nobody asks for again stays until it is overwritten or expires.

Risk

  • Existence checks under verify_size. A copy of the wrong size is now reported missing instead of present, which causes a re-upload. That is the intended effect. If a store below verify reported a size other than what a read delivers, its blobs would be re-uploaded on every check. The stores listed above don't. One that did would show up as a rising size_verification_failures count and the new warning.
  • Deletes in Azure. The store now deletes in one more case: an empty object under a non-empty digest, conditional on its ETag. If a SAS lacks delete permission, the delete fails, the read path logs it, and the blob stays until the re-upload replaces it. The Azure how-to now asks for read, write and delete.
  • Shard removal. A repair request to a shard now reaches its stores, so a memory or filesystem tier under a shard is evicted on a failed read. That already happened without a shard. Wrap such a tier in cache_metrics with allow_remove: false if it holds the only copy. The sharding docs now say so.
  • Not covered. A durable copy that is truncated but not empty, or corrupt at the right length, is still kept. It needs targeted repair, as the docs say.
  • Overlap with Bound the requests an Azure store keeps in flight, uploads included #2985. That PR adds a request limit to the Azure store and edits the same file. Whichever merges second should make remove take a request permit.

AI assistance

Claude Code (Claude Opus 5.5) traced the repair path through the store composition, rendered the chart, wrote the change, tests and doc updates, ran the checks above and drafted this description. Marcus Eagan directed the scope.

🤖 Generated with Claude Code

https://claude.ai/code/session_01HBxQtV9S561AzBo7aPCYFj

An upload that lost its body on a retry could leave an empty Azure blob
under a digest that names content. The Azure store reported it present,
so clients never uploaded it again, and every read of it failed size
verification. The removal verification asks for never reached Azure:
ShardStore did not forward remove(), and AzureBlobStore had none.

- VerifyStore::has with verify_size reports a copy whose stored size
  disagrees with its digest as missing, so the client uploads the blob
  again and the upload replaces the bad copy in every tier.
- ShardStore::remove forwards to the one store the key is sharded to.
- AzureBlobStore::remove deletes only an empty object under a non-empty
  digest, conditional on the ETag it read. Any object holding bytes is
  kept: a store above may have written it shorter than its digest.

Fixes TraceMachina#2970

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HBxQtV9S561AzBo7aPCYFj
@vercel

vercel Bot commented Oct 20, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
nativelink Ready Ready Preview Oct 21, 2026 3:19pm UTC
nativelink-aidm Ready Ready Preview Oct 21, 2026 3:19pm UTC

Request Review

This branch was successfully deployed

2 active deployments
Preview – nativelink — 3c56e51a Deployed Oct 21, 2026 by vercel[bot]
Preview – nativelink-aidm — 3c56e51a Deployed Oct 21, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Forward cache repair through ShardStore to the owning shard

1 participant