Skip to content

Store: resume a short fast-tier read from the slow store, and name the digest of a missing input - #2855

Merged
amankrx merged 2 commits into
TraceMachina:mainfrom
amankrx:fix/missing-input-digest
Oct 1, 2026
Merged

amankrx merged 2 commits into
TraceMachina:mainfrom
amankrx:fix/missing-input-digest

Conversation

@amankrx

@amankrx amankrx commented Oct 1, 2026 •

Copy link
Copy Markdown
Member

What and why

A Redis key evicted under a read now ends the read with NotFound instead of a truncated blob (#2844), and on the test bed that turned into a failed build: the worker reported the action FAILED_PRECONDITION, but without the PreconditionFailure detail naming the blob, so Bazel did not re-upload and retry, and a link step failed with Remote Execution Failure. The blob was in the object store the whole time.

Two changes. The fast-slow store resumes from the slow store when the fast tier ends short on a digest: a digest names an immutable blob, so what the fast tier sent is a correct prefix, and the slow store serves the rest from the byte it stopped at, repopulating the fast tier on the way (a metric counts it). A short read of any other key still fails, since the tiers could hold different content for it. And a missing input names its digest: the store attaches ErrorContext::MissingDigest when the slow store loses a blob under the read, as it already did for a blob that was never there, and the worker loop keeps that context when it turns the error into FAILED_PRECONDITION, so to_execute_response emits the detail and the client re-uploads exactly that blob.

How was this verified?

fast_slow_store_test: a fast tier that sends sixteen bytes and then NotFound on a digest serves the whole blob, and a ranged read resumes within the range; the same on a plain key stays NotFound; a slow store that loses a blob under the read raises NotFound with the digest as context. local_worker_test: the FAILED_PRECONDITION the worker sends carries the context through. Seen on the bed: T1 on the release candidate failed one link on exactly this read.

Risk

A read that used to fail after partial bytes from the fast tier now completes from the slow tier; the prefix is trusted only for digest keys, and a verify store above this one still checks the whole blob against its digest, so a wrong prefix is caught there and the copy dropped (#2844). Nothing changes for a blob missing from both tiers.

AI assistance

An agent drafted the change and I reviewed every line.

…store, and a missing input names its digest so the client re-uploads it
@vercel

vercel Bot commented Oct 1, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
nativelink Ready Ready Preview Oct 1, 2026 4:32am UTC
nativelink-aidm Ready Ready Preview Oct 1, 2026 4:32am UTC

Request Review

@amankrx
amankrx merged commit bd1bb34 into TraceMachina:main Oct 1, 2026
43 of 45 checks passed

This branch was successfully deployed

2 active deployments
Preview – nativelink — 3ddc0c66 Deployed Oct 1, 2026 by vercel[bot]
Preview – nativelink-aidm — 3ddc0c66 Deployed Oct 1, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants