Repository navigation
Store: resume a short fast-tier read from the slow store, and name the digest of a missing input - #2855
Merged
Conversation
…store, and a missing input names its digest so the client re-uploads it
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
corcillo
approved these changes
Oct 1, 2026
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
A Redis key evicted under a read now ends the read with
NotFoundinstead of a truncated blob (#2844), and on the test bed that turned into a failed build: the worker reported the actionFAILED_PRECONDITION, but without thePreconditionFailuredetail naming the blob, so Bazel did not re-upload and retry, and a link step failed withRemote Execution Failure. The blob was in the object store the whole time.Two changes. The fast-slow store resumes from the slow store when the fast tier ends short on a digest: a digest names an immutable blob, so what the fast tier sent is a correct prefix, and the slow store serves the rest from the byte it stopped at, repopulating the fast tier on the way (a metric counts it). A short read of any other key still fails, since the tiers could hold different content for it. And a missing input names its digest: the store attaches
ErrorContext::MissingDigestwhen the slow store loses a blob under the read, as it already did for a blob that was never there, and the worker loop keeps that context when it turns the error intoFAILED_PRECONDITION, soto_execute_responseemits the detail and the client re-uploads exactly that blob.How was this verified?
fast_slow_store_test: a fast tier that sends sixteen bytes and thenNotFoundon a digest serves the whole blob, and a ranged read resumes within the range; the same on a plain key staysNotFound; a slow store that loses a blob under the read raisesNotFoundwith the digest as context.local_worker_test: theFAILED_PRECONDITIONthe worker sends carries the context through. Seen on the bed: T1 on the release candidate failed one link on exactly this read.Risk
A read that used to fail after partial bytes from the fast tier now completes from the slow tier; the prefix is trusted only for digest keys, and a verify store above this one still checks the whole blob against its digest, so a wrong prefix is caught there and the copy dropped (#2844). Nothing changes for a blob missing from both tiers.
AI assistance
An agent drafted the change and I reviewed every line.