Repository navigation
fix(error): classify transient Redis failures as retryable - #2657
Merged
amankrx merged 3 commits intoAug 3, 2026
Merged
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
palfrey
approved these changes
Aug 3, 2026
Redis failures reach REAPI clients as gRPC statuses, and Bazel treats INVALID_ARGUMENT as permanent — it fails the build instead of retrying. The mapping sent several purely transient conditions there, so an ordinary sentinel failover ended in-flight builds. Sentinel and cluster topology errors, dropped connections, and transient server states (BUSYLOADING, CLUSTERDOWN, MASTERDOWN, TRYAGAIN, READONLY) are now UNAVAILABLE. Parse and UnexpectedReturnType become INTERNAL: a malformed reply is our fault or the server's, never the caller's argument. Genuine misconfiguration — an unparseable client config, an empty sentinel list — stays INVALID_ARGUMENT, since retrying cannot help there. The connect path separately forced InvalidArgument onto every failure via err_tip_with_code, which overrode the classification entirely; it now keeps the underlying code. This is what made MasterNameNotFoundBySentinel permanent no matter how the conversion classified it. The sentinel test that pinned that behaviour is updated: an unknown master name is indistinguishable from a failover in progress, and guessing between them is not worth failing recoverable builds — a genuinely wrong name keeps failing and stays visible.
test_sentinel_connect_with_url_specified_master asserts that the sentinelServiceName URL parameter resolves, but allowed only 100ms to do it — a value copied from the fail-fast bad-master test directly above, where a tight budget is the actual subject. That 100ms has to cover two TCP connects and two handshakes with a SENTINEL MASTERS round trip in between. A loaded macOS runner took ~125ms and the test failed on DeadlineExceeded. It is the only success-path test in this file with a sub-second budget; the others allow 1s or 5s. Unrelated to the error-classification change in this PR, but it fails on its CI.
amankrx
force-pushed
the
fix/redis-error-classification
branch
from
August 3, 2026 15:17
f3dcce4 to
f6040ad
Compare
2 of 4 tasks
b7r6
added a commit
to b7r6/nativelink
that referenced
this pull request
Sep 29, 2026
tonic codes a transport failure it cannot classify further (a connection reset or broken pipe mid-stream) as Code::Unknown with message "transport error"; tonic's own Status::from_error maps the shapes it does recognize (e.g. a connect refusal) to Unavailable already. The gRPC status model reserves UNKNOWN for genuinely unmapped errors and codes transport failures UNAVAILABLE. Left as Unknown, such a blip is treated as permanent by REAPI clients — Bazel retries UNAVAILABLE but not UNKNOWN — and by any retry configuration keyed to Unavailable, so a retryable failure becomes terminal and cacheable uploads are silently dropped mid-write. Detection is downcast-first: walk the Status source chain for a concrete tonic::transport::Error. The fallback, for a Status reconstructed from the wire where only the message survives, is anchored to the two shapes that actually occur: tonic's exact "transport error" rendering, and "transport error : <context>" as produced by this crate's own From<Error> for Status when a peer re-encodes and forwards one. An app-level Unknown that merely mentions the phrase mid-message is not reclassified. Upstream's TraceMachina#2657 made the same argument for transient Redis failures; this applies it to the adjacent tonic transport surface. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
b7r6
added a commit
to b7r6/nativelink
that referenced
this pull request
Sep 29, 2026
tonic codes a transport failure it cannot classify further (a connection reset or broken pipe mid-stream) as Code::Unknown with message "transport error"; tonic's own Status::from_error maps the shapes it does recognize (e.g. a connect refusal) to Unavailable already. The gRPC status model reserves UNKNOWN for genuinely unmapped errors and codes transport failures UNAVAILABLE. Left as Unknown, such a blip is treated as permanent by REAPI clients — Bazel retries UNAVAILABLE but not UNKNOWN — and by any retry configuration keyed to Unavailable, so a retryable failure becomes terminal and cacheable uploads are silently dropped mid-write. Detection is downcast-first: walk the Status source chain for a concrete tonic::transport::Error. The fallback, for a Status reconstructed from the wire where only the message survives, is anchored to the two shapes that actually occur: tonic's exact "transport error" rendering, and "transport error : <context>" as produced by this crate's own From<Error> for Status when a peer re-encodes and forwards one. An app-level Unknown that merely mentions the phrase mid-message is not reclassified. Upstream's TraceMachina#2657 made the same argument for transient Redis failures; this applies it to the adjacent tonic transport surface. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
b7r6
added a commit
to b7r6/nativelink
that referenced
this pull request
Oct 7, 2026
tonic codes a transport failure it cannot classify further (a connection reset or broken pipe mid-stream) as Code::Unknown with message "transport error"; tonic's own Status::from_error maps the shapes it does recognize (e.g. a connect refusal) to Unavailable already. The gRPC status model reserves UNKNOWN for genuinely unmapped errors and codes transport failures UNAVAILABLE. Left as Unknown, such a blip is treated as permanent by REAPI clients — Bazel retries UNAVAILABLE but not UNKNOWN — and by any retry configuration keyed to Unavailable, so a retryable failure becomes terminal and cacheable uploads are silently dropped mid-write. Detection is downcast-first: walk the Status source chain for a concrete tonic::transport::Error. The fallback, for a Status reconstructed from the wire where only the message survives, is anchored to the two shapes that actually occur: tonic's exact "transport error" rendering, and "transport error : <context>" as produced by this crate's own From<Error> for Status when a peer re-encodes and forwards one. An app-level Unknown that merely mentions the phrase mid-message is not reclassified. Upstream's TraceMachina#2657 made the same argument for transient Redis failures; this applies it to the adjacent tonic transport surface. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
b7r6
added a commit
to b7r6/nativelink
that referenced
this pull request
Oct 7, 2026
tonic codes a transport failure it cannot classify further (a connection reset or broken pipe mid-stream) as Code::Unknown with message "transport error"; tonic's own Status::from_error maps the shapes it does recognize (e.g. a connect refusal) to Unavailable already. The gRPC status model reserves UNKNOWN for genuinely unmapped errors and codes transport failures UNAVAILABLE. Left as Unknown, such a blip is treated as permanent by REAPI clients — Bazel retries UNAVAILABLE but not UNKNOWN — and by any retry configuration keyed to Unavailable, so a retryable failure becomes terminal and cacheable uploads are silently dropped mid-write. Detection is downcast-first: walk the Status source chain for a concrete tonic::transport::Error. The fallback, for a Status reconstructed from the wire where only the message survives, is anchored to the two shapes that actually occur: tonic's exact "transport error" rendering, and "transport error : <context>" as produced by this crate's own From<Error> for Status when a peer re-encodes and forwards one. An app-level Unknown that merely mentions the phrase mid-message is not reclassified. Upstream's TraceMachina#2657 made the same argument for transient Redis failures; this applies it to the adjacent tonic transport surface. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
b7r6
added a commit
to b7r6/nativelink
that referenced
this pull request
Oct 7, 2026
tonic codes a transport failure it cannot classify further (a connection reset or broken pipe mid-stream) as Code::Unknown with message "transport error"; tonic's own Status::from_error maps the shapes it does recognize (e.g. a connect refusal) to Unavailable already. The gRPC status model reserves UNKNOWN for genuinely unmapped errors and codes transport failures UNAVAILABLE. Left as Unknown, such a blip is treated as permanent by REAPI clients — Bazel retries UNAVAILABLE but not UNKNOWN — and by any retry configuration keyed to Unavailable, so a retryable failure becomes terminal and cacheable uploads are silently dropped mid-write. Detection is downcast-first: walk the Status source chain for a concrete tonic::transport::Error. The fallback, for a Status reconstructed from the wire where only the message survives, is anchored to the two shapes that actually occur: tonic's exact "transport error" rendering, and "transport error : <context>" as produced by this crate's own From<Error> for Status when a peer re-encodes and forwards one. An app-level Unknown that merely mentions the phrase mid-message is not reclassified. Upstream's TraceMachina#2657 made the same argument for transient Redis failures; this applies it to the adjacent tonic transport surface. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
b7r6
added a commit
to b7r6/nativelink
that referenced
this pull request
Oct 8, 2026
tonic codes a transport failure it cannot classify further (a connection reset or broken pipe mid-stream) as Code::Unknown with message "transport error"; tonic's own Status::from_error maps the shapes it does recognize (e.g. a connect refusal) to Unavailable already. The gRPC status model reserves UNKNOWN for genuinely unmapped errors and codes transport failures UNAVAILABLE. Left as Unknown, such a blip is treated as permanent by REAPI clients — Bazel retries UNAVAILABLE but not UNKNOWN — and by any retry configuration keyed to Unavailable, so a retryable failure becomes terminal and cacheable uploads are silently dropped mid-write. Detection is downcast-first: walk the Status source chain for a concrete tonic::transport::Error. The fallback, for a Status reconstructed from the wire where only the message survives, is anchored to the two shapes that actually occur: tonic's exact "transport error" rendering, and "transport error : <context>" as produced by this crate's own From<Error> for Status when a peer re-encodes and forwards one. An app-level Unknown that merely mentions the phrase mid-message is not reclassified. Upstream's TraceMachina#2657 made the same argument for transient Redis failures; this applies it to the adjacent tonic transport surface. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
b7r6
added a commit
to b7r6/nativelink
that referenced
this pull request
Oct 14, 2026
tonic codes a transport failure it cannot classify further (a connection reset or broken pipe mid-stream) as Code::Unknown with message "transport error"; tonic's own Status::from_error maps the shapes it does recognize (e.g. a connect refusal) to Unavailable already. The gRPC status model reserves UNKNOWN for genuinely unmapped errors and codes transport failures UNAVAILABLE. Left as Unknown, such a blip is treated as permanent by REAPI clients — Bazel retries UNAVAILABLE but not UNKNOWN — and by any retry configuration keyed to Unavailable, so a retryable failure becomes terminal and cacheable uploads are silently dropped mid-write. Detection is downcast-first: walk the Status source chain for a concrete tonic::transport::Error. The fallback, for a Status reconstructed from the wire where only the message survives, is anchored to the two shapes that actually occur: tonic's exact "transport error" rendering, and "transport error : <context>" as produced by this crate's own From<Error> for Status when a peer re-encodes and forwards one. An app-level Unknown that merely mentions the phrase mid-message is not reclassified. Upstream's TraceMachina#2657 made the same argument for transient Redis failures; this applies it to the adjacent tonic transport surface. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
b7r6
added a commit
to b7r6/nativelink
that referenced
this pull request
Oct 15, 2026
tonic codes a transport failure it cannot classify further (a connection reset or broken pipe mid-stream) as Code::Unknown with message "transport error"; tonic's own Status::from_error maps the shapes it does recognize (e.g. a connect refusal) to Unavailable already. The gRPC status model reserves UNKNOWN for genuinely unmapped errors and codes transport failures UNAVAILABLE. Left as Unknown, such a blip is treated as permanent by REAPI clients — Bazel retries UNAVAILABLE but not UNKNOWN — and by any retry configuration keyed to Unavailable, so a retryable failure becomes terminal and cacheable uploads are silently dropped mid-write. Detection is downcast-first: walk the Status source chain for a concrete tonic::transport::Error. The fallback, for a Status reconstructed from the wire where only the message survives, is anchored to the two shapes that actually occur: tonic's exact "transport error" rendering, and "transport error : <context>" as produced by this crate's own From<Error> for Status when a peer re-encodes and forwards one. An app-level Unknown that merely mentions the phrase mid-message is not reclassified. Upstream's TraceMachina#2657 made the same argument for transient Redis failures; this applies it to the adjacent tonic transport surface. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
b7r6
added a commit
to b7r6/nativelink
that referenced
this pull request
Oct 15, 2026
tonic codes a transport failure it cannot classify further (a connection reset or broken pipe mid-stream) as Code::Unknown with message "transport error"; tonic's own Status::from_error maps the shapes it does recognize (e.g. a connect refusal) to Unavailable already. The gRPC status model reserves UNKNOWN for genuinely unmapped errors and codes transport failures UNAVAILABLE. Left as Unknown, such a blip is treated as permanent by REAPI clients — Bazel retries UNAVAILABLE but not UNKNOWN — and by any retry configuration keyed to Unavailable, so a retryable failure becomes terminal and cacheable uploads are silently dropped mid-write. Detection is downcast-first: walk the Status source chain for a concrete tonic::transport::Error. The fallback, for a Status reconstructed from the wire where only the message survives, is anchored to the two shapes that actually occur: tonic's exact "transport error" rendering, and "transport error : <context>" as produced by this crate's own From<Error> for Status when a peer re-encodes and forwards one. An app-level Unknown that merely mentions the phrase mid-message is not reclassified. Upstream's TraceMachina#2657 made the same argument for transient Redis failures; this applies it to the adjacent tonic transport surface. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
palfrey
pushed a commit
that referenced
this pull request
Oct 15, 2026
tonic codes a transport failure it cannot classify further (a connection reset or broken pipe mid-stream) as Code::Unknown with message "transport error"; tonic's own Status::from_error maps the shapes it does recognize (e.g. a connect refusal) to Unavailable already. The gRPC status model reserves UNKNOWN for genuinely unmapped errors and codes transport failures UNAVAILABLE. Left as Unknown, such a blip is treated as permanent by REAPI clients — Bazel retries UNAVAILABLE but not UNKNOWN — and by any retry configuration keyed to Unavailable, so a retryable failure becomes terminal and cacheable uploads are silently dropped mid-write. Detection is downcast-first: walk the Status source chain for a concrete tonic::transport::Error. The fallback, for a Status reconstructed from the wire where only the message survives, is anchored to the two shapes that actually occur: tonic's exact "transport error" rendering, and "transport error : <context>" as produced by this crate's own From<Error> for Status when a peer re-encodes and forwards one. An app-level Unknown that merely mentions the phrase mid-message is not reclassified. Upstream's #2657 made the same argument for transient Redis failures; this applies it to the adjacent tonic transport surface.
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Redis failures reach REAPI clients as gRPC statuses. Bazel treats
INVALID_ARGUMENTas permanent — it does not retry, it fails the build. The current mapping sends several purely transient conditions there, so an ordinary sentinel failover (1-2 minutes, fully self-healing) ends every in-flight build instead of causing a brief stall.TRA-292 during a customer stress test: a Redis roll during a
helm upgradekilled a running build withMasterNameNotFoundBySentinel.Two separate causes:
From<RedisError>mapping putParse | UnexpectedReturnType | InvalidClientConfiginInvalidArgument, and everything topology-relatedfell through to
Unknown.IobecameInternaleven for a plain droppedconnection.
RedisStore's connect path wrapped every failure witherr_tip_with_code(Code::InvalidArgument, ...), which overrode theclassification entirely. Even a correct mapping could not survive it.
Type of change
How Has This Been Tested?
Full
nativelink-errorandnativelink-storesuites pass.Checklist
bazel test //...passes locallyThis change is
This change is