Skip to content

fix(error): classify transient Redis failures as retryable - #2657

Merged
amankrx merged 3 commits into
TraceMachina:mainfrom
amankrx:fix/redis-error-classification
Aug 3, 2026
Merged

amankrx merged 3 commits into
TraceMachina:mainfrom
amankrx:fix/redis-error-classification

Conversation

@amankrx

@amankrx amankrx commented Aug 3, 2026 •

Copy link
Copy Markdown
Member

Problem

Redis failures reach REAPI clients as gRPC statuses. Bazel treats INVALID_ARGUMENT as permanent — it does not retry, it fails the build. The current mapping sends several purely transient conditions there, so an ordinary sentinel failover (1-2 minutes, fully self-healing) ends every in-flight build instead of causing a brief stall.

TRA-292 during a customer stress test: a Redis roll during a helm upgrade killed a running build with MasterNameNotFoundBySentinel.

Two separate causes:

  1. The From<RedisError> mapping put Parse | UnexpectedReturnType | InvalidClientConfig in InvalidArgument, and everything topology-related
    fell through to Unknown. Io became Internal even for a plain dropped
    connection.
  2. RedisStore's connect path wrapped every failure with
    err_tip_with_code(Code::InvalidArgument, ...), which overrode the
    classification entirely. Even a correct mapping could not survive it.
Type of change
  • Bug fix (non-breaking change which fixes an issue)

How Has This Been Tested?

Full nativelink-error and nativelink-store suites pass.

Checklist

  • Updated documentation if needed
  • bazel test //... passes locally
  • PR is contained in a single commit

This change is Reviewable


This change is Reviewable

@vercel

vercel Bot commented Aug 3, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
nativelink Ready Ready Preview Aug 3, 2026 3:40pm
nativelink-aidm Ready Ready Preview Aug 3, 2026 3:40pm

Request Review

amankrx added 2 commits August 3, 2026 16:17
Redis failures reach REAPI clients as gRPC statuses, and Bazel treats
INVALID_ARGUMENT as permanent — it fails the build instead of retrying. The
mapping sent several purely transient conditions there, so an ordinary
sentinel failover ended in-flight builds.

Sentinel and cluster topology errors, dropped connections, and transient
server states (BUSYLOADING, CLUSTERDOWN, MASTERDOWN, TRYAGAIN, READONLY) are
now UNAVAILABLE. Parse and UnexpectedReturnType become INTERNAL: a malformed
reply is our fault or the server's, never the caller's argument. Genuine
misconfiguration — an unparseable client config, an empty sentinel list —
stays INVALID_ARGUMENT, since retrying cannot help there.

The connect path separately forced InvalidArgument onto every failure via
err_tip_with_code, which overrode the classification entirely; it now keeps
the underlying code. This is what made MasterNameNotFoundBySentinel
permanent no matter how the conversion classified it.

The sentinel test that pinned that behaviour is updated: an unknown master
name is indistinguishable from a failover in progress, and guessing between
them is not worth failing recoverable builds — a genuinely wrong name keeps
failing and stays visible.
test_sentinel_connect_with_url_specified_master asserts that the
sentinelServiceName URL parameter resolves, but allowed only 100ms to do
it — a value copied from the fail-fast bad-master test directly above,
where a tight budget is the actual subject.

That 100ms has to cover two TCP connects and two handshakes with a
SENTINEL MASTERS round trip in between. A loaded macOS runner took ~125ms
and the test failed on DeadlineExceeded. It is the only success-path test
in this file with a sub-second budget; the others allow 1s or 5s.

Unrelated to the error-classification change in this PR, but it fails on
its CI.
@amankrx
amankrx merged commit 56aa6f7 into TraceMachina:main Aug 3, 2026
49 of 51 checks passed
@corcillo corcillo mentioned this pull request Aug 4, 2026
2 of 4 tasks
b7r6 added a commit to b7r6/nativelink that referenced this pull request Sep 29, 2026
tonic codes a transport failure it cannot classify further (a connection
reset or broken pipe mid-stream) as Code::Unknown with message
"transport error"; tonic's own Status::from_error maps the shapes it
does recognize (e.g. a connect refusal) to Unavailable already. The
gRPC status model reserves UNKNOWN for genuinely unmapped errors and
codes transport failures UNAVAILABLE. Left as Unknown, such a blip is
treated as permanent by REAPI clients — Bazel retries UNAVAILABLE but
not UNKNOWN — and by any retry configuration keyed to Unavailable, so
a retryable failure becomes terminal and cacheable uploads are
silently dropped mid-write.

Detection is downcast-first: walk the Status source chain for a
concrete tonic::transport::Error. The fallback, for a Status
reconstructed from the wire where only the message survives, is
anchored to the two shapes that actually occur: tonic's exact
"transport error" rendering, and "transport error : <context>" as
produced by this crate's own From<Error> for Status when a peer
re-encodes and forwards one. An app-level Unknown that merely mentions
the phrase mid-message is not reclassified.

Upstream's TraceMachina#2657 made the same argument for transient Redis failures;
this applies it to the adjacent tonic transport surface.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
b7r6 added a commit to b7r6/nativelink that referenced this pull request Sep 29, 2026
tonic codes a transport failure it cannot classify further (a connection
reset or broken pipe mid-stream) as Code::Unknown with message
"transport error"; tonic's own Status::from_error maps the shapes it
does recognize (e.g. a connect refusal) to Unavailable already. The
gRPC status model reserves UNKNOWN for genuinely unmapped errors and
codes transport failures UNAVAILABLE. Left as Unknown, such a blip is
treated as permanent by REAPI clients — Bazel retries UNAVAILABLE but
not UNKNOWN — and by any retry configuration keyed to Unavailable, so
a retryable failure becomes terminal and cacheable uploads are
silently dropped mid-write.

Detection is downcast-first: walk the Status source chain for a
concrete tonic::transport::Error. The fallback, for a Status
reconstructed from the wire where only the message survives, is
anchored to the two shapes that actually occur: tonic's exact
"transport error" rendering, and "transport error : <context>" as
produced by this crate's own From<Error> for Status when a peer
re-encodes and forwards one. An app-level Unknown that merely mentions
the phrase mid-message is not reclassified.

Upstream's TraceMachina#2657 made the same argument for transient Redis failures;
this applies it to the adjacent tonic transport surface.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
b7r6 added a commit to b7r6/nativelink that referenced this pull request Oct 7, 2026
tonic codes a transport failure it cannot classify further (a connection
reset or broken pipe mid-stream) as Code::Unknown with message
"transport error"; tonic's own Status::from_error maps the shapes it
does recognize (e.g. a connect refusal) to Unavailable already. The
gRPC status model reserves UNKNOWN for genuinely unmapped errors and
codes transport failures UNAVAILABLE. Left as Unknown, such a blip is
treated as permanent by REAPI clients — Bazel retries UNAVAILABLE but
not UNKNOWN — and by any retry configuration keyed to Unavailable, so
a retryable failure becomes terminal and cacheable uploads are
silently dropped mid-write.

Detection is downcast-first: walk the Status source chain for a
concrete tonic::transport::Error. The fallback, for a Status
reconstructed from the wire where only the message survives, is
anchored to the two shapes that actually occur: tonic's exact
"transport error" rendering, and "transport error : <context>" as
produced by this crate's own From<Error> for Status when a peer
re-encodes and forwards one. An app-level Unknown that merely mentions
the phrase mid-message is not reclassified.

Upstream's TraceMachina#2657 made the same argument for transient Redis failures;
this applies it to the adjacent tonic transport surface.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
b7r6 added a commit to b7r6/nativelink that referenced this pull request Oct 7, 2026
tonic codes a transport failure it cannot classify further (a connection
reset or broken pipe mid-stream) as Code::Unknown with message
"transport error"; tonic's own Status::from_error maps the shapes it
does recognize (e.g. a connect refusal) to Unavailable already. The
gRPC status model reserves UNKNOWN for genuinely unmapped errors and
codes transport failures UNAVAILABLE. Left as Unknown, such a blip is
treated as permanent by REAPI clients — Bazel retries UNAVAILABLE but
not UNKNOWN — and by any retry configuration keyed to Unavailable, so
a retryable failure becomes terminal and cacheable uploads are
silently dropped mid-write.

Detection is downcast-first: walk the Status source chain for a
concrete tonic::transport::Error. The fallback, for a Status
reconstructed from the wire where only the message survives, is
anchored to the two shapes that actually occur: tonic's exact
"transport error" rendering, and "transport error : <context>" as
produced by this crate's own From<Error> for Status when a peer
re-encodes and forwards one. An app-level Unknown that merely mentions
the phrase mid-message is not reclassified.

Upstream's TraceMachina#2657 made the same argument for transient Redis failures;
this applies it to the adjacent tonic transport surface.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
b7r6 added a commit to b7r6/nativelink that referenced this pull request Oct 7, 2026
tonic codes a transport failure it cannot classify further (a connection
reset or broken pipe mid-stream) as Code::Unknown with message
"transport error"; tonic's own Status::from_error maps the shapes it
does recognize (e.g. a connect refusal) to Unavailable already. The
gRPC status model reserves UNKNOWN for genuinely unmapped errors and
codes transport failures UNAVAILABLE. Left as Unknown, such a blip is
treated as permanent by REAPI clients — Bazel retries UNAVAILABLE but
not UNKNOWN — and by any retry configuration keyed to Unavailable, so
a retryable failure becomes terminal and cacheable uploads are
silently dropped mid-write.

Detection is downcast-first: walk the Status source chain for a
concrete tonic::transport::Error. The fallback, for a Status
reconstructed from the wire where only the message survives, is
anchored to the two shapes that actually occur: tonic's exact
"transport error" rendering, and "transport error : <context>" as
produced by this crate's own From<Error> for Status when a peer
re-encodes and forwards one. An app-level Unknown that merely mentions
the phrase mid-message is not reclassified.

Upstream's TraceMachina#2657 made the same argument for transient Redis failures;
this applies it to the adjacent tonic transport surface.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
b7r6 added a commit to b7r6/nativelink that referenced this pull request Oct 8, 2026
tonic codes a transport failure it cannot classify further (a connection
reset or broken pipe mid-stream) as Code::Unknown with message
"transport error"; tonic's own Status::from_error maps the shapes it
does recognize (e.g. a connect refusal) to Unavailable already. The
gRPC status model reserves UNKNOWN for genuinely unmapped errors and
codes transport failures UNAVAILABLE. Left as Unknown, such a blip is
treated as permanent by REAPI clients — Bazel retries UNAVAILABLE but
not UNKNOWN — and by any retry configuration keyed to Unavailable, so
a retryable failure becomes terminal and cacheable uploads are
silently dropped mid-write.

Detection is downcast-first: walk the Status source chain for a
concrete tonic::transport::Error. The fallback, for a Status
reconstructed from the wire where only the message survives, is
anchored to the two shapes that actually occur: tonic's exact
"transport error" rendering, and "transport error : <context>" as
produced by this crate's own From<Error> for Status when a peer
re-encodes and forwards one. An app-level Unknown that merely mentions
the phrase mid-message is not reclassified.

Upstream's TraceMachina#2657 made the same argument for transient Redis failures;
this applies it to the adjacent tonic transport surface.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
b7r6 added a commit to b7r6/nativelink that referenced this pull request Oct 14, 2026
tonic codes a transport failure it cannot classify further (a connection
reset or broken pipe mid-stream) as Code::Unknown with message
"transport error"; tonic's own Status::from_error maps the shapes it
does recognize (e.g. a connect refusal) to Unavailable already. The
gRPC status model reserves UNKNOWN for genuinely unmapped errors and
codes transport failures UNAVAILABLE. Left as Unknown, such a blip is
treated as permanent by REAPI clients — Bazel retries UNAVAILABLE but
not UNKNOWN — and by any retry configuration keyed to Unavailable, so
a retryable failure becomes terminal and cacheable uploads are
silently dropped mid-write.

Detection is downcast-first: walk the Status source chain for a
concrete tonic::transport::Error. The fallback, for a Status
reconstructed from the wire where only the message survives, is
anchored to the two shapes that actually occur: tonic's exact
"transport error" rendering, and "transport error : <context>" as
produced by this crate's own From<Error> for Status when a peer
re-encodes and forwards one. An app-level Unknown that merely mentions
the phrase mid-message is not reclassified.

Upstream's TraceMachina#2657 made the same argument for transient Redis failures;
this applies it to the adjacent tonic transport surface.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
b7r6 added a commit to b7r6/nativelink that referenced this pull request Oct 15, 2026
tonic codes a transport failure it cannot classify further (a connection
reset or broken pipe mid-stream) as Code::Unknown with message
"transport error"; tonic's own Status::from_error maps the shapes it
does recognize (e.g. a connect refusal) to Unavailable already. The
gRPC status model reserves UNKNOWN for genuinely unmapped errors and
codes transport failures UNAVAILABLE. Left as Unknown, such a blip is
treated as permanent by REAPI clients — Bazel retries UNAVAILABLE but
not UNKNOWN — and by any retry configuration keyed to Unavailable, so
a retryable failure becomes terminal and cacheable uploads are
silently dropped mid-write.

Detection is downcast-first: walk the Status source chain for a
concrete tonic::transport::Error. The fallback, for a Status
reconstructed from the wire where only the message survives, is
anchored to the two shapes that actually occur: tonic's exact
"transport error" rendering, and "transport error : <context>" as
produced by this crate's own From<Error> for Status when a peer
re-encodes and forwards one. An app-level Unknown that merely mentions
the phrase mid-message is not reclassified.

Upstream's TraceMachina#2657 made the same argument for transient Redis failures;
this applies it to the adjacent tonic transport surface.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
b7r6 added a commit to b7r6/nativelink that referenced this pull request Oct 15, 2026
tonic codes a transport failure it cannot classify further (a connection
reset or broken pipe mid-stream) as Code::Unknown with message
"transport error"; tonic's own Status::from_error maps the shapes it
does recognize (e.g. a connect refusal) to Unavailable already. The
gRPC status model reserves UNKNOWN for genuinely unmapped errors and
codes transport failures UNAVAILABLE. Left as Unknown, such a blip is
treated as permanent by REAPI clients — Bazel retries UNAVAILABLE but
not UNKNOWN — and by any retry configuration keyed to Unavailable, so
a retryable failure becomes terminal and cacheable uploads are
silently dropped mid-write.

Detection is downcast-first: walk the Status source chain for a
concrete tonic::transport::Error. The fallback, for a Status
reconstructed from the wire where only the message survives, is
anchored to the two shapes that actually occur: tonic's exact
"transport error" rendering, and "transport error : <context>" as
produced by this crate's own From<Error> for Status when a peer
re-encodes and forwards one. An app-level Unknown that merely mentions
the phrase mid-message is not reclassified.

Upstream's TraceMachina#2657 made the same argument for transient Redis failures;
this applies it to the adjacent tonic transport surface.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
palfrey pushed a commit that referenced this pull request Oct 15, 2026
tonic codes a transport failure it cannot classify further (a connection
reset or broken pipe mid-stream) as Code::Unknown with message
"transport error"; tonic's own Status::from_error maps the shapes it
does recognize (e.g. a connect refusal) to Unavailable already. The
gRPC status model reserves UNKNOWN for genuinely unmapped errors and
codes transport failures UNAVAILABLE. Left as Unknown, such a blip is
treated as permanent by REAPI clients — Bazel retries UNAVAILABLE but
not UNKNOWN — and by any retry configuration keyed to Unavailable, so
a retryable failure becomes terminal and cacheable uploads are
silently dropped mid-write.

Detection is downcast-first: walk the Status source chain for a
concrete tonic::transport::Error. The fallback, for a Status
reconstructed from the wire where only the message survives, is
anchored to the two shapes that actually occur: tonic's exact
"transport error" rendering, and "transport error : <context>" as
produced by this crate's own From<Error> for Status when a peer
re-encodes and forwards one. An app-level Unknown that merely mentions
the phrase mid-message is not reclassified.

Upstream's #2657 made the same argument for transient Redis failures;
this applies it to the adjacent tonic transport surface.

This branch was successfully deployed

2 active deployments
Preview – nativelink — 63023e73 Deployed Aug 3, 2026 by vercel[bot]
Preview – nativelink-aidm — 63023e73 Deployed Aug 3, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants