Skip to content

Map transport-originated Unknown to retryable Unavailable - #2816

Merged
palfrey merged 2 commits into
TraceMachina:mainfrom
b7r6:pr/transport-unknown-retryable
Oct 15, 2026
Merged

palfrey merged 2 commits into
TraceMachina:mainfrom
b7r6:pr/transport-unknown-retryable

Conversation

@b7r6

@b7r6 b7r6 commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

What and why

When tonic cannot classify a transport failure further, such as a connection reset or broken pipe mid-stream, it reports Code::Unknown with the message "transport error". The gRPC status model reserves UNKNOWN for errors nobody mapped and expects transport failures to be UNAVAILABLE, and tonic already maps the shapes it does recognize (a refused connect, for example) that way. Left as Unknown, a network blip looks permanent: Bazel retries UNAVAILABLE but not UNKNOWN, and so does any retry config keyed on Unavailable. A retryable failure becomes terminal, and an upload is dropped mid-write.

The conversion in From<tonic::Status> for Error now walks the status's source chain for a tonic::transport::Error and maps it to Unavailable. A status rebuilt from the wire keeps only its message, so there is also a narrow fallback on the message. It matches tonic's exact "transport error" text, and the "transport error : " form this crate produces when a peer re-encodes one and forwards it. An application-level Unknown that merely mentions the phrase is left alone. This applies the reasoning of #2657 (transient Redis failures) to the tonic transport path.

How was this verified?

nativelink-error/tests/error_tests.rs adds six tests:

  • Remapped to Unavailable: a transport error in the source chain, a bare "transport error" status, a forwarded "transport error : " status, and a round trip through Error and back to Status.
  • Left as Unknown: a genuine Unknown, and an Unknown that mentions "transport error" mid-message.

On 8bf6be1 without the fix, the four remap tests fail (left: Unknown, right: Unavailable). The two must-stay-Unknown tests pass both ways, by design.

Tom and Marcus both approved this before I withdrew it (2026-09-29 and 2026-09-30). Apart from the rebase, the only change in the diff is context lines that moved under the licence header added since.

Risk

Low. Only Unknown statuses that come from the transport change code, and the effect is that clients retry them. A misclassification would turn a real Unknown into a retry, and the fallback is anchored to the exact message forms to avoid that.

AI assistance

Claude wrote the fix and tests under my direction, checked that they fail without the fix, and drafted this description.

@vercel

vercel Bot commented Sep 29, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
nativelink Ready Ready Preview Oct 15, 2026 2:01pm UTC
nativelink-aidm Ready Ready Preview Oct 15, 2026 2:01pm UTC

Request Review

Comment thread nativelink-error/src/lib.rs Resolved
palfrey
palfrey previously approved these changes Sep 29, 2026

@MarcusSorealheis MarcusSorealheis left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the source-chain classification, the narrow message fallback for proxied transport failures, and preservation of ordinary Unknown statuses. No blocking findings. Ran cargo test -p nativelink-error at c8cb8ca: all 44 tests passed, including a real tonic transport error, appended proxy context, preservation of application Unknown, and the error round trip.

@b7r6

b7r6 commented Oct 3, 2026

Copy link
Copy Markdown
Contributor Author

Withdrawing this from the queue to respect the team's review bandwidth. It's a verified fix (fails-without regression, retained on our fork) — rather than keep a flood of drafts open, we'll re-propose one clean, reviewed PR at a time. Happy to prioritize any of these on request.

@b7r6 b7r6 closed this Oct 3, 2026
tonic codes a transport failure it cannot classify further (a connection
reset or broken pipe mid-stream) as Code::Unknown with message
"transport error"; tonic's own Status::from_error maps the shapes it
does recognize (e.g. a connect refusal) to Unavailable already. The
gRPC status model reserves UNKNOWN for genuinely unmapped errors and
codes transport failures UNAVAILABLE. Left as Unknown, such a blip is
treated as permanent by REAPI clients — Bazel retries UNAVAILABLE but
not UNKNOWN — and by any retry configuration keyed to Unavailable, so
a retryable failure becomes terminal and cacheable uploads are
silently dropped mid-write.

Detection is downcast-first: walk the Status source chain for a
concrete tonic::transport::Error. The fallback, for a Status
reconstructed from the wire where only the message survives, is
anchored to the two shapes that actually occur: tonic's exact
"transport error" rendering, and "transport error : <context>" as
produced by this crate's own From<Error> for Status when a peer
re-encodes and forwards one. An app-level Unknown that merely mentions
the phrase mid-message is not reclassified.

Upstream's TraceMachina#2657 made the same argument for transient Redis failures;
this applies it to the adjacent tonic transport surface.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@palfrey
palfrey marked this pull request as ready for review October 15, 2026 13:11
@b7r6
b7r6 deleted the pr/transport-unknown-retryable branch October 15, 2026 19:11

This branch was successfully deployed

2 active deployments
Preview – nativelink — 9886a84b Deployed Oct 15, 2026 by vercel[bot]
Preview – nativelink-aidm — 9886a84b Deployed Oct 15, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants