Repository navigation
Map transport-originated Unknown to retryable Unavailable - #2816
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
c9f4ea1 to
4e6bf02
Compare
4e6bf02 to
138d75e
Compare
MarcusSorealheis
left a comment
There was a problem hiding this comment.
Reviewed the source-chain classification, the narrow message fallback for proxied transport failures, and preservation of ordinary Unknown statuses. No blocking findings. Ran cargo test -p nativelink-error at c8cb8ca: all 44 tests passed, including a real tonic transport error, appended proxy context, preservation of application Unknown, and the error round trip.
|
Withdrawing this from the queue to respect the team's review bandwidth. It's a verified fix (fails-without regression, retained on our fork) — rather than keep a flood of drafts open, we'll re-propose one clean, reviewed PR at a time. Happy to prioritize any of these on request. |
tonic codes a transport failure it cannot classify further (a connection reset or broken pipe mid-stream) as Code::Unknown with message "transport error"; tonic's own Status::from_error maps the shapes it does recognize (e.g. a connect refusal) to Unavailable already. The gRPC status model reserves UNKNOWN for genuinely unmapped errors and codes transport failures UNAVAILABLE. Left as Unknown, such a blip is treated as permanent by REAPI clients — Bazel retries UNAVAILABLE but not UNKNOWN — and by any retry configuration keyed to Unavailable, so a retryable failure becomes terminal and cacheable uploads are silently dropped mid-write. Detection is downcast-first: walk the Status source chain for a concrete tonic::transport::Error. The fallback, for a Status reconstructed from the wire where only the message survives, is anchored to the two shapes that actually occur: tonic's exact "transport error" rendering, and "transport error : <context>" as produced by this crate's own From<Error> for Status when a peer re-encodes and forwards one. An app-level Unknown that merely mentions the phrase mid-message is not reclassified. Upstream's TraceMachina#2657 made the same argument for transient Redis failures; this applies it to the adjacent tonic transport surface. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
c8cb8ca to
9f48f1f
Compare
What and why
When tonic cannot classify a transport failure further, such as a connection reset or broken pipe mid-stream, it reports
Code::Unknownwith the message "transport error". The gRPC status model reservesUNKNOWNfor errors nobody mapped and expects transport failures to beUNAVAILABLE, and tonic already maps the shapes it does recognize (a refused connect, for example) that way. Left asUnknown, a network blip looks permanent: Bazel retriesUNAVAILABLEbut notUNKNOWN, and so does any retry config keyed onUnavailable. A retryable failure becomes terminal, and an upload is dropped mid-write.The conversion in
From<tonic::Status> for Errornow walks the status's source chain for atonic::transport::Errorand maps it toUnavailable. A status rebuilt from the wire keeps only its message, so there is also a narrow fallback on the message. It matches tonic's exact "transport error" text, and the "transport error : " form this crate produces when a peer re-encodes one and forwards it. An application-levelUnknownthat merely mentions the phrase is left alone. This applies the reasoning of #2657 (transient Redis failures) to the tonic transport path.How was this verified?
nativelink-error/tests/error_tests.rsadds six tests:Unavailable: a transport error in the source chain, a bare "transport error" status, a forwarded "transport error : " status, and a round trip throughErrorand back toStatus.Unknown: a genuineUnknown, and anUnknownthat mentions "transport error" mid-message.On 8bf6be1 without the fix, the four remap tests fail (
left: Unknown, right: Unavailable). The two must-stay-Unknowntests pass both ways, by design.Tom and Marcus both approved this before I withdrew it (2026-09-29 and 2026-09-30). Apart from the rebase, the only change in the diff is context lines that moved under the licence header added since.
Risk
Low. Only
Unknownstatuses that come from the transport change code, and the effect is that clients retry them. A misclassification would turn a realUnknowninto a retry, and the fallback is anchored to the exact message forms to avoid that.AI assistance
Claude wrote the fix and tests under my direction, checked that they fail without the fix, and drafted this description.