Skip to content

Honor Retry-After on throttled Azure blob responses - #2931

Merged
amankrx merged 2 commits into
TraceMachina:mainfrom
amankrx:fix/azure-throttle-retry
Oct 13, 2026
Merged

amankrx merged 2 commits into
TraceMachina:mainfrom
amankrx:fix/azure-throttle-retry

Conversation

@amankrx

@amankrx amankrx commented Oct 13, 2026 •

Copy link
Copy Markdown
Member

What and why

A storage account over its egress limit answers 503 ServerBusy for minutes, and the Azure store treated that like any transient error: the retrier's budget (about 38 s with 6 retries from 0.3 s) ran out, the action failed and was re-dispatched to fetch the same blobs again. One customer's workers failed 786,837 input fetches that way in 18 hours, all on the seventh attempt.

The Azure store classifies 503 ServerBusy and 429 as throttles and reads retry-after-ms, x-ms-retry-after-ms or retry-after. The retrier gains RetryResult::RetryAfter(err, wait) and a throttle budget: within throttle_retry_budget_s (default 120) of one operation each throttle waits what the server asked, between throttle_retry_delay_s (2) and throttle_retry_max_delay_s (60, a longer header is capped and logged), and spends none of retry.max_retries. Past the budget, throttles spend attempts and backoff like any other error, so a throttle that never ends still fails. Counter store.object.throttles{provider,op}.

How was this verified?

Retrier tests for the hint and the backoff, throttles inside and past the budget, and a zero wait; Azure store tests under paused time for the header, the floor, the cap and three throttles on zero allowed retries. nativelink-util, config and store suites and clippy pass.

Risk

Low. Non-throttled errors keep their path; a throttled fetch holds its slot for up to the budget, bounded.

AI assistance

An agent drafted the change and I reviewed every line.

@vercel

vercel Bot commented Oct 13, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
nativelink Ready Ready Preview Oct 13, 2026 4:12pm UTC
nativelink-aidm Ready Ready Preview Oct 13, 2026 4:12pm UTC

Request Review

Comment thread nativelink-util/src/retry.rs Resolved
corcillo
corcillo previously approved these changes Oct 13, 2026
@amankrx
amankrx merged commit 00d3b7f into TraceMachina:main Oct 13, 2026
43 checks passed
MarcusSorealheis added a commit to corcillo/nativelink that referenced this pull request Oct 14, 2026
Resolve conflicts with Retry-After support (TraceMachina#2931): the retrier keeps
both its client label and outcome hook and the throttle budget, and
Azure sets both. A throttle waited out within the budget reports
`retried`, and a success after one reports `recovered`, though it spends
no attempt. Regenerate the metrics reference.
b7r6 added a commit to b7r6/nativelink that referenced this pull request Oct 14, 2026
…pl added upstream (TraceMachina#2931)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch was successfully deployed

2 active deployments
Preview – nativelink — 6bd7b0f0 Deployed Oct 13, 2026 by vercel[bot]
Preview – nativelink-aidm — 6bd7b0f0 Deployed Oct 13, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants