Skip to content

Bound the requests an Azure store keeps in flight, uploads included - #2985

Merged
amankrx merged 3 commits into
TraceMachina:mainfrom
MarcusSorealheis:bugfix/azure-request-limit
Oct 26, 2026
Merged

amankrx merged 3 commits into
TraceMachina:mainfrom
MarcusSorealheis:bugfix/azure-request-limit

Conversation

@MarcusSorealheis

@MarcusSorealheis MarcusSorealheis commented Oct 20, 2026 •

Copy link
Copy Markdown
Member

What and why

A CAS pod ran out of local ports toward one storage endpoint, and connect() failed with AddrNotAvailable. Nothing limited the connections one Azure store opens:

  • Azure Blob Storage speaks HTTP/1.1, and the store's TLS config offers no ALPN, so every request in flight holds a connection.
  • The only limit on requests in flight was the process-wide open-file permit (24,576 by default). That is close to Linux's default range of 28,232 local ports.
  • max_concurrent_reads (Bound object store reads with an adaptive permit pool, Azure first #2941) limits downloads only. Existence checks, uploads, block staging and commits had no limit, and has_with_results sends a HEAD for every key in a batch at once.

This adds max_concurrent_requests to CommonObjectSpec, default 0 (no limit, unchanged behavior). When it is set, every Azure request takes a permit first:

  • HEAD;
  • GET, held through the body;
  • the single PUT for small uploads;
  • each staged block;
  • the block list commit;
  • the cleanup after a failed upload.

At the limit, requests wait in the process, first come first served. Nothing is sent or refused, so the cost is latency. Waits are counted on store.object.request_waits.

A download holds its permit while its body streams, so the store refuses to start with max_concurrent_requests set unless max_concurrent_reads is between 1 and one less than it: downloads then hold at most that many permits and existence checks and uploads keep the rest. The store's HTTP client now has the SDK's own 20s connect and 60s read timeouts, so a hung request fails and is retried instead of holding its permit. A failed staged upload discards its blocks from the guard in the background, so the error returns at once and a cancelled caller still discards.

A request that fails with AddrNotAvailable anywhere in its error chain now logs a warning, at most once a minute per store. The warning says the host is out of local ports and names the setting: set it when unset, lower it when set. Like max_concurrent_reads, only the Azure store honors the setting so far.

This is meant for 1.8.1, not 1.8.0. Please merge after v1.8.0 is tagged.

Are idle connections holding ports, and are they never deleted? From the vendored sources:

  • reqwest keeps idle connections for 90s with no limit per host (reqwest-0.13.4/src/async_impl/client.rs:301-302), and the store changes neither setting.
  • hyper-util evicts idle connections after that timeout (hyper-util-0.1.17/src/client/legacy/pool.rs:477-494).
  • A ranged GET returns its connection to the pool once its last byte is read (hyper-1.7.0/src/proto/h1/conn.rs:373-378).
  • A body dropped early is drained or the connection is closed (dispatch.rs:209-213). The local side then holds the port in TIME_WAIT.

So nothing is leaked. But after a burst, every connection used at the peak holds a port for about 90s idle plus TIME_WAIT. With this limit set, the pool can't grow beyond it.

How was this verified?

Four new tests in azure_blob_store_test. They use the mock transport, extended to hold each request and count how many are in flight.

  • 64 concurrent existence checks: at most 4 in flight with a limit of 4, and all 64 without a limit.
  • 16 small uploads plus a 4-block upload together: at most 3 in flight with a limit of 3, and more without one.
  • Repeated out-of-ports failures log the warning once per store, with the right wording for each case.
  • A real reqwest client bound to an unassigned address (192.0.2.1) fails with AddrNotAvailable through the same hyper-util/reqwest/SDK error chain as a host out of ports, and the check detects it. Connection-refused and HTTP 503 errors are not detected.

With the pool and the warning disabled, the 3 tests that cover them fail. With the change, all 39 tests in the file pass. The nativelink-config tests, clippy on the changed crates and pre-commit also pass. The config and metrics references were regenerated.

Not verified:

  • a real Azure account or a real port-exhaustion reproduction;
  • the metric (no test checks it), and the timeouts (the mock transport bypasses reqwest);
  • Bazel, which was not run locally.

Risk

The setting is off by default, so existing deployments behave as before. The one exception is small uploads, which now read their content before taking the open-file permit, so a slow producer holds no permit.

When the limit is set:

  • Slow readers: downloads are held to max_concurrent_reads, which must be below the limit, so slow readers only block other downloads. Tested: eight held downloads at reads 2 / requests 4 leave an existence check waiting one request, 30ms behind them without the gap.
  • Shards: each store in a shard has its own limit, so N shards can hold N times this.
  • Timeouts: the 20s connect and 60s read timeouts now apply to every Azure request, set or not. The read timer runs only while the body is polled, so a download a slow reader holds back is not cut off.

The AddrNotAvailable check only reads errors. #2986 adds AzureBlobStore::remove in the same file; whichever merges second should make remove take a request permit.

AI assistance

Claude Code wrote the change, the tests and this description at Marcus Eagan's direction, and checked the client pool behavior against the vendored crate sources. A maintainer should review the permit order and the hung-request risk.

🤖 Generated with Claude Code

https://claude.ai/code/session_01HBxQtV9S561AzBo7aPCYFj

@vercel

vercel Bot commented Oct 20, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
nativelink Ready Ready Preview Oct 26, 2026 11:47am UTC
nativelink-aidm Ready Ready Preview Oct 26, 2026 11:47am UTC

Request Review

…ts, and discard a failed upload's blocks in the background
corcillo
corcillo previously approved these changes Oct 26, 2026
@amankrx
amankrx merged commit 4a6f505 into TraceMachina:main Oct 26, 2026
16 of 17 checks passed

This branch was successfully deployed

2 active deployments
Preview – nativelink — 8b66d744 Deployed Oct 26, 2026 by vercel[bot]
Preview – nativelink-aidm — 8b66d744 Deployed Oct 26, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants