Repository navigation
Bound the requests an Azure store keeps in flight, uploads included - #2985
Merged
amankrx merged 3 commits intoOct 26, 2026
Merged
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
amankrx
force-pushed
the
bugfix/azure-request-limit
branch
from
October 26, 2026 11:11
e794709 to
9c222ac
Compare
corcillo
reviewed
Oct 26, 2026
corcillo
reviewed
Oct 26, 2026
corcillo
reviewed
Oct 26, 2026
…ts, and discard a failed upload's blocks in the background
corcillo
previously approved these changes
Oct 26, 2026
corcillo
approved these changes
Oct 26, 2026
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
A CAS pod ran out of local ports toward one storage endpoint, and
connect()failed withAddrNotAvailable. Nothing limited the connections one Azure store opens:max_concurrent_reads(Bound object store reads with an adaptive permit pool, Azure first #2941) limits downloads only. Existence checks, uploads, block staging and commits had no limit, andhas_with_resultssends a HEAD for every key in a batch at once.This adds
max_concurrent_requeststoCommonObjectSpec, default 0 (no limit, unchanged behavior). When it is set, every Azure request takes a permit first:At the limit, requests wait in the process, first come first served. Nothing is sent or refused, so the cost is latency. Waits are counted on
store.object.request_waits.A download holds its permit while its body streams, so the store refuses to start with
max_concurrent_requestsset unlessmax_concurrent_readsis between 1 and one less than it: downloads then hold at most that many permits and existence checks and uploads keep the rest. The store's HTTP client now has the SDK's own 20s connect and 60s read timeouts, so a hung request fails and is retried instead of holding its permit. A failed staged upload discards its blocks from the guard in the background, so the error returns at once and a cancelled caller still discards.A request that fails with
AddrNotAvailableanywhere in its error chain now logs a warning, at most once a minute per store. The warning says the host is out of local ports and names the setting: set it when unset, lower it when set. Likemax_concurrent_reads, only the Azure store honors the setting so far.This is meant for 1.8.1, not 1.8.0. Please merge after v1.8.0 is tagged.
Are idle connections holding ports, and are they never deleted? From the vendored sources:
reqwest-0.13.4/src/async_impl/client.rs:301-302), and the store changes neither setting.hyper-util-0.1.17/src/client/legacy/pool.rs:477-494).hyper-1.7.0/src/proto/h1/conn.rs:373-378).dispatch.rs:209-213). The local side then holds the port in TIME_WAIT.So nothing is leaked. But after a burst, every connection used at the peak holds a port for about 90s idle plus TIME_WAIT. With this limit set, the pool can't grow beyond it.
How was this verified?
Four new tests in
azure_blob_store_test. They use the mock transport, extended to hold each request and count how many are in flight.AddrNotAvailablethrough the same hyper-util/reqwest/SDK error chain as a host out of ports, and the check detects it. Connection-refused and HTTP 503 errors are not detected.With the pool and the warning disabled, the 3 tests that cover them fail. With the change, all 39 tests in the file pass. The
nativelink-configtests, clippy on the changed crates and pre-commit also pass. The config and metrics references were regenerated.Not verified:
Risk
The setting is off by default, so existing deployments behave as before. The one exception is small uploads, which now read their content before taking the open-file permit, so a slow producer holds no permit.
When the limit is set:
max_concurrent_reads, which must be below the limit, so slow readers only block other downloads. Tested: eight held downloads at reads 2 / requests 4 leave an existence check waiting one request, 30ms behind them without the gap.shardhas its own limit, so N shards can hold N times this.The
AddrNotAvailablecheck only reads errors. #2986 addsAzureBlobStore::removein the same file; whichever merges second should makeremovetake a request permit.AI assistance
Claude Code wrote the change, the tests and this description at Marcus Eagan's direction, and checked the client pool behavior against the vendored crate sources. A maintainer should review the permit order and the hung-request risk.
🤖 Generated with Claude Code
https://claude.ai/code/session_01HBxQtV9S561AzBo7aPCYFj