Skip to content

Finish an upload's commit on its own task when the client leaves after its last byte, instead of parking it unpolled in the store's write queue - #3044

Merged
amankrx merged 4 commits into
TraceMachina:mainfrom
amankrx:fix/bytestream-finish-commit-on-disconnect
Oct 26, 2026
Merged

amankrx merged 4 commits into
TraceMachina:mainfrom
amankrx:fix/bytestream-finish-commit-on-disconnect

Conversation

@amankrx

@amankrx amankrx commented Oct 25, 2026 •

Copy link
Copy Markdown
Member

What and why

A ByteStream client that leaves after its last byte leaves its upload parked, with the store update unpolled, until the sweeper drops it. A filesystem store waits for a max_concurrent_writes permit at exactly that point. The parked waiter keeps its place in the queue and holds the permit it gets until swept. Enough of those and no upload commits; a production CAS had 98% of writes over 60 s until a restart.

After the last byte, a dropped handler now hands the commit to its own task, bounded at 15 minutes, and parks a future that waits for it. A store update that already returned is never polled again: a commit stays parked as its result for a resume, a failure is dropped.

How was this verified?

New tests cover the abandoned commit, a store failing after EOF, a retry that sees the commit finish and then disconnects, and a resume at the full size. All but the last fail without the change. On a test cluster with one write permit and 32 clients abandoning uploads, a fresh upload took 90 s before and 1.8 s after.

Risk

Low. Partial uploads park and resume as before.

AI assistance

An agent drafted the change and I reviewed every line.

…r its last byte, instead of parking it unpolled in the store's write queue
@vercel

vercel Bot commented Oct 25, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
nativelink Ready Ready Preview Oct 26, 2026 1:54am UTC
nativelink-aidm Ready Ready Preview Oct 26, 2026 1:54am UTC

Request Review

@corcillo

Copy link
Copy Markdown
Collaborator

The fix is right for the stall it targets, and I checked the new test fails without it. But the new branch in Drop also runs after a store error that happens after the last byte, and then it re-polls a finished future. I reproduced that as a panic, with a misleading Internal error on retry. That needs a fix and a test before merge.

@corcillo

Copy link
Copy Markdown
Collaborator

Moving this to request changes for the panic at line 472. Under load with the slow store paused for 2 minutes, this branch logged 36 async fn resumed after completion panics and the base logged none, so it isn't a rare path. The rest of the fix holds up well: probe writes on uploads abandoned after the last byte went from p50 74 s to 54 ms, and the server recovered after the flood where the base never did.

@amankrx
amankrx merged commit 779c3e7 into TraceMachina:main Oct 26, 2026
14 checks passed

This branch was successfully deployed

2 active deployments
Preview – nativelink — 95492cc5 Deployed Oct 26, 2026 by vercel[bot]
Preview – nativelink-aidm — 95492cc5 Deployed Oct 26, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants