Skip to content

fix(store): wake in-flight slow-write waiters when any upload completes the digest - #36

Open
b7r6 wants to merge 2 commits into
mainfrom
b7r6/fastslow-inflight-waiter-fresh-upload
Open

b7r6 wants to merge 2 commits into
mainfrom
b7r6/fastslow-inflight-waiter-fresh-upload

Conversation

@b7r6

@b7r6 b7r6 commented Oct 7, 2026 •

Copy link
Copy Markdown
Owner

What and why

Lane A stability fix from the findings catalog (see nativelink-findings-2026-10-05.csv
on the meta/findings-catalog-2026-10-05 branch), rebased onto OSS main
@bf4b3aa855. This fork PR exists to validate the change on CI that is
representative of upstream PR lanes (public read-only cache, same endpoints)
before any upstream submission.

How was this verified?

The branch carries its own regression tests (fails-without where the fix
class allows it — see the branch's test commits). This PR's purpose is the
full representative CI run; green here rehearses the exact checks an
upstream PR would face.

Risk

Fork-internal validation PR; nothing lands anywhere by merging this. The
underlying fix's risk assessment happens at upstream-submission time.

AI assistance

Agent-assisted pipeline (Claude) for rebase and PR mechanics under my
direction; the fix itself carries verification in its tests.

@github-actions

This comment has been minimized.

@b7r6
b7r6 force-pushed the b7r6/fastslow-inflight-waiter-fresh-upload branch 3 times, most recently from 11d0614 to 77dc359 Compare October 8, 2026 17:51
@b7r6
b7r6 force-pushed the b7r6/fastslow-inflight-waiter-fresh-upload branch 2 times, most recently from 89091d2 to f6fa0da Compare October 15, 2026 11:19
Completes the no-per-PR-OCI-builds intent: image-off-pr moved image.yaml off
PRs and TraceMachina#2877 moved the nix integration job off PRs, but main.yaml's
integration-tests (docker-compose image build + run_integration_tests.sh)
fell between them and still ran on every PR. Defer to nightly + merge-to-main,
matching TraceMachina#2880's pattern. Fork-local until folded into the upstream OCI/nightly
set.
…es the digest

An existence check that found a slow-store write in flight parked on that
specific writer's completion mutex. A ByteStream upload that is interrupted
(cancel, half-close, client deadline) is parked for resumption, keeping its
FastSlowStore::update() future -- and thus the locked mutex -- alive. A later
registration for the same key overwrote the map entry, so completing the
digest under a fresh upload id never woke the original waiters: they starved
until their client deadline even though the blob was present and new lookups
succeeded.

Replace the per-writer mutex with a per-key slot shared by all concurrent
writers of that key, broadcasting Pending/Done over a watch channel. Any
writer completing the digest wakes every waiter; the entry is removed (and
waiters released with "not written") only when the last writer deregisters,
which also fixes the stale-remove where a second writer's guard evicted a
slot the first writer still occupied.

Regression: stalled_upload_waiter_released_by_fresh_upload_of_same_digest --
fails on v1.7.3 (waiter hangs past a 2s deadline), passes with the fix; also
asserts a genuinely-absent digest still resolves promptly.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant