Skip to content

Write every upload through the existence cache instead of discarding one it believes present - #2951

Merged
amankrx merged 2 commits into
TraceMachina:mainfrom
amankrx:fix/existence-cache-write-through
Oct 16, 2026
Merged

amankrx merged 2 commits into
TraceMachina:mainfrom
amankrx:fix/existence-cache-write-through

Conversation

@amankrx

@amankrx amankrx commented Oct 15, 2026

Copy link
Copy Markdown
Member

What and why

ExistenceCacheStore::update drained and discarded any upload for a digest the cache believed present. When the tier under it loses the blob and the remove callback never arrives (a Redis under allkeys-lru whose keyspace events are dropped, as seen on a production CAS), the cache answers present for the rest of its TTL, and a client that uploads anyway loses the blob too. The read then fails not found in either fast or slow store and Bazel exits 39.

Every upload now writes through, whatever the cache believes, and refreshes the entry. Clients only upload what they were told is missing, so nothing is wasted in the normal flow. The callback miss log drops to debug: the subscription delivers every key event on the server.

How was this verified?

A new test loses a blob below the cache without a callback, re-uploads it and asserts it lands and reads back; it fails on main. A second pins that a positive older than max_seconds asks the inner store again. On a three-pod test cluster an upload to a pod with a stale positive never reached Redis before, and does now.

Risk

Low. Only the early return goes; a stale positive is still bounded by max_seconds.

AI assistance

An agent drafted the change and I reviewed every line.

@vercel

vercel Bot commented Oct 15, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
nativelink Ready Ready Preview Oct 15, 2026 8:42pm UTC
nativelink-aidm Ready Ready Preview Oct 15, 2026 8:42pm UTC

Request Review

return Ok(size);
}
// Every upload reaches the inner store, whatever this cache believes.
// A client only uploads what it was told is missing, so the write is

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Problem: "A client only uploads what it was told is missing, so the write is never wasted in the normal flow" doesn't hold for concurrent uploads. Two clients told "missing" at the same time both upload. Before, the second was usually dropped, because inner_has_with_results asked the inner store for a digest the cache didn't hold. Now both write through to every tier, including the object store. Content addressing makes that harmless, but it's a cost the comment and the description say isn't there.

Fix: reword so the trade-off is stated:

// Every upload reaches the inner store, whatever this cache believes:
// when the tier under it lost the blob and the remove callback did not
// arrive, discarding the upload turns a stale answer into a blob nothing
// holds. Concurrent uploads of the same blob now both write, which
// content addressing makes harmless. The entry is refreshed on success.

Alternatively, keep the deduplication without trusting the cache: skip the cache and ask self.inner_store.has(digest) directly, dropping the upload only when the inner store confirms it. FastSlowStore::has_with_results never checks the fast tier, so that answer comes from the durable store. The cost is one has per upload, which the old code already paid for every digest the cache didn't hold. I'm fine either way; it's your call.

// never wasted in the normal flow; and the one time the cache is
// wrong (the tier under it lost the blob and the remove callback did
// not arrive), discarding the upload is what turns a stale answer
// into a blob nothing holds. The entry is refreshed on success.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Other commenting nit: Problem: the entry is refreshed when an upload succeeds, but nothing removes it when a read proves it wrong. After a silent eviction, has keeps answering present for up to max_seconds. A client that trusts FindMissingBlobs never re-uploads, so its build fails reading the blob for the whole window. This PR covers the client that uploads anyway, not that one.

Fix: in get_part (outside this diff), drop the entry when the inner store says the blob is gone:

let result = self
    .inner_store
    .get_part(digest, writer, offset, length)
    .await;
match &result {
    Ok(()) => {
        let _ = self
            .existence_cache
            .insert(digest, ExistenceItem(digest.size_bytes()))
            .await;
    }
    Err(err) if err.code == Code::NotFound => {
        self.existence_cache.remove(&digest).await;
    }
    Err(_) => {}
}


/// A store that loses entries without telling anyone: Redis under
/// allkeys-lru whose keyspace events never reach the subscriber, which is
/// what the September 2026 capture showed. The remove callback is accepted

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shouldnt reference this instead describe thing that was observed or nothing

@amankrx
amankrx merged commit d05d910 into TraceMachina:main Oct 16, 2026
40 of 42 checks passed

This branch was successfully deployed

2 active deployments
Preview – nativelink — 51721bba Deployed Oct 15, 2026 by vercel[bot]
Preview – nativelink-aidm — 51721bba Deployed Oct 15, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants