Repository navigation
Write every upload through the existence cache instead of discarding one it believes present - #2951
Conversation
…one it believes present
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
| return Ok(size); | ||
| } | ||
| // Every upload reaches the inner store, whatever this cache believes. | ||
| // A client only uploads what it was told is missing, so the write is |
There was a problem hiding this comment.
Problem: "A client only uploads what it was told is missing, so the write is never wasted in the normal flow" doesn't hold for concurrent uploads. Two clients told "missing" at the same time both upload. Before, the second was usually dropped, because inner_has_with_results asked the inner store for a digest the cache didn't hold. Now both write through to every tier, including the object store. Content addressing makes that harmless, but it's a cost the comment and the description say isn't there.
Fix: reword so the trade-off is stated:
// Every upload reaches the inner store, whatever this cache believes:
// when the tier under it lost the blob and the remove callback did not
// arrive, discarding the upload turns a stale answer into a blob nothing
// holds. Concurrent uploads of the same blob now both write, which
// content addressing makes harmless. The entry is refreshed on success.Alternatively, keep the deduplication without trusting the cache: skip the cache and ask self.inner_store.has(digest) directly, dropping the upload only when the inner store confirms it. FastSlowStore::has_with_results never checks the fast tier, so that answer comes from the durable store. The cost is one has per upload, which the old code already paid for every digest the cache didn't hold. I'm fine either way; it's your call.
| // never wasted in the normal flow; and the one time the cache is | ||
| // wrong (the tier under it lost the blob and the remove callback did | ||
| // not arrive), discarding the upload is what turns a stale answer | ||
| // into a blob nothing holds. The entry is refreshed on success. |
There was a problem hiding this comment.
Other commenting nit: Problem: the entry is refreshed when an upload succeeds, but nothing removes it when a read proves it wrong. After a silent eviction, has keeps answering present for up to max_seconds. A client that trusts FindMissingBlobs never re-uploads, so its build fails reading the blob for the whole window. This PR covers the client that uploads anyway, not that one.
Fix: in get_part (outside this diff), drop the entry when the inner store says the blob is gone:
let result = self
.inner_store
.get_part(digest, writer, offset, length)
.await;
match &result {
Ok(()) => {
let _ = self
.existence_cache
.insert(digest, ExistenceItem(digest.size_bytes()))
.await;
}
Err(err) if err.code == Code::NotFound => {
self.existence_cache.remove(&digest).await;
}
Err(_) => {}
}|
|
||
| /// A store that loses entries without telling anyone: Redis under | ||
| /// allkeys-lru whose keyspace events never reach the subscriber, which is | ||
| /// what the September 2026 capture showed. The remove callback is accepted |
There was a problem hiding this comment.
shouldnt reference this instead describe thing that was observed or nothing
What and why
ExistenceCacheStore::updatedrained and discarded any upload for a digest the cache believed present. When the tier under it loses the blob and the remove callback never arrives (a Redis under allkeys-lru whose keyspace events are dropped, as seen on a production CAS), the cache answers present for the rest of its TTL, and a client that uploads anyway loses the blob too. The read then failsnot found in either fast or slow storeand Bazel exits 39.Every upload now writes through, whatever the cache believes, and refreshes the entry. Clients only upload what they were told is missing, so nothing is wasted in the normal flow. The callback miss log drops to debug: the subscription delivers every key event on the server.
How was this verified?
A new test loses a blob below the cache without a callback, re-uploads it and asserts it lands and reads back; it fails on main. A second pins that a positive older than
max_secondsasks the inner store again. On a three-pod test cluster an upload to a pod with a stale positive never reached Redis before, and does now.Risk
Low. Only the early return goes; a stale positive is still bounded by
max_seconds.AI assistance
An agent drafted the change and I reviewed every line.