Repository navigation
Redis: don't silently lose pubsub subscriptions on reconnect or a transient init error - #2839
Merged
palfrey merged 2 commits intoOct 15, 2026
Merged
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
b7r6
force-pushed
the
fix/redis-subscription-lifecycle
branch
from
September 30, 2026 04:59
4eb6717 to
b931ef6
Compare
Contributor
Author
|
Withdrawing this from the queue to respect the team's review bandwidth. It's a verified fix (fails-without regression, retained on our fork) — rather than keep a flood of drafts open, we'll re-propose one clean, reviewed PR at a time. Happy to prioritize any of these on request. |
…ents Two confirmed bugs left the RedisStore keyspace/pattern subscriptions recorded as active while never actually established on the live connection. Both come with a deterministic repro test that fails on the old code and passes on the fix. 1. psubscribe lost across a concurrent reconnect (StandardRedisManager). `psubscribe` used to fetch a connection, then insert the pattern into the tracked `subscriptions` set. If a concurrent `reconnect` snapshotted+replayed that set on the new connection in between, the pattern was PSUBSCRIBEd only on the old, about-to-be-demoted connection and recorded as active — so every PMessage for it was dropped until an unrelated later reconnect replayed the now-complete set. Fix: order registration against the reconnect snapshot. `psubscribe` now inserts the pattern BEFORE fetching a connection, and `reconnect` recomputes the missing delta and subscribes it while holding the `connection_manager` write lock just before publishing the new handle. A racing `psubscribe` then either lands in the replayed snapshot or blocks on the write lock and subscribes on the already-published live connection — no window where a pattern is "active" but subscribed only on a stale connection. 2. transient init error permanently disabled remove callbacks. `register_remove_callback` returned Ok(()) and deferred the CONFIG GET + PSUBSCRIBE "__key*__:*" to a background one-shot `OnceCell::get_or_try_init`. A single transient Redis error (e.g. -LOADING during a restart/failover) was only logged; the OnceCell stayed empty and nothing retried, so eviction/expiry callbacks were silently dead forever (ExistenceCacheStore never invalidated → stale `exists` hits). Fix: retry the init with capped backoff until it succeeds, and demote the advisory notify-keyspace-events CONFIG GET check to non-fatal (log misconfig, subscribe anyway) so a transient read failure can no longer permanently disable the subscription. Both repro tests fail on the unfixed code and pass on the fix (verified both directions on the build fleet). Full `cargo test -p nativelink-store` green, clippy clean, nightly rustfmt applied. Provenance: adversarial review fleet. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
b7r6
force-pushed
the
fix/redis-subscription-lifecycle
branch
from
October 15, 2026 12:03
b931ef6 to
25bfa43
Compare
palfrey
approved these changes
Oct 15, 2026
palfrey
marked this pull request as ready for review
October 15, 2026 13:27
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
There are two ways
RedisStoreends up believing a pattern subscription is active when it is not subscribed on the live connection. In both, events are dropped with no error.A
psubscriberacing a reconnect.StandardRedisManager::psubscribefetched a connection and only then added the pattern tosubscriptions. A concurrentreconnectthat snapshottedsubscriptionsin between would replay the old set on the new connection. The new pattern was subscribed only on the connection being replaced, yet recorded as active. Every message for it was lost until some later, unrelated reconnect. The fix has two parts:psubscribenow records the pattern before it fetches a connection.reconnectsubscribes whatever patterns were added since its snapshot, under the same write lock it publishes with.A racing
psubscribetherefore either lands in the replay, or blocks on the lock and subscribes on the new connection itself.A transient error at init disabled remove callbacks for good.
register_remove_callbackreturnsOk(())and sets up the__key*__:*subscription in the background, through a one-shotOnceCell::get_or_try_init. One transient error, such as-LOADINGduring a restart or failover, was logged and never retried. Eviction and expiry callbacks then stayed dead for the life of the process, so anExistenceCacheStorein front of Redis kept answering "exists" for evicted keys. The init now retries with backoff from 50 ms up to 5 s. Thenotify-keyspace-eventscheck is advisory: if reading the config fails, the store logs it and subscribes anyway.How was this verified?
Each bug has a deterministic regression test. I ran both on 8bf6be1 with the fix reverted:
psubscribe_lost_across_reconnect(inredis_store.rs) uses a small fake RESP3 server to parkreconnectafter its snapshot, runs apsubscribein that window, then lets the reconnect finish. Without the fix it fails withpattern "pat" is recorded as an active subscription but the live post-reconnect connection was only subscribed to ["pre"].transient_init_error_permanently_disables_remove_callbacks(inredis_store_test.rs) answers the init'sCONFIG GETwith-LOADING, then checks that the__key*__:*subscription is still established. Without the fix it fails.Both pass with the fix, and the rest of
redis_store_testpasses.Risk
Moderate, contained to Redis subscription setup:
reconnectnow holds the connection write lock while it subscribes any patterns added during the reconnect. The set is normally empty, but a slowPSUBSCRIBEthere delays callers waiting onget_connection.AI assistance
Claude found both windows in a review of the Redis store, wrote the fixes and tests under my direction, checked that each test fails without its fix, and drafted this description.