Skip to content

Redis: don't silently lose pubsub subscriptions on reconnect or a transient init error - #2839

Merged
palfrey merged 2 commits into
TraceMachina:mainfrom
b7r6:fix/redis-subscription-lifecycle
Oct 15, 2026
Merged

palfrey merged 2 commits into
TraceMachina:mainfrom
b7r6:fix/redis-subscription-lifecycle

Conversation

@b7r6

@b7r6 b7r6 commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

What and why

There are two ways RedisStore ends up believing a pattern subscription is active when it is not subscribed on the live connection. In both, events are dropped with no error.

  1. A psubscribe racing a reconnect. StandardRedisManager::psubscribe fetched a connection and only then added the pattern to subscriptions. A concurrent reconnect that snapshotted subscriptions in between would replay the old set on the new connection. The new pattern was subscribed only on the connection being replaced, yet recorded as active. Every message for it was lost until some later, unrelated reconnect. The fix has two parts:

    • psubscribe now records the pattern before it fetches a connection.
    • Before publishing the new handle, reconnect subscribes whatever patterns were added since its snapshot, under the same write lock it publishes with.

    A racing psubscribe therefore either lands in the replay, or blocks on the lock and subscribes on the new connection itself.

  2. A transient error at init disabled remove callbacks for good. register_remove_callback returns Ok(()) and sets up the __key*__:* subscription in the background, through a one-shot OnceCell::get_or_try_init. One transient error, such as -LOADING during a restart or failover, was logged and never retried. Eviction and expiry callbacks then stayed dead for the life of the process, so an ExistenceCacheStore in front of Redis kept answering "exists" for evicted keys. The init now retries with backoff from 50 ms up to 5 s. The notify-keyspace-events check is advisory: if reading the config fails, the store logs it and subscribes anyway.

How was this verified?

Each bug has a deterministic regression test. I ran both on 8bf6be1 with the fix reverted:

  • psubscribe_lost_across_reconnect (in redis_store.rs) uses a small fake RESP3 server to park reconnect after its snapshot, runs a psubscribe in that window, then lets the reconnect finish. Without the fix it fails with pattern "pat" is recorded as an active subscription but the live post-reconnect connection was only subscribed to ["pre"].
  • transient_init_error_permanently_disables_remove_callbacks (in redis_store_test.rs) answers the init's CONFIG GET with -LOADING, then checks that the __key*__:* subscription is still established. Without the fix it fails.

Both pass with the fix, and the rest of redis_store_test passes.

Risk

Moderate, contained to Redis subscription setup:

  • reconnect now holds the connection write lock while it subscribes any patterns added during the reconnect. The set is normally empty, but a slow PSUBSCRIBE there delays callers waiting on get_connection.
  • While Redis is unreachable, the background init keeps retrying and logs an error every 5 s at most.
  • No wire or config changes.

AI assistance

Claude found both windows in a review of the Redis store, wrote the fixes and tests under my direction, checked that each test fails without its fix, and drafted this description.

@vercel

vercel Bot commented Sep 30, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
nativelink Ready Ready Preview Oct 15, 2026 2:58pm UTC
nativelink-aidm Ready Ready Preview Oct 15, 2026 2:58pm UTC

Request Review

@b7r6

b7r6 commented Oct 3, 2026

Copy link
Copy Markdown
Contributor Author

Withdrawing this from the queue to respect the team's review bandwidth. It's a verified fix (fails-without regression, retained on our fork) — rather than keep a flood of drafts open, we'll re-propose one clean, reviewed PR at a time. Happy to prioritize any of these on request.

@b7r6 b7r6 closed this Oct 3, 2026
…ents

Two confirmed bugs left the RedisStore keyspace/pattern subscriptions
recorded as active while never actually established on the live
connection. Both come with a deterministic repro test that fails on the
old code and passes on the fix.

1. psubscribe lost across a concurrent reconnect
   (StandardRedisManager). `psubscribe` used to fetch a connection, then
   insert the pattern into the tracked `subscriptions` set. If a
   concurrent `reconnect` snapshotted+replayed that set on the new
   connection in between, the pattern was PSUBSCRIBEd only on the old,
   about-to-be-demoted connection and recorded as active — so every
   PMessage for it was dropped until an unrelated later reconnect
   replayed the now-complete set. Fix: order registration against the
   reconnect snapshot. `psubscribe` now inserts the pattern BEFORE
   fetching a connection, and `reconnect` recomputes the missing delta
   and subscribes it while holding the `connection_manager` write lock
   just before publishing the new handle. A racing `psubscribe` then
   either lands in the replayed snapshot or blocks on the write lock and
   subscribes on the already-published live connection — no window where
   a pattern is "active" but subscribed only on a stale connection.

2. transient init error permanently disabled remove callbacks.
   `register_remove_callback` returned Ok(()) and deferred the CONFIG
   GET + PSUBSCRIBE "__key*__:*" to a background one-shot
   `OnceCell::get_or_try_init`. A single transient Redis error (e.g.
   -LOADING during a restart/failover) was only logged; the OnceCell
   stayed empty and nothing retried, so eviction/expiry callbacks were
   silently dead forever (ExistenceCacheStore never invalidated → stale
   `exists` hits). Fix: retry the init with capped backoff until it
   succeeds, and demote the advisory notify-keyspace-events CONFIG GET
   check to non-fatal (log misconfig, subscribe anyway) so a transient
   read failure can no longer permanently disable the subscription.

Both repro tests fail on the unfixed code and pass on the fix (verified
both directions on the build fleet). Full `cargo test -p
nativelink-store` green, clippy clean, nightly rustfmt applied.

Provenance: adversarial review fleet.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@palfrey
palfrey marked this pull request as ready for review October 15, 2026 13:27
@palfrey
palfrey merged commit 89ea706 into TraceMachina:main Oct 15, 2026
38 checks passed
@b7r6
b7r6 deleted the fix/redis-subscription-lifecycle branch October 15, 2026 19:11

This branch was successfully deployed

2 active deployments
Preview – nativelink — cc37faeb Deployed Oct 15, 2026 by vercel[bot]
Preview – nativelink-aidm — cc37faeb Deployed Oct 15, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants