Repository navigation
Read a filter pass's listed records one at a time again: the 64-way reads broke the scheduler's Redis connection under a queue burst - #3021
Merged
amankrx merged 3 commits intoOct 22, 2026
Conversation
…eads broke the scheduler's Redis connection under a queue burst
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
corcillo
approved these changes
Oct 22, 2026
corcillo
reviewed
Oct 22, 2026
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
#3001 read a filter pass's listed records 64 at a time. On the Redis backend each read is a round trip and, for a queued action, a keepalive write, and every write is a keyspace notification back to the scheduler's own subscription, which shares the connection with its commands. Under a burst of 6,000 queued actions on a test cluster that traffic broke the scheduler's Redis connection: Redis closed it once for its pubsub output-buffer limit, and each break cost a reconnect of one to three minutes during which the queue froze, the admin listing hung and workers' updates failed with broken pipe. Three such runs stalled two or three times each; four runs on v1.8.0, and runs on this branch, did not.
The listing reads one record at a time again, as before #3001. #3013's memory-store change stays; the admin listing keeps its corrected comments.
How was this verified?
The scheduler crate's tests pass (the concurrency test is removed with the behaviour), clippy is clean and the binary checks on 1.98.1 and 1.97.1. On the test cluster the same 6,000-action burst ran clean on this branch where main stalled, with the stall watcher and the Redis slowlog attached.
Risk
The admin demand listing goes back to about 0.6 s at 6,000 queued, well inside the provisioner's 5 s budget.
AI assistance
An agent drafted the change and I reviewed every line.