Skip to content

Read a filter pass's listed records one at a time again: the 64-way reads broke the scheduler's Redis connection under a queue burst - #3021

Merged
amankrx merged 3 commits into
TraceMachina:mainfrom
amankrx:fix/redis-listing-serial-reads
Oct 22, 2026
Merged

amankrx merged 3 commits into
TraceMachina:mainfrom
amankrx:fix/redis-listing-serial-reads

Conversation

@amankrx

@amankrx amankrx commented Oct 22, 2026

Copy link
Copy Markdown
Member

What and why

#3001 read a filter pass's listed records 64 at a time. On the Redis backend each read is a round trip and, for a queued action, a keepalive write, and every write is a keyspace notification back to the scheduler's own subscription, which shares the connection with its commands. Under a burst of 6,000 queued actions on a test cluster that traffic broke the scheduler's Redis connection: Redis closed it once for its pubsub output-buffer limit, and each break cost a reconnect of one to three minutes during which the queue froze, the admin listing hung and workers' updates failed with broken pipe. Three such runs stalled two or three times each; four runs on v1.8.0, and runs on this branch, did not.

The listing reads one record at a time again, as before #3001. #3013's memory-store change stays; the admin listing keeps its corrected comments.

How was this verified?

The scheduler crate's tests pass (the concurrency test is removed with the behaviour), clippy is clean and the binary checks on 1.98.1 and 1.97.1. On the test cluster the same 6,000-action burst ran clean on this branch where main stalled, with the stall watcher and the Redis slowlog attached.

Risk

The admin demand listing goes back to about 0.6 s at 6,000 queued, well inside the provisioner's 5 s budget.

AI assistance

An agent drafted the change and I reviewed every line.

…eads broke the scheduler's Redis connection under a queue burst
@vercel

vercel Bot commented Oct 22, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
nativelink Ready Ready Preview Oct 22, 2026 10:27pm UTC
nativelink-aidm Ready Ready Preview Oct 22, 2026 10:27pm UTC

Request Review

@amankrx
amankrx merged commit 86c2923 into TraceMachina:main Oct 22, 2026
28 checks passed

This branch was successfully deployed

2 active deployments
Preview – nativelink — 1e486e15 Deployed Oct 22, 2026 by vercel[bot]
Preview – nativelink-aidm — 1e486e15 Deployed Oct 22, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants