Skip to content

Reconnect and retry the Redis length check after a failover - #2961

Merged
MarcusSorealheis merged 5 commits into
TraceMachina:mainfrom
amankrx:fix/redis-strlen-failover
Oct 20, 2026
Merged

MarcusSorealheis merged 5 commits into
TraceMachina:mainfrom
amankrx:fix/redis-strlen-failover

Conversation

@amankrx

@amankrx amankrx commented Oct 19, 2026

Copy link
Copy Markdown
Member

What and why

RedisStore::update writes its chunks, checks the temp key's length, renames it and sets the TTL. Every step but the length check reconnected on a failover error. When the master died between the chunk writes and that check, the STRLEN waited out the command timeout on the dead connection and the upload failed. On a bed this made every CAS upload fail for about forty seconds after a Redis master kill, and Bazel gave up on its input uploads with "errors during bulk transfer".

The length check now re-resolves the master and retries on a retryable error, within the attempt budget the loop already has.

How was this verified?

New test in redis_store_test.rs: the STRLEN fails once with a timed out IO error, the store reconnects, and the second STRLEN and the RENAME succeed. The existing Redis store tests pass. On the bed a Redis master kill mid-build went from failing the build to a clean failover with no CAS error.

Risk

Low: the new branch runs only on a connection-level error that the chunk writes, the rename and the TTL already handle the same way.

AI assistance

An agent drafted the change and I reviewed every line.

@vercel

vercel Bot commented Oct 19, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
nativelink Ready Ready Preview Oct 20, 2026 3:16am UTC
nativelink-aidm Ready Ready Preview Oct 20, 2026 3:16am UTC

Request Review

Comment thread nativelink-store/src/redis_store.rs Resolved

@MarcusSorealheis MarcusSorealheis left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@MarcusSorealheis
MarcusSorealheis merged commit 11dc077 into TraceMachina:main Oct 20, 2026
29 checks passed

This branch was successfully deployed

2 active deployments
Preview – nativelink — d2e27259 Deployed Oct 20, 2026 by vercel[bot]
Preview – nativelink-aidm — d2e27259 Deployed Oct 20, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants