Skip to content

fix: give a caller-built Redis client the same half-open protection - #23

Merged
adhikjoshi merged 3 commits into
mainfrom
fix/caller-redis-client-timeouts
Nov 9, 2026
Merged

adhikjoshi merged 3 commits into
mainfrom
fix/caller-redis-client-timeouts

Conversation

@adhikjoshi

@adhikjoshi adhikjoshi commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Problem

Servers pass their own Redis(host, port, password) as redis_client=. That skips _connect_to_redis(), so the pool has no socket_timeout, keepalive or health check. When a connection is dropped silently, a worker in a blocking pop waits forever. No exception is raised, so _RedisWithRetry never runs. Only a container restart brings the worker back. This is the "stuck until restart" mode on the TTS, SFX and Song servers.

Change

  • ModelQ.__init__ calls _apply_liveness_defaults() on a caller-built client. It fills socket_timeout, socket_connect_timeout, health_check_interval and keepalive only where the caller left them unset. Then it resets the pool.
  • Values that the caller set are kept. Unix-socket pools are not changed.
  • Version 1.0.22.

Tests

  • 4 new tests in tests/test_connection_liveness.py. One opens a real TCP socket that never answers.
  • Full suite: 143 passed, 14 skipped. When the call is removed, 2 new tests fail.
  • End to end, with a proxy that silently drops every connection: 1.0.16 with a bare client stayed stuck for 200 s. This branch recovered and ran the next task 68 s after the drop.

Autofix with [code]smith
Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.

Servers build `Redis(host=..., port=..., password=...)` and pass it as
`ModelQ(redis_client=...)`, which skips `_connect_to_redis()`. Their pool
had no socket timeout, keepalive or health check, so a worker parked in a
blocking pop on a silently dropped connection waited forever: nothing was
raised, `_RedisWithRetry` never ran, and only a container restart brought
the worker back. TTS, SFX and Song generation all build their client this
way.

ModelQ now fills socket_timeout, socket_connect_timeout,
health_check_interval and keepalive on a caller's TCP pool when the caller
left them unset, and resets the pool so no connection keeps the old
settings. Values the caller set are kept; unix-socket pools are left alone.

Reproduced with a TCP proxy that silently drops every connection: on
1.0.16 with a bare client the worker stayed stuck past 200 s; with this
change it raised a TimeoutError and ran the next task 68 s after the drop.
Two of the new tests fail when the call is removed.
@adhikjoshi
adhikjoshi merged commit 19eaf5b into main Nov 9, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant