Repository navigation
fix: give a caller-built Redis client the same half-open protection - #23
Merged
Merged
Conversation
Servers build `Redis(host=..., port=..., password=...)` and pass it as `ModelQ(redis_client=...)`, which skips `_connect_to_redis()`. Their pool had no socket timeout, keepalive or health check, so a worker parked in a blocking pop on a silently dropped connection waited forever: nothing was raised, `_RedisWithRetry` never ran, and only a container restart brought the worker back. TTS, SFX and Song generation all build their client this way. ModelQ now fills socket_timeout, socket_connect_timeout, health_check_interval and keepalive on a caller's TCP pool when the caller left them unset, and resets the pool so no connection keeps the old settings. Values the caller set are kept; unix-socket pools are left alone. Reproduced with a TCP proxy that silently drops every connection: on 1.0.16 with a bare client the worker stayed stuck past 200 s; with this change it raised a TimeoutError and ran the next task 68 s after the drop. Two of the new tests fail when the call is removed.
…8 does not mask them
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Servers pass their own
Redis(host, port, password)asredis_client=. That skips_connect_to_redis(), so the pool has nosocket_timeout, keepalive or health check. When a connection is dropped silently, a worker in a blocking pop waits forever. No exception is raised, so_RedisWithRetrynever runs. Only a container restart brings the worker back. This is the "stuck until restart" mode on the TTS, SFX and Song servers.Change
ModelQ.__init__calls_apply_liveness_defaults()on a caller-built client. It fillssocket_timeout,socket_connect_timeout,health_check_intervaland keepalive only where the caller left them unset. Then it resets the pool.Tests
tests/test_connection_liveness.py. One opens a real TCP socket that never answers.Open in [code]smith · Need help on this PR? Tag
@codesmith-botwith what you need.