Repository navigation
Don't acknowledge stale delivery tags after a channel reopen - #759
Merged
Merged
Conversation
eglitobias
force-pushed
the
stale-delivery-tags-after-reopen
branch
from
October 30, 2026 14:27
280393a to
eb15408
Compare
eglitobias
marked this pull request as ready for review
October 30, 2026 17:24
eglitobias
force-pushed
the
stale-delivery-tags-after-reopen
branch
from
November 9, 2026 08:02
4da9922 to
ae3d17b
Compare
Delivery tags restart at 1 on a reopened channel, so an acknowledgement from a handler that started before the reopen closed the channel again. Closes #710.
A concurrent Channel#open could change it between the check and the subtraction. Also shortens the ChangeLog entry.
eglitobias
force-pushed
the
stale-delivery-tags-after-reopen
branch
from
November 9, 2026 08:22
ae3d17b to
34c0b4a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #710.
After
Channel#reopendelivery tags restart at 1, but handlers that started before the closure still acknowledge with their old tags. A tag the reopened channel doesn't know closes it again withPRECONDITION_FAILED - unknown delivery tag, and a lower one acknowledges a different delivery.How we ran into it: after a RabbitMQ node came back from a migration, a Hutch 2.0 worker (prefetch 10) had its consumer channel closed with
unknown delivery tag. Hutch's channel recovery (ruby-amqp/hutch@f664026) reopened it. The handlers still working through the prefetched deliveries then acknowledged their old tags on the reopened channel, which closed it again, and so on. That made about 30,000 reopens, each leaking a consumer work pool thread, until creating a thread failed withThreadErrorand killed the reader loop. The process kept running without a consumer.This shifts delivery tags by the highest tag seen before the channel is opened again, like
RecoveryAwareChannelNin the Java client, and dropsbasic.ack,basic.nackandbasic.rejectfor tags from before that. The offset is set inChannel#open, so connection recovery gets the same treatment. Topology recovery also no longer replaces the work pool#reopenhas just started.Tags seen by applications now keep increasing across a reopen or recovery instead of restarting at 1.
Repro with Hutch 2.0.0, prefetch 10, 500 messages and one bogus ack: