Skip to content

Don't acknowledge stale delivery tags after a channel reopen - #759

Merged
eglitobias merged 2 commits into
mainfrom
stale-delivery-tags-after-reopen
Nov 9, 2026
Merged

eglitobias merged 2 commits into
mainfrom
stale-delivery-tags-after-reopen

Conversation

@eglitobias

@eglitobias eglitobias commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

Closes #710.

After Channel#reopen delivery tags restart at 1, but handlers that started before the closure still acknowledge with their old tags. A tag the reopened channel doesn't know closes it again with PRECONDITION_FAILED - unknown delivery tag, and a lower one acknowledges a different delivery.

How we ran into it: after a RabbitMQ node came back from a migration, a Hutch 2.0 worker (prefetch 10) had its consumer channel closed with unknown delivery tag. Hutch's channel recovery (ruby-amqp/hutch@f664026) reopened it. The handlers still working through the prefetched deliveries then acknowledged their old tags on the reopened channel, which closed it again, and so on. That made about 30,000 reopens, each leaking a consumer work pool thread, until creating a thread failed with ThreadError and killed the reader loop. The process kept running without a consumer.

This shifts delivery tags by the highest tag seen before the channel is opened again, like RecoveryAwareChannelN in the Java client, and drops basic.ack, basic.nack and basic.reject for tags from before that. The offset is set in Channel#open, so connection recovery gets the same treatment. Topology recovery also no longer replaces the work pool #reopen has just started.

Tags seen by applications now keep increasing across a reopen or recovery instead of restarting at 1.

Repro with Hutch 2.0.0, prefetch 10, 500 messages and one bogus ack:

reopens extra threads deliveries processed
3.5.0 300-540 380-690 1300-2000
this branch 1 0 510

@eglitobias
eglitobias force-pushed the stale-delivery-tags-after-reopen branch from 280393a to eb15408 Compare October 30, 2026 14:27
@eglitobias
eglitobias marked this pull request as ready for review October 30, 2026 17:24
@eglitobias eglitobias added this to the 3.6.0 milestone Nov 4, 2026
@eglitobias
eglitobias force-pushed the stale-delivery-tags-after-reopen branch from 4da9922 to ae3d17b Compare November 9, 2026 08:02
Delivery tags restart at 1 on a reopened channel, so an acknowledgement from a handler
that started before the reopen closed the channel again. Closes #710.
A concurrent Channel#open could change it between the check and the subtraction. Also shortens the ChangeLog entry.
@eglitobias
eglitobias force-pushed the stale-delivery-tags-after-reopen branch from ae3d17b to 34c0b4a Compare November 9, 2026 08:22
@eglitobias
eglitobias merged commit 66fe740 into main Nov 9, 2026
17 checks passed
@eglitobias
eglitobias deleted the stale-delivery-tags-after-reopen branch November 9, 2026 08:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Replace versioned delivery tags with a similar concept in the Java client

1 participant