Skip to content

fix(driver): memory leaks on drop - #769

Merged
Berrysoft merged 2 commits into
compio-rs:masterfrom
johnnyshields:fix-asan-memleaks
Mar 17, 2026
Merged

Berrysoft merged 2 commits into
compio-rs:masterfrom
johnnyshields:fix-asan-memleaks

Conversation

@johnnyshields

@johnnyshields johnnyshields commented Mar 14, 2026 •

Copy link
Copy Markdown
Contributor

I used asan to look for leaks in compio-driver and found 3 sources of memory leaks in compio-driver when the driver or its resources are dropped without all operations completing. Found via AddressSanitizer + LeakSanitizer (-Zsanitizer=address with -Zbuild-std).

Before: 12,952 bytes leaked in 18 allocations across the test suite
After: 0 bytes leaked

Changes

1. io_uring Driver — in-flight key tracking + Drop impl

push_raw_with_key calls key.into_raw() to leak the ThinCell<RawOp> allocation as a u64 user_data into the io_uring submission queue. On completion, ErasedKey::from_raw() reconstructs ownership. But if the Driver drops before all completions arrive (which happens on every normal process exit), the raw pointers are never reconstructed and the allocations leak.

Fix: Track in-flight raw pointers in a HashSet<usize>. Remove on completion. In Drop, drain the completion queue, then free any remaining tracked keys via ErasedKey::from_raw().

I've also committed the asan script I used.

Edit: These have been reverted

2. FrozenKey — missing Drop impl

~~FrozenKey wraps ManuallyDrop<ErasedKey> and has no Drop impl. It's used in push_blocking to send operations to the thread pool. If the thread pool drops the closure without running it (e.g., on shutdown), the inner ErasedKey leaks.

Fix: Add impl Drop for FrozenKey that drops the inner ErasedKey. Modify into_inner() to wrap self in ManuallyDrop first, preventing double-drop on the normal (non-leak) path.~~

~~ 3. BufferPoolInner — IoUringBufRing leaks on drop~~

IoUringBufRing internally stores its mmap region and backing buffers in ManuallyDrop fields. It requires an explicit release() call (which unregisters from io_uring) to free them. If a BufferPool is dropped without calling release_buffer_pool, the mmap and buffer Vecs leak.

Fix: Wrap buf_ring in ManuallyDrop inside BufferPoolInner and add a Drop impl that calls IoUringBufRing::drop() (the crate's provided unsafe cleanup function that frees memory without unregistering). Update into_inner() to take the ring out and mem::forget the wrapper to prevent double-free when the caller intends to call release().

@George-Miao

George-Miao commented Mar 14, 2026 •

Copy link
Copy Markdown
Member

Thanks for the PR.
For the 3 changes you mentioned:
1 looks good, go ahead. For 2 it was on purpose, since FrozenKey is used to unsafely sending non-Send data to other thread, if it's dropped, we can't run ThinCell's drop on other thread since that will cause UB. And for 3 we're planning for a major refactor for buffer pool, so please leave it as is.

@Berrysoft Berrysoft changed the title Fix memory leaks in compio-driver on drop fix(driver): memory leaks on drop Mar 14, 2026
@Berrysoft

Copy link
Copy Markdown
Member

For 3, there might be ops still operating, referencing this bufing. Therefore, it's not safe to drop the buffers before releasing it.

@johnnyshields

Copy link
Copy Markdown
Contributor Author

Reverted 2 and 3. Please give it a second look 🙇‍♂️

Comment thread compio-driver/scripts/memleak/asan.sh Resolved
Comment thread compio-driver/scripts/memleak/asan.sh Resolved
Comment thread compio-driver/scripts/memleak/asan.sh Resolved
@Berrysoft
Berrysoft requested a review from George-Miao March 16, 2026 04:09
@Berrysoft
Berrysoft merged commit 6a85e2e into compio-rs:master Mar 17, 2026
120 of 121 checks passed
@github-actions github-actions Bot mentioned this pull request Mar 17, 2026
This was referenced Apr 21, 2026
Xerxes-2 added a commit to Xerxes-2/compio that referenced this pull request Aug 21, 2026
`drop_with_inflight_ops` pushed an `Asyncify` op that completes immediately,
so there was never an in-flight op to leak, and the test asserted nothing: it
passed with or without the `Drop for Driver` fix from compio-rs#769, even under ASan.

Push a recv on an idle TCP socket so the op is genuinely in-flight, assert that
`push` returns `Pending`, and detect a leaked op by checking the driver dropped
the `SharedFd` clone that op owned.

Verified against a mutant with `impl Drop for Driver` removed: io_uring fails,
poll still passes because its registry owns the keys.

The IOCP driver has no equivalent handling -- it calls `key.into_raw()` for
pending overlapped ops and never reclaims them on drop -- so this test is
expected to fail on Windows until that is fixed.
Xerxes-2 added a commit to Xerxes-2/compio that referenced this pull request Aug 21, 2026
`drop_with_inflight_ops` pushed an `Asyncify` op that completes immediately, so
there was never an in-flight op to leak, and the test asserted nothing: it
passed with or without the `Drop for Driver` fix from compio-rs#769, even under ASan.

Push a recv on a bound UDP socket nobody ever sends to, so the op is genuinely
in-flight, assert that `push` returns `Pending`, and detect a leaked op by
checking that dropping the driver also dropped the `SharedFd` clone that op
owned.

Verified against a mutant with `impl Drop for Driver` removed: io_uring fails,
poll still passes because its registry owns the keys.

The IOCP driver has no equivalent handling -- it calls `key.into_raw()` for
pending overlapped ops and never reclaims them on drop -- so this test is
expected to fail on Windows until that is fixed.
Xerxes-2 added a commit to Xerxes-2/compio that referenced this pull request Aug 21, 2026
`drop_with_inflight_ops` pushed an `Asyncify` op that completes immediately, so
there was never an in-flight op to leak, and the test asserted nothing: it
passed with or without the `Drop for Driver` fix from compio-rs#769, even under ASan.

Push a recv on a bound UDP socket nobody ever sends to, so the op is genuinely
in-flight, assert that `push` returns `Pending`, and detect a leaked op by
checking that dropping the driver also dropped the `SharedFd` clone that op
owned.

Verified against a mutant with `impl Drop for Driver` removed: io_uring fails,
poll still passes because its registry owns the keys.

The IOCP driver has no equivalent handling -- it calls `key.into_raw()` for
pending overlapped ops and never reclaims them on drop -- so this test is
expected to fail on Windows until that is fixed.
Xerxes-2 added a commit to Xerxes-2/compio that referenced this pull request Aug 21, 2026
`drop_with_inflight_ops` pushed an `Asyncify` op that completes immediately, so
there was never an in-flight op to leak, and the test asserted nothing: it
passed with or without the `Drop for Driver` fix from compio-rs#769, even under ASan.

Push a recv on a bound UDP socket nobody ever sends to, so the op is genuinely
in-flight, assert that `push` returns `Pending`, and detect a leaked op by
checking that dropping the driver also dropped the `SharedFd` clone that op
owned.

Verified against a mutant with `impl Drop for Driver` removed: io_uring fails,
poll still passes because its registry owns the keys.

The IOCP driver has no equivalent handling -- it calls `key.into_raw()` for
pending overlapped ops and never reclaims them on drop -- so this test is
expected to fail on Windows until that is fixed.
Xerxes-2 added a commit to Xerxes-2/compio that referenced this pull request Aug 21, 2026
`drop_with_inflight_ops` pushed an `Asyncify` op that completes immediately, so
there was never an in-flight op to observe, and the test asserted nothing: it
passed with or without the `Drop for Driver` fix from compio-rs#769, even under ASan.

Push a recv on a bound UDP socket nobody ever sends to, so the op is genuinely
in-flight, assert that `push` returns `Pending`, and then check what the driver
did with the op by looking at the `SharedFd` clone the op owned.

The correct outcome differs per driver, so assert both sides of it. io_uring
closes the ring on drop, which makes the kernel cancel the op, and the poll
driver owns its keys outright, so both must release the op. IOCP is a real
proactor: dropping the port does not cancel a submitted op, so the kernel may
still write into its buffer, and releasing the key or the fd would be a
use-after-free -- there the op must stay leaked.

Verified against mutants: removing `impl Drop for Driver` fails the io_uring
assertion, and taking the non-IOCP branch on Windows fails the IOCP assertion.
Xerxes-2 added a commit to Xerxes-2/compio that referenced this pull request Aug 22, 2026
`drop_with_inflight_ops` pushed an `Asyncify` op that completes immediately, so
there was never an in-flight op to observe, and the test asserted nothing: it
passed with or without the `Drop for Driver` fix from compio-rs#769, even under ASan.

Push a recv on a bound UDP socket nobody ever sends to, so the op is genuinely
in-flight, assert that `push` returns `Pending`, and then check what the driver
did with the op by looking at the `SharedFd` clone the op owned.

The correct outcome differs per driver, so assert both sides of it. io_uring
closes the ring on drop, which makes the kernel cancel the op, and the poll
driver owns its keys outright, so both must release the op. IOCP is a real
proactor: dropping the port does not cancel a submitted op, so the kernel may
still write into its buffer, and releasing the key or the fd would be a
use-after-free -- there the op must stay leaked.

Verified against mutants: removing `impl Drop for Driver` fails the io_uring
assertion, and taking the non-IOCP branch on Windows fails the IOCP assertion.
Berrysoft pushed a commit that referenced this pull request Aug 22, 2026
…1018)

`drop_with_inflight_ops` pushed an `Asyncify` op that completes immediately, so
there was never an in-flight op to observe, and the test asserted nothing: it
passed with or without the `Drop for Driver` fix from #769, even under ASan.

Push a recv on a bound UDP socket nobody ever sends to, so the op is genuinely
in-flight, assert that `push` returns `Pending`, and then check what the driver
did with the op by looking at the `SharedFd` clone the op owned.

The correct outcome differs per driver, so assert both sides of it. io_uring
closes the ring on drop, which makes the kernel cancel the op, and the poll
driver owns its keys outright, so both must release the op. IOCP is a real
proactor: dropping the port does not cancel a submitted op, so the kernel may
still write into its buffer, and releasing the key or the fd would be a
use-after-free -- there the op must stay leaked.

Verified against mutants: removing `impl Drop for Driver` fails the io_uring
assertion, and taking the non-IOCP branch on Windows fails the IOCP assertion.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants