You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
IO Uring supports registration of file descriptors. Instead of maintaining reference counts for each IO operation Linux constructs user provided indirection array for each thread.
Registration is beneficial for long lived descriptors. Refcounting overhead is relatively significant for small submit batches.
I'm personally interested in driver-level support of registered file descriptors.
I see the following approaches:
define driver specific operations with registered fds. User either prefers a single platform or writes 2 IO data paths for regular and registered file descriptors. IO Uring could expose tokio-uring Fd or FixedFd, platform independent ops code wraps RawFd into Fd.
define common registration interface. For platforms that don't support registration driver could maintain indirection array in userspace.
separate operations for regular/registered fds, no runtime overhead (branching) at operation construction time, more duplicate code
I don't have much attention to this feature now, and AFAIK, neither tokio-uring or monoio uses this feature, am I right? Welcome PRs to implement this. It you implement this, I'll review the PR and make sure the IOCP side will not break.
There's already a method attach which does nothing now. We can use that to register fd, maybe with a little change.
For efficiency we need registration/unregistration/update of fixed fd array. User calculates maximum number of fds at compile time. Like LISTEN_SOCKET + MAX_NUMBER_OF_CLIENTS + JOURNAL_FILE_FD.
For efficiency we need registration/unregistration/update of fixed fd array.
Yes, you're right.
If you want to work on this, I suggest making this feature as a must. In other words, all high-level File and Socket should use registered fd, and all low-level OpCode accept only registered fd. That will simplify the APIs. We can give a map between registered index and raw fd, or simply defining our own RawFd as the registered index.
I think I'll focus on building project-specific callback-based runtime using regular fds. If I find time for optimization I could try to use registered fds, registered buffers and measure gains if there will be any.
Anyway which approach would you prefer? I'm personally biased towards 2.i with implementation only for IO uring.
Redefining RawFd as the registered index. Don't use regular fds at all.
However, it's up to the implementation author to choose the approach. 2.i is also OK, I think, but it will make the API of Poller more complex. I would like to keep a simple API.
For efficiency we need registration/unregistration/update of fixed fd array.
Yes, you're right.
If you want to work on this, I suggest making this feature as a must. In other words, all high-level File and Socket should use registered fd, and all low-level OpCode accept only registered fd. That will simplify the APIs. We can give a map between registered index and raw fd, or simply defining our own RawFd as the registered index.
For thread-per-core runtime that makes sense. Newer kernels allow to pass data across thread-local rings, including registered descriptors or sets of descriptors - in case user will want coordinated cross-thread IO work.
I could try to implement indirection array to establish API and continue to use regular fds internally. Later it's possible to implement actual registration/deregistration for IO uring, potentially as a pluggable feature.
I think we can use 1024 fds as a default array size - it's a default limit of open fds for Linux.
compio aims at thread-per-core runtime. I prefer not considering multi-threading and multi-rings now. However, feel free to PR if you would like a thread safe driver.
For multiple drivers working together, I prefer that users don't register an fd everywhere, or move them quite often:). One reason is that on Windows, file handle could only attach one IOCP. (It's tricky to detach it because that needs undocumented API.)
I would like a Driver API that is (nearly) consistent on every supported platform, without requiring users writing #[cfg(target_os = "linux")] here and there. If we provide both regular and registered fd API, the best practice to use the driver is to use regular fd on Windows (because registering needs more space and time to lookup), and use registered fd on Linux (for ref-counting reasons). That will make it painful to use driver.
Yes, common interface is more user friendly. I'm not interested in cross-thread fds either but would like to offload blocking RocksDB IO work into separate thread. I'd prefer to use some notification mechanism like eventfd on Linux to wake up IO thread waiting for uring completions.
Well, if you want notifications, the current solution might be signals. Channels may not work because the driver might be waiting forever. And thank you for your remind, we still need a compatible channel for compio:)
Well, if you want notifications, the current solution might be signals. Channels may not work because the driver might be waiting forever. And thank you for your remind, we still need a compatible channel for compio:)
This multi-thread channel is fast according to the benchmark.
It uses async-event for notifications. I'd prefer to make it configurable to support non async runtimes. For example, notify via eventfd.
What do you think about changing signal implementation to eventfd for Linux?
Yes, that's possible. Actually signal itself will interrupt io-uring and make it return EINTR.
However I'm thinking about Poller::post. It is complicated if we want it also interrupt the thread.
Channel is another problem. I think there should be an occassion that, when the runtime is waiting io-uring forever, a message comes from the channel. However, the thread is still stuck.
What do you think about changing signal implementation to eventfd for Linux?
Yes, that's possible. Actually signal itself will interrupt io-uring and make it return EINTR.
However I'm thinking about Poller::post. It is complicated if we want it also interrupt the thread.
Channel is another problem. I think there should be an occassion that, when the runtime is waiting io-uring forever, a message comes from the channel. However, the thread is still stuck.
It's possible to mask signals in the IO thread and have a dedicated thread for background work. In this case it's possible handle signals only when IO is driven.
When I experimented with compio-rs' fork, I exposed FixedFd and Driver::register_fd in the public Driver API. I needed this for a custom event loop without an async layer.
Some data point for this issue, in case it helps prioritise.
I prototyped fixed files on compio 0.19.2 (compio-driver 0.12.5) for a DNS server: the listening UDP socket of
each thread registered with Runtime::register_files, and RecvMsgMulti (behind recv_from_multi) and the SendMsg of send_to built with types::Fixed(index) instead of types::Fd when the fd is registered (a
thread-local fd → index map, a hack for measuring only).
dnsperf against a server answering from its cache, one core, Linux 7.0, io_uring, three alternating rounds:
488k-513k queries/s with plain fds, 501k-517k with fixed files: about +1%, within noise. Each thread has its own SO_REUSEPORT socket, so the file refcount the kernel takes per operation is never contended, and the
multishot receive takes it once for many datagrams. So, for socket workloads like this one, fixed files seem to
matter little; they might matter more with a socket shared between threads or many short operations per fd.
What the prototype showed about the API: the registration already exists (register_files), but no op can
use it, since every op builds its SQE from as_fd().as_raw_fd(). A small, opt-in way would be for SharedFd
(or a wrapper type) to carry an optional registered index that the io_uring ops check when building their
entry, falling back to the raw fd otherwise, so the other drivers and the public API stay unchanged.
Activity
Berrysoft commentedon Sep 8, 2023
There's already a method
attachwhich does nothing now. We can use that to register fd, maybe with a little change.Berrysoft commentedon Sep 8, 2023
I don't have much attention to this feature now, and AFAIK, neither
tokio-uringormonoiouses this feature, am I right? Welcome PRs to implement this. It you implement this, I'll review the PR and make sure the IOCP side will not break.DXist commentedon Sep 8, 2023
For efficiency we need registration/unregistration/update of fixed fd array. User calculates maximum number of fds at compile time. Like LISTEN_SOCKET + MAX_NUMBER_OF_CLIENTS + JOURNAL_FILE_FD.
Berrysoft commentedon Sep 8, 2023
Yes, you're right.
If you want to work on this, I suggest making this feature as a must. In other words, all high-level
FileandSocketshould use registered fd, and all low-levelOpCodeaccept only registered fd. That will simplify the APIs. We can give a map between registered index and raw fd, or simply defining our ownRawFdas the registered index.DXist commentedon Sep 8, 2023
I think I'll focus on building project-specific callback-based runtime using regular fds. If I find time for optimization I could try to use registered fds, registered buffers and measure gains if there will be any.
Anyway which approach would you prefer? I'm personally biased towards 2.i with implementation only for IO uring.
Berrysoft commentedon Sep 8, 2023
My preference is, TL;DR:
Redefining RawFd as the registered index. Don't use regular fds at all.
However, it's up to the implementation author to choose the approach. 2.i is also OK, I think, but it will make the API of
Pollermore complex. I would like to keep a simple API.DXist commentedon Sep 8, 2023
For thread-per-core runtime that makes sense. Newer kernels allow to pass data across thread-local rings, including registered descriptors or sets of descriptors - in case user will want coordinated cross-thread IO work.
DXist commentedon Sep 8, 2023
I could try to implement indirection array to establish API and continue to use regular fds internally. Later it's possible to implement actual registration/deregistration for IO uring, potentially as a pluggable feature.
I think we can use 1024 fds as a default array size - it's a default limit of open fds for Linux.
Berrysoft commentedon Sep 8, 2023
compioaims at thread-per-core runtime. I prefer not considering multi-threading and multi-rings now. However, feel free to PR if you would like a thread safe driver.For multiple drivers working together, I prefer that users don't register an fd everywhere, or move them quite often:). One reason is that on Windows, file handle could only attach one IOCP. (It's tricky to detach it because that needs undocumented API.)
I would like a
DriverAPI that is (nearly) consistent on every supported platform, without requiring users writing#[cfg(target_os = "linux")]here and there. If we provide both regular and registered fd API, the best practice to use the driver is to use regular fd on Windows (because registering needs more space and time to lookup), and use registered fd on Linux (for ref-counting reasons). That will make it painful to use driver.DXist commentedon Sep 8, 2023
Yes, common interface is more user friendly. I'm not interested in cross-thread fds either but would like to offload blocking RocksDB IO work into separate thread. I'd prefer to use some notification mechanism like eventfd on Linux to wake up IO thread waiting for uring completions.
Berrysoft commentedon Sep 8, 2023
Well, if you want notifications, the current solution might be signals. Channels may not work because the driver might be waiting forever. And thank you for your remind, we still need a compatible channel for
compio:)Berrysoft commentedon Sep 8, 2023
The
Poller::postforio-uringcould not wake up the IO thread until now. I think we need a simple solution. Maybe eventfd.DXist commentedon Sep 8, 2023
What do you think about changing signal implementation to eventfd for Linux?
In this case sync_queue won't be needed for Linux target.
DXist commentedon Sep 8, 2023
This multi-thread channel is fast according to the benchmark.
It uses async-event for notifications. I'd prefer to make it configurable to support non async runtimes. For example, notify via eventfd.
Berrysoft commentedon Sep 8, 2023
Yes, that's possible. Actually signal itself will interrupt io-uring and make it return EINTR.
However I'm thinking about
Poller::post. It is complicated if we want it also interrupt the thread.Channel is another problem. I think there should be an occassion that, when the runtime is waiting io-uring forever, a message comes from the channel. However, the thread is still stuck.
DXist commentedon Sep 8, 2023
It's possible to mask signals in the IO thread and have a dedicated thread for background work. In this case it's possible handle signals only when IO is driven.
DXist commentedon Sep 8, 2023
I've pushed initial interface for registered fds - https://github.com/Berrysoft/compio/pull/30/files
George-Miao commentedon Oct 20, 2023
Close due to lack of activity. Re-open if furthur progress is made.
George-Miao commentedon Apr 13, 2026
Reopen since I'm investigating on supporting more io-uring features in compio lately.
DXist commentedon Apr 13, 2026
When I experimented with compio-rs' fork, I exposed
FixedFdandDriver::register_fdin the public Driver API. I needed this for a custom event loop without an async layer.spnngl commentedon Oct 29, 2026
Some data point for this issue, in case it helps prioritise.
I prototyped fixed files on compio 0.19.2 (compio-driver 0.12.5) for a DNS server: the listening UDP socket of
each thread registered with
Runtime::register_files, andRecvMsgMulti(behindrecv_from_multi) and theSendMsgofsend_tobuilt withtypes::Fixed(index)instead oftypes::Fdwhen the fd is registered (athread-local fd → index map, a hack for measuring only).
dnsperf against a server answering from its cache, one core, Linux 7.0, io_uring, three alternating rounds:
488k-513k queries/s with plain fds, 501k-517k with fixed files: about +1%, within noise. Each thread has its own
SO_REUSEPORTsocket, so the file refcount the kernel takes per operation is never contended, and themultishot receive takes it once for many datagrams. So, for socket workloads like this one, fixed files seem to
matter little; they might matter more with a socket shared between threads or many short operations per fd.
What the prototype showed about the API: the registration already exists (
register_files), but no op canuse it, since every op builds its SQE from
as_fd().as_raw_fd(). A small, opt-in way would be forSharedFd(or a wrapper type) to carry an optional registered index that the io_uring ops check when building their
entry, falling back to the raw fd otherwise, so the other drivers and the public API stay unchanged.