Skip to content

refactor(buf): better IoBuf - #555

Merged
George-Miao merged 26 commits into
compio-rs:masterfrom
George-Miao:refactor/buf
Dec 5, 2025
Merged

George-Miao merged 26 commits into
compio-rs:masterfrom
George-Miao:refactor/buf

Conversation

@George-Miao

Copy link
Copy Markdown
Member

This PR refactors compio-buf with following changes:

  • Remove platform-dependent IoSlice{,Mut}. Use IoBuffer{,Mut} instead
  • IoBuf only requires single unsafe fn buffer(&self) -> IoBuffer and IoBufMut only requires fn uninit_len(&self) -> usize.
  • Platform-dependent IoSlice{,Mut} is being moved to driver, renamed to SysSlice and can be converted from IoBuffer{,Mut}.
  • Tidy up IoVectoredBuf{,Mut}:
    • IoVectoredBuf now only requires unsafe fn iter_buffer(&self) -> impl Iterator<Item = IoBuffer>;
    • IoVectoredBufMut now only requires fn uninit_len_of(&self, idx: usize) -> usize; which returns uninit_len of given index corresponding to index in iter_buffer.

George-Miao and others added 25 commits December 1, 2025 17:54
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
@George-Miao George-Miao mentioned this pull request Dec 3, 2025
5 of 7 tasks
@George-Miao

George-Miao commented Dec 3, 2025 •

Copy link
Copy Markdown
Member Author

I'm thinking of removing the unsafe requirement for constructing IoBuffer{,Mut}, i.e., IoBuf::buffer. We treat it as something similar to as_ptr, where constructing the pointer is harmless (or even better, let it return a &[u8] directly). It's only unsafe at the consumption site, which is IoBuffer::slice (if we stick with IoBuffer) and all the OpCode within compio-driver.

What do you think @Berrysoft

@George-Miao
George-Miao marked this pull request as draft December 3, 2025 07:10
@Berrysoft

Copy link
Copy Markdown
Member

It's OK. Actually I'm considering making the trait OpCode unsafe, so that we can leverage the unsafety to the driver.

@Berrysoft

Berrysoft commented Dec 3, 2025 •

Copy link
Copy Markdown
Member

Another reason we previously make it unsafe is that IoBuf{,Mut} is not guaranteed to be pinned, so the address might change after moving. But as you've said, as_ptr also don't ensure the stability of the pointer.

@George-Miao
George-Miao force-pushed the refactor/buf branch 8 times, most recently from 8beabb6 to 7430913 Compare December 4, 2025 10:23
@George-Miao
George-Miao marked this pull request as ready for review December 4, 2025 10:44
@George-Miao
George-Miao requested a review from Berrysoft December 4, 2025 10:44
@George-Miao George-Miao self-assigned this Dec 4, 2025
@George-Miao George-Miao added package: buf Related to compio-buf package: driver Related to compio-driver refactor Refactoring existing code labels Dec 4, 2025
Comment thread compio-buf/src/io_buf.rs Resolved
@George-Miao
George-Miao requested a review from Berrysoft December 4, 2025 21:09

@Berrysoft Berrysoft left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@George-Miao
George-Miao merged commit 65917da into compio-rs:master Dec 5, 2025
@George-Miao
George-Miao deleted the refactor/buf branch December 5, 2025 03:50
This was referenced Jan 28, 2026
fantix pushed a commit to fantix/compio that referenced this pull request Mar 3, 2026
---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
@github-actions github-actions Bot mentioned this pull request Aug 4, 2026
FalkWoldmann pushed a commit to FalkWoldmann/compio that referenced this pull request Sep 20, 2026
…ompio-rs#555

Issue compio-rs#220 raised exactly this in 2024 and it was fixed by making IoBuf
and IoBufMut unsafe traits. PR compio-rs#555 removed the marker as part of a
refactor whose description promised to replace it with a single
`unsafe fn buffer`, which never landed. The recommended fix is a
restoration of a property the project already decided it wanted.
FalkWoldmann added a commit to FalkWoldmann/compio that referenced this pull request Sep 21, 2026
Bug 2 -- unsafe code trusting safe trait methods to agree with each other --
has been fixed four times on this branch, once per call site, because the
traits gave unsafe code nothing to rely on. This fixes the cause instead.

`IoBuf`, `IoBufMut`, `IoVectoredBuf`, `IoVectoredBufMut` and `SetLen` are
`unsafe trait`s again, each stating the obligations unsafe code already
depended on: stability of the pointer and length across calls, containment of
the initialized prefix within the buffer's extent, an initialized prefix, and
idempotent vectored iteration. `IoVectoredBuf`'s idempotency was already
written down as a "Note for implementors" -- unsafe code builds an iovec array
from one traversal and resolves completions against another, so it was a
safety obligation wearing a convention's clothes.

This restores the state the crate was in from compio-rs#220 until compio-rs#555 removed the
marker in favour of an `unsafe fn buffer` that never landed.

BREAKING CHANGE: downstream implementors must write `unsafe impl` and uphold
the documented obligations. Code that only uses buffers is unaffected.

Two things the change turned up, both verified with Miri:

- `BytesMut::as_uninit` built a `capacity()`-long slice from a pointer that
  resolves through `DerefMut` and so carries provenance for only `len()`
  bytes. Undefined behaviour whenever the buffer has spare capacity, which for
  a read buffer is always. Its SAFETY comment asserted the opposite as a
  "DEPENDENCY LEMMA". The pointer now comes from `spare_capacity_mut`, which
  `bytes` derives from its own owning pointer.

- A first draft of `SetLen`'s contract required `as_init().len() == n` after
  `set_len(n)`. Three in-tree impls -- `[u8]`, `[u8; N]` and `MmapMut` -- are
  no-ops, because a fixed-size always-initialized buffer has no boundary to
  move. The contract now requires only what memory safety needs: that
  `as_init` never reports an uninitialized byte and never exceeds `as_uninit`.
  A guarantee the crate's own code violates is worse than none.

The defensive measures added earlier all stay. They are no longer load-bearing
-- a correct implementation cannot reach them -- but they keep a broken
implementation merely wrong rather than unsafe. The adversarial test types are
now marked as deliberately contract-violating.

Two new guards:

- Static assertions that each trait is still `unsafe`. compio-rs#555 dropped the marker
  and shipped because nothing failed; these fail `cargo check` with E0199.
- A contract test that checks stability and containment empirically for every
  in-tree buffer type, including a partially filled `BytesMut` and a spilled
  `SmallVec`. It is what found the `BytesMut` bug.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

package: buf Related to compio-buf package: driver Related to compio-driver refactor Refactoring existing code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants