Skip to content

femu-cxl-ssd: a userfaultfd DER mode (der=uffd) for direct hits without a host kernel patch #214

Open

Description

@wnsah814

femu-cxl-ssd can let the guest reach cached pages in three ways, and each gives up something:

  • der=off (the default) sends every access through MMIO. A hit costs 6.5 us here, and a guest lock-prefixed read-modify-write reaches the device as a separate read and write, so it is not atomic.
  • der=memslot makes every map and unmap a memory transaction: a miss that maps a page costs 3.3 ms with a 256-page cache (52 us for der=off). Past the 1,024-alias cap the other cached pages stay on MMIO; with a 2,048-page hot set on a 4,096-page cache, half of the accesses went through MMIO. It cannot see writes through an alias, so it programs every evicted page it mapped, even under a read-only load.
  • der=cylon is fast, but it needs a patched host kernel: the Cylon kernel with the fixes on MoatLab/Cylon master (see "Host kernel" in cxlssd.md).

I would like to propose a fourth mode, der=uffd, and ask whether you would take it before I turn the prototype into a PR. Prototype: https://github.com/wnsah814/FEMU/tree/cxl-der-uffd (one commit on 159dc55).

Design

  • One mapping. The whole window is one RAM alias of a shared memfd backend. A page is cached when it is present in the backend's page tables. Eviction zaps it with MADV_DONTNEED: KVM's MMU notifier drops the EPT entry, and the data stays in the page cache.
  • Miss. Touching a zapped page raises a userfaultfd minor fault, whether from the guest through KVM or from QEMU itself. A handler thread charges the miss to the FTL, under the media lock as a linked NVMe controller does, and resolves it with UFFDIO_CONTINUE once the media time has passed. Fills wait in a timer queue, so misses to different pages overlap. Prefetch works as on the MMIO path.
  • Dirty tracking. Pages are continued write-protected (UFFDIO_CONTINUE_MODE_WP). The first write of each residency raises a WP fault that marks the page dirty, so only written pages are programmed.
  • No emulation. Hits, guest atomics, instruction fetch and page tables on CXL memory run on the hardware as on RAM. There is no kernel change.
  • Pages in use. A filled page is not evicted until the thread that faulted on it faults elsewhere (UFFD_FEATURE_THREAD_ID) or 1 ms passes. An instruction that needs several pages at once must have all of them mapped, so the device refuses cache-policy=lifo and sets of fewer than 4 ways, and stays on MMIO for them.
  • Ownership. The first decoded access maps the window, but only when no other access is in the gate: with concurrent-misses, another MMIO miss could still be filling the cache (qtest cxl-wait-uffd). Flush, invalidation, a way change and teardown stop the handler, wake any fault still waiting, and hand the cache back to the MMIO path; the next access maps the window again. The alias is used only when one HDM decoder covers the whole window with no DPA skip.

Requirements

  • A shared, preallocated memory-backend-memfd.
  • Linux 6.4 or later, for UFFDIO_CONTINUE_MODE_WP; without it the device stays on MMIO. We ran 6.4.6 and, nested, Ubuntu's stock 6.8.
  • Under KVM, access to /dev/userfaultfd or CAP_SYS_PTRACE (we ran as root). Under TCG and qtest no privilege is needed.
  • A guest with HLT passed through (-overcommit cpu-pm=on) and PV async PF off (kvm-asyncpf=off). Otherwise KVM's async-PF worker faults pages in for write, every read looks like a first write, and read-only accesses took 269 us on average instead of 25 us.

Measured

Xeon E5-2699 v3, guest with 4 vCPUs (8 for the 8-thread row), 1 GiB window, FIFO fully associative, read-ns 40 us, program-ns 200 us, random reads over 64 MiB. Every comparison ran twice, in alternating order.

With a 256-page cache:

der=off der=off, concurrent-misses=on memslot uffd
hit latency 6.46 us 5.31 us 0.09 us 0.05 us
random reads, 1 / 4 threads (kops) 19.4 / 22.2 19.7 / 60.4 0.3 / 0.5 14.9 / 54.3
programs per eviction, 0% / 50% writes 0.000 / 0.501 0.000 / 0.501 1.000 / 1.000 0.000 / 0.501
lock add lost, 4 threads 0.1% 54.9% 0.0% 0.0%

With a 4,096-page cache and four threads, uffd does 70.9 kops, against 88.9 for concurrent-misses=on, 28.1 for der=off and 0.3 for memslot.

Against der=cylon, with the host booted into the fixed Cylon kernel (Cylon master plus MoatLab/Cylon#2) and the same build, 256-page cache:

der=off cylon uffd
hit latency 6.23 us 0.07 us 0.05 us
per miss, one thread, hits taken out 51.5 us 57.3 us 67.9 us
random reads, 4 threads (kops) 21.3 to 21.4 49.9 to 59.5 53.1 to 53.5
random reads, 8 threads, 8 vCPUs (kops) 20.1 to 21.4 58.8 to 60.2 77.3 to 77.7
lock add lost, 4 threads 0.2% 0.0% 0.0%

A uffd miss costs about 10 us more than a cylon miss, and the first write to a resident, clean page costs a 17 us WP fault (about 0.5 us under cylon). With 8 threads concurrent-misses=on reached 111 kops; cylon did not gain after 4 threads on this host, and we have not looked into why.

Correctness under KVM: 4,096 pages written and read back three times with no mismatch; no lock add lost with 1 and 4 threads; 200 invalidations while three threads read, with no hang and no data change; the guest's region destroyed and re-created with no data change; 200 two-page rep movsb copies done with FIFO at 4 and 256 ways. The data, atomics, invalidation and copy checks also passed on Ubuntu's stock 6.8 kernel, nested. The FEMU qtests pass (536 of 536, also under ASan/UBSan), including four new ones that run under TCG without privilege.

Limits

  • Misses cost more than der=off with concurrent misses: 16 us per miss, and 54.3 against 60.4 kops with four threads.
  • Each eviction costs a VM-wide TLB flush; cylon after femu/cxl: drop Cylon TLB flushes that have nothing to remove #212 skips almost all of them through the accessed bit. With 8 guest threads the single handler is 92% busy. Several handlers on the same userfaultfd did not help in our try, and each zap got slower, which points at this flush path.
  • No LIFO and no sets under 4 ways (above), so the policy comparison in the Cylon paper (Fig. 8, Table 3) cannot be repeated with uffd. The handler sees no hits, so clock acts close to FIFO, as it does with direct hits under memslot and cylon.
  • Not supported yet: CCA, a linked NVMe controller, direct ratios.
  • HLT passthrough keeps every idle vCPU's host CPU busy.
  • memfd backend only: hugetlb minor faults would come at 2 MiB.
  • Timed on one Haswell host; the code is prototype quality.

Questions

  1. Would you consider a userfaultfd-based mode for FEMU, as a new der value next to memslot and cylon?
  2. The guest CPU flags above belong to the guest's configuration, not the device's. Should the device refuse to activate without them, or only document them?

Activity

  1. huaicheng commented on Oct 3, 2026

    @huaicheng
    Contributor

    Thanks @wnsah814 for the work and the measurements. They help clarify the trade-offs.

    Yes, we'd like to add der=uffd. It provides an easier option alongside der=cylon, since it doesn't require a patched host kernel. Correct handling of atomics on cache hits also addresses a problem with der=off. Misses and first writes are more expensive, so cylon remains the choice for latency studies.

    The policy and CCA interface should be independent of the mechanism. Cache policies, ways, prefetch, CCA commands and counters should have the same semantics across der values. Users should be able to compare mechanisms without changing the rest of the configuration.

    Please keep that logic in shared code, with small backend ops for memslot, cylon and uffd: map, unmap, sample dirty/accessed state, and flush. Reject unsupported configurations at startup, with a message identifying a der value that supports them. This includes LIFO, small sets, uncached ranges, ratios and the NVMe link where uffd cannot support them.

    I'm concerned about forward progress with the 4-way rule. It's a heuristic: an instruction that needs more pages could repeatedly evict pages it needs to complete. Could you add detection for a thread faulting again on pages it just lost, and pin those pages so it can make progress? Please also add a stress test with guest page tables on CXL memory.

    On question 2, a warning and documentation of the guest CPU settings should be sufficient. Those settings affect performance, not correctness.

    A few optimizations seem worth investigating without kernel changes:

    • Map a write miss writable immediately, avoiding the second fault. A write miss currently pays a minor
      fault and then a write-protect fault; the userfaultfd event already says whether the access was a write,
      so the page can be continued writable and marked dirty in one step.
    • Use KVM's dirty ring (PML) to track dirty pages instead of taking write-protect faults. PML logs the GPA
      of the first write in hardware, so the ~17 us WP round trip on a resident clean page goes away and only
      the ring has to be harvested at eviction.
    • Split the window across several userfaultfds, each with its own handler. One fd funnels every fault
      through one thread, which you measured at 92% busy with 8 threads; disjoint fds spread the faults, and
      only the short FTL charge still serializes.
    • Evict in batches in the background, so misses don't wait for the zap and TLB flush. Each MADV_DONTNEED
      goes through the MMU notifier and flushes the whole VM; keeping a few free slots takes that off the
      miss path, and contiguous victims can share one invalidation.
    • Use idle page tracking to provide CLOCK with actual access information. The handler never sees hits, so
      CLOCK behaves like FIFO; page_idle reaches the EPT accessed bits through KVM's test_young notifier
      without a kernel change.

    We'll benchmark your branch against cylon on our CXL test host once the current runs finish. We're also investigating why cylon stops scaling beyond four threads. We suspect the global lock in its miss path, which your handler avoids. We'll post the results here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions