Repository navigation
Description
femu-cxl-ssd can let the guest reach cached pages in three ways, and each gives up something:
der=off(the default) sends every access through MMIO. A hit costs 6.5 us here, and a guestlock-prefixed read-modify-write reaches the device as a separate read and write, so it is not atomic.der=memslotmakes every map and unmap a memory transaction: a miss that maps a page costs 3.3 ms with a 256-page cache (52 us forder=off). Past the 1,024-alias cap the other cached pages stay on MMIO; with a 2,048-page hot set on a 4,096-page cache, half of the accesses went through MMIO. It cannot see writes through an alias, so it programs every evicted page it mapped, even under a read-only load.der=cylonis fast, but it needs a patched host kernel: the Cylon kernel with the fixes on MoatLab/Cylonmaster(see "Host kernel" incxlssd.md).
I would like to propose a fourth mode, der=uffd, and ask whether you would take it before I turn the prototype into a PR. Prototype: https://github.com/wnsah814/FEMU/tree/cxl-der-uffd (one commit on 159dc55).
Design
- One mapping. The whole window is one RAM alias of a shared memfd backend. A page is cached when it is present in the backend's page tables. Eviction zaps it with
MADV_DONTNEED: KVM's MMU notifier drops the EPT entry, and the data stays in the page cache. - Miss. Touching a zapped page raises a userfaultfd minor fault, whether from the guest through KVM or from QEMU itself. A handler thread charges the miss to the FTL, under the media lock as a linked NVMe controller does, and resolves it with
UFFDIO_CONTINUEonce the media time has passed. Fills wait in a timer queue, so misses to different pages overlap. Prefetch works as on the MMIO path. - Dirty tracking. Pages are continued write-protected (
UFFDIO_CONTINUE_MODE_WP). The first write of each residency raises a WP fault that marks the page dirty, so only written pages are programmed. - No emulation. Hits, guest atomics, instruction fetch and page tables on CXL memory run on the hardware as on RAM. There is no kernel change.
- Pages in use. A filled page is not evicted until the thread that faulted on it faults elsewhere (
UFFD_FEATURE_THREAD_ID) or 1 ms passes. An instruction that needs several pages at once must have all of them mapped, so the device refusescache-policy=lifoand sets of fewer than 4 ways, and stays on MMIO for them. - Ownership. The first decoded access maps the window, but only when no other access is in the gate: with
concurrent-misses, another MMIO miss could still be filling the cache (qtestcxl-wait-uffd). Flush, invalidation, a way change and teardown stop the handler, wake any fault still waiting, and hand the cache back to the MMIO path; the next access maps the window again. The alias is used only when one HDM decoder covers the whole window with no DPA skip.
Requirements
- A shared, preallocated
memory-backend-memfd. - Linux 6.4 or later, for
UFFDIO_CONTINUE_MODE_WP; without it the device stays on MMIO. We ran 6.4.6 and, nested, Ubuntu's stock 6.8. - Under KVM, access to
/dev/userfaultfdorCAP_SYS_PTRACE(we ran as root). Under TCG and qtest no privilege is needed. - A guest with HLT passed through (
-overcommit cpu-pm=on) and PV async PF off (kvm-asyncpf=off). Otherwise KVM's async-PF worker faults pages in for write, every read looks like a first write, and read-only accesses took 269 us on average instead of 25 us.
Measured
Xeon E5-2699 v3, guest with 4 vCPUs (8 for the 8-thread row), 1 GiB window, FIFO fully associative, read-ns 40 us, program-ns 200 us, random reads over 64 MiB. Every comparison ran twice, in alternating order.
With a 256-page cache:
der=off |
der=off, concurrent-misses=on |
memslot |
uffd |
|
|---|---|---|---|---|
| hit latency | 6.46 us | 5.31 us | 0.09 us | 0.05 us |
| random reads, 1 / 4 threads (kops) | 19.4 / 22.2 | 19.7 / 60.4 | 0.3 / 0.5 | 14.9 / 54.3 |
| programs per eviction, 0% / 50% writes | 0.000 / 0.501 | 0.000 / 0.501 | 1.000 / 1.000 | 0.000 / 0.501 |
lock add lost, 4 threads |
0.1% | 54.9% | 0.0% | 0.0% |
With a 4,096-page cache and four threads, uffd does 70.9 kops, against 88.9 for concurrent-misses=on, 28.1 for der=off and 0.3 for memslot.
Against der=cylon, with the host booted into the fixed Cylon kernel (Cylon master plus MoatLab/Cylon#2) and the same build, 256-page cache:
der=off |
cylon |
uffd |
|
|---|---|---|---|
| hit latency | 6.23 us | 0.07 us | 0.05 us |
| per miss, one thread, hits taken out | 51.5 us | 57.3 us | 67.9 us |
| random reads, 4 threads (kops) | 21.3 to 21.4 | 49.9 to 59.5 | 53.1 to 53.5 |
| random reads, 8 threads, 8 vCPUs (kops) | 20.1 to 21.4 | 58.8 to 60.2 | 77.3 to 77.7 |
lock add lost, 4 threads |
0.2% | 0.0% | 0.0% |
A uffd miss costs about 10 us more than a cylon miss, and the first write to a resident, clean page costs a 17 us WP fault (about 0.5 us under cylon). With 8 threads concurrent-misses=on reached 111 kops; cylon did not gain after 4 threads on this host, and we have not looked into why.
Correctness under KVM: 4,096 pages written and read back three times with no mismatch; no lock add lost with 1 and 4 threads; 200 invalidations while three threads read, with no hang and no data change; the guest's region destroyed and re-created with no data change; 200 two-page rep movsb copies done with FIFO at 4 and 256 ways. The data, atomics, invalidation and copy checks also passed on Ubuntu's stock 6.8 kernel, nested. The FEMU qtests pass (536 of 536, also under ASan/UBSan), including four new ones that run under TCG without privilege.
Limits
- Misses cost more than
der=offwith concurrent misses: 16 us per miss, and 54.3 against 60.4 kops with four threads. - Each eviction costs a VM-wide TLB flush;
cylonafter femu/cxl: drop Cylon TLB flushes that have nothing to remove #212 skips almost all of them through the accessed bit. With 8 guest threads the single handler is 92% busy. Several handlers on the same userfaultfd did not help in our try, and each zap got slower, which points at this flush path. - No LIFO and no sets under 4 ways (above), so the policy comparison in the Cylon paper (Fig. 8, Table 3) cannot be repeated with
uffd. The handler sees no hits, soclockacts close to FIFO, as it does with direct hits undermemslotandcylon. - Not supported yet: CCA, a linked NVMe controller, direct ratios.
- HLT passthrough keeps every idle vCPU's host CPU busy.
- memfd backend only: hugetlb minor faults would come at 2 MiB.
- Timed on one Haswell host; the code is prototype quality.
Questions
- Would you consider a userfaultfd-based mode for FEMU, as a new
dervalue next tomemslotandcylon? - The guest CPU flags above belong to the guest's configuration, not the device's. Should the device refuse to activate without them, or only document them?
Activity
huaicheng commentedon Oct 3, 2026
Thanks @wnsah814 for the work and the measurements. They help clarify the trade-offs.
Yes, we'd like to add
der=uffd. It provides an easier option alongsideder=cylon, since it doesn't require a patched host kernel. Correct handling of atomics on cache hits also addresses a problem withder=off. Misses and first writes are more expensive, socylonremains the choice for latency studies.The policy and CCA interface should be independent of the mechanism. Cache policies, ways, prefetch, CCA commands and counters should have the same semantics across
dervalues. Users should be able to compare mechanisms without changing the rest of the configuration.Please keep that logic in shared code, with small backend ops for memslot, cylon and uffd: map, unmap, sample dirty/accessed state, and flush. Reject unsupported configurations at startup, with a message identifying a
dervalue that supports them. This includes LIFO, small sets, uncached ranges, ratios and the NVMe link where uffd cannot support them.I'm concerned about forward progress with the 4-way rule. It's a heuristic: an instruction that needs more pages could repeatedly evict pages it needs to complete. Could you add detection for a thread faulting again on pages it just lost, and pin those pages so it can make progress? Please also add a stress test with guest page tables on CXL memory.
On question 2, a warning and documentation of the guest CPU settings should be sufficient. Those settings affect performance, not correctness.
A few optimizations seem worth investigating without kernel changes:
fault and then a write-protect fault; the userfaultfd event already says whether the access was a write,
so the page can be continued writable and marked dirty in one step.
of the first write in hardware, so the ~17 us WP round trip on a resident clean page goes away and only
the ring has to be harvested at eviction.
through one thread, which you measured at 92% busy with 8 threads; disjoint fds spread the faults, and
only the short FTL charge still serializes.
goes through the MMU notifier and flushes the whole VM; keeping a few free slots takes that off the
miss path, and contiguous victims can share one invalidation.
CLOCK behaves like FIFO; page_idle reaches the EPT accessed bits through KVM's test_young notifier
without a kernel change.
We'll benchmark your branch against
cylonon our CXL test host once the current runs finish. We're also investigating whycylonstops scaling beyond four threads. We suspect the global lock in its miss path, which your handler avoids. We'll post the results here.