The scheduler policy suite's last three cases: nested Operation, adversarial placement, and what a storm costs its waker

issues/kernel/an-idle-cpu-that-slept-before-the-surplus-is-never-probed.md

@@ -0,0 +1,53 @@
+---
+status: open
+kind: finding
+opened: 2026-08-22
+---
+
+# An idle CPU that slept before the surplus appeared is never probed again
+
+The balance path is pull-only. `SchedPass::post_steal_probe`
+(`toyos-sched/src/cpu.rs`) runs on the idle path, picks the CPU publishing the
+most surplus, and returns without posting anything if that surplus is under two.
+The probe node is one-shot, so there is at most one outstanding per CPU, and
+nothing re-posts it: `driver::execute`'s `Action::Idle` arm halts the CPU, and a
+CPU with no run queue, no deadline and no device work stays halted until an
+interrupt arrives. There is no push half — a CPU that gains surplus tells
+nobody.
+
+So a CPU that reached its idle pass while every other CPU still published a
+surplus of zero, and then halted, is not given work by the balance path however
+much surplus appears afterwards. What recovers it is a kick from somewhere else,
+and every path that produces one is a path that was already giving that CPU the
+work: spawn placement kicks the CPU it placed on, a wake kicks the waiter's home
+CPU.
+
+Measured, `toyos-sched/sim/tests/policy.rs`'s
+`the_balance_path_drains_the_cpu_an_adversary_loaded` over 20 seeds per width,
+with every thread spawned onto cpu0 (`workload::PlacementShape::AllOn`):
+
+| cpus | threads | seeds reaching every CPU | seeds reaching a second CPU |
+|---|---|---|---|
+| 2 |  8 | 9/20 |  9/20 |
+| 4 | 16 | 2/20 | 13/20 |
+| 4 | 64 | 2/20 | 13/20 |
+| 8 | 64 | 0/20 | 15/20 |
+
+Where a thief was awake the drain is complete — the best seed at every width
+moves exactly `threads − 2` tasks, cpu0's whole surplus above the floor
+`answer_steal_requests` stops at. The first column is the CPUs that lost the
+race, and it falls with width because all of them have to win it.
+
+**Reachable in the kernel, not only in the model.** A wake posts to the waiter's
+*home* CPU whatever that CPU's load is, and a home is where spawn placement put
+the thread when it was created. A process whose threads were all placed on one
+CPU while the machine was otherwise busy, parked, and then woken together, is
+exactly the state above: one CPU with N runnable threads and the sleepers with
+no probe outstanding. Nothing has measured how often that composition occurs on
+a real boot.
+
+Not filed as a defect because no shipped workload has been shown to reach it,
+and the fix is a design decision the balance path does not currently make: a
+push on surplus, or a bounded re-arm of the probe, both of which cost an idle
+CPU wakes it is currently right to be without
+(`kernel/CLAUDE.md`: anything added to the idle loop is an audio change).

kernel/src/actuator.rs

@@ -177,6 +177,26 @@ actuators! {
     /// See `kernel/src/nvme_gate.rs`.
     nvme_spent_budget = "nvme-spent-budget";
 
+    /// Establish three nested `scheduler::Operation`s with known deadlines and
+    /// report what every level observed and what each drop restored — once from
+    /// a boot phase, which has no task, and once from `iod`'s body, which is
+    /// one.
+    ///
+    /// **The law it reads has no host-side reader and cannot get one.**
+    /// `Operation::begin` stores `outer.min(until)`, so an inner establishment
+    /// may only narrow; the type reaches `percpu::cpu_id` and
+    /// `driver::current_handle`, and `kernel/` is excluded from the host
+    /// workspace, so nothing off a booted machine can construct one. The
+    /// establishments the other gates drive are incidental to what those gates
+    /// are about — `nvme-spent-budget` proves a narrowing happened by the
+    /// refusal it produces and reads none of the values — and every one of them
+    /// is the task-less kind.
+    ///
+    /// It measures and stages nothing: no driver behaviour changes with it on,
+    /// no device is touched, and every guard is dropped before the function
+    /// returns. See `kernel/src/sched_gate.rs`.
+    sched_operation_nesting = "sched-operation-nesting";
+
     /// Answer SYNCHRONIZE CACHE with the ILLEGAL REQUEST / INVALID COMMAND
     /// OPERATION CODE a stick with no write cache gives. QEMU's `scsi-disk`
     /// implements 0x35 for every front end that reaches it, so the difference

kernel/src/iod.rs

@@ -55,6 +55,18 @@ pub fn start() {
 extern "C" fn body(_arg: u64) -> ! {
     let parkable = scheduler::Parkable::at_entry();
     let handle = crate::sched::driver::current_handle().expect("iod runs as a task");
+    // The task half of the operation-nesting gate, and this thread is where it
+    // lives because of the sentence above: `iod` is the one context in this
+    // kernel that is a task, exists on every boot, and reaches its loop with
+    // nothing to do — so asking the question here displaces no work and needs
+    // no thread of its own. Every establishment any other gate drives is a
+    // boot phase's, which is the *other* slot (`kernel/src/sched_gate.rs`).
+    // Between `at_entry` above and the arm below because it must not be inside
+    // an establishment, which is what `at_entry` refuses, and it leaves none.
+    #[cfg(feature = "boot-actuators")]
+    if crate::actuator::sched_operation_nesting() {
+        crate::sched_gate::run("iod");
+    }
     // Armed once and held across the loop — §5.3a's edge contract: a producer
     // that pushes while this thread is draining must find the watch still
     // armed.

kernel/src/main.rs

@@ -56,6 +56,8 @@ mod input_merge_test;
 mod usb_gate;
 #[cfg(feature = "boot-actuators")]
 mod nvme_gate;
+#[cfg(feature = "boot-actuators")]
+mod sched_gate;
 mod block;
 mod gpt;
 mod page_cache;
@@ -591,6 +593,15 @@ unsafe fn kernel_main(kernel_args: &KernelArgs) -> ! {
     vfs::init();
     process::init();
     scheduler::init();
+    // The task-less half of the operation-nesting gate: a boot phase runs on
+    // the BSP with no current task, so what it establishes in is the per-CPU
+    // slot. `iod`'s body runs the other half. Here rather than earlier because
+    // this is the phase that owns the scheduler; it touches no device, waits
+    // for nothing and leaves no establishment behind.
+    #[cfg(feature = "boot-actuators")]
+    if actuator::sched_operation_nesting() {
+        sched_gate::run("boot");
+    }
     pipe::init();
     inbox::init();
 

kernel/src/sched_gate.rs

@@ -0,0 +1,100 @@
+//! `sched-operation-nesting`: the in-guest gate on what nesting an
+//! [`Operation`] may and may not do.
+//!
+//! **The law is one line and nothing host-side can reach it.**
+//! `scheduler::Operation::begin` stores `outer.min(until)` and its `Drop`
+//! restores what it displaced, so an inner establishment can only *narrow* — a
+//! caller cannot buy itself more device time by starting a second operation
+//! inside the first, which is the failure `block::OPERATION` exists to stop
+//! arriving one layer lower. The type reaches `percpu::cpu_id` and
+//! `driver::current_handle`, and `kernel/` is excluded from the host workspace,
+//! so nothing outside a booted machine can construct one. This is what a test
+//! can read instead.
+//!
+//! **It measures and stages nothing.** No driver behaviour changes with it
+//! armed, no device is touched and no deadline it establishes outlives the
+//! function: three guards on one stack, dropped in reverse, and the slot is
+//! back to where the caller left it. `io_depth_probe` is the same shape of
+//! actuator — an instrument rather than an injection.
+//!
+//! **Both homes, because the slot a context establishes in is a decision too.**
+//! A task's word lives on its [`TaskHandle`](crate::sched::payload::TaskHandle)
+//! and a context with no task uses one slot per CPU, and until this ran in two
+//! places every establishment any test had ever driven was the second kind:
+//! `nvme_gate` and `usb_gate` both establish from a boot phase, where there is
+//! no current task at all. So [`run`] is called twice — once from
+//! `kernel_main`, and once from `iod`'s body, which is the kernel's own task
+//! that reaches its loop with nothing to do (`crate::iod`) and is therefore the
+//! one context that can ask this question without displacing work.
+//!
+//! What each line carries is the *offset* from the instant the sequence
+//! started, not an absolute deadline, so the numbers a test asserts on are the
+//! numbers this file asks for and not a function of when the machine booted.
+
+use crate::clock;
+use crate::scheduler::Operation;
+use crate::time::{Deadline, Duration};
+
+/// The three deadlines, as offsets from the base instant.
+///
+/// Chosen so that each nesting step is a different question: [`INNER`] is
+/// earlier than [`OUTER`], so level 2 must narrow; [`WIDER`] is later than
+/// both, so level 3 must change nothing. Their spacing is arbitrary and their
+/// magnitudes are irrelevant — nothing waits for any of them.
+const OUTER: u64 = 1_000_000_000;
+const INNER: u64 = 250_000_000;
+const WIDER: u64 = 4_000_000_000;
+
+/// Establish three nested operations and report what every level observed.
+///
+/// `site` names the context, which is the whole of what distinguishes the two
+/// calls: `boot` has no task and establishes in its CPU's slot, `iod` is a task
+/// and establishes in its own handle's.
+pub fn run(site: &str) {
+    let base = clock::now();
+    log!(
+        "sched-op: {site} outside established={}",
+        Operation::established(),
+    );
+
+    let level = |until: u64| Deadline::at(base + Duration::from_nanos(until));
+    let observed = || Operation::deadline().nanos() - base.nanos_since_boot();
+
+    let outer = Operation::begin(level(OUTER));
+    log!(
+        "sched-op: {site} begin level=1 asked={OUTER} observed={}",
+        observed(),
+    );
+    {
+        let inner = Operation::begin(level(INNER));
+        log!(
+            "sched-op: {site} begin level=2 asked={INNER} observed={}",
+            observed(),
+        );
+        {
+            // The one that matters: an establishment asking for *more* than the
+            // frame above it, which must change nothing at all.
+            let _wider = Operation::begin(level(WIDER));
+            log!(
+                "sched-op: {site} begin level=3 asked={WIDER} observed={}",
+                observed(),
+            );
+        }
+        log!(
+            "sched-op: {site} end level=3 observed={} established={}",
+            observed(),
+            Operation::established(),
+        );
+        drop(inner);
+        log!(
+            "sched-op: {site} end level=2 observed={} established={}",
+            observed(),
+            Operation::established(),
+        );
+    }
+    drop(outer);
+    log!(
+        "sched-op: {site} end level=1 established={}",
+        Operation::established(),
+    );
+}

kernel/src/scheduler.rs

@@ -218,6 +218,13 @@ impl Parkable {
 /// exactly the failure `block::OPERATION` exists to stop, arriving one layer
 /// lower. The guard restores what it displaced rather than clearing the slot,
 /// so the outer operation survives the inner one ending.
+///
+/// **Both halves of that paragraph are gated in a guest and can only be.** This
+/// type reaches [`percpu::cpu_id`] and [`driver::current_handle`], and `kernel/`
+/// is outside the host workspace, so nothing off a booted machine can construct
+/// one: `kernel/src/sched_gate.rs` establishes three of them with known
+/// deadlines in both homes and prints what every level observed, and the
+/// `operation_nesting` test recomputes the running minimum from what it printed.
 #[must_use = "an operation lasts exactly as long as this guard"]
 pub struct Operation {
     /// The handle whose slot this establishment wrote, or `None` for the

tests/test-durations

@@ -238,6 +238,7 @@ nvme_home_roundtrip 13
 nvme_image_is_held_by_one_guest 0
 nvme_large_device 6052
 nvme_wide_sector 3500
+operation_nesting 5075
 panic_recovery 130
 poller_capacity 36
 port_poll_churn 84
@@ -306,6 +307,7 @@ usb_transport_break 5408
 va_exhaustion 6159
 virtio_net_no_msix 2039
 virtio_used_ring 4189
+wake_storm_cost 337
 wall_clock_file 4642
 wall_clock_now 13
 wall_clock_refusals 25658