diff --git a/issues/kernel/an-idle-cpu-that-slept-before-the-surplus-is-never-probed.md b/issues/kernel/an-idle-cpu-that-slept-before-the-surplus-is-never-probed.md new file mode 100644 index 00000000000..f625a5c6cc0 --- /dev/null +++ b/issues/kernel/an-idle-cpu-that-slept-before-the-surplus-is-never-probed.md @@ -0,0 +1,53 @@ +--- +status: open +kind: finding +opened: 2026-08-22 +--- + +# An idle CPU that slept before the surplus appeared is never probed again + +The balance path is pull-only. `SchedPass::post_steal_probe` +(`toyos-sched/src/cpu.rs`) runs on the idle path, picks the CPU publishing the +most surplus, and returns without posting anything if that surplus is under two. +The probe node is one-shot, so there is at most one outstanding per CPU, and +nothing re-posts it: `driver::execute`'s `Action::Idle` arm halts the CPU, and a +CPU with no run queue, no deadline and no device work stays halted until an +interrupt arrives. There is no push half — a CPU that gains surplus tells +nobody. + +So a CPU that reached its idle pass while every other CPU still published a +surplus of zero, and then halted, is not given work by the balance path however +much surplus appears afterwards. What recovers it is a kick from somewhere else, +and every path that produces one is a path that was already giving that CPU the +work: spawn placement kicks the CPU it placed on, a wake kicks the waiter's home +CPU. + +Measured, `toyos-sched/sim/tests/policy.rs`'s +`the_balance_path_drains_the_cpu_an_adversary_loaded` over 20 seeds per width, +with every thread spawned onto cpu0 (`workload::PlacementShape::AllOn`): + +| cpus | threads | seeds reaching every CPU | seeds reaching a second CPU | +|---|---|---|---| +| 2 | 8 | 9/20 | 9/20 | +| 4 | 16 | 2/20 | 13/20 | +| 4 | 64 | 2/20 | 13/20 | +| 8 | 64 | 0/20 | 15/20 | + +Where a thief was awake the drain is complete — the best seed at every width +moves exactly `threads − 2` tasks, cpu0's whole surplus above the floor +`answer_steal_requests` stops at. The first column is the CPUs that lost the +race, and it falls with width because all of them have to win it. + +**Reachable in the kernel, not only in the model.** A wake posts to the waiter's +*home* CPU whatever that CPU's load is, and a home is where spawn placement put +the thread when it was created. A process whose threads were all placed on one +CPU while the machine was otherwise busy, parked, and then woken together, is +exactly the state above: one CPU with N runnable threads and the sleepers with +no probe outstanding. Nothing has measured how often that composition occurs on +a real boot. + +Not filed as a defect because no shipped workload has been shown to reach it, +and the fix is a design decision the balance path does not currently make: a +push on surplus, or a bounded re-arm of the probe, both of which cost an idle +CPU wakes it is currently right to be without +(`kernel/CLAUDE.md`: anything added to the idle loop is an audio change). diff --git a/issues/kernel/scheduler-policy-behavior-has-no-quantified-suite.md b/issues/kernel/scheduler-policy-behavior-has-no-quantified-suite.md deleted file mode 100644 index 6e175af9d0c..00000000000 --- a/issues/kernel/scheduler-policy-behavior-has-no-quantified-suite.md +++ /dev/null @@ -1,54 +0,0 @@ ---- -status: open -kind: track -opened: 2026-08-20 ---- - -# Scheduler policy behavior has no quantified suite - -From the external review of 2026-08-20: the model checking and SMP stress -methodology verify functional correctness; the POLICY — "threads execute, -processes own fair share" — is encoded in implementation intent and validated -nowhere empirically. - -`toyos-sched/sim/tests/policy.rs` is the suite, host-side, no guest. Every case -measures and then asserts, and every bound came out as the same expression — -`(runnable threads on one CPU + 1) × (QUANTUM + 2 × RUN_CHUNK)`, the fair band's -insertion-key staleness, which turns out to govern all four: - -- **thread-count asymmetry** — one thread against N, both processes pure CPU: - the single-threaded one finishes exactly `N+1` quanta late at every N from 2 to - 64, and the deficit is unchanged when the window doubles twice. Its share over - the window it would take against one rival falls from 500‰ to 77‰ at N=64; - over four times that window it is 453‰ at N=4. So the floor is independent of - N asymptotically and is a *granularity*, not a per-window guarantee. Negative - control: per-thread shares take the same figure to 1,020 ms against a 336 ms - bound. -- **mixed interactive/background** — worst wake 637 ms at 64 hogs, which is the - spawn burst's stale keys; every other wake in a run is served inside one - quantum (worst per-run runner-up 8 ms at one CPU, mean 0.36 ms at one hog). -- **wakeup storms** — 64 waiters drain in 27.75 ms on one CPU, 3.1× the - 16-waiter figure: linear in the waiters one run queue holds, and the balance - path is moving tasks while it happens (43 migrations at eight CPUs). -- **starvation** — a runnable task waits at most one dispatch per rival: 50 ms at - 4 runnable threads, 680 ms at 65, at 0.83–0.97 of the derived bound. - -What is left, and what blocks it: - -- **Nested `Operation` narrowing.** The type is `kernel/src/scheduler.rs`'s and - reaches `percpu::cpu_id` and `driver::current_handle`; `kernel/` is excluded - from the host workspace, so nothing host-side can construct one. The law is one - line — `Operation::begin` stores `outer.min(until)` and its `Drop` restores - what it displaced, so an inner establishment can only narrow — and it needs a - guest gate. -- **Adversarial placement.** Spawn placement is least-loaded-with-rotation and - the simulator has no knob for anything else, so a lopsided machine cannot be - staged; what the storm exercises is the idle-probe steal path only. -- **The cost of raising a storm.** `wake_all` claims N waiters in one loop on the - waker's CPU, and the model's clock does not advance inside a step, so that loop - is free here. It is a guest measurement. - -Two findings the suite records rather than gates. The storm's worst drain *rises* -with machine width — 9 ms at one CPU, 57 ms at eight, with the mean flat at -~3.5 ms — so what the width costs is the tail. And at two CPUs the spawn burst's -residue outlives one interactive wake (30 ms runner-up, against 8 ms at one CPU). diff --git a/kernel/src/actuator.rs b/kernel/src/actuator.rs index 94c53a85b4e..bd30f644d20 100644 --- a/kernel/src/actuator.rs +++ b/kernel/src/actuator.rs @@ -177,6 +177,26 @@ actuators! { /// See `kernel/src/nvme_gate.rs`. nvme_spent_budget = "nvme-spent-budget"; + /// Establish three nested `scheduler::Operation`s with known deadlines and + /// report what every level observed and what each drop restored — once from + /// a boot phase, which has no task, and once from `iod`'s body, which is + /// one. + /// + /// **The law it reads has no host-side reader and cannot get one.** + /// `Operation::begin` stores `outer.min(until)`, so an inner establishment + /// may only narrow; the type reaches `percpu::cpu_id` and + /// `driver::current_handle`, and `kernel/` is excluded from the host + /// workspace, so nothing off a booted machine can construct one. The + /// establishments the other gates drive are incidental to what those gates + /// are about — `nvme-spent-budget` proves a narrowing happened by the + /// refusal it produces and reads none of the values — and every one of them + /// is the task-less kind. + /// + /// It measures and stages nothing: no driver behaviour changes with it on, + /// no device is touched, and every guard is dropped before the function + /// returns. See `kernel/src/sched_gate.rs`. + sched_operation_nesting = "sched-operation-nesting"; + /// Answer SYNCHRONIZE CACHE with the ILLEGAL REQUEST / INVALID COMMAND /// OPERATION CODE a stick with no write cache gives. QEMU's `scsi-disk` /// implements 0x35 for every front end that reaches it, so the difference diff --git a/kernel/src/iod.rs b/kernel/src/iod.rs index 910d088f208..2a991eb783a 100644 --- a/kernel/src/iod.rs +++ b/kernel/src/iod.rs @@ -55,6 +55,18 @@ pub fn start() { extern "C" fn body(_arg: u64) -> ! { let parkable = scheduler::Parkable::at_entry(); let handle = crate::sched::driver::current_handle().expect("iod runs as a task"); + // The task half of the operation-nesting gate, and this thread is where it + // lives because of the sentence above: `iod` is the one context in this + // kernel that is a task, exists on every boot, and reaches its loop with + // nothing to do — so asking the question here displaces no work and needs + // no thread of its own. Every establishment any other gate drives is a + // boot phase's, which is the *other* slot (`kernel/src/sched_gate.rs`). + // Between `at_entry` above and the arm below because it must not be inside + // an establishment, which is what `at_entry` refuses, and it leaves none. + #[cfg(feature = "boot-actuators")] + if crate::actuator::sched_operation_nesting() { + crate::sched_gate::run("iod"); + } // Armed once and held across the loop — §5.3a's edge contract: a producer // that pushes while this thread is draining must find the watch still // armed. diff --git a/kernel/src/main.rs b/kernel/src/main.rs index 6d33b5aedce..c251eea6808 100644 --- a/kernel/src/main.rs +++ b/kernel/src/main.rs @@ -56,6 +56,8 @@ mod input_merge_test; mod usb_gate; #[cfg(feature = "boot-actuators")] mod nvme_gate; +#[cfg(feature = "boot-actuators")] +mod sched_gate; mod block; mod gpt; mod page_cache; @@ -591,6 +593,15 @@ unsafe fn kernel_main(kernel_args: &KernelArgs) -> ! { vfs::init(); process::init(); scheduler::init(); + // The task-less half of the operation-nesting gate: a boot phase runs on + // the BSP with no current task, so what it establishes in is the per-CPU + // slot. `iod`'s body runs the other half. Here rather than earlier because + // this is the phase that owns the scheduler; it touches no device, waits + // for nothing and leaves no establishment behind. + #[cfg(feature = "boot-actuators")] + if actuator::sched_operation_nesting() { + sched_gate::run("boot"); + } pipe::init(); inbox::init(); diff --git a/kernel/src/sched_gate.rs b/kernel/src/sched_gate.rs new file mode 100644 index 00000000000..e61dfb24722 --- /dev/null +++ b/kernel/src/sched_gate.rs @@ -0,0 +1,100 @@ +//! `sched-operation-nesting`: the in-guest gate on what nesting an +//! [`Operation`] may and may not do. +//! +//! **The law is one line and nothing host-side can reach it.** +//! `scheduler::Operation::begin` stores `outer.min(until)` and its `Drop` +//! restores what it displaced, so an inner establishment can only *narrow* — a +//! caller cannot buy itself more device time by starting a second operation +//! inside the first, which is the failure `block::OPERATION` exists to stop +//! arriving one layer lower. The type reaches `percpu::cpu_id` and +//! `driver::current_handle`, and `kernel/` is excluded from the host workspace, +//! so nothing outside a booted machine can construct one. This is what a test +//! can read instead. +//! +//! **It measures and stages nothing.** No driver behaviour changes with it +//! armed, no device is touched and no deadline it establishes outlives the +//! function: three guards on one stack, dropped in reverse, and the slot is +//! back to where the caller left it. `io_depth_probe` is the same shape of +//! actuator — an instrument rather than an injection. +//! +//! **Both homes, because the slot a context establishes in is a decision too.** +//! A task's word lives on its [`TaskHandle`](crate::sched::payload::TaskHandle) +//! and a context with no task uses one slot per CPU, and until this ran in two +//! places every establishment any test had ever driven was the second kind: +//! `nvme_gate` and `usb_gate` both establish from a boot phase, where there is +//! no current task at all. So [`run`] is called twice — once from +//! `kernel_main`, and once from `iod`'s body, which is the kernel's own task +//! that reaches its loop with nothing to do (`crate::iod`) and is therefore the +//! one context that can ask this question without displacing work. +//! +//! What each line carries is the *offset* from the instant the sequence +//! started, not an absolute deadline, so the numbers a test asserts on are the +//! numbers this file asks for and not a function of when the machine booted. + +use crate::clock; +use crate::scheduler::Operation; +use crate::time::{Deadline, Duration}; + +/// The three deadlines, as offsets from the base instant. +/// +/// Chosen so that each nesting step is a different question: [`INNER`] is +/// earlier than [`OUTER`], so level 2 must narrow; [`WIDER`] is later than +/// both, so level 3 must change nothing. Their spacing is arbitrary and their +/// magnitudes are irrelevant — nothing waits for any of them. +const OUTER: u64 = 1_000_000_000; +const INNER: u64 = 250_000_000; +const WIDER: u64 = 4_000_000_000; + +/// Establish three nested operations and report what every level observed. +/// +/// `site` names the context, which is the whole of what distinguishes the two +/// calls: `boot` has no task and establishes in its CPU's slot, `iod` is a task +/// and establishes in its own handle's. +pub fn run(site: &str) { + let base = clock::now(); + log!( + "sched-op: {site} outside established={}", + Operation::established(), + ); + + let level = |until: u64| Deadline::at(base + Duration::from_nanos(until)); + let observed = || Operation::deadline().nanos() - base.nanos_since_boot(); + + let outer = Operation::begin(level(OUTER)); + log!( + "sched-op: {site} begin level=1 asked={OUTER} observed={}", + observed(), + ); + { + let inner = Operation::begin(level(INNER)); + log!( + "sched-op: {site} begin level=2 asked={INNER} observed={}", + observed(), + ); + { + // The one that matters: an establishment asking for *more* than the + // frame above it, which must change nothing at all. + let _wider = Operation::begin(level(WIDER)); + log!( + "sched-op: {site} begin level=3 asked={WIDER} observed={}", + observed(), + ); + } + log!( + "sched-op: {site} end level=3 observed={} established={}", + observed(), + Operation::established(), + ); + drop(inner); + log!( + "sched-op: {site} end level=2 observed={} established={}", + observed(), + Operation::established(), + ); + } + drop(outer); + log!( + "sched-op: {site} end level=1 established={}", + Operation::established(), + ); +} diff --git a/kernel/src/scheduler.rs b/kernel/src/scheduler.rs index 934eeb3e014..92fcbb8e1cc 100644 --- a/kernel/src/scheduler.rs +++ b/kernel/src/scheduler.rs @@ -218,6 +218,13 @@ impl Parkable { /// exactly the failure `block::OPERATION` exists to stop, arriving one layer /// lower. The guard restores what it displaced rather than clearing the slot, /// so the outer operation survives the inner one ending. +/// +/// **Both halves of that paragraph are gated in a guest and can only be.** This +/// type reaches [`percpu::cpu_id`] and [`driver::current_handle`], and `kernel/` +/// is outside the host workspace, so nothing off a booted machine can construct +/// one: `kernel/src/sched_gate.rs` establishes three of them with known +/// deadlines in both homes and prints what every level observed, and the +/// `operation_nesting` test recomputes the running minimum from what it printed. #[must_use = "an operation lasts exactly as long as this guard"] pub struct Operation { /// The handle whose slot this establishment wrote, or `None` for the diff --git a/tests/test-durations b/tests/test-durations index 30d80dd020b..ad463576e5b 100644 --- a/tests/test-durations +++ b/tests/test-durations @@ -238,6 +238,7 @@ nvme_home_roundtrip 13 nvme_image_is_held_by_one_guest 0 nvme_large_device 6052 nvme_wide_sector 3500 +operation_nesting 5075 panic_recovery 130 poller_capacity 36 port_poll_churn 84 @@ -306,6 +307,7 @@ usb_transport_break 5408 va_exhaustion 6159 virtio_net_no_msix 2039 virtio_used_ring 4189 +wake_storm_cost 337 wall_clock_file 4642 wall_clock_now 13 wall_clock_refusals 25658 diff --git a/tests/toyos-rust-tests/src/bin/wake_storm_cost.rs b/tests/toyos-rust-tests/src/bin/wake_storm_cost.rs new file mode 100644 index 00000000000..17103b712c6 --- /dev/null +++ b/tests/toyos-rust-tests/src/bin/wake_storm_cost.rs @@ -0,0 +1,227 @@ +//! **What raising a wake storm costs the thread that raises it.** +//! +//! `completion::post_n` walks the waiters armed on one word, claims each one +//! and posts a message to its home CPU — one loop, on the caller's CPU, before +//! the syscall returns. `toyos-sched/sim`'s wakeup-storm case measures how long +//! the *waiters* then take to reach a CPU and can say nothing whatever about +//! that loop: the model's clock does not advance inside a step, so raising a +//! storm of any size is free there. The cost is a guest measurement, and this +//! is it. +//! +//! **The derivation, off the loop itself.** The body is a token comparison, a +//! record store and one `claim_wake` per waiter, so the cost is +//! `fixed + N × per-waiter` — linear in the waiters on the word, with a +//! constant that has nothing to do with N. That shape is what is asserted: +//! quadrupling the storm may cost four times the loop, and a quarter more on +//! top, which is the same allowance `a_wakeup_storm_drains_without_serializing` +//! states about the drain. Anything past it is a term that grows with the +//! storm and is not the walk — a lock somebody takes per waiter, a scan of the +//! whole bucket per claim. +//! +//! **The absolute numbers are printed and never asserted**, for +//! `syscall_cost`'s reason: a threshold measured under TCG is meaningless — +//! QEMU prices an uncontended atomic read-modify-write nothing like silicon, +//! and this loop is made of them — and one measured on metal drifts. What +//! transfers is the *ratio*, which is dimensionless. +//! +//! **The instrument is `rdtsc` around the syscall, and it is the tree's +//! own.** `SYS_PROCESS_STATS` answers per *process*, and the waker and its 64 +//! waiters are one process here; the scheduler dump reports queues rather than +//! costs; and a per-waiter kernel counter would be a kernel addition. What a +//! syscall costs its caller is exactly what `syscall_cost` reads this way. +//! +//! **What it measured**, on this host's TCG guest at `--smp 2`, minimum of nine +//! timed calls per width, two runs of the same tree: +//! +//! | waiters | run 1 | run 2 | +//! |---|---|---| +//! | 1 | 7,000 | 7,000 | +//! | 2 | 7,000 | 8,000 | +//! | 4 | 7,000 | 11,000 | +//! | 8 | 10,000 | 18,000 | +//! | 16 | 14,000 | 20,000 | +//! | 32 | 23,000 | 21,000 | +//! | 64 | 33,000 | 38,000 | +//! +//! So **raising a 64-waiter storm costs its waker 33–38 µs of its own CPU** at +//! this guest's 998 MHz, which is a third of one percent of a 10 ms quantum, +//! and the marginal cost is 412–492 cycles per waiter claimed. Quadrupling the +//! storm from 16 to 64 costs 2.4× and 1.9× the walk rather than 4×, because the +//! fixed part of the call is a third of the 16-waiter figure. +//! +//! **The middle of the sweep is not monotone between runs** — 32 waiters read +//! under 8 in the second — and the reason is the instrument rather than the +//! kernel: this guest's TSC advances in steps of about a thousand cycles, so +//! the resolution is a microsecond and the whole per-waiter term below eight +//! waiters is inside it, and a minimum of nine calls on a two-CPU guest sharing +//! a host with eleven others still carries the host's descheduling. The two +//! endpoints are five times apart, which is why the assertions are stated at 1, +//! 16 and 64 and never between neighbours. +//! +//! **Every waiter is proved parked before the call that is timed**, by the +//! answer that call gives: a wake with no limit over `want` parked, unclaimed +//! waiters answers `want` under every arithmetic, and one that answers anything +//! else is discarded rather than measured. The word never changes while the +//! storms are raised, so a claimed waiter re-derives its predicate, finds it +//! false and re-parks — which is what makes the same word measurable at seven +//! widths in one run. + +use std::sync::atomic::{AtomicU32, Ordering}; +use std::thread; +use std::time::Duration; + +use toyos_abi::syscall::{self, clock_nanos}; + +/// The widths measured. 64 is the storm the simulator's case raises, and the +/// powers of two below it are what makes the shape readable rather than a +/// single number. +const WIDTHS: [usize; 7] = [1, 2, 4, 8, 16, 32, 64]; + +/// Timed calls kept per width. The **minimum** is reported, for +/// `syscall_cost`'s reason: it is the repetition with the least interference, +/// and on a host running eleven other guests interference is all a mean +/// measures. +const REPS: usize = 9; + +/// Calls to make per width before giving up on getting [`REPS`] of them to +/// answer `want`. A call that answers anything else means a waiter had not +/// re-parked yet, which costs a repetition and never a wrong number. +const ATTEMPTS: usize = 400; + +/// How long a waiter is given to re-park between two attempts. +const REPARK: Duration = Duration::from_millis(2); + +/// The word every waiter parks on. It never changes until the run is over, so +/// every wake is a claim the waiter re-parks from. +static WORD: AtomicU32 = AtomicU32::new(0); + +/// Set once at the end, to send every waiter home. +static DONE: AtomicU32 = AtomicU32::new(0); + +fn rdtsc() -> u64 { + let lo: u32; + let hi: u32; + unsafe { + core::arch::asm!( + "lfence", + "rdtsc", + out("eax") lo, + out("edx") hi, + options(nomem, nostack), + ); + } + ((hi as u64) << 32) | lo as u64 +} + +/// One waiter: park on the word, and come home when it finally changes. +fn waiter() { + while DONE.load(Ordering::SeqCst) == 0 { + unsafe { syscall::futex_wait(WORD.as_ptr(), 0, None) }; + } +} + +/// The cheapest `futex_wake` that claimed exactly `want` waiters, in cycles. +/// +/// `None` means the arrangement never held — every attempt found somebody not +/// yet re-parked — which is a measurement that did not happen and is reported +/// as one rather than averaged into the table. +fn storm_cycles(want: usize) -> Option { + let mut best = None; + let mut taken = 0; + for _ in 0..ATTEMPTS { + let start = rdtsc(); + let woken = unsafe { syscall::futex_wake(WORD.as_ptr(), u32::MAX) }; + let end = rdtsc(); + if woken as usize == want { + best = Some(best.map_or(end - start, |b: u64| b.min(end - start))); + taken += 1; + if taken == REPS { + return best; + } + } + thread::sleep(REPARK); + } + best +} + +/// Wait until a wake with no limit answers `want` — the proof that every +/// waiter has reached its park. +fn wait_until_parked(want: usize) { + for _ in 0..500 { + if unsafe { syscall::futex_wake(WORD.as_ptr(), u32::MAX) } as usize == want { + return; + } + thread::sleep(Duration::from_millis(10)); + } + panic!("{want} waiters never parked on the word"); +} + +fn main() { + let mut waiters = Vec::new(); + let mut table: Vec<(usize, u64)> = Vec::new(); + + for width in WIDTHS { + while waiters.len() < width { + waiters.push(thread::spawn(waiter)); + } + wait_until_parked(width); + let cycles = storm_cycles(width) + .unwrap_or_else(|| panic!("no wake of {width} waiters ever found all of them parked")); + println!("wake_storm_cost: {width} waiters, {cycles} cycles"); + table.push((width, cycles)); + } + + // The clock alongside the cycles, for `syscall_cost`'s reason: a TSC that + // does not tick at a fixed rate makes the counts incomparable and this is + // the only thing in the run that would show it. + let t0 = clock_nanos(); + let c0 = rdtsc(); + while clock_nanos() - t0 < 20_000_000 {} + let hz = (rdtsc() - c0) * 1_000_000_000 / (clock_nanos() - t0); + println!("wake_storm_cost: tsc {} MHz", hz / 1_000_000); + + let at = |want: usize| { + table + .iter() + .find(|&&(width, _)| width == want) + .map(|&(_, cycles)| cycles) + .unwrap_or_else(|| panic!("{want} waiters were not measured")) + }; + let (one, small, large) = (at(1), at(16), at(64)); + let marginal = (large - one) / 63; + println!( + "wake_storm_cost: {one} cycles at one waiter, {large} at 64 — {marginal} cycles per \ + waiter over the walk, {} ns per waiter", + marginal * 1_000_000_000 / hz, + ); + + // **The loop is not free**, which is the whole of what the model cannot + // say: its clock does not advance inside a step, so a 64-waiter storm and a + // one-waiter wake cost the same there. If they cost the same here, this + // instrument is not reading the walk at all. + assert!( + large > one, + "a wake claiming 64 waiters cost {large} cycles and one claiming a single waiter \ + {one} — the per-waiter term of the walk is not measurable through this instrument, \ + so nothing below is a statement about it", + ); + // **And it is linear.** Four times the waiters may cost four times the + // walk — that is what a loop over them *is* — and this allows a quarter + // more on top, which is the allowance the simulator's storm case states + // about the drain for the same reason. + assert!( + large <= small * 5, + "quadrupling the storm from 16 waiters to 64 took the waker's own cost from {small} \ + cycles to {large}. `post_n` walks the waiters once and does a constant amount per \ + claim, so four times the waiters may cost four times the loop and no more; past \ + this, something in the claim grows with the size of the storm", + ); + + DONE.store(1, Ordering::SeqCst); + WORD.store(1, Ordering::SeqCst); + while unsafe { syscall::futex_wake(WORD.as_ptr(), u32::MAX) } > 0 {} + for waiter in waiters { + waiter.join().expect("a waiter panicked"); + } + println!("raising a storm costs the waker one walk of its waiters, and it is linear"); +} diff --git a/tests/toyos.rs b/tests/toyos.rs index ae5f31edd9c..6fa0318b3ae 100644 --- a/tests/toyos.rs +++ b/tests/toyos.rs @@ -510,6 +510,13 @@ const MACHINE_TESTS: &[(&str, Sched, Tier)] = &[ // it reds. Host contention moves this guest's median by a factor of eight, // which is the definition of a test that must have the machine to itself. ("sched_check_build", Sched::Serial, Tier::Fast), + // What nesting a `scheduler::Operation` may and may not do, which is a law + // with no host-side reader: the type reaches `percpu::cpu_id` and + // `driver::current_handle`, so nothing outside a booted machine can + // construct one. Parallel, and nothing in it is a duration — every verdict + // is a comparison between two numbers the kernel printed, both of them + // offsets it chose itself. + ("operation_nesting", Sched::Parallel, Tier::Fast), ("short_sleep_livelock", Sched::Parallel, Tier::Fast), // The kernel thread and the row that says what its panic means. Two // boots, both headless: the second one halts on purpose — and the pair @@ -9600,6 +9607,144 @@ fn run_machine_test( eprintln!(" [i8042] {health} idle-health lines — the CPU still halts"); Ok(()) } + "operation_nesting" => { + // **An inner `scheduler::Operation` may only narrow, and its drop + // restores what it displaced.** `Operation::begin` stores + // `outer.min(until)`, so a caller cannot buy itself more device + // time by starting a second operation inside the first — which is + // the failure `block::OPERATION` exists to stop, arriving one layer + // lower. + // + // Nothing host-side can read it: the type reaches `percpu::cpu_id` + // and `driver::current_handle`, and `kernel/` is excluded from the + // host workspace, so a `Operation` cannot be constructed off a + // booted machine. The other gates that drive an establishment — + // `cache_eviction`'s `nvme-spent-budget`, `usb_storage_gate`'s — + // prove a narrowing happened by the refusal it produces and read + // none of the values, and all of them establish from a boot phase, + // which is the *task-less* slot. `kernel/src/sched_gate.rs` runs + // three nested establishments with known deadlines in both homes + // and prints what every level saw. + // + // **The bound is derived here and not read off the kernel's + // verdict**: the kernel prints what each level *asked* for and what + // it *observed*, and this recomputes the running minimum. A kernel + // that printed a verdict would be marking its own paper. + let qemu = QemuInstance::boot_with_options( + test_config, + c_bins, + rust_bins, + BootOptions { + kernel_params: &["sched-operation-nesting"], + ..Default::default() + }, + ); + let log = qemu.boot_log().to_string(); + + /// One `key=value` off a gate line, as a number. + fn number(line: &str, key: &str) -> Result { + line.split_whitespace() + .find_map(|word| word.strip_prefix(key)?.strip_prefix('=')?.parse().ok()) + .ok_or_else(|| format!("no numeric {key} in {line:?}")) + } + /// One `key=value` off a gate line, as a flag. + fn flag(line: &str, key: &str) -> Result { + line.split_whitespace() + .find_map(|word| word.strip_prefix(key)?.strip_prefix('=')?.parse().ok()) + .ok_or_else(|| format!("no boolean {key} in {line:?}")) + } + + // Both homes. A task's word is on its `TaskHandle` and a context + // with no task uses one slot per CPU, and the two are reached by + // different arms of `operation_slot` — so a gate that ran in one + // place would leave the other arm unexecuted by any test at all. + for site in ["boot", "iod"] { + let say = |what: &str| -> Result { + let needle = format!("sched-op: {site} {what}"); + log.lines() + .find(|line| line.contains(&needle)) + .map(str::to_string) + .ok_or_else(|| format!("no {needle:?} line on this boot:\n{log}")) + }; + + let outside = say("outside")?; + if flag(&outside, "established")? { + return Err(format!( + "{site}: an operation was already established before the gate began, \ + so nothing below is about the nesting it made: {outside}" + )); + } + + // Every level: what it asked for, and what the depth below it + // recovered. The bound is the running minimum — an inner + // establishment takes the earlier of its own deadline and its + // parent's, and only that. + let mut asked = Vec::new(); + let mut narrowest = u64::MAX; + for level in 1..=3 { + let line = say(&format!("begin level={level}"))?; + let want = number(&line, "asked")?; + let saw = number(&line, "observed")?; + narrowest = narrowest.min(want); + if saw != narrowest { + return Err(format!( + "{site}: level {level} asked for {want} ns and the depth inside it \ + recovered {saw} ns, against the {narrowest} ns that is the \ + earliest of it and every level above it. An establishment that \ + observes more than its parent allowed is a caller buying itself \ + device time by nesting: {line}" + )); + } + asked.push(want); + } + // The widening attempt has to have been a real one, or the line + // above is satisfied by a scenario in which nothing was asked. + if asked[2] <= asked[1] { + return Err(format!( + "{site}: level 3 asked for {} ns inside a level 2 of {} ns, so the \ + gate never attempted to widen and the narrowing it reports is vacuous", + asked[2], asked[1], + )); + } + + // And the restore: each drop puts back what that establishment + // displaced rather than clearing the slot, so the operation + // above it survives the one below ending. + for (level, restored) in [(3, asked[1].min(asked[0])), (2, asked[0])] { + let line = say(&format!("end level={level}"))?; + let saw = number(&line, "observed")?; + if saw != restored { + return Err(format!( + "{site}: with level {level} dropped the depth recovered {saw} ns \ + and the frame above it established {restored} ns — a guard that \ + restores something else has ended an operation its caller is \ + still inside: {line}" + )); + } + if !flag(&line, "established")? { + return Err(format!( + "{site}: dropping level {level} left no operation established at \ + all, and its caller is still inside one: {line}" + )); + } + } + + let last = say("end level=1")?; + if flag(&last, "established")? { + return Err(format!( + "{site}: the outermost guard dropped and an operation is still \ + established — the slot was restored rather than cleared, so the next \ + depth to ask would be answered a deadline nobody set: {last}" + )); + } + eprintln!( + " [operation] {site}: {} ns narrowed to {} ns, a {} ns request changed \ + nothing, and both drops restored", + asked[0], asked[1], asked[2], + ); + } + Ok(()) + } "virtio_used_ring" => { // Both fields of a virtqueue used-ring element are written by the // device, and on virtio-sound's control and event queues the ring diff --git a/toyos-sched/sim/src/explore.rs b/toyos-sched/sim/src/explore.rs index ccd7a94cac1..38d7a3aebfc 100644 --- a/toyos-sched/sim/src/explore.rs +++ b/toyos-sched/sim/src/explore.rs @@ -98,6 +98,9 @@ pub struct Outcome { pub process_finish_ns: Vec>, /// How many tasks the balance path moved between CPUs. pub migrations: u64, + /// Per CPU: when it first took an execution step — see + /// [`crate::vm::Vm::first_exec_ns`]. + pub first_exec_ns: Vec>, /// What each CPU's passes cost, as the core's `feature = "check"` recorder /// measured them — the on-target instrument, driven here by the scenario's /// modelled `pass_cost_ns`. @@ -131,6 +134,20 @@ impl Outcome { self.run_wait[process].get(cause) } + /// When the whole machine was working: the last CPU's first execution step, + /// or `None` if some CPU never took one. + /// + /// `None` is the answer that matters and it is not the same as a large + /// number: a CPU that never ran anything is a CPU the balance path never + /// reached, and a caller that folded that into a maximum would report the + /// *other* CPUs' recovery as the machine's. + pub fn machine_working_at_ns(&self) -> Option { + self.first_exec_ns + .iter() + .copied() + .try_fold(0, |worst, at| Some(worst.max(at?))) + } + pub fn report(&self) -> String { if self.passed() { return format!( @@ -268,6 +285,7 @@ fn outcome_of(scenario: &'static str, vm: &Vm<'_>, choices: &ChoiceStream) -> Ou process_service_ns: vm.service_ns.clone(), process_finish_ns: vm.finish_ns.clone(), migrations: vm.migrations, + first_exec_ns: vm.first_exec_ns.clone(), pass_costs: (0..vm.handles.len()) .map(|cpu| { let cpu = CpuId(cpu as u32); diff --git a/toyos-sched/sim/src/scenarios.rs b/toyos-sched/sim/src/scenarios.rs index b8caf8343eb..56104033039 100644 --- a/toyos-sched/sim/src/scenarios.rs +++ b/toyos-sched/sim/src/scenarios.rs @@ -10,8 +10,8 @@ use toyos_sched::task::WaitClass; use crate::vm::RUN_CHUNK_NS; use crate::workload::{ - AgeShape, BlockShape, ChargeShape, IrqSpec, MigrateShape, Op, ParkShape, ProcSpec, Protocol, - QueueSpec, Scenario, Script, ShareShape, WindowShape, + AgeShape, BalanceShape, BlockShape, ChargeShape, IrqSpec, MigrateShape, Op, ParkShape, + PlacementShape, ProcSpec, Protocol, QueueSpec, Scenario, Script, ShareShape, WindowShape, }; const MS: u64 = 1_000_000; @@ -40,6 +40,8 @@ fn scenario( age: AgeShape::BoundedDeferral, share: ShareShape::PerProcess, charge: ChargeShape::Honest, + placement: PlacementShape::LeastLoadedRotating, + balance: BalanceShape::Pull, order: FairOrder::InsertSequence, pass_cost_ns: 0, fair_allowance_ns: 0, @@ -1084,6 +1086,12 @@ pub fn by_name(name: &str) -> Option { if let Some(rest) = name.strip_prefix("wakeup_storm:") { return two_numbers(rest).map(|(cpus, waiters)| wakeup_storm(cpus, waiters)); } + // `lopsided_placement::` at the suite's own per-thread work, + // for the same reason: the recovery it measures is a curve in both numbers. + if let Some(rest) = name.strip_prefix("lopsided_placement:") { + return two_numbers(rest) + .map(|(cpus, threads)| lopsided_placement(cpus, threads, WORK)); + } match name { "old_steal_port" => Some(old_steal_port()), "old_migrate_kept_the_corpse" => Some(old_migrate_kept_the_corpse()), @@ -1359,6 +1367,54 @@ pub fn wakeup_storm(cpus: usize, waiters: usize) -> Scenario { /// How many storms one [`wakeup_storm`] run raises. pub const STORM_ROUNDS: usize = 4; +/// **The adversarial machine**: `threads` pure-CPU threads, every one of them +/// spawned onto cpu0 of a `cpus`-wide machine, and nothing that ever blocks. +/// +/// Spawn placement is least-loaded-with-rotation, so under the shipped policy a +/// burst is spread the instant it is made and a lopsided machine cannot arise +/// from a workload at all. That is what left the balance path measured only by +/// what [`wakeup_storm`] happens to produce — an idle CPU probing a busy one a +/// few times per run. This stages the state the path exists for and nothing +/// else does: every runnable thread on one CPU, the rest of the machine with +/// nothing to run, and the *only* mechanism that can change that the pull half +/// of §7.7. +/// +/// Shape, and why each part of it: +/// +/// * **Nothing blocks and nothing wakes.** A blocked thread is placed again +/// when it wakes, so any blocking would let the wake path launder the +/// adversary's placement into a legal one and the recovery being measured +/// would be somebody else's. +/// * **One process.** The threads share a fair share, so what is measured is +/// the *machine* finishing a fixed amount of work and never a split between +/// two claimants — `share_gain` is where that question lives. +/// * **Every thread carries the same `work`.** The makespan is then a rate: +/// `threads × work` of CPU delivered by `cpus` CPUs, against the +/// `threads × work / cpus` a work-conserving machine would take. +/// +/// `sim/tests/policy.rs` measures how long the machine takes to start working +/// and how long it takes to finish, against a bound derived from the protocol, +/// with [`BalanceShape::None`] as the control. +pub fn lopsided_placement(cpus: usize, threads: usize, work: u64) -> Scenario { + let mut scenario = scenario( + "lopsided_placement", + cpus, + // No wait queues: a queue nobody blocks on would only be scaffolding. + Vec::new(), + vec![process("crowd", vec![0; threads], vec![Script::new(vec![Op::Run(work)])])], + ) + .with_placement(PlacementShape::AllOn(0)); + scenario.max_tasks = threads + 1; + // `share_gain`'s term, with `threads` in place of its `threads + 1`: every + // thread's work is chopped into `RUN_CHUNK_NS` execution steps and each + // quantum boundary costs a pass on top. The idle passes and steal probes a + // wide machine spends on top of that are what the fourfold reserve is for — + // measured over the sweep `sim/tests/policy.rs` runs: 677 steps at 2 CPUs + // and 8 threads, 1,681 at 4 and 16, 6,851 at 4 and 64, 8,897 at 8 and 64. + scenario.max_steps = 20_000 + 4 * threads * (work / RUN_CHUNK_NS) as usize; + scenario +} + /// Negative gate for invariant I9: [`lend_then_block`] under commit `9c2fc4d`'s /// park, which cleared the borrowed window only `if now >= until`. /// diff --git a/toyos-sched/sim/src/vm.rs b/toyos-sched/sim/src/vm.rs index dcc64a7d576..f2d412946af 100644 --- a/toyos-sched/sim/src/vm.rs +++ b/toyos-sched/sim/src/vm.rs @@ -38,8 +38,8 @@ use crate::payload::{ MockAddressSpace, SimCtx, SimPayload, SimPreempt, SimShareLock, SimWaitList, StdLock, }; use crate::workload::{ - AgeShape, BlockShape, ChargeShape, MigrateShape, Op, ParkShape, Protocol, Scenario, Script, - ShareShape, WindowShape, + AgeShape, BalanceShape, BlockShape, ChargeShape, MigrateShape, Op, ParkShape, PlacementShape, + Protocol, Scenario, Script, ShareShape, WindowShape, }; /// How finely a `Run(ns)` op is chopped. Small enough that a 10 ms quantum @@ -469,6 +469,18 @@ pub struct Vm<'q> { /// than judged: a wakeup storm that drains in parallel across a machine is /// only evidence about the balance path if the balance path ran. pub migrations: u64, + /// Per CPU: the instant it first took an execution step, or `None` for a + /// CPU that never ran anything at all. + /// + /// **How long a machine took to start working**, which is a different + /// question from every latency beside it: those are asked of a task, and + /// this is asked of the CPU. It exists for the adversarial placement case + /// (`scenarios::lopsided_placement`), where every thread is spawned onto + /// one CPU and the only thing that can put work on the others is the pull + /// half of the balance path — so the last of these is when that path + /// finished recovering the machine, and a `None` in it is a CPU it never + /// reached. + pub first_exec_ns: Vec>, } /// One contention window: a maximal interval over which the same set of @@ -589,6 +601,7 @@ impl<'q> Vm<'q> { run_wait: vec![RunWait::default(); process_count], finish_ns: vec![None; process_count], migrations: 0, + first_exec_ns: vec![None; n], next_irq, next_key: 1, next_spawn_cpu: 0, @@ -657,12 +670,20 @@ impl<'q> Vm<'q> { // what used to misread contention as emptiness. Ties rotate, or every // task of a freshly booted system would land on cpu0 and the // scenarios would never see two CPUs at once. + // + // `AllOn` is the adversary and not a policy — see [`PlacementShape`]. + // The rotation counter still advances under it, so the two answers + // differ in where a task lands and in nothing else. let base = self.next_spawn_cpu; - let dst = (0..self.scenario.cpus) + let least_loaded = (0..self.scenario.cpus) .map(|offset| (base + offset) % self.scenario.cpus) .min_by_key(|&c| self.handles.get(CpuId(c as u32)).load()) .expect("at least one cpu"); - self.next_spawn_cpu = (dst + 1) % self.scenario.cpus; + self.next_spawn_cpu = (least_loaded + 1) % self.scenario.cpus; + let dst = match self.scenario.placement { + PlacementShape::LeastLoadedRotating => least_loaded, + PlacementShape::AllOn(cpu) => cpu, + }; let builder = TaskBuilder { key, share, @@ -868,6 +889,11 @@ impl<'q> Vm<'q> { Step::Exec(cpu) => Some(cpu), _ => None, }; + // Stamped from the same reading, and *before* the step runs: the + // instant this CPU began working, not the instant it stopped. + if let Some(cpu) = executed { + self.first_exec_ns[cpu].get_or_insert(self.clock.0); + } self.execute_inner(step, choices); let owed = self.hw.with(|s| s.need_resched.clone()); // A CPU number: it reads `owed` and writes `self.resched_at` at the @@ -1038,6 +1064,10 @@ impl<'q> Vm<'q> { // Copied out before the borrow: the injection below runs while the // pass holds `CpuSched`, and these are the only fields it needs. let queues = self.queues; + // The one policy bit in the `Env` (`sched::driver::env`), read off the + // scenario for the same reason as the shapes above it — see + // [`BalanceShape`]. + let steal = self.scenario.balance == BalanceShape::Pull; let mut injected = None; let (action, parked, end) = { let Vm { @@ -1052,7 +1082,7 @@ impl<'q> Vm<'q> { cpus: handles, frontier, preempt: &SimPreempt, - steal: true, + steal, }; let pass = SchedPass::begin(&mut cpus[cpu], env, now); match dispose { diff --git a/toyos-sched/sim/src/workload.rs b/toyos-sched/sim/src/workload.rs index b4489df9cf6..b17f985c664 100644 --- a/toyos-sched/sim/src/workload.rs +++ b/toyos-sched/sim/src/workload.rs @@ -228,6 +228,51 @@ pub enum ChargeShape { Double { process: &'static str }, } +/// Where a spawn is placed. +/// +/// A scenario dimension rather than a constant because the shipped answer is a +/// *policy*, and it is the one that decides which machines the rest of the +/// scheduler is ever measured on: least-loaded-with-rotation spreads every +/// burst by construction, so under it a machine whose runnable threads all sit +/// on one CPU is not expressible from a workload at all — and that machine is +/// exactly what the pull half of §7.7 exists for. Both answers below are the +/// kernel's own code either way; what this decides is which of the two numbers +/// `Vm::spawn` places by. +#[derive(Clone, Copy, PartialEq, Eq, Debug)] +pub enum PlacementShape { + /// Spec §9.4, and `kernel::sched::driver`'s `placement`: the least-loaded + /// CPU from the published counters, ties rotating so a freshly booted + /// system does not put everything on cpu0. + LeastLoadedRotating, + /// Every spawn onto one named CPU, whatever the counters say. + /// + /// **Not a policy this kernel has ever had, and that is the point.** It is + /// the adversary: the lopsided machine no legal placement produces and no + /// workload can otherwise stage, so that what the balance path does with + /// one is a measurement rather than a belief. See + /// `scenarios::lopsided_placement`. + AllOn(usize), +} + +/// Whether the pull half of the balance path runs. +/// +/// A scenario dimension for [`PlacementShape`]'s reason and with the same two +/// roles: the shipped answer, and the control that says what the measurement +/// would be without it. `kernel::sched::driver::env` sets `steal: true` and +/// never clears it, so [`BalanceShape::None`] is a kernel this tree has not +/// shipped — `toyos-sched/loom`'s `loom_retire` already builds an `Env` with +/// `steal: false` for the same reason, to state what the rest of the protocol +/// does without it. +#[derive(Clone, Copy, PartialEq, Eq, Debug)] +pub enum BalanceShape { + /// Spec §7.7 and §9.4's pull half: an idle pass probes the CPU with the + /// most surplus, and a loaded pass answers a probe out of `pop_surplus`. + Pull, + /// No probe and no answer. A task woken or placed onto a busy CPU waits + /// there until that CPU's own queue reaches it. + None, +} + /// Which teardown/balance algorithm the VM drives the core with. #[derive(Clone, Copy, PartialEq, Eq, Debug)] pub enum Protocol { @@ -256,6 +301,8 @@ pub struct Scenario { pub age: AgeShape, pub share: ShareShape, pub charge: ChargeShape, + pub placement: PlacementShape, + pub balance: BalanceShape, /// How the fair band picks between two ready threads of one share. A /// scenario dimension for the same reason [`ShareShape`] is, and the type is /// the core's own rather than a parallel copy of it: the broken orderings @@ -348,6 +395,28 @@ impl Scenario { self } + /// **Checked here rather than at the placement site**, which runs once per + /// spawn: a CPU index outside the machine is a scenario that was written + /// wrong, and a panic at the first spawn of a sweep would name the VM + /// instead of the scenario that asked for it. + pub fn with_placement(mut self, placement: PlacementShape) -> Self { + if let PlacementShape::AllOn(cpu) = placement { + assert!( + cpu < self.cpus, + "{}: placement names cpu{cpu} on a {}-cpu machine", + self.name, + self.cpus, + ); + } + self.placement = placement; + self + } + + pub fn with_balance(mut self, balance: BalanceShape) -> Self { + self.balance = balance; + self + } + pub fn with_order(mut self, order: FairOrder) -> Self { self.order = order; self diff --git a/toyos-sched/sim/tests/policy.rs b/toyos-sched/sim/tests/policy.rs index d93cc182b7f..40fd461ec6b 100644 --- a/toyos-sched/sim/tests/policy.rs +++ b/toyos-sched/sim/tests/policy.rs @@ -38,6 +38,15 @@ //! | interactive wake | 637 ms worst at 64 hogs; every wake but one at 0 ns | `(rivals+1) × 12 ms` = 792 ms | 0.80 | //! | wakeup storm drain | 27.75 ms for 64 waiters on one CPU, 3.1× the 16-waiter figure | `(per queue+1) × 12 ms` = 780 ms | 0.04 | //! | starvation | 70 ms at 5 runnable threads, 680 ms at 65 | `(threads+1) × 12 ms` | 0.97 / 0.86 | +//! | adversarial placement | a CPU an adversary loaded is drained to the surplus floor, and only the CPUs still awake are reached | `threads − 2` stealable | 1.00 | +//! +//! The last case is the only one whose bound is a *count* rather than a +//! duration, and that is the model's limit rather than a choice: the clock +//! advances on the executing CPU's step, so a run's makespan is what one CPU +//! would take at every width and no wall-clock statement about a wide machine +//! can be read off it. What survives is the protocol — how many tasks the +//! balance path moved, and which CPUs ever got one — and that is enough to +//! separate a machine that recovers from one that does not. //! //! Every number came from this file at 16–40 seeds per point: //! `cargo test -p toyos-sched-sim --test policy -- --nocapture`. The tables in @@ -74,7 +83,7 @@ use toyos_sched_sim::explore::{run, Outcome}; use toyos_sched_sim::latency::{Latency, ReadyCause}; use toyos_sched_sim::scenarios; use toyos_sched_sim::vm::RUN_CHUNK_NS; -use toyos_sched_sim::workload::{Scenario, ShareShape}; +use toyos_sched_sim::workload::{BalanceShape, Scenario, ShareShape}; const MS: u64 = 1_000_000; @@ -552,6 +561,176 @@ fn a_wakeup_storm_drains_without_serializing() { println!("storm drain 16→64 waiters on one cpu: {small} ns → {large} ns"); } +/// Seeds per point for the two adversarial-placement cases. +const LOPSIDED_SEEDS: u64 = 20; + +/// The widths the adversarial machine is measured at: CPUs, and threads all +/// spawned onto cpu0 of them. +const LOPSIDED: [(usize, usize); 4] = [(2, 8), (4, 16), (4, 64), (8, 64)]; + +/// How far the balance path can drain the CPU an adversary loaded, derived. +/// +/// Both halves of the pull path stop at the same inequality: +/// `answer_steal_requests` refuses below `fair_len() > 1` and `post_steal_probe` +/// will not probe a victim publishing a surplus under two — and `fair_len` is +/// sampled after the pick, so it excludes the task the CPU is about to run. So +/// the floor a loaded CPU can be drained to is one running task plus one queued +/// behind it, and everything above that floor is stealable, one task per +/// answered probe. +/// +/// It is also the ceiling on the *whole run's* migrations, because in this +/// workload a thief never becomes a victim: a CPU holding one task publishes a +/// surplus of zero, so no task in flight here is ever moved twice. +fn stealable(threads: usize) -> u64 { + threads as u64 - 2 +} + +/// **Adversarial placement**: every thread of a saturating workload spawned onto +/// one CPU, and what the balance path does with the machine that leaves. +/// +/// Spawn placement is least-loaded-with-rotation, so this state arises from no +/// workload at all — which is why the simulator grew a placement knob for it +/// (`workload::PlacementShape`), and why until it did, the only balance path +/// anything had measured was the handful of probes a wakeup storm happens to +/// provoke. What this stages is the machine the pull half of §7.7 exists for: +/// every runnable thread on cpu0, every other CPU with nothing, and no wake, no +/// block and no second placement anywhere in the run. +/// +/// **What it found**, 20 seeds per point at 60 ms of work per thread: +/// +/// | cpus | threads | seeds reaching every CPU | seeds reaching a second CPU | best seed's migrations | the floor | +/// |---|---|---|---|---|---| +/// | 2 | 8 | 9/20 | 9/20 | 6 | 6 | +/// | 4 | 16 | 2/20 | 13/20 | 14 | 14 | +/// | 4 | 64 | 2/20 | 13/20 | 62 | 62 | +/// | 8 | 64 | 0/20 | 15/20 | 62 | 62 | +/// +/// **Two findings, and the second is the one nobody had written down.** +/// +/// Where the balance path runs at all it drains the loaded CPU *completely*: +/// the best seed at every width moves exactly [`stealable`] tasks — all but the +/// one cpu0 is running and the one queued behind it — so a machine an adversary +/// piled onto one CPU is emptied to the protocol's own floor, one task per +/// answered probe. +/// +/// And **it only ever reaches the CPUs that are still awake.** The probe is +/// posted from an idle pass, one per idle trip, and only against a victim that +/// has already *published* a surplus of two; a CPU whose idle pass ran before +/// the loaded one published halts with no probe outstanding, and nothing in this +/// protocol wakes it — there is no push half, and a loaded CPU never announces +/// that it has surplus. That is why the first column falls with width: at eight +/// CPUs all seven have to lose the same race and none of the twenty seeds +/// did. `issues/kernel/an-idle-cpu-that-slept-before-the-surplus-is-never-probed.md` +/// carries it; the assertions below are about what the path does when it runs, +/// which is the half that is a law. +#[test] +fn the_balance_path_drains_the_cpu_an_adversary_loaded() { + let mut table = Vec::new(); + for (cpus, threads) in LOPSIDED { + let scenario = scenarios::lopsided_placement(cpus, threads, WORK); + let floor = stealable(threads); + let (mut best, mut widest, mut full, mut reached) = (0, 0, 0, 0); + + sweep(&scenario, LOPSIDED_SEEDS, |outcome| { + let ran = outcome.first_exec_ns.iter().filter(|at| at.is_some()).count(); + // **Every CPU beyond cpu0 that ran was handed its first task by the + // balance path**, because nothing else in this workload can put one + // there: the placement is `AllOn(0)`, nothing blocks and nothing + // wakes. An answered probe hands over exactly one task, so a machine + // in which `ran` CPUs did work owes at least `ran - 1` migrations. + assert!( + outcome.migrations + 1 >= ran as u64, + "at {cpus} cpus {ran} cpu(s) executed a step on a machine whose every thread \ + was spawned onto cpu0, and the balance path moved only {} task(s) — one of \ + those CPUs was given work by something that is not the balance path", + outcome.migrations, + ); + // And the ceiling: no task is moved twice, so the whole run cannot + // exceed what cpu0 held above the surplus floor. + assert!( + outcome.migrations <= floor, + "at {cpus} cpus the balance path moved {} of {threads} tasks, past the {floor} \ + that stand above cpu0's surplus floor — a task is being migrated more than \ + once, which is a thief that became a victim", + outcome.migrations, + ); + best = best.max(outcome.migrations); + widest = widest.max(ran); + full += usize::from(ran == cpus); + reached += usize::from(ran > 1); + }); + + println!( + "lopsided cpus={cpus} threads={threads}: {full}/{LOPSIDED_SEEDS} seeds reached \ + every cpu, {reached}/{LOPSIDED_SEEDS} reached a second, widest={widest}, best \ + seed moved {best} of a {floor} floor", + ); + // **The drain, asserted where it is a law.** The best seed is the one in + // which a thief was awake for the whole run, and there the balance path + // has to empty cpu0 down to the floor and no further. Anything less is a + // path that gives up while surplus is still published; anything more is + // the ceiling above. + assert_eq!( + best, floor, + "at {cpus} cpus and {threads} threads on cpu0, the best of {LOPSIDED_SEEDS} \ + schedules moved {best} tasks and the surplus floor leaves {floor} stealable. A \ + machine with a thief awake for the whole run must be drained to that floor: one \ + task per answered probe, until `fair_len() > 1` stops being true", + ); + table.push((cpus, threads, full, reached, best, floor)); + } + + for &(cpus, threads, full, reached, best, floor) in &table { + println!( + "lopsided cpus={cpus} threads={threads}: every-cpu {full}/{LOPSIDED_SEEDS}, \ + second-cpu {reached}/{LOPSIDED_SEEDS}, drained {best}/{floor}", + ); + } +} + +/// The negative control for the case above: the same lopsided machine with the +/// pull half of the balance path switched off. +/// +/// Without it "the balance path drains the CPU an adversary loaded" is a number +/// with nothing to compare it to — the simulator would report the same drain if +/// the tasks had been placed by the shipped policy in the first place, and the +/// case above would be measuring the placement knob rather than the balance +/// path. With `Env::steal` off nothing can move a task off cpu0 at all. +/// +/// **Measured**: at every width the drain is 0 against a floor of 6, 14, 62 and +/// 62, and exactly one CPU of the machine ever executes a step — so the +/// assertion next door reds on its first width. +#[test] +fn without_the_balance_path_a_lopsided_machine_stays_lopsided() { + for (cpus, threads) in LOPSIDED { + let scenario = + scenarios::lopsided_placement(cpus, threads, WORK).with_balance(BalanceShape::None); + let floor = stealable(threads); + let (mut most, mut widest) = (0, 0); + + sweep(&scenario, LOPSIDED_SEEDS, |outcome| { + most = most.max(outcome.migrations); + widest = widest.max(outcome.first_exec_ns.iter().filter(|at| at.is_some()).count()); + }); + + println!( + "lopsided-control cpus={cpus} threads={threads}: {most} migrations against a \ + {floor} floor, {widest} of {cpus} cpus ran", + ); + assert_eq!( + most, 0, + "with `Env::steal` off the balance path still moved {most} task(s) at {cpus} cpus \ + — nothing else in this protocol migrates, so this is not a control", + ); + assert_eq!( + widest, 1, + "with the balance path off, {widest} of {cpus} cpus executed a step on a machine \ + whose every thread was spawned onto cpu0 — the placement knob is not staging the \ + machine the case above is about", + ); + } +} + /// **The starvation bound**: the worst wait of a runnable task under saturation, /// measured, against the fair band's own granularity. ///