From 400f999c67c37bc725de0a05e450705b3bd19a93 Mon Sep 17 00:00:00 2001 From: japabu Date: Sat, 22 Aug 2026 14:35:29 +0200 Subject: [PATCH 1/2] sched: both cures for the one-shot probe, measured, and neither shipped MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `issues/kernel/an-idle-cpu-that-slept-before-the-surplus-is-never-probed.md` named two ways out of a pull-only, one-shot balance path and had numbers for neither. Both are now implemented in the pure crate behind `cpu::Balance`, which the simulator selects and the kernel does not: `sched::driver::env` selects `Balance::Pull` and every scheduling decision the kernel makes is byte-identical to before. `Env::steal: bool` becomes `Env::balance: Balance` — `None`, `Pull`, `PullWithRearm { every_ns, times }`, `PushOnSurplus { threshold }`. The sim's `BalanceShape` is deleted and the scenario carries the core's own type, for the reason `Scenario::order` already carries `FairOrder`: every setting is a policy the core implements, so the simulator drives the shipped code whichever one a scenario names. The re-arm is a bounded one. A CPU that halts with nothing to run arms its one-shot timer `every_ns` ahead and probes again when it fires, up to `times` times per idle period, the allowance refilling at every dispatch — so a machine with nothing to hand out stops ticking after `times × every_ns` instead of waking for ever, and the simulator's runs terminate. The counter counts *up* from zero rather than down from an allowance, and the first draft got that wrong in the one way that matters: a CPU has no policy until a pass hands it one, so a countdown initialised in `CpuSched::new` starts empty on a CPU that has never run a pass, and the CPU the re-arm exists for is the one CPU it never fires on. Measured that way round it reproduced `Pull` exactly, 0 of 20 seeds at eight CPUs. The push rings the doorbell of one sleeping CPU per pass, with no message behind it: `CpuHandle::poke` is `Doorbell::ring(Urgency::Normal)`, so §7.3's edge-coalescing already prices it, and the woken CPU posts an ordinary probe. A cursor makes consecutive pushes walk the machine — without it every push goes to the lowest-numbered sleeper, which posts its probe and halts again with SLEEPING still set, so the next pass re-pokes the CPU that is already coming and the one behind it waits for the first one's probe to be *answered*. Two passes per sleeper: 130,000,000 ns of probe gap at eight CPUs against 66,000,000 ns with the cursor. **The push costs an ordering the pull path does not have, and that is the finding this change did not set out to make.** It is the store-buffer litmus test: the pusher stores its surplus and loads the sleeper's SLEEPING bit while the sleeper stores SLEEPING and loads the surplus, and with anything weaker than a `SeqCst` fence between each side's store and its load both may miss — the same defect in a window nanoseconds wide. `cpu::balance_fence` is that fence, an `mfence` on the exit of every pass that has surplus, and the sleeper's half of the pair is a second read of the surplus after `arm_sleep` has published SLEEPING. `toyos-sched/loom/tests/loom_push.rs` models it and `push-fence-relaxed` weakens the fence to a release one, under which the model reds at `a_cpu_that_halts_without_seeing_the_surplus_was_pushed`; the control is wired into `host-tests.yml` beside the other eight, which `build.rs`'s `every_loom_control_is_wired_into_host_tests` requires anyway. Two instruments carry the measurement. `Vm::idle_wakes` counts wakes of a halted CPU whose first pass reaches the idle disposition again — a property of the run and not of the policy, which is what lets `Pull`'s own figure be a baseline, and it is zero on every workload measured. `Vm::probe_gap_ns` is the defect as a duration: the longest a halted CPU sat beside a published surplus with no probe of its own outstanding. Recovery is counted in CPUs and timed in probe gaps because the model does not oblige a CPU with a task loaded to take an execution step, so a first-exec instant measures the explorer as much as the protocol. `sim/tests/policy.rs` gains three cases, 20 seeds per point. Both cures take the lopsided machine from 9/20, 2/20, 2/20, 0/20 to 20/20 at every width. The re-arm's probe gap is 10,000,000 ns against a derived `every_ns + 3 × RUN_CHUNK` = 13,000,000 ns; the push's is 66,000,000 ns at eight CPUs against a derived `(cpus−1) × DISPATCH + IPI + 2 × RUN_CHUNK` = 86,200,000 ns. Both assertions red with `Balance::Pull` substituted and nothing else changed, at the first width, and the red lines are pasted into the cases' docs. The cost case prices the two against `Pull` on the interactive, storm and audio workloads; on `audio_pipeline(4)` the cheapest re-arm costs 153.85 idle wakes per second and the push costs **zero**, because that workload never has a surplus to announce. Under a wakeup storm at eight CPUs it is the other way round — 748 wakes against 521 — and both rows are in the table for that reason. The wake latency does not move under any policy, and the model cannot bill a wake: it charges a pass zero nanoseconds. What a wake costs the CPU it wakes is a kernel measurement nobody has taken, and it is one of the two things the decision is short of. The issue becomes a `question` at `status: owner`. Nothing about it is an agent's to decide — it is an idle-path change, which `kernel/CLAUDE.md` makes an audio change — and the section it now carries is the two cures, the measured recovery and cost of each, and a recommendation. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01Dpa781ehanBhUgrycmXcPW --- .github/workflows/host-tests.yml | 38 +- ...lept-before-the-surplus-is-never-probed.md | 80 ++- kernel/src/sched/driver.rs | 18 +- toyos-sched/Cargo.toml | 8 + toyos-sched/loom/Cargo.toml | 10 + toyos-sched/loom/tests/loom_push.rs | 187 ++++++ toyos-sched/loom/tests/loom_retire.rs | 4 +- toyos-sched/sim/src/explore.rs | 20 + toyos-sched/sim/src/scenarios.rs | 7 +- toyos-sched/sim/src/vm.rs | 105 +++- toyos-sched/sim/src/workload.rs | 35 +- toyos-sched/sim/tests/deadline_claim_race.rs | 4 +- toyos-sched/sim/tests/policy.rs | 565 +++++++++++++++++- toyos-sched/src/cpu.rs | 363 ++++++++++- toyos-sched/src/sync.rs | 4 +- toyos-sched/src/timer.rs | 17 + 16 files changed, 1374 insertions(+), 91 deletions(-) create mode 100644 toyos-sched/loom/tests/loom_push.rs diff --git a/.github/workflows/host-tests.yml b/.github/workflows/host-tests.yml index 9d13abd12b4..73cfb04842d 100644 --- a/.github/workflows/host-tests.yml +++ b/.github/workflows/host-tests.yml @@ -549,6 +549,39 @@ jobs: exit 1 fi + # **The balance path's push half, which is a candidate and not shipped.** + # `cpu::Balance::PushOnSurplus` rests on one CPU observing another's + # SLEEPING bit while that CPU observes its surplus — the store-buffer + # litmus test — and `cpu::balance_fence`'s `SeqCst` fence is what forbids + # both sides missing. `push-fence-relaxed` weakens it to a release fence, + # which orders stores against stores and nothing against a later load, and + # `loom_push` must red. + # + # It is wired here rather than left to whoever adopts the policy, for the + # reason `no-preempt-guard` and `doorbell-kick-relaxed` were: a control + # nothing runs is a control nobody can trust on the day it matters. This + # model's panic unwinds cleanly — nothing is queued in it — so unlike the + # step above it does print a `FAILED` line; the verdict is still grepped + # by the assertion's own words, because a non-zero exit alone would accept + # a compile error. + - name: toyos-sched-loom push-on-surplus fence has teeth + run: | + set -o pipefail + echo "=== toyos-sched-loom --features push-fence-relaxed --test loom_push (must red)" \ + | tee -a /tmp/host.log + if cargo test -p toyos-sched-loom \ + --features push-fence-relaxed --test loom_push 2>&1 \ + | tee /tmp/push-fence-relaxed.log | tee -a /tmp/host.log; then + echo "loom_push passed with the balance fence relaxed: the model has no teeth" \ + | tee -a /tmp/host.log + exit 1 + fi + if ! grep -q 'published and no push behind it' /tmp/push-fence-relaxed.log; then + echo "the fence-relaxed build failed without a_cpu_that_halts_without_seeing_the_surplus_was_pushed reaching its verdict: this step proved nothing" \ + | tee -a /tmp/host.log + exit 1 + fi + # `userland/.cargo/config.toml` cross-compiles by default, so a userland # crate's host tests have to name the host. `calc` is here for the reason # the pure crates above are: its arithmetic is decided without a guest, @@ -601,9 +634,10 @@ jobs: | tee -a /tmp/host.log # The root package's `--lib`, one `--workspace` over the other host-tested - # members, `kernel-loom`'s no-loom command, eight negative-control runs + # members, `kernel-loom`'s no-loom command, nine negative-control runs # (`wake-fence-off` plus the five `kernel-loom` gained 2026-08-17, and - # `toyos-sched-loom`'s `no-preempt-guard` and `doorbell-kick-relaxed`), + # `toyos-sched-loom`'s `no-preempt-guard`, `doorbell-kick-relaxed` and + # `push-fence-relaxed`), # the three userland crates and the SDK. A red is still one line somewhere among them, # and `--workspace` prints a `Running ...` line naming the binary rather # than the `=== ` the other steps echo, so the summary matches both diff --git a/issues/kernel/an-idle-cpu-that-slept-before-the-surplus-is-never-probed.md b/issues/kernel/an-idle-cpu-that-slept-before-the-surplus-is-never-probed.md index f625a5c6cc0..4e5ce1c522d 100644 --- a/issues/kernel/an-idle-cpu-that-slept-before-the-surplus-is-never-probed.md +++ b/issues/kernel/an-idle-cpu-that-slept-before-the-surplus-is-never-probed.md @@ -1,6 +1,6 @@ --- -status: open -kind: finding +status: owner +kind: question opened: 2026-08-22 --- @@ -51,3 +51,79 @@ and the fix is a design decision the balance path does not currently make: a push on surplus, or a bounded re-arm of the probe, both of which cost an idle CPU wakes it is currently right to be without (`kernel/CLAUDE.md`: anything added to the idle loop is an audio change). + +## Decision owed: which cure, if either, the balance path takes + +Both cures are now implemented in `toyos-sched` behind `cpu::Balance`, which the +simulator selects and `kernel::sched::driver::env` does not: it selects +`Balance::Pull` and the kernel's behaviour is unchanged. The measurements are +`toyos-sched/sim/tests/policy.rs`'s three new cases, 20 seeds per point +(`cargo test -p toyos-sched-sim --test policy -- --nocapture`). + +**`Balance::PullWithRearm { every_ns, times }`** — a CPU that halts with nothing +to run arms its one-shot timer `every_ns` ahead and probes again when it fires, +up to `times` times per idle period, the allowance refilling at every dispatch. +It observes nothing and needs nobody's cooperation. + +**`Balance::PushOnSurplus { threshold }`** — a pass that publishes a surplus of +`threshold` or more rings the doorbell of one CPU that reads SLEEPING, walking +them in turn. No task moves on the ring; the woken CPU posts an ordinary probe. + +Recovery on the lopsided machine, at `every_ns = QUANTUM_NS` and `threshold = 2` +— seeds reaching every CPU, and the worst time a halted CPU sat beside a +published surplus with no probe outstanding: + +| cpus/threads | pull | re-arm ×1 | re-arm ×4 | push ≥2 | +|---|---|---|---|---| +| 2 / 8 | 9/20, 480 ms | 20/20, 10 ms | 20/20, 10 ms | 20/20, 1 ms | +| 4 / 16 | 2/20, 960 ms | 20/20, 10 ms | 20/20, 10 ms | 20/20, 22 ms | +| 4 / 64 | 2/20, 3.84 s | 20/20, 10 ms | 20/20, 10 ms | 20/20, 22 ms | +| 8 / 64 | 0/20, 3.84 s | 20/20, 10 ms | 20/20, 10 ms | 20/20, 66 ms | + +Cost, in wakes of a halted CPU that found nothing to do — the whole sweep's +count, and the worst single run's rate against simulated time: + +| workload | pull | re-arm ×1 | re-arm ×4 | push ≥2 | +|---|---|---|---|---| +| `interactive_mix(2,4)` | 0, 0.00/s | 44, 31.58/s | 171, 88.00/s | 9, 11.76/s | +| `interactive_mix(2,16)` | 0, 0.00/s | 49, 11.94/s | 174, 30.14/s | 2, 3.08/s | +| `interactive_mix(4,16)` | 0, 0.00/s | 104, 20.90/s | 363, 63.01/s | 53, 24.62/s | +| `wakeup_storm(4,16)` | 0, 0.00/s | 251, 92.13/s | 1004, 306.22/s | 80, 97.56/s | +| `wakeup_storm(8,64)` | 0, 0.00/s | 521, 150.67/s | 2075, 534.78/s | 748, 380.95/s | +| `audio_pipeline(4)` | 0, 0.00/s | 111, 153.85/s | 351, 243.90/s | **0, 0.00/s** | + +The audio-relevant latency does not move: the watched thread's worst wake is the +same nanosecond under all four policies at every point but `wakeup_storm(8,64)`, +where the push's 50,750,000 ns beats the shipped path's 57,000,000 ns, and the +means move by under 2%. The model charges a pass zero nanoseconds, so it counts +an extra wake and cannot bill one — **what a wake costs the CPU it wakes is a +kernel measurement nobody has taken.** That is the one number this decision is +short of. + +**The push costs an ordering the pull path does not have.** It is the +store-buffer litmus test: the pusher stores its surplus and loads the sleeper's +SLEEPING bit while the sleeper stores SLEEPING and loads the surplus, and without +a `SeqCst` fence between each side's store and its load both may miss — the same +defect in a window nanoseconds wide. `cpu::balance_fence` is that fence, an +`mfence` on the exit of every pass that has surplus, and +`toyos-sched/loom/tests/loom_push.rs` is the model; +`cargo test -p toyos-sched-loom --features push-fence-relaxed --test loom_push` +weakens it and the model reds. The re-arm needs no such edge at all: a timer +fires whether or not anybody published anything. + +**Recommendation: `PushOnSurplus { threshold: 2 }`, or nothing.** It recovers the +machine at every width, it costs the audio pipeline exactly zero extra idle +wakes because a machine with no surplus never pushes, and it is self-limiting in +the direction that matters — the wake only ever happens where there is work to +come and get. The re-arm buys the same recovery with a periodic tick on every +idle CPU, including on a machine that has nothing at all to hand out; on the +audio workload that is 154 wakes per second bought for nothing. What the push +costs instead is an `mfence` on the pass exit of a CPU with surplus and a second +read of the surplus on the idle path — both off the audio pipeline's hot path, +because that workload never has surplus. Doing nothing also remains defensible: +no shipped workload has been shown to reach the state, and `kernel/CLAUDE.md`'s +rule is that the idle path stays empty until something forces it open. + +Two things would change the answer, and neither is measured yet: what one wake +actually costs a woken CPU in the kernel (the model cannot say), and how often a +real boot produces the composition in "Reachable in the kernel" above. diff --git a/kernel/src/sched/driver.rs b/kernel/src/sched/driver.rs index 356832728f8..8b444790e4c 100644 --- a/kernel/src/sched/driver.rs +++ b/kernel/src/sched/driver.rs @@ -37,7 +37,7 @@ use core::cell::UnsafeCell; use core::ptr; use core::sync::atomic::{AtomicBool, AtomicPtr, AtomicU64, Ordering}; -use toyos_sched::cpu::{Action, CpuHandle, CpuHandles, CpuSched, Env, SchedPass}; +use toyos_sched::cpu::{Action, Balance, CpuHandle, CpuHandles, CpuSched, Env, SchedPass}; use toyos_sched::fair::Frontier; use toyos_sched::hw::{CpuId, Hw, Kicker, Machine, Nanos}; use toyos_sched::mailbox::{mailbox, Kick, PreemptGuard, Urgency}; @@ -661,10 +661,16 @@ pub enum Dispose { /// The environment every pass runs against. /// -/// `steal` is the one policy bit in it, and it is on: an idle pass probes the -/// busiest CPU for work and a loaded pass answers probes from surplus (spec -/// §7.7, §9.4's pull half). Without it a task woken onto a busy CPU waits -/// there until the owner yields. +/// `balance` is the one policy value in it, and it is [`Balance::Pull`]: an idle +/// pass probes the busiest CPU for work and a loaded pass answers probes from +/// surplus (spec §7.7, §9.4's pull half). Without it a task woken onto a busy +/// CPU waits there until the owner yields. +/// +/// The other settings of that knob are the simulator's, and deliberately not +/// this line's: [`Balance::PullWithRearm`] and [`Balance::PushOnSurplus`] both +/// add wakes to the idle path, which `kernel/CLAUDE.md` makes an audio change, +/// and what they buy and cost is measured in `toyos-sched/sim/tests/policy.rs` +/// rather than decided here. /// /// The guard comes in by reference because its lifetime is the pass's and it /// belongs to the caller that raised the count. @@ -674,7 +680,7 @@ fn env(preempt: &PreemptOff) -> Env<'_, crate::hw::KernelHw, PreemptOff> { cpus: cpus(), frontier: &FRONTIER, preempt, - steal: true, + balance: Balance::Pull, } } diff --git a/toyos-sched/Cargo.toml b/toyos-sched/Cargo.toml index fbc4ec1a84b..6f2b228582d 100644 --- a/toyos-sched/Cargo.toml +++ b/toyos-sched/Cargo.toml @@ -41,3 +41,11 @@ loom = [] # negative control (spec §7.3, §7.5). No kernel build reaches it; the crate # declares it only so `cfg` checking knows the name. doorbell-kick-relaxed = [] +# Declared, never enabled here: `toyos-sched-loom` turns this on to weaken the +# `SeqCst` fence `cpu.rs`'s `balance_fence` puts between the push half's surplus +# store and its read of a target's SLEEPING bit, and +# `loom/tests/loom_push.rs` must red when it does — `Balance::PushOnSurplus`'s +# own negative control. No kernel build reaches it, and no kernel build selects +# that policy either; the crate declares the name only so `cfg` checking knows +# it. +push-fence-relaxed = [] diff --git a/toyos-sched/loom/Cargo.toml b/toyos-sched/loom/Cargo.toml index c3be25723a2..e788ffd3cfb 100644 --- a/toyos-sched/loom/Cargo.toml +++ b/toyos-sched/loom/Cargo.toml @@ -45,6 +45,16 @@ no-preempt-guard = [] # Never on by default and never reachable from a kernel build, which declares # the same name only so `cfg` checking knows it. doorbell-kick-relaxed = [] +# The negative control for `Balance::PushOnSurplus`'s ordering. It weakens +# `cpu.rs`'s `balance_fence` from a `SeqCst` fence to a release one, which +# orders stores against stores and nothing against a later load — so the push's +# surplus store and the sleeper's SLEEPING store may each be missed by the +# other's read, and the CPU halts with a surplus published and no probe +# outstanding. `loom_push.rs` must red under it: +# +# cargo test -p toyos-sched-loom --features push-fence-relaxed --test loom_push +# +push-fence-relaxed = [] [dependencies] loom = "0.7" diff --git a/toyos-sched/loom/tests/loom_push.rs b/toyos-sched/loom/tests/loom_push.rs new file mode 100644 index 00000000000..4278b632307 --- /dev/null +++ b/toyos-sched/loom/tests/loom_push.rs @@ -0,0 +1,187 @@ +//! Loom: the publish/observe edge `Balance::PushOnSurplus` is made of. +//! +//! The pull half of the balance path needs no ordering between two CPUs at all: +//! an idle CPU reads a surplus, and if it reads a stale one it simply does not +//! probe. The push half is different in kind, because it is a *liveness* claim +//! that rests on one CPU observing another — the CPU that gained surplus reads +//! the SLEEPING bit of the CPU that is going to sleep, and decides from it +//! whether anybody needs telling. +//! +//! That makes it the store-buffer litmus test: +//! +//! ```text +//! pusher sleeper +//! ------ ------- +//! surplus.store(2) doorbell.fetch_or(SLEEPING) +//! balance_fence() balance_fence() +//! doorbell.load() -> asleep? surplus.load() -> anything to steal? +//! ``` +//! +//! Each side stores to one location and loads the other, and with anything +//! weaker than a `SeqCst` fence between the two, **both loads may miss**: the +//! pusher decides nobody is asleep, the sleeper decides there is nothing to come +//! for, and the machine is back in +//! `issues/kernel/an-idle-cpu-that-slept-before-the-surplus-is-never-probed.md` +//! in a window nanoseconds wide instead of milliseconds. Nothing in a guest test +//! can find that — on x86 the sleeper's side of it is a `lock or`, a full fence +//! already, so only the pusher's half is even reorderable and only under a +//! store-buffer delay no test can schedule. +//! +//! The property below is the one the push actually needs: **a CPU that halts +//! without having seen the surplus has a poke coming.** It is the same shape as +//! `loom_sleep.rs`'s I2 — halted with work queued implies an IPI in flight — +//! asked of the push's own pair rather than of the mailbox's. +//! +//! The negative control is a cargo feature rather than a comment: +//! +//! ```text +//! cargo test -p toyos-sched-loom --features push-fence-relaxed --test loom_push +//! ``` +//! +//! makes `cpu::balance_fence` a release fence — which orders stores against +//! stores and nothing against a later load — and this file must red at +//! [`a_cpu_that_halts_without_seeing_the_surplus_was_pushed`]. Verified +//! 2026-08-22, both ways round. + +use loom::sync::atomic::{AtomicUsize, Ordering}; +use loom::sync::Arc; +use toyos_sched_loom::cpu::{balance_fence, CpuHandle}; +use toyos_sched_loom::hw::CpuId; +use toyos_sched_loom::mailbox::{mailbox, Kick, MailboxConsumer}; +use toyos_sched_loom::model::{model, Msg}; + +/// The surplus at which a pass pushes and at which an idle pass probes — one +/// inequality, `SchedPass::best_victim`'s. +const THRESHOLD: u32 = 2; + +/// Two CPUs' shared faces: the one that gains surplus, and the one going to +/// sleep. The `CpuHandle` is the real one, so `publish_surplus`, `surplus`, +/// `doorbell` and `poke` are the shipped code and not a restatement of it. +struct World { + /// The CPU that gains surplus. Only its `surplus` matters here. + victim: CpuHandle, + /// The CPU going to sleep. Only its doorbell matters here. + thief: CpuHandle, + /// Targeted IPIs the push sent. As in `loom_sleep.rs`, an IPI's only + /// relevant effect is "the target cannot stay halted". + ipis: AtomicUsize, + /// Doorbell rings the push made, kicked or coalesced. A ring that elides its + /// IPI is still an observation — it can only elide because an IPI is already + /// on its way — so the property counts rings and not kicks. + pokes: AtomicUsize, +} + +impl World { + /// `SchedPass::push_on_surplus`, as the pass runs it: publish, fence, look + /// for a sleeper, ring it. + fn publish_and_push(&self) { + self.victim.publish_surplus(THRESHOLD); + balance_fence(); + if self.thief.doorbell().sleeping() { + self.pokes.fetch_add(1, Ordering::AcqRel); + if self.thief.poke() == Kick::Send { + self.ipis.fetch_add(1, Ordering::AcqRel); + } + } + } + + /// `SchedPass::try_sleep` under `Balance::PushOnSurplus`, with the part that + /// matters: the probe's own read of the surplus found nothing, SLEEPING is + /// published, and the re-read behind the fence is the load that pairs with + /// the pusher's read of that bit. + /// + /// Returns `true` if the CPU halted **without** having seen the surplus — + /// the state the push has to make impossible without a poke behind it. + fn try_sleep(&self, rx: &MailboxConsumer) -> bool { + self.thief.doorbell().begin_pass(); + // `post_steal_probe`'s read, before SLEEPING is published. A stale + // answer here is exactly what the re-read below exists to catch. + let probed = self.victim.surplus() >= THRESHOLD; + let arm = self.thief.doorbell().arm_sleep(); + balance_fence(); + if self.victim.surplus() >= THRESHOLD { + // Observed: the pass goes round again and posts a probe. + arm.abandon(); + return false; + } + let halted = arm.confirm(rx).is_ok(); + halted && !probed + } +} + +/// The push's liveness: a CPU cannot halt believing there is nothing to steal +/// while another CPU has already published a surplus and believes nobody is +/// asleep. +/// +/// One of the two has to observe the other, and the `SeqCst` fence in +/// [`balance_fence`] is the whole of why. The assertion is stated on the poke +/// and not on the IPI for the reason `Doorbell::ring` gives: a ring that returns +/// `Kick::Elide` did so because the kick edge was already raised, i.e. because a +/// kick is already in flight. +#[test] +fn a_cpu_that_halts_without_seeing_the_surplus_was_pushed() { + model(|| { + let (victim_tx, _victim_rx) = mailbox::(); + let (thief_tx, thief_rx) = mailbox::(); + let world = Arc::new(World { + victim: CpuHandle::new(CpuId(0), victim_tx), + thief: CpuHandle::new(CpuId(1), thief_tx), + ipis: AtomicUsize::new(0), + pokes: AtomicUsize::new(0), + }); + + let pusher = { + let world = world.clone(); + loom::thread::spawn(move || world.publish_and_push()) + }; + let blind = world.try_sleep(&thief_rx); + pusher.join().unwrap(); + + assert!( + !blind || world.pokes.load(Ordering::SeqCst) >= 1, + "cpu1 halted with cpu0's surplus of {THRESHOLD} published and no push behind it — \ + both sides of the store-buffer pair missed, which is \ + `an-idle-cpu-that-slept-before-the-surplus-is-never-probed` in a window a fence \ + would have closed", + ); + }); +} + +/// The other direction, and the one that says the fence is not simply making +/// everything look asleep: the push does not ring a CPU that is awake. +/// +/// A ring on a busy target sets the kick edge with no message behind it, which +/// costs that target's next idle disposition a spurious extra pass — harmless, +/// and exactly what the push is allowed to cost. What it may not do is ring a +/// CPU that never armed a sleep at all, because then the count this policy is +/// priced by is measuring something that is not a wake. +#[test] +fn a_push_rings_nobody_when_nobody_is_asleep() { + model(|| { + let (victim_tx, _victim_rx) = mailbox::(); + let (thief_tx, _thief_rx) = mailbox::(); + let world = Arc::new(World { + victim: CpuHandle::new(CpuId(0), victim_tx), + thief: CpuHandle::new(CpuId(1), thief_tx), + ipis: AtomicUsize::new(0), + pokes: AtomicUsize::new(0), + }); + + // The target runs a pass and stays awake: `begin_pass` clears SLEEPING + // and nothing arms one. + let target = { + let world = world.clone(); + loom::thread::spawn(move || world.thief.doorbell().begin_pass()) + }; + world.publish_and_push(); + target.join().unwrap(); + + assert_eq!( + world.pokes.load(Ordering::SeqCst), + 0, + "the push rang a CPU that never published SLEEPING — a wake charged to a policy \ + that did not need to send it", + ); + assert_eq!(world.ipis.load(Ordering::SeqCst), 0, "and no IPI either"); + }); +} diff --git a/toyos-sched/loom/tests/loom_retire.rs b/toyos-sched/loom/tests/loom_retire.rs index db4cb88d4a7..9d5fd28c37f 100644 --- a/toyos-sched/loom/tests/loom_retire.rs +++ b/toyos-sched/loom/tests/loom_retire.rs @@ -10,7 +10,7 @@ //! node. use loom::sync::Arc; -use toyos_sched_loom::cpu::{CpuHandle, CpuHandles, CpuSched, Env, RunToken, SchedPass}; +use toyos_sched_loom::cpu::{Balance, CpuHandle, CpuHandles, CpuSched, Env, RunToken, SchedPass}; use toyos_sched_loom::fair::{FairShare, Frontier, ShareState}; use toyos_sched_loom::hw::{CpuId, Hw, Kicker, Machine, Nanos, TraceEvent}; use toyos_sched_loom::mailbox::{mailbox, MailboxConsumer, Urgency}; @@ -428,7 +428,7 @@ fn the_retire_arm_never_loses_a_parked_task_to_a_racing_wake() { cpus: &owner.cpus, frontier: &owner.frontier, preempt: &guard, - steal: false, + balance: Balance::None, }; let pass = |cpu: &mut CpuSched| { let _ = SchedPass::begin(cpu, env, NOW).dispose_none().finish(); diff --git a/toyos-sched/sim/src/explore.rs b/toyos-sched/sim/src/explore.rs index 38d7a3aebfc..20f8caa57e1 100644 --- a/toyos-sched/sim/src/explore.rs +++ b/toyos-sched/sim/src/explore.rs @@ -101,6 +101,12 @@ pub struct Outcome { /// Per CPU: when it first took an execution step — see /// [`crate::vm::Vm::first_exec_ns`]. pub first_exec_ns: Vec>, + /// Per CPU: wakes out of `hlt` that found nothing to do — what a balance + /// policy costs the idle path. See [`crate::vm::Vm::idle_wakes`]. + pub idle_wakes: Vec, + /// The longest a halted CPU sat beside a published surplus with no probe + /// outstanding — see [`crate::vm::Vm::probe_gap_ns`]. + pub probe_gap_ns: u64, /// What each CPU's passes cost, as the core's `feature = "check"` recorder /// measured them — the on-target instrument, driven here by the scenario's /// modelled `pass_cost_ns`. @@ -148,6 +154,18 @@ impl Outcome { .try_fold(0, |worst, at| Some(worst.max(at?))) } + /// How many CPUs of the machine ever executed a step. + pub fn cpus_reached(&self) -> usize { + self.first_exec_ns.iter().filter(|at| at.is_some()).count() + } + + /// Wakes out of `hlt` that found nothing to do, summed over the machine — + /// the price of a balance policy over one run. Divide by [`Self::elapsed`] + /// for a rate; `sim/tests/policy.rs` does. + pub fn idle_wakes_total(&self) -> u64 { + self.idle_wakes.iter().sum() + } + pub fn report(&self) -> String { if self.passed() { return format!( @@ -286,6 +304,8 @@ fn outcome_of(scenario: &'static str, vm: &Vm<'_>, choices: &ChoiceStream) -> Ou process_finish_ns: vm.finish_ns.clone(), migrations: vm.migrations, first_exec_ns: vm.first_exec_ns.clone(), + idle_wakes: vm.idle_wakes.clone(), + probe_gap_ns: vm.probe_gap_ns, pass_costs: (0..vm.handles.len()) .map(|cpu| { let cpu = CpuId(cpu as u32); diff --git a/toyos-sched/sim/src/scenarios.rs b/toyos-sched/sim/src/scenarios.rs index 56104033039..132eb11a554 100644 --- a/toyos-sched/sim/src/scenarios.rs +++ b/toyos-sched/sim/src/scenarios.rs @@ -5,12 +5,13 @@ //! workload, and a scenario that takes ten thousand steps to quiesce buys one //! schedule per second instead of a thousand. +use toyos_sched::cpu::Balance; use toyos_sched::queue::FairOrder; use toyos_sched::task::WaitClass; use crate::vm::RUN_CHUNK_NS; use crate::workload::{ - AgeShape, BalanceShape, BlockShape, ChargeShape, IrqSpec, MigrateShape, Op, ParkShape, + AgeShape, BlockShape, ChargeShape, IrqSpec, MigrateShape, Op, ParkShape, PlacementShape, ProcSpec, Protocol, QueueSpec, Scenario, Script, ShareShape, WindowShape, }; @@ -41,7 +42,7 @@ fn scenario( share: ShareShape::PerProcess, charge: ChargeShape::Honest, placement: PlacementShape::LeastLoadedRotating, - balance: BalanceShape::Pull, + balance: Balance::Pull, order: FairOrder::InsertSequence, pass_cost_ns: 0, fair_allowance_ns: 0, @@ -1394,7 +1395,7 @@ pub const STORM_ROUNDS: usize = 4; /// /// `sim/tests/policy.rs` measures how long the machine takes to start working /// and how long it takes to finish, against a bound derived from the protocol, -/// with [`BalanceShape::None`] as the control. +/// with [`Balance::None`] as the control. pub fn lopsided_placement(cpus: usize, threads: usize, work: u64) -> Scenario { let mut scenario = scenario( "lopsided_placement", diff --git a/toyos-sched/sim/src/vm.rs b/toyos-sched/sim/src/vm.rs index f2d412946af..fa0441fb4fa 100644 --- a/toyos-sched/sim/src/vm.rs +++ b/toyos-sched/sim/src/vm.rs @@ -38,7 +38,7 @@ use crate::payload::{ MockAddressSpace, SimCtx, SimPayload, SimPreempt, SimShareLock, SimWaitList, StdLock, }; use crate::workload::{ - AgeShape, BalanceShape, BlockShape, ChargeShape, MigrateShape, Op, ParkShape, PlacementShape, + AgeShape, BlockShape, ChargeShape, MigrateShape, Op, ParkShape, PlacementShape, Protocol, Scenario, Script, ShareShape, WindowShape, }; @@ -481,8 +481,53 @@ pub struct Vm<'q> { /// finished recovering the machine, and a `None` in it is a CPU it never /// reached. pub first_exec_ns: Vec>, + /// Per CPU: how many times it was woken out of `hlt` and had nothing to do + /// when it got there. + /// + /// **The price of a balance policy, in the one unit the idle path is charged + /// in.** `kernel/CLAUDE.md` makes anything added to the idle loop an audio + /// change, and what [`Balance::PullWithRearm`] and [`Balance::PushOnSurplus`] + /// add is wakes: a re-armed timer firing on a CPU with nothing queued, or a + /// doorbell ring with no message behind it. Both land here, and so does + /// anything else that wakes a halted CPU for nothing — the count is a + /// property of the *run* and not of the policy, which is what lets + /// [`Balance::Pull`]'s own figure be the baseline the others are read + /// against. + /// + /// "Had nothing to do" is the pass's own answer rather than a guess: the + /// wake is counted when the first pass after it ends in `Action::Idle` + /// again. A wake that dispatched something was worth taking, whoever sent + /// it. + pub idle_wakes: Vec, + /// Per CPU: woken out of `hlt` and not yet through the pass that says + /// whether the wake was worth anything. + woken_from_halt: Vec, + /// The longest any one CPU sat **halted, with a sibling publishing a surplus + /// of two or more, and no probe of its own outstanding**. + /// + /// This is the defect + /// `issues/kernel/an-idle-cpu-that-slept-before-the-surplus-is-never-probed.md` + /// names, as a duration. Under [`Balance::Pull`] the interval ends only when + /// the surplus does, because nothing in that protocol can end it; under a + /// cure it is bounded by the cure's own period. Both bounds are derived in + /// `sim/tests/policy.rs`, and it is the quantity those derivations are + /// asserted against — a count of CPUs reached says whether the machine + /// recovered, and this says how long it was blind. + /// + /// Read at step boundaries, which is exactly the model's resolution: the + /// clock moves only on an execution step or a clock jump, so no interval can + /// open and close between two readings. + pub probe_gap_ns: u64, + /// Per CPU: when its current probe gap opened, or `None` if it is not in + /// one. + probe_gap_since: Vec>, } +/// The surplus at which a victim is worth probing — +/// `SchedPass::post_steal_probe`'s own inequality, restated here because +/// [`Vm::probe_gap_ns`] has to ask the same question from outside the core. +const PROBE_WORTH_IT: u32 = 2; + /// One contention window: a maximal interval over which the same set of /// fair-band processes was continuously runnable. /// @@ -602,6 +647,10 @@ impl<'q> Vm<'q> { finish_ns: vec![None; process_count], migrations: 0, first_exec_ns: vec![None; n], + idle_wakes: vec![0; n], + woken_from_halt: vec![false; n], + probe_gap_ns: 0, + probe_gap_since: vec![None; n], next_irq, next_key: 1, next_spawn_cpu: 0, @@ -933,6 +982,34 @@ impl<'q> Vm<'q> { (false, None, _) => {} } } + self.note_probe_gaps(); + } + + /// [`Vm::probe_gap_ns`]: how long a halted CPU has sat next to a surplus it + /// has no probe out for. + /// + /// The maximum is refreshed on every step the gap is still open rather than + /// only when it closes, so a run that quiesces — or stops at a violation — + /// with a gap standing still reports it. + fn note_probe_gaps(&mut self) { + let halted = self.hw.with(|s| s.halted.clone()); + // A CPU number, for [`Vm::execute`]'s reason: it reads `halted`, asks + // `self.cpus` and writes `self.probe_gap_since` at one index, and the + // pairing across those three is what this loop is about. + #[allow(clippy::needless_range_loop)] + for cpu in 0..self.scenario.cpus { + let surplus_next_door = (0..self.scenario.cpus).any(|victim| { + victim != cpu && self.handles.get(CpuId(victim as u32)).surplus() >= PROBE_WORTH_IT + }); + let blind = halted[cpu] && surplus_next_door && !self.cpus[cpu].probe_outstanding(); + match (blind, self.probe_gap_since[cpu]) { + (true, None) => self.probe_gap_since[cpu] = Some(self.clock), + (true, Some(since)) => { + self.probe_gap_ns = self.probe_gap_ns.max(self.clock.since(since)); + } + (false, _) => self.probe_gap_since[cpu] = None, + } + } } /// Record how long one unwind stamp stood before an execution step (or the @@ -949,19 +1026,24 @@ impl<'q> Vm<'q> { self.run_pass(cpu, Dispose::None); } Step::DeliverIpi(cpu) => { - self.hw.with(|s| { + // Whether this ended a `hlt` is read here and judged later: the + // pass that follows says whether the wake was worth taking. See + // [`Vm::idle_wakes`]. + let was_halted = self.hw.with(|s| { s.pending_ipi[cpu] -= 1; s.need_resched[cpu] = true; - s.halted[cpu] = false; + core::mem::replace(&mut s.halted[cpu], false) }); + self.woken_from_halt[cpu] |= was_halted; self.ipi_due[cpu] = None; } Step::FireTimer(cpu) => { - self.hw.with(|s| { + let was_halted = self.hw.with(|s| { s.armed[cpu] = None; s.need_resched[cpu] = true; - s.halted[cpu] = false; + core::mem::replace(&mut s.halted[cpu], false) }); + self.woken_from_halt[cpu] |= was_halted; } Step::DeviceIrq(index) => self.device_irq(index), Step::Advance => { @@ -1064,10 +1146,10 @@ impl<'q> Vm<'q> { // Copied out before the borrow: the injection below runs while the // pass holds `CpuSched`, and these are the only fields it needs. let queues = self.queues; - // The one policy bit in the `Env` (`sched::driver::env`), read off the + // The one policy value in the `Env` (`sched::driver::env`), read off the // scenario for the same reason as the shapes above it — see - // [`BalanceShape`]. - let steal = self.scenario.balance == BalanceShape::Pull; + // [`Scenario::balance`]. + let balance = self.scenario.balance; let mut injected = None; let (action, parked, end) = { let Vm { @@ -1082,7 +1164,7 @@ impl<'q> Vm<'q> { cpus: handles, frontier, preempt: &SimPreempt, - steal, + balance, }; let pass = SchedPass::begin(&mut cpus[cpu], env, now); match dispose { @@ -1168,6 +1250,11 @@ impl<'q> Vm<'q> { if end == BlockEnd::Killed { self.killed_at_park += 1; } + // The verdict on whatever wake ended this CPU's `hlt`: a pass that + // reaches the idle disposition again found nothing to do with it. + if core::mem::take(&mut self.woken_from_halt[cpu]) && matches!(action, Action::Idle(_)) { + self.idle_wakes[cpu] += 1; + } self.apply(action); self.hw.leave_pass(); self.note_kicks(kicks_before); diff --git a/toyos-sched/sim/src/workload.rs b/toyos-sched/sim/src/workload.rs index b17f985c664..f7e5d8dcf53 100644 --- a/toyos-sched/sim/src/workload.rs +++ b/toyos-sched/sim/src/workload.rs @@ -11,6 +11,7 @@ //! `Futex`. Giving it a second opcode would be modelling a second wake path //! — the very thing §8.2 removes. +use toyos_sched::cpu::Balance; use toyos_sched::queue::FairOrder; use toyos_sched::task::WaitClass; @@ -254,25 +255,6 @@ pub enum PlacementShape { AllOn(usize), } -/// Whether the pull half of the balance path runs. -/// -/// A scenario dimension for [`PlacementShape`]'s reason and with the same two -/// roles: the shipped answer, and the control that says what the measurement -/// would be without it. `kernel::sched::driver::env` sets `steal: true` and -/// never clears it, so [`BalanceShape::None`] is a kernel this tree has not -/// shipped — `toyos-sched/loom`'s `loom_retire` already builds an `Env` with -/// `steal: false` for the same reason, to state what the rest of the protocol -/// does without it. -#[derive(Clone, Copy, PartialEq, Eq, Debug)] -pub enum BalanceShape { - /// Spec §7.7 and §9.4's pull half: an idle pass probes the CPU with the - /// most surplus, and a loaded pass answers a probe out of `pop_surplus`. - Pull, - /// No probe and no answer. A task woken or placed onto a busy CPU waits - /// there until that CPU's own queue reaches it. - None, -} - /// Which teardown/balance algorithm the VM drives the core with. #[derive(Clone, Copy, PartialEq, Eq, Debug)] pub enum Protocol { @@ -302,7 +284,18 @@ pub struct Scenario { pub share: ShareShape, pub charge: ChargeShape, pub placement: PlacementShape, - pub balance: BalanceShape, + /// What the balance path does. A scenario dimension for [`PlacementShape`]'s + /// reason and with the same roles: the shipped answer + /// ([`Balance::Pull`], which is what `kernel::sched::driver::env` selects), + /// the control that says what the measurement would be without it + /// ([`Balance::None`]), and the two candidate cures for the one-shot probe + /// that neither the kernel nor any default here selects. + /// + /// **The core's own type rather than a parallel copy of it**, exactly as + /// [`Scenario::order`] carries `FairOrder`: every setting is a policy the + /// core implements, so the simulator drives the real code whichever one a + /// scenario names. + pub balance: Balance, /// How the fair band picks between two ready threads of one share. A /// scenario dimension for the same reason [`ShareShape`] is, and the type is /// the core's own rather than a parallel copy of it: the broken orderings @@ -412,7 +405,7 @@ impl Scenario { self } - pub fn with_balance(mut self, balance: BalanceShape) -> Self { + pub fn with_balance(mut self, balance: Balance) -> Self { self.balance = balance; self } diff --git a/toyos-sched/sim/tests/deadline_claim_race.rs b/toyos-sched/sim/tests/deadline_claim_race.rs index a9ecf814170..c1562985039 100644 --- a/toyos-sched/sim/tests/deadline_claim_race.rs +++ b/toyos-sched/sim/tests/deadline_claim_race.rs @@ -20,7 +20,7 @@ use std::sync::Arc as StdArc; -use toyos_sched::cpu::{Action, CpuHandle, CpuHandles, CpuSched, Env, SchedPass}; +use toyos_sched::cpu::{Action, Balance, CpuHandle, CpuHandles, CpuSched, Env, SchedPass}; use toyos_sched::fair::{FairShare, Frontier, ShareState}; use toyos_sched::hw::{CpuId, Kicker, Nanos}; use toyos_sched::mailbox::{mailbox, Kick, Urgency}; @@ -55,7 +55,7 @@ impl Machine1 { cpus: &self.handles, frontier: &self.frontier, preempt: &SimPreempt, - steal: false, + balance: Balance::None, } } } diff --git a/toyos-sched/sim/tests/policy.rs b/toyos-sched/sim/tests/policy.rs index 40fd461ec6b..e971223b34e 100644 --- a/toyos-sched/sim/tests/policy.rs +++ b/toyos-sched/sim/tests/policy.rs @@ -39,14 +39,21 @@ //! | wakeup storm drain | 27.75 ms for 64 waiters on one CPU, 3.1× the 16-waiter figure | `(per queue+1) × 12 ms` = 780 ms | 0.04 | //! | starvation | 70 ms at 5 runnable threads, 680 ms at 65 | `(threads+1) × 12 ms` | 0.97 / 0.86 | //! | adversarial placement | a CPU an adversary loaded is drained to the surplus floor, and only the CPUs still awake are reached | `threads − 2` stealable | 1.00 | +//! | a bounded re-arm | every CPU is reached at every width; 10 ms of probe gap | `every_ns + 3 × RUN_CHUNK` = 13 ms | 0.77 | +//! | a push on surplus | every CPU is reached at every width; 66 ms of probe gap at eight | `(cpus−1) × 12 ms + IPI + 2 × RUN_CHUNK` | 0.77 | +//! | what the cures cost | 0 idle wakes/s shipped, 154/s for the cheapest re-arm and **0/s** for the push on `audio_pipeline` | wake latency unchanged, under its own bound | — | //! -//! The last case is the only one whose bound is a *count* rather than a -//! duration, and that is the model's limit rather than a choice: the clock -//! advances on the executing CPU's step, so a run's makespan is what one CPU -//! would take at every width and no wall-clock statement about a wide machine -//! can be read off it. What survives is the protocol — how many tasks the -//! balance path moved, and which CPUs ever got one — and that is enough to -//! separate a machine that recovers from one that does not. +//! The adversarial-placement case is the only one whose bound is a *count* +//! rather than a duration, and that is the model's limit rather than a choice: +//! the clock advances on the executing CPU's step, so a run's makespan is what +//! one CPU would take at every width and no wall-clock statement about a wide +//! machine can be read off it. What survives is the protocol — how many tasks +//! the balance path moved, and which CPUs ever got one — and that is enough to +//! separate a machine that recovers from one that does not. The two cure cases +//! below it inherit the same limit and answer it the same way: recovery is +//! counted in CPUs, and timed in **probe gaps** — how long a halted CPU sat +//! beside a published surplus with no probe of its own outstanding, which is a +//! quantity the protocol owns end to end. //! //! Every number came from this file at 16–40 seeds per point: //! `cargo test -p toyos-sched-sim --test policy -- --nocapture`. The tables in @@ -64,6 +71,18 @@ //! form of "processes own fair share", and it is measured here rather than //! asserted anywhere. //! +//! **The balance path is pull-only and one-shot, and both cures for that work.** +//! An idle CPU posts one probe on its way to `hlt` and nothing re-posts it, so a +//! CPU that halted before any neighbour published surplus is never probed again: +//! 0 of 20 seeds reach every CPU at eight. A bounded re-arm of the probe and a +//! push on surplus each take that to 20 of 20 at every width, and they cost +//! opposite things — the timer ticks whether or not there is anything to come +//! for, the push fires only where there is. On the audio pipeline that is 154 +//! extra idle wakes per second against **zero**. Neither is shipped; +//! `issues/kernel/an-idle-cpu-that-slept-before-the-surplus-is-never-probed.md` +//! is where the decision sits, and these three cases are the numbers it is owed +//! against. +//! //! # Determinism, seeds, and what varies //! //! The simulator is deterministic in its decision stream: a seed is a schedule. @@ -77,13 +96,14 @@ //! a reader is owed that. The multi-CPU cases do vary, and there the sweep is //! doing what a sweep is for. +use toyos_sched::cpu::Balance; use toyos_sched::fair::QUANTUM_NS; use toyos_sched_sim::choice::ChoiceStream; use toyos_sched_sim::explore::{run, Outcome}; use toyos_sched_sim::latency::{Latency, ReadyCause}; use toyos_sched_sim::scenarios; -use toyos_sched_sim::vm::RUN_CHUNK_NS; -use toyos_sched_sim::workload::{BalanceShape, Scenario, ShareShape}; +use toyos_sched_sim::vm::{IPI_LATENCY_NS, RUN_CHUNK_NS}; +use toyos_sched_sim::workload::{Scenario, ShareShape}; const MS: u64 = 1_000_000; @@ -704,7 +724,7 @@ fn the_balance_path_drains_the_cpu_an_adversary_loaded() { fn without_the_balance_path_a_lopsided_machine_stays_lopsided() { for (cpus, threads) in LOPSIDED { let scenario = - scenarios::lopsided_placement(cpus, threads, WORK).with_balance(BalanceShape::None); + scenarios::lopsided_placement(cpus, threads, WORK).with_balance(Balance::None); let floor = stealable(threads); let (mut most, mut widest) = (0, 0); @@ -731,6 +751,531 @@ fn without_the_balance_path_a_lopsided_machine_stays_lopsided() { } } +/// How long a cure waits before probing again, and how many times. +/// +/// One quantum, because that is the interval the rest of this scheduler already +/// works in: a loaded CPU reaches a pass at every quantum boundary, so a thief +/// that re-probes faster than that is asking a question whose answer cannot have +/// changed. Four, because the re-arm has to be **bounded** — `times × every_ns` +/// is the whole window a cure of this shape can see a surplus in, and past it +/// the CPU halts for good rather than ticking for ever on a machine with nothing +/// to run. +const REARM_EVERY_NS: u64 = QUANTUM_NS; +const REARM_TIMES: u32 = 4; + +/// The surplus a push fires at: `SchedPass::post_steal_probe`'s own inequality, +/// so a push never wakes a CPU whose victim would refuse it. +const PUSH_THRESHOLD: u32 = 2; + +/// The four policies the two cure cases and the cost table are read across. +fn policies() -> [(&'static str, Balance); 4] { + [ + ("pull (ships)", Balance::Pull), + ( + "re-arm ×1", + Balance::PullWithRearm { + every_ns: REARM_EVERY_NS, + times: 1, + }, + ), + ( + "re-arm ×4", + Balance::PullWithRearm { + every_ns: REARM_EVERY_NS, + times: REARM_TIMES, + }, + ), + ( + "push ≥2", + Balance::PushOnSurplus { + threshold: PUSH_THRESHOLD, + }, + ), + ] +} + +/// What one policy did with one lopsided machine, over a whole sweep. +#[derive(Default)] +struct Recovery { + /// Seeds in which every CPU of the machine executed a step. + full: usize, + /// Seeds in which more than cpu0 did. + second: usize, + /// The most tasks the balance path moved in any seed. + best_migrations: u64, + /// The worst [`Outcome::probe_gap_ns`] any seed produced. + gap_ns: u64, + /// The worst and best `machine_working_at_ns` over the seeds that reached + /// every CPU; `None` if no seed did. + working_worst: Option, + working_best: Option, + /// Idle wakes summed over the sweep, and the worst single run's rate. + wakes: u64, + worst_rate: f64, +} + +/// Wakes of a halted CPU that found nothing to do, per second of simulated time +/// — the unit a balance policy's cost is quoted in. +fn wake_rate(outcome: &Outcome) -> f64 { + if outcome.elapsed == 0 { + return 0.0; + } + outcome.idle_wakes_total() as f64 * 1e9 / outcome.elapsed as f64 +} + +/// Run the lopsided machine under one policy and fold what it did. +/// +/// The two ceilings the shipped path already obeys are asserted here rather than +/// in each case, because they are claims about the *pull* half and every policy +/// below is built on it: no CPU is given work by anything but the balance path, +/// and no task is moved twice. +fn recover(cpus: usize, threads: usize, balance: Balance) -> Recovery { + let scenario = scenarios::lopsided_placement(cpus, threads, WORK).with_balance(balance); + let floor = stealable(threads); + let mut r = Recovery::default(); + sweep(&scenario, LOPSIDED_SEEDS, |outcome| { + let ran = outcome.cpus_reached(); + assert!( + outcome.migrations + 1 >= ran as u64, + "{balance:?} at {cpus} cpus: {ran} cpu(s) executed a step on a machine whose every \ + thread was spawned onto cpu0, and the balance path moved only {} task(s) — one of \ + those CPUs was given work by something that is not the balance path", + outcome.migrations, + ); + assert!( + outcome.migrations <= floor, + "{balance:?} at {cpus} cpus: the balance path moved {} of {threads} tasks, past the \ + {floor} that stand above cpu0's surplus floor — a task is being migrated more than \ + once, which is a thief that became a victim", + outcome.migrations, + ); + r.full += usize::from(ran == cpus); + r.second += usize::from(ran > 1); + r.best_migrations = r.best_migrations.max(outcome.migrations); + r.gap_ns = r.gap_ns.max(outcome.probe_gap_ns); + r.wakes += outcome.idle_wakes_total(); + r.worst_rate = r.worst_rate.max(wake_rate(outcome)); + if let Some(at) = outcome.machine_working_at_ns() { + r.working_worst = Some(r.working_worst.unwrap_or(0).max(at)); + r.working_best = Some(r.working_best.unwrap_or(u64::MAX).min(at)); + } + }); + r +} + +/// The condition under which "every CPU is reached" is a statement about the +/// balance path rather than about arithmetic: cpu0 can be drained to +/// [`stealable`] and every other CPU needs one task, so there has to be at least +/// one task per starved CPU above the floor. +fn enough_to_go_round(cpus: usize, threads: usize) { + assert!( + stealable(threads) >= cpus as u64 - 1, + "at {cpus} cpus and {threads} threads only {} task(s) stand above cpu0's surplus floor \ + and {} CPU(s) need one each — no balance policy can reach every CPU here, so a case \ + asserting that it does would be asserting arithmetic", + stealable(threads), + cpus - 1, + ); +} + +/// **Cure one, measured: a bounded re-arm of the probe.** +/// +/// `issues/kernel/an-idle-cpu-that-slept-before-the-surplus-is-never-probed.md` +/// names two ways out of the one-shot probe, and this is the one that needs no +/// observation of anything: a CPU that halts with nothing to run programs its +/// one-shot timer [`REARM_EVERY_NS`] ahead and probes again when it fires, up to +/// [`REARM_TIMES`] times per idle period. Nothing has to notice it, nothing has +/// to publish anything, and the timer fires whether or not a surplus ever +/// appeared — which is both why it works and what it costs. +/// +/// **The derivation, and it is the assertion.** A CPU that halted at `H` is +/// woken at `H + every_ns`: the model's step relation forbids an execution step +/// anywhere while a CPU owes a timer delivery, so the clock cannot run past the +/// armed instant unpunished. Three [`RUN_CHUNK_NS`] chunks of granularity sit on +/// top, and each is a step the model permits before the probe is posted — the +/// execution step that carries the clock over the armed instant, the one chunk +/// of grace `Vm::enabled` gives a CPU that owes a rescheduling pass, and the +/// execution step that carries the clock over *that*. So no CPU may sit halted +/// beside a published surplus with no probe outstanding for longer than +/// `every_ns + 3 × RUN_CHUNK` — [`Outcome::probe_gap_ns`] is that quantity, and +/// the surplus in this workload is published at clock 0, before any CPU can have +/// halted, so one re-arm is inside the window at every width. +/// +/// **What it found**, 20 seeds per width at 60 ms of work per thread and +/// `every_ns = QUANTUM_NS`: +/// +/// | cpus | threads | every CPU, ×1 | every CPU, ×4 | pull | probe gap | pull's gap | idle wakes/s, worst run (×1 / ×4) | +/// |---|---|---|---|---|---|---|---| +/// | 2 | 8 | 20/20 | 20/20 | 9/20 | 10.0 ms | 480 ms | 8.16 / 21.15 | +/// | 4 | 16 | 20/20 | 20/20 | 2/20 | 10.0 ms | 960 ms | 8.25 / 23.00 | +/// | 4 | 64 | 20/20 | 20/20 | 2/20 | 10.0 ms | 3,840 ms | 3.38 / 6.44 | +/// | 8 | 64 | 20/20 | 20/20 | 0/20 | 10.0 ms | 3,840 ms | 6.75 / 12.89 | +/// +/// The derived bound is 13,000,000 ns and the measurement is 10,000,000 ns at +/// every width — 0.77 of it, which is where the rest of this file's numbers sit. +/// It is *exactly* `every_ns`, because this workload's clock is a multiple of +/// `RUN_CHUNK_NS` throughout and none of the three granularity chunks is ever +/// spent; they stay in the bound because the model permits them, not because a +/// schedule was found that needs them. `Balance::Pull`'s own gap is the whole +/// run at every width, because nothing in that protocol can close it. +/// +/// **One re-arm is enough here**: the ×1 and ×4 columns are the same recovery +/// for two and a half times the wakes. What this workload needs is *a* second +/// look, not a periodic one — and `times` is what decides how much of the second +/// kind is bought with it. +/// +/// **The negative control.** With the cure off and everything else identical +/// (`Balance::Pull` substituted for the policy under test) this case reds on its +/// first width, at the recovery assertion: +/// +/// ```text +/// with a re-arm every 10000000 ns, 11 of 20 schedules left a CPU of 2 asleep on +/// a machine whose whole workload sits on cpu0. [...] left: 9 right: 20 +/// ``` +/// +/// and with that assertion elided, at the derived timing one behind it: +/// +/// ```text +/// a CPU sat halted beside a published surplus with no probe outstanding for +/// 480000000 ns, against a derived 13000000 ns +/// ``` +/// +/// A CPU's first execution step is reported and **not** asserted: the model does +/// not oblige a CPU that has a task loaded to take one, so `working_at` measures +/// the explorer's freedom as much as the protocol's recovery — 20,000,000 ns in +/// the best seed at eight CPUs against 3,780,000,000 ns in the worst, on runs +/// whose probe gap is 10,000,000 ns either way. That is the same limit the case +/// above states about makespan, and it is why recovery is counted in CPUs and +/// timed in probe gaps. +#[test] +fn a_bounded_re_arm_reaches_every_cpu_the_pull_path_left_asleep() { + let bound = REARM_EVERY_NS + 3 * RUN_CHUNK_NS; + let mut table = Vec::new(); + for (cpus, threads) in LOPSIDED { + enough_to_go_round(cpus, threads); + let floor = stealable(threads); + let control = recover(cpus, threads, Balance::Pull); + for times in [1, REARM_TIMES] { + let balance = Balance::PullWithRearm { + every_ns: REARM_EVERY_NS, + times, + }; + let cured = recover(cpus, threads, balance); + println!( + "re-arm ×{times} cpus={cpus} threads={threads}: every-cpu \ + {}/{LOPSIDED_SEEDS} (pull {}/{LOPSIDED_SEEDS}), gap={} ns (pull {} ns, bound \ + {bound} ns), drained {}/{floor}, working_at {:?}..{:?} ns, {} idle wakes over \ + the sweep, worst run {:.2}/s", + cured.full, + control.full, + cured.gap_ns, + control.gap_ns, + cured.best_migrations, + cured.working_best, + cured.working_worst, + cured.wakes, + cured.worst_rate, + ); + + // **The law.** A cure that re-probes every `every_ns` cannot leave a + // CPU asleep beside a surplus that was published before it halted, + // and in this workload every surplus is. + assert_eq!( + cured.full, LOPSIDED_SEEDS as usize, + "with a re-arm every {REARM_EVERY_NS} ns, {} of {LOPSIDED_SEEDS} schedules left \ + a CPU of {cpus} asleep on a machine whose whole workload sits on cpu0. cpu0 \ + publishes its surplus at clock 0 — before any CPU can have halted, since no \ + execution step is enabled until it has taken its first pass — so every halt is \ + inside the first re-arm's window and every CPU must be probed.", + LOPSIDED_SEEDS as usize - cured.full, + ); + assert!( + cured.gap_ns <= bound, + "a CPU sat halted beside a published surplus with no probe outstanding for {} \ + ns, against a derived {bound} ns — one re-arm period at {REARM_EVERY_NS} ns \ + plus three {RUN_CHUNK_NS} ns chunks of the model's own granularity. Past this \ + the timer is not what is waking the CPU.", + cured.gap_ns, + ); + // And the drain is unchanged: a cure that reaches every CPU by + // moving more tasks than cpu0 has to give would be a different + // machine, not a repaired one. + assert_eq!( + cured.best_migrations, floor, + "the best of {LOPSIDED_SEEDS} schedules moved {} tasks and the surplus floor \ + leaves {floor} stealable — the re-arm changes when a probe is posted and \ + nothing about what answering one hands over", + cured.best_migrations, + ); + table.push((cpus, threads, times, cured, control.full, control.gap_ns)); + } + } + + // The negative control, stated where the comparison is: with the cure off, + // the same assertion has to fail. It does — at eight CPUs not one of the + // twenty schedules reaches every CPU. + let (_, _, _, _, worst_control_full, _) = table + .iter() + .map(|&(c, t, times, ref cured, full, gap)| (c, t, times, cured.full, full, gap)) + .min_by_key(|&(_, _, _, _, full, _)| full) + .expect("the sweep ran"); + assert!( + worst_control_full < LOPSIDED_SEEDS as usize, + "`Balance::Pull` reached every CPU in every one of {LOPSIDED_SEEDS} schedules at every \ + width — then the case above is not measuring a cure, because there is nothing left to \ + cure", + ); +} + +/// **Cure two, measured: a push on surplus.** +/// +/// The other way out of the one-shot probe, and the one that costs almost +/// nothing: a pass that publishes a surplus of [`PUSH_THRESHOLD`] or more rings +/// the doorbell of one CPU that reads SLEEPING. No task moves on that ring — the +/// woken CPU runs its own idle pass and posts an ordinary probe — so the push +/// adds no second way to migrate anything, only a way to make the pull half run +/// on a CPU that had stopped asking. +/// +/// **The derivation, and it is the assertion.** A pass pushes to one CPU, and +/// `SchedPass::push_on_surplus`'s cursor makes consecutive pushes walk the +/// machine, so `k` sleeping CPUs are reached in `k` passes of the CPU holding the +/// surplus. That CPU is running a task, so it reaches a pass at each quantum +/// boundary: `k × DISPATCH_NS`. The kick is an IPI ([`IPI_LATENCY_NS`], which the +/// model enforces by disabling every execution step while a delivery is overdue) +/// and the pass it asks for is one chunk of grace plus the chunk that carries the +/// clock over it. Here `k` is `cpus − 1`, every CPU but the loaded one, so +/// +/// ```text +/// (cpus − 1) × DISPATCH_NS + IPI_LATENCY_NS + 2 × RUN_CHUNK_NS +/// ``` +/// +/// **What it found**, 20 seeds per width: +/// +/// | cpus | threads | seeds reaching every CPU | pull | probe gap | derived | measured/bound | idle wakes/s, worst run | +/// |---|---|---|---|---|---|---|---| +/// | 2 | 8 | 20/20 | 9/20 | 1.0 ms | 14.2 ms | 0.07 | 2.08 | +/// | 4 | 16 | 20/20 | 2/20 | 22.0 ms | 38.2 ms | 0.58 | 3.12 | +/// | 4 | 64 | 20/20 | 2/20 | 22.0 ms | 38.2 ms | 0.58 | 0.78 | +/// | 8 | 64 | 20/20 | 0/20 | 66.0 ms | 86.2 ms | 0.77 | 1.82 | +/// +/// The bound is loose at two CPUs for the reason it is tight at eight: there is +/// one sleeper to reach and the first pass reaches it, so `k = 1` and the `k` +/// term has nothing to say. The wakes it costs are 11, 36, 36 and 74 over a +/// twenty-seed sweep — between three and eleven times fewer than the cheapest +/// re-arm buys the same recovery for. +/// +/// **The cursor is load-bearing and the measurement is what said so.** Without +/// it every push goes to the lowest-numbered sleeper, which posts its probe and +/// halts again with SLEEPING still set — so the next pass re-pokes the CPU that +/// is already coming, and the one behind it waits for the first one's probe to +/// be *answered*. Two passes per sleeper: 130,000,000 ns of probe gap at eight +/// CPUs against the 66,000,000 ns above. +/// +/// **The negative control**, `Balance::Pull` substituted for the policy under +/// test and nothing else changed — the recovery assertion first: +/// +/// ```text +/// with a push at a surplus of 2, 11 of 20 schedules left a CPU of 2 asleep on a +/// machine whose whole workload sits on cpu0. [...] left: 9 right: 20 +/// ``` +/// +/// and the derived timing one behind it: +/// +/// ```text +/// a CPU sat halted beside a published surplus with no probe outstanding for +/// 480000000 ns, against a derived 14200000 ns +/// ``` +#[test] +fn a_push_on_surplus_reaches_every_cpu_the_pull_path_left_asleep() { + let balance = Balance::PushOnSurplus { + threshold: PUSH_THRESHOLD, + }; + let mut reached_all = 0; + for (cpus, threads) in LOPSIDED { + enough_to_go_round(cpus, threads); + let floor = stealable(threads); + let sleepers = cpus as u64 - 1; + let bound = sleepers * DISPATCH_NS + IPI_LATENCY_NS + 2 * RUN_CHUNK_NS; + let control = recover(cpus, threads, Balance::Pull); + let cured = recover(cpus, threads, balance); + + println!( + "push ≥{PUSH_THRESHOLD} cpus={cpus} threads={threads}: every-cpu \ + {}/{LOPSIDED_SEEDS} (pull {}/{LOPSIDED_SEEDS}), gap={} ns (pull {} ns, bound \ + {bound} ns), drained {}/{floor}, working_at {:?}..{:?} ns, {} idle wakes over the \ + sweep, worst run {:.2}/s", + cured.full, + control.full, + cured.gap_ns, + control.gap_ns, + cured.best_migrations, + cured.working_best, + cured.working_worst, + cured.wakes, + cured.worst_rate, + ); + + assert_eq!( + cured.full, LOPSIDED_SEEDS as usize, + "with a push at a surplus of {PUSH_THRESHOLD}, {} of {LOPSIDED_SEEDS} schedules \ + left a CPU of {cpus} asleep on a machine whose whole workload sits on cpu0. cpu0 \ + publishes a surplus of {} at clock 0 and pushes to one sleeping CPU per pass, \ + walking them in turn, so every one of them is rung inside {sleepers} passes.", + LOPSIDED_SEEDS as usize - cured.full, + threads - 1, + ); + assert!( + cured.gap_ns <= bound, + "a CPU sat halted beside a published surplus with no probe outstanding for {} ns, \ + against a derived {bound} ns — {sleepers} sleeping CPU(s) at one push per pass and \ + {DISPATCH_NS} ns a pass, plus {IPI_LATENCY_NS} ns of IPI latency and two \ + {RUN_CHUNK_NS} ns chunks for the pass the kick asks for. Past this the push is \ + reaching the same CPU twice instead of walking the machine.", + cured.gap_ns, + ); + assert_eq!( + cured.best_migrations, floor, + "the best of {LOPSIDED_SEEDS} schedules moved {} tasks and the surplus floor leaves \ + {floor} stealable — the push changes which CPU asks and nothing about what \ + answering hands over", + cured.best_migrations, + ); + reached_all += usize::from(control.full == LOPSIDED_SEEDS as usize); + } + assert!( + reached_all < LOPSIDED.len(), + "`Balance::Pull` reached every CPU in every schedule at every width — then the case \ + above is not measuring a cure", + ); +} + +/// **What the two cures cost the idle path**, on the workloads a desktop +/// actually runs. +/// +/// `kernel/CLAUDE.md` makes anything added to the idle loop an audio change, and +/// both cures add exactly one kind of thing: a wake of a CPU that had halted. +/// [`Outcome::idle_wakes`] counts them — a wake whose first pass reaches the idle +/// disposition again — so the count is a property of the run rather than of the +/// policy, and `Balance::Pull`'s own figure is the baseline the others are read +/// against. It is **zero on every workload here**, which is what makes the +/// comparison one. +/// +/// **What it found**, 20 seeds per point. The first figure is the whole sweep's +/// wakes, the second the worst single run's rate against simulated time: +/// +/// | workload | pull | re-arm ×1 | re-arm ×4 | push ≥2 | +/// |---|---|---|---|---| +/// | `interactive_mix(2,4)` | 0, 0.00/s | 44, 31.58/s | 171, 88.00/s | 9, 11.76/s | +/// | `interactive_mix(2,16)` | 0, 0.00/s | 49, 11.94/s | 174, 30.14/s | 2, 3.08/s | +/// | `interactive_mix(4,16)` | 0, 0.00/s | 104, 20.90/s | 363, 63.01/s | 53, 24.62/s | +/// | `wakeup_storm(4,16)` | 0, 0.00/s | 251, 92.13/s | 1004, 306.22/s | 80, 97.56/s | +/// | `wakeup_storm(8,64)` | 0, 0.00/s | 521, 150.67/s | 2075, 534.78/s | 748, 380.95/s | +/// | `audio_pipeline(4)` | 0, 0.00/s | 111, 153.85/s | 351, 243.90/s | **0, 0.00/s** | +/// +/// **The last row is the decision.** `audio_pipeline` is four threads on four +/// CPUs, so no CPU ever holds a fair band two deep and no CPU ever has surplus to +/// announce: the push fires **not once** in the whole sweep and costs the idle +/// path literally nothing, while the re-arm ticks every idle CPU regardless, +/// because a timer cannot ask whether there is anything to come for. That is the +/// difference between the two cures stated in the unit the owner has to decide +/// in, on the workload the owner cares about. +/// +/// **And the row above it is the qualification.** Under a wakeup storm at eight +/// CPUs the push costs 748 wakes against the cheapest re-arm's 521: a storm is +/// a machine that keeps producing surplus beside CPUs that keep going idle, so +/// the observation the push rests on keeps coming back true. The push is cheap +/// where nothing is queued and dear where a great deal is — which is the +/// opposite way round from the re-arm, and the reason the two rows are both here. +/// +/// **The wake latency does not move**, which is the other half of the same +/// question. The watched thread's worst wake is the same nanosecond under all +/// four policies at every point but one (`wakeup_storm(8,64)`, where the push's +/// 50,750,000 ns is *better* than the shipped path's 57,000,000 ns), and the mean +/// moves by under 2%. That is a schedule perturbation and not a price: the model +/// charges a scheduler pass zero nanoseconds (`SimHwState::pass_cost_ns`), so it +/// can *count* an extra wake and cannot bill one. The count is what goes to the +/// owner; what a wake costs the CPU it wakes is the kernel's own measurement to +/// make. +#[test] +fn the_two_cures_are_priced_against_the_pull_path() { + const COST_SEEDS: u64 = 20; + let points: [(&str, usize, usize, Scenario); 6] = [ + ("interactive_mix(2,4)", 2, 4, scenarios::interactive_mix(2, 4)), + ("interactive_mix(2,16)", 2, 16, scenarios::interactive_mix(2, 16)), + ("interactive_mix(4,16)", 4, 16, scenarios::interactive_mix(4, 16)), + ("wakeup_storm(4,16)", 4, 16, scenarios::wakeup_storm(4, 16)), + ("wakeup_storm(8,64)", 8, 64, scenarios::wakeup_storm(8, 64)), + ("audio_pipeline(4)", 4, 0, scenarios::audio_pipeline(4)), + ]; + for (label, cpus, hogs, scenario) in points { + // Whose wake latency this workload is about: the interactive thread, the + // storm's waiters, or the audio clients. + let watched = scenario + .procs + .iter() + .position(|p| matches!(p.name, "sleeper" | "waiters" | "client")) + .expect("every workload here has a thread whose wake latency is the point"); + // One run queue's worth of rivals plus the leader, exactly as + // `an_interactive_wake_waits_out_at_most_the_band_it_is_queued_behind` + // derives it. `audio_pipeline` carries its own thread count rather than + // a hog parameter, so it is counted from the scenario. + let rivals = if hogs > 0 { + hogs.div_ceil(cpus) as u64 + 1 + } else { + scenario.procs.iter().map(|p| p.initial.len() as u64).sum::() / cpus as u64 + 1 + }; + let bound = (rivals + 1) * DISPATCH_NS; + + for (name, balance) in policies() { + let scenario = scenario.clone().with_balance(balance); + let mut woken = Latency::default(); + let (mut wakes, mut worst_rate) = (0u64, 0.0f64); + sweep(&scenario, COST_SEEDS, |outcome| { + woken.merge(outcome.wait(watched, ReadyCause::Woken)); + wakes += outcome.idle_wakes_total(); + worst_rate = worst_rate.max(wake_rate(outcome)); + }); + println!( + "cost {label} [{name}]: idle wakes {wakes} over {COST_SEEDS} runs, worst run \ + {worst_rate:.2}/s; woken[{}] bound={bound} ns", + woken.summary(), + ); + + assert!( + woken.count() >= COST_SEEDS, + "{label} [{name}]: the watched thread was woken {} time(s) over {COST_SEEDS} \ + runs — the distribution below is a measurement of nothing", + woken.count(), + ); + // The audio-relevant claim, asserted under every policy and not only + // the shipped one: a cure that lengthened the wake path would be + // paying for the balance path out of the interactive one. + assert!( + woken.max_ns() <= bound, + "{label} [{name}]: a wake waited {} ns against a derived {bound} ns — {rivals} \ + runnable thread(s) on one run queue plus the leader, at {DISPATCH_NS} ns a \ + dispatch. A balance policy may cost idle wakes; it may not cost wake latency. \ + Distribution: {}", + woken.max_ns(), + woken.summary(), + ); + // And the baseline is a baseline. Under `Pull` nothing arms a timer + // on a CPU with an empty queue and nothing rings a doorbell without + // a message behind it, so every wake of a halted CPU here carries + // work — if this stops being zero the column the cures are read + // against has stopped meaning "what the shipped path costs". + if balance == Balance::Pull { + assert_eq!( + wakes, 0, + "{label}: the shipped balance path woke a halted CPU {wakes} time(s) for \ + nothing over {COST_SEEDS} runs. Every column beside it is quoted as a cost \ + *over* this one", + ); + } + } + } +} + /// **The starvation bound**: the worst wait of a runnable task under saturation, /// measured, against the fair band's own granularity. /// diff --git a/toyos-sched/src/cpu.rs b/toyos-sched/src/cpu.rs index 0c9b0fbb73a..72f8979d234 100644 --- a/toyos-sched/src/cpu.rs +++ b/toyos-sched/src/cpu.rs @@ -31,7 +31,7 @@ use crate::mailbox::{ }; use crate::msg::Msg; use crate::queue::RunQueue; -use crate::sync::{Arc, AtomicU32, Ordering}; +use crate::sync::{fence, Arc, AtomicU32, Ordering}; #[cfg(feature = "check")] use crate::sync::AtomicU64; use crate::task::{ @@ -273,6 +273,36 @@ pub struct CpuSched { /// bounded by the quantum arm, which is what keeps invariant I4's new term /// exactly one chunk wide. aged_grant: bool, + /// How many times this CPU has re-armed its timer purely to probe again + /// since it last dispatched anything — [`Balance::PullWithRearm`]'s bound. + /// + /// **Counted up from zero and cleared at every dispatch, rather than + /// counted down from an allowance.** The allowance lives in the policy, and + /// a CPU has no policy until a pass hands it one: a countdown initialized + /// here would start at zero on a CPU that had never run a pass, so the one + /// CPU the re-arm exists for — the one that halted at boot before any + /// sibling published a surplus — would be the one CPU it never fired on. + /// Measured that way round: 0 of 20 seeds at eight CPUs, which is + /// [`Balance::Pull`]'s own number exactly. + /// + /// Clearing it at every dispatch is what makes the bound per *idle period* + /// rather than per run: a CPU that is given work and later goes idle again + /// gets the same allowance, and one that has spent it halts for good until + /// something real wakes it — so a machine with nothing to run stops ticking + /// after `times × every_ns` instead of waking for ever. + idle_probes_spent: u32, + /// Where [`Balance::PushOnSurplus`]'s next push starts looking for a + /// sleeping CPU. + /// + /// A pass pushes to **one** CPU, and without this it is the same CPU every + /// time: the target posts its probe and halts again with SLEEPING still + /// published, so the next pass re-pokes the CPU that is already coming and + /// the CPU behind it waits for the first one's probe to be *answered*. Two + /// passes per sleeper instead of one, measured at 130,000,000 ns of probe + /// gap on an eight-CPU lopsided machine against the 44,000,000 ns four CPUs + /// cost. The cursor makes consecutive pushes walk the machine, so `k` + /// sleeping CPUs are reached in `k` passes. + push_cursor: u32, loaded: Loaded, loaded_ctx: *mut X::Ctx, idle_ctx: Box, @@ -310,6 +340,8 @@ impl CpuSched { steal_requests: Vec::new(), quantum_end: Nanos::ZERO, aged_grant: false, + idle_probes_spent: 0, + push_cursor: 0, loaded: Loaded::Idle, loaded_ctx, idle_ctx, @@ -396,6 +428,18 @@ impl CpuSched { self.mailbox.is_empty() } + /// Is a steal probe from this CPU still on its way to a victim? + /// + /// The node's in-flight flag *is* the answer (spec §7.7), so this asks the + /// mechanism rather than a shadow of it. It exists for one reader: the + /// simulator's probe-gap instrument, which measures how long a halted CPU + /// sits with a surplus published next door and no probe outstanding — the + /// quantity [`Balance::PullWithRearm`] and [`Balance::PushOnSurplus`] are + /// judged on. + pub fn probe_outstanding(&self) -> bool { + self.steal_probe.in_flight() + } + /// The number of ready tasks, republished to the handle every pass for /// spawn placement (spec §9.4). pub fn ready_len(&self) -> usize { @@ -500,6 +544,8 @@ impl CpuSched { (core::mem::offset_of!(Self, steal_requests), "steal_requests"), (core::mem::offset_of!(Self, quantum_end), "quantum_end"), (core::mem::offset_of!(Self, aged_grant), "aged_grant"), + (core::mem::offset_of!(Self, idle_probes_spent), "idle_probes_spent"), + (core::mem::offset_of!(Self, push_cursor), "push_cursor"), (core::mem::offset_of!(Self, loaded), "loaded"), (core::mem::offset_of!(Self, loaded_ctx), "loaded_ctx"), (core::mem::offset_of!(Self, idle_ctx), "idle_ctx"), @@ -577,6 +623,131 @@ impl CpuSched { } } +/// What the balance path does — the one policy value in [`Env`]. +/// +/// [`Balance::Pull`] is what ships and what spec §7.7 and §9.4's pull half +/// describe: an idle pass probes the CPU publishing the most surplus, a loaded +/// pass answers a probe out of `pop_surplus`, and nothing else moves a task +/// between CPUs. [`Balance::None`] is the control that says what the rest of the +/// protocol does without it. +/// +/// **The other two are candidate cures, and neither is shipped.** The pull path +/// is one-shot: [`SchedPass::post_steal_probe`] posts at most one probe per idle +/// trip and returns without posting anything if no CPU publishes a surplus of +/// two, so a CPU that reached its idle pass while every sibling still published +/// zero halts with no probe outstanding and nothing in this protocol wakes it — +/// there is no push half, and a loaded CPU never announces that it has surplus. +/// Measured on a lopsided machine at 20 seeds per width, 0 of 20 seeds reach +/// every CPU at eight; `sim/tests/policy.rs` carries the tables and +/// `issues/kernel/an-idle-cpu-that-slept-before-the-surplus-is-never-probed.md` +/// carries the decision the two below are owed against. +/// +/// Both cost an idle CPU wakes it is currently right to be without +/// (`kernel/CLAUDE.md`: anything added to the idle loop is an audio change), so +/// they exist here to be **measured**, behind a knob the simulator selects and +/// the kernel does not. +#[derive(Clone, Copy, PartialEq, Eq, Debug)] +pub enum Balance { + /// No probe and no answer. A task woken or placed onto a busy CPU waits + /// there until that CPU's own queue reaches it. + None, + /// Spec §7.7 and §9.4's pull half, one-shot — what `kernel::sched::driver` + /// selects. + Pull, + /// Pull, plus a **bounded re-arm**: a CPU that halts with nothing to run + /// programs its one-shot timer `every_ns` ahead so it wakes and probes + /// again, up to `times` times per idle period. The counter is refilled the + /// moment the CPU dispatches anything, so the bound is per idle period and + /// a machine that stays idle past `times × every_ns` stops ticking + /// altogether. + /// + /// It needs no observation of anything: the timer fires whether or not a + /// sibling ever published a surplus, which is the whole difference between + /// this and the push below. + PullWithRearm { every_ns: u64, times: u32 }, + /// Pull, plus a **push**: a pass that publishes a surplus of at least + /// `threshold` rings the doorbell of one CPU that reads SLEEPING, which + /// makes that CPU's own idle pass run again and probe. + /// + /// **It rests on an observation, and the observation is one half of + /// Dekker's pair** — see [`balance_fence`]. The pusher stores its surplus + /// and then loads the sleeper's SLEEPING bit; the sleeper stores SLEEPING + /// and then loads the surplus. Without a `SeqCst` fence between each side's + /// store and its load, both may miss, and the CPU sleeps through a surplus + /// exactly as it does under [`Balance::Pull`] — a narrower window on the + /// same defect. `toyos-sched/loom/tests/loom_push.rs` is the model. + PushOnSurplus { threshold: u32 }, +} + +impl Balance { + /// Does the pull half run at all? Every cure is built on it — a push and a + /// re-arm both end in a `StealRequest` that a loaded pass has to answer. + pub fn pulls(self) -> bool { + !matches!(self, Balance::None) + } + + /// The re-arm's period and repeat count, or `None` for a policy without + /// one. + pub fn rearm(self) -> Option<(u64, u32)> { + match self { + Balance::PullWithRearm { every_ns, times } => Some((every_ns, times)), + _ => None, + } + } + + /// The surplus at which a pass pushes, or `None` for a policy that never + /// pushes. + pub fn push_threshold(self) -> Option { + match self { + Balance::PushOnSurplus { threshold } => Some(threshold), + _ => None, + } + } +} + +/// The store/load barrier [`Balance::PushOnSurplus`] is made of. +/// +/// The push is the store-buffer litmus test wearing scheduler clothes. The CPU +/// that gained surplus does +/// +/// ```text +/// surplus.store(n); ... doorbell.load() // "is anybody asleep?" +/// ``` +/// +/// and the CPU going to sleep does +/// +/// ```text +/// doorbell.store(SLEEPING); ... surplus.load() // "is there anything to steal?" +/// ``` +/// +/// Each side stores to one location and loads the other. With plain accesses +/// both loads may return the pre-store value — on x86 because a store sits in +/// the store buffer while a later load bypasses it, and in the C11 model because +/// nothing orders a store against a subsequent load at all. Both sides then +/// decide "nothing to do", and the sleeper halts with a surplus published and no +/// probe outstanding: [`Balance::Pull`]'s defect reproduced in a window a few +/// nanoseconds wide instead of a few milliseconds. +/// +/// A `SeqCst` fence between each side's store and its load is what forbids that +/// outcome, and it is a real instruction on the push path — `mfence` on x86 — +/// which is part of what the push costs. **A cargo feature rather than a +/// comment, because a model that has never failed proves nothing**: +/// `toyos-sched-loom`'s `push-fence-relaxed` weakens it to a release fence, +/// which orders stores against stores and nothing against a later load, and +/// `loom/tests/loom_push.rs` must red under it. No kernel build can turn the +/// name on; the crate declares it only so `cfg` checking knows it. +#[cfg(not(feature = "push-fence-relaxed"))] +const PUSH_FENCE: Ordering = Ordering::SeqCst; +#[cfg(feature = "push-fence-relaxed")] +const PUSH_FENCE: Ordering = Ordering::Release; + +/// The barrier [`PUSH_FENCE`] describes, as the one function both halves of the +/// push call — so the loom model exercises the shipped ordering rather than a +/// restatement of it. +pub fn balance_fence() { + fence(PUSH_FENCE); +} + /// The environment a pass runs against. One value, threaded by reference, so /// that a pass cannot be constructed without the pieces that make its effects /// deliverable. @@ -587,10 +758,9 @@ pub struct Env<'e, H: Hw, P: PreemptGuard> { /// The pass runs preempt-disabled (spec §6.2), which is also what its own /// mailbox pushes need (N3). pub preempt: &'e P, - /// Whether an idle pass probes for work and a loaded pass answers probes - /// (spec §7.7, §9.4's pull half). A field and not a `cfg` so that both - /// settings stay compiled and simulatable. - pub steal: bool, + /// What the balance path does (spec §7.7, §9.4's pull half). A field and not + /// a `cfg` so that every setting stays compiled and simulatable. + pub balance: Balance, } impl Clone for Env<'_, H, P> { @@ -1494,9 +1664,14 @@ impl SchedPass<'_, '_, H, P, Disposed> { // Both are sampled after the pick, so neither counts the task this // CPU is about to run. let handle = self.env.cpus.get(self.cpu.id); + let surplus = self.cpu.rq.fair_len() as u32; handle.publish_load((self.cpu.rq.len() + self.cpu.dying.len()) as u32); - handle.publish_surplus(self.cpu.rq.fair_len() as u32); + handle.publish_surplus(surplus); + self.push_on_surplus(surplus); if self.cpu.running.is_some() { + // The re-arm's allowance is per idle period, so a CPU that is + // given work starts its next one with a full one. + self.cpu.idle_probes_spent = 0; return self.switch_to_current(); } if !self.cpu.is_idle() { @@ -1694,7 +1869,7 @@ impl SchedPass<'_, '_, H, P, Disposed> { /// per-CPU scheduler record reading as a value no operation on it can /// produce is inside what that yields. fn answer_steal_requests(&mut self) { - if !self.env.steal { + if !self.env.balance.pulls() { self.cpu.steal_requests.clear(); return; } @@ -1712,9 +1887,22 @@ impl SchedPass<'_, '_, H, P, Disposed> { } fn apply_timer(&mut self) -> TimerApplied { + self.apply_timer_no_later_than(None) + } + + /// [`Self::apply_timer`], with a wake this CPU wants for a reason the + /// deadline machinery knows nothing about — [`Balance::PullWithRearm`]'s + /// re-arm, and nothing else. + /// + /// It can only move the arming **earlier**, which is why it cannot break + /// invariant T: the invariant says the armed instant is no later than the + /// earliest event this CPU owes ([`crate::invariants::check_timer`]), and an + /// extra wake before that instant is a spurious pass, not a missed + /// deadline. + fn apply_timer_no_later_than(&mut self, extra: Option) -> TimerApplied { let deadline = self.cpu.earliest_deadline(); let quantum = self.cpu.running.as_ref().map(|_| self.cpu.quantum_end); - let plan = TimerPlan::compute(quantum, deadline); + let plan = TimerPlan::compute(quantum, deadline).no_later_than(extra); match plan { TimerPlan::Arm(at) => self.env.hw.set_timer(at), TimerPlan::Stop => self.env.hw.stop_timer(), @@ -1778,8 +1966,21 @@ impl SchedPass<'_, '_, H, P, Disposed> { /// arrived in between — stay awake and decide again. fn try_sleep(&mut self) -> Result::Payload>, ()> { self.post_steal_probe(); - let timer = self.apply_timer(); + let rearm = self.rearm_deadline(); + let timer = self.apply_timer_no_later_than(rearm); let arm: SleepArm<'_> = self.env.cpus.get(self.cpu.id).doorbell().arm_sleep(); + // [`Balance::PushOnSurplus`]'s other half, and the half that makes the + // push an ordering rather than a hope: SLEEPING is published above, so + // reading the surplus *here*, behind the fence, is the load that pairs + // with the pusher's read of that bit. `post_steal_probe`'s own read is + // before the store and answers nothing about it. + if self.probe_still_owed() { + arm.abandon(); + self.env.cpus.get(self.cpu.id).doorbell().begin_pass(); + self.cpu.drain(self.env, self.now); + self.cpu.fire_deadlines(self.env, self.now); + return Err(()); + } match arm.confirm(&self.cpu.mailbox) { Ok(quiesced) => Ok(Action::Idle(SleepToken::new(quiesced, timer))), Err(_awake) => { @@ -1791,31 +1992,51 @@ impl SchedPass<'_, '_, H, P, Disposed> { } } + /// When [`Balance::PullWithRearm`] wants this CPU back, and the allowance it + /// spends to ask. + /// + /// `None` under every other policy and once the allowance is gone, which is + /// what makes the tick stop: a machine with nothing to run halts for good + /// after `times` probes rather than waking for ever. + fn rearm_deadline(&mut self) -> Option { + let (every_ns, times) = self.env.balance.rearm()?; + if self.cpu.idle_probes_spent >= times { + return None; + } + self.cpu.idle_probes_spent += 1; + Some(self.now.after(every_ns)) + } + + /// Did a surplus appear after this CPU published SLEEPING? + /// + /// Only [`Balance::PushOnSurplus`] asks, and only to close its own Dekker + /// window (see [`balance_fence`]): if the pusher missed our SLEEPING bit, we + /// must not also miss its surplus. Answering `true` sends the pass round + /// again, and the second trip's `post_steal_probe` reads the surplus behind + /// the same fence and posts — which is why this terminates: a probe in + /// flight is an answer, so the question is asked at most once per halt. + fn probe_still_owed(&self) -> bool { + if self.env.balance.push_threshold().is_none() { + return false; + } + if self.cpu.probe_outstanding() { + return false; + } + balance_fence(); + self.best_victim().is_some() + } + /// One probe at a time (spec §7.7): if the previous one is still in /// flight the claim fails and we simply do not post another — the /// outstanding probe will be answered, and this CPU sleeps with its /// doorbell armed. fn post_steal_probe(&mut self) { - if !self.env.steal { + if !self.env.balance.pulls() { return; } - // **Chosen by surplus and not by load**, which are two numbers because - // they answer two questions — see [`CpuHandle::surplus`]. The guard is - // the same inequality [`SchedPass::answer_steal_requests`] enforces - // (`fair_len() > 1`), read one CPU away: a victim that would refuse the - // probe is not probed, and the probe is spent on a CPU that can answer - // it. - let Some((victim, _)) = (0..self.env.cpus.len()) - .map(|i| CpuId(i as u32)) - .filter(|&cpu| cpu != self.cpu.id) - .map(|cpu| (cpu, self.env.cpus.get(cpu).surplus())) - .max_by_key(|&(_, surplus)| surplus) - else { + let Some(victim) = self.best_victim() else { return; }; - if self.env.cpus.get(victim).surplus() < 2 { - return; - } let Some(slot) = self.cpu.steal_probe.claim() else { return; }; @@ -1830,6 +2051,66 @@ impl SchedPass<'_, '_, H, P, Disposed> { self.env.hw.kick(victim); } } + + /// The CPU a probe would be spent on, or `None` if none is worth probing. + /// + /// **Chosen by surplus and not by load**, which are two numbers because they + /// answer two questions — see [`CpuHandle::surplus`]. The guard is the same + /// inequality [`SchedPass::answer_steal_requests`] enforces + /// (`fair_len() > 1`), read one CPU away: a victim that would refuse the + /// probe is not probed, and the probe is spent on a CPU that can answer it. + /// + /// Factored out of [`SchedPass::post_steal_probe`] because + /// [`SchedPass::probe_still_owed`] must ask the *same* question after + /// publishing SLEEPING; two spellings of one inequality is how a push that + /// wakes a CPU its victim would refuse gets written. + fn best_victim(&self) -> Option { + let (victim, surplus) = (0..self.env.cpus.len()) + .map(|i| CpuId(i as u32)) + .filter(|&cpu| cpu != self.cpu.id) + .map(|cpu| (cpu, self.env.cpus.get(cpu).surplus())) + .max_by_key(|&(_, surplus)| surplus)?; + (surplus >= 2).then_some(victim) + } + + /// [`Balance::PushOnSurplus`]: this pass has just published `surplus`, so + /// tell one sleeping CPU that there is something to come and get. + /// + /// **One CPU per pass, and a doorbell ring rather than a message.** The ring + /// is what [`crate::mailbox::Doorbell`] already does for every producer, and + /// its edge-coalescing does the rest: only the 0→1 kick edge on a target + /// that reads SLEEPING costs an IPI, so a target that already has one coming + /// is not kicked twice. The woken CPU finds an empty mailbox, runs its idle + /// pass again and posts an ordinary probe — the push adds no second way to + /// move a task, only a way to make the pull path run. + /// + /// The fence is [`balance_fence`]'s, and it is the price: an `mfence` on the + /// exit of every pass that has surplus, which under [`Balance::Pull`] costs + /// nothing because this returns before reaching it. + fn push_on_surplus(&mut self, surplus: u32) { + let Some(threshold) = self.env.balance.push_threshold() else { + return; + }; + if surplus < threshold { + return; + } + balance_fence(); + let n = self.env.cpus.len(); + let me = self.cpu.id.0 as usize; + let base = self.cpu.push_cursor as usize; + let Some(target) = (0..n) + .map(|offset| (base + offset) % n) + .filter(|&cpu| cpu != me) + .map(|cpu| CpuId(cpu as u32)) + .find(|&cpu| self.env.cpus.get(cpu).doorbell().sleeping()) + else { + return; + }; + self.cpu.push_cursor = (target.0 + 1) % n as u32; + if self.env.cpus.get(target).poke() == Kick::Send { + self.env.hw.kick(target); + } + } } /// The globally shared, `Sync` face of a CPU, and the whole remote surface: @@ -1927,6 +2208,23 @@ impl CpuHandle { self.doorbell.ring(urgency) } + /// Ring the doorbell with **no message behind it** — the whole of + /// [`Balance::PushOnSurplus`]'s effect on the target. + /// + /// A wake with nothing queued is exactly what the doorbell already tolerates + /// on the consumer side: `begin_pass` clears the edge, the drain finds an + /// empty mailbox, and the pass decides again from scratch. What the target + /// gains is that its idle disposition runs a second time, with the pusher's + /// surplus now visible to it. + /// + /// The returned [`Kick`] is the caller's obligation exactly as + /// [`Self::post`]'s is. `Urgency::Normal`, so the coalescing rule of §7.3 + /// applies: a target that already has an IPI coming costs nothing, and a + /// busy target costs nothing at all. + pub fn poke(&self) -> Kick { + self.doorbell.ring(Urgency::Normal) + } + /// Post a message that carries its own node — the ownership-transferring /// `Adopt` (spec §7.2). pub fn post_owned( @@ -2087,9 +2385,10 @@ mod tests { frontier: Frontier, preempt: NoPreempt, next_key: u64, - /// `Env::steal`. Off by default because most tests want a CPU's queue - /// to stay where they put it; the probe's own tests turn it on. - steal: bool, + /// `Env::balance`. [`Balance::None`] by default because most tests want + /// a CPU's queue to stay where they put it; the probe's own tests turn + /// the pull half on. + balance: Balance, } impl World { @@ -2108,7 +2407,7 @@ mod tests { frontier: Frontier::new(), preempt: NoPreempt, next_key: 1, - steal: false, + balance: Balance::None, } } @@ -2128,7 +2427,7 @@ mod tests { cpus: &self.handles, frontier: &self.frontier, preempt: &self.preempt, - steal: self.steal, + balance: self.balance, }, ) } @@ -3029,7 +3328,7 @@ mod tests { fn a_steal_probe_goes_to_surplus_and_not_to_a_cpu_full_of_corpses() { const C2: CpuId = CpuId(2); let mut w = World::new(3); - w.steal = true; + w.balance = Balance::Pull; // cpu1: one live task running, three corpses queued behind it. Three // units of work, none of them stealable. @@ -3112,7 +3411,7 @@ mod tests { #[test] fn a_cpu_does_not_hand_over_the_context_it_is_still_standing_on() { let mut w = World::new(2); - w.steal = true; + w.balance = Balance::Pull; let tasks: Vec<_> = (0..3).map(|_| w.spawn(C1)).collect(); w.run_a_pass(C1); diff --git a/toyos-sched/src/sync.rs b/toyos-sched/src/sync.rs index f9e649de0cc..f144a9f4491 100644 --- a/toyos-sched/src/sync.rs +++ b/toyos-sched/src/sync.rs @@ -11,10 +11,10 @@ #[cfg(not(feature = "loom"))] pub use alloc::sync::Arc; #[cfg(not(feature = "loom"))] -pub use core::sync::atomic::{AtomicBool, AtomicPtr, AtomicU32, AtomicU64, Ordering}; +pub use core::sync::atomic::{fence, AtomicBool, AtomicPtr, AtomicU32, AtomicU64, Ordering}; #[cfg(feature = "loom")] -pub use loom::sync::atomic::{AtomicBool, AtomicPtr, AtomicU32, AtomicU64, Ordering}; +pub use loom::sync::atomic::{fence, AtomicBool, AtomicPtr, AtomicU32, AtomicU64, Ordering}; #[cfg(feature = "loom")] pub use loom::sync::Arc; diff --git a/toyos-sched/src/timer.rs b/toyos-sched/src/timer.rs index 83f62e2ac24..5eea3669c76 100644 --- a/toyos-sched/src/timer.rs +++ b/toyos-sched/src/timer.rs @@ -36,6 +36,23 @@ impl TimerPlan { } } + /// Pull the arming forward to `at`, if a caller has a reason to be woken + /// that no parked deadline and no quantum accounts for — today + /// [`crate::cpu::Balance::PullWithRearm`]'s re-probe, and nothing else. + /// + /// One direction only, and that is what keeps invariant T out of it: the + /// invariant bounds the armed instant from *above* by the earliest event the + /// CPU owes, so a plan that only ever moves earlier cannot violate it. A + /// method that could move the arming later would be a missed deadline + /// waiting for a caller. + pub fn no_later_than(self, at: Option) -> Self { + match (self, at) { + (_, None) => self, + (TimerPlan::Stop, Some(at)) => TimerPlan::Arm(at), + (TimerPlan::Arm(planned), Some(at)) => TimerPlan::Arm(planned.min(at)), + } + } + pub fn armed(&self) -> Option { match self { TimerPlan::Arm(at) => Some(*at), From bfd69d068d7126f7fd032de94dde69deeadb0046 Mon Sep 17 00:00:00 2001 From: japabu Date: Sat, 22 Aug 2026 14:39:46 +0200 Subject: [PATCH 2/2] timer.rs: the header says an arming need not be a deadline `TimerPlan::no_later_than` arrived with the re-arm and the module header did not: a reader of that header would take every arming to be a parked task's deadline or a quantum, and one of them now is not. The one-directional rule is the part that has to be in the header rather than only at the method, because it is what keeps invariant T out of the new arming's business. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01Dpa781ehanBhUgrycmXcPW --- toyos-sched/src/timer.rs | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/toyos-sched/src/timer.rs b/toyos-sched/src/timer.rs index 5eea3669c76..78e15998335 100644 --- a/toyos-sched/src/timer.rs +++ b/toyos-sched/src/timer.rs @@ -10,6 +10,14 @@ //! have to remember to retire both. It did not — `fire_deadlines` discarded the //! index entry of a claim it lost and left the entry's copy standing, which is //! a CPU reporting a deadline nothing will fire. +//! +//! **Not every arming is a deadline.** [`TimerPlan::no_later_than`] lets a caller +//! pull the arming forward for a reason nothing in this module knows about — a +//! CPU that wants waking to look for work again. It moves the instant in one +//! direction only, which is what keeps invariant T out of it: the invariant +//! bounds the armed instant from above by the earliest event the CPU owes, so +//! an extra wake before that instant is a spurious pass and never a missed +//! deadline. use crate::hw::Nanos;