Repository navigation
Description
Implementation Milestones
Implement unchecked items in dependency order. Checked items preserve history; later corrections supersede older wording. The desired concept is specified in documentation/desired-concept/; this issue owns the project plan. Research inventories and rejected/deferred ideas are outside implementation scope.
A milestone includes its specified public/direct/runner/CLI behavior, types/exports, identity/records, failure/cancellation/cleanup paths, documentation, and acceptance checks. Platform-dependent features require a validated supported path and explicit compatibility rejection elsewhere; a facade or universal unsupported result does not complete a feature.
-
Monorepo Foundation
- Workspace Layout Implement workspace layout foundation #195
- Dependencies And Tooling: update dependencies, scripts, TS config, lint/test/build wiring, and CI for the new shared-source layout
-
First Usable Engine
- Public
@overkill-dev/engineEntry Point: one supported import surface for defining suites/tests and executing them without repo-internal imports - Direct Engine Execution Flow: create suites/tests, execute them through the public API, and get a result back
- Structured Outcomes And Run Result: stable
Pass/Fail/Skip/Inconclusiveprotocol plus run summary/result shapes - Stable Identity And Run Counts: engine-owned test/case identity, discovered vs executed counts, and orphan detection
- Reporter Event Stream Contract: engine-owned structured event surface for reporters and machine consumers
- Engine Self-Tests Through Public API: rewrite engine tests so they exercise the public engine entry point, not internal source paths
- Public
-
First Human Output
- Reporter Lifecycle And Sink Semantics: real-time vs final-result reporters plus sink declarations as required by the concept
- Define
RealTimeReporterandFinalResultReporteras separate lifecycles - Require declared reporter sinks and validate direct-engine conflicts before
run-start - Document lifecycle and sink behavior on the engine package surface
- Define
-
@overkill-dev/reporter-line: first standalone human reporter package exposed through its own facade - Human Failure Rendering: useful terminal rendering for pass/fail/summary output
-
@overkill-dev/reporter-dot: minimal progress reporter for compact local/CI output
- Reporter Lifecycle And Sink Semantics: real-time vs final-result reporters plus sink declarations as required by the concept
-
Structured Assertions
- Engine-Owned Assertion Protocol: keep assertion semantics, injected
case.assert/case.require, and assertion counting in@overkill-dev/engine - Built-In Assertion Catalog: first useful built-in assertion set
-
assert/requireSemantics: define non-gating vs gating assertion behavior -
done(),plan(n), And Zero-Assertion Detection: explicit assertion-count contract and default no-assertions failure - Custom Assertion Protocol: decide the public shape for custom nodes, composite assertion boundaries, and foreign assertion bridges
- Assertion Source Locations: capture call-site locations for built-in, custom, composite, and foreign assertion failures
- Shared Source Location Relativization: centralize project-root path rendering so every reporter benefits from one path policy
- Adapt reporters to display source locations
- Helper And Macro Location Forwarding: provide an explicit API for wrapped assertions to report the caller location
- Failed Check And Diff Model: structured mismatch payloads and diff shapes
- Bounded Assertion Serialization: cap depth, node count, entry count, string bytes, and serialized bytes before reporter delivery
- Error Matcher Contract: explicit matcher shape for
throws/rejects -
@overkill-dev/assert: reusable assertion-extension helpers such asdefineCompositeAssertion(...), foreign-assertion bridges, and packaged assertion adapters
- Engine-Owned Assertion Protocol: keep assertion semantics, injected
-
Runtime Assertion Integrity
- Durable Contract Failures: persist assertion-contract violations before throwing so catches, error assertions, and foreign bridges cannot erase them or turn engine control signals into expected application failures.
- Assertion Context Lifetime: reject recording after body settlement or interruption, preserve originating attempt/trial identity, and deliver durable late-activity diagnostics without changing published verdicts.
- Assertion-Time Operand Meaning: preserve injected builder checks and diagnostics against later operand mutation while retaining the public low-level lazy protocol contract.
- Returned Builder Assertion Ownership: reject nodes from other cases, retries, or trials and prevent repeated recorded node identities from inflating assertion counts or satisfying
plan(n). - Assertion Protocol Validation: validate node/child structure and reject malformed values or structural cycles as durable contract failures while allowing supported cyclic application operands.
- Nested Assertion Context Ownership: keep property-trial checks and custom assertion diagnostics in their injected contexts, including supported asynchronous composition.
- Test Entry Export Validation: reject runtime exports besides the selected engine's
testNodeduring discovery; preserve support-module exports and erased type-only exports.
-
Doubles
-
@overkill-dev/doubles: first doubles package facade -
testDouble(): primary public double creation API - Rule-Based Behaviors: configurable responses and behavior rules
- Typed Pattern Matchers aka integrate
ts-pattern - Call History Introspection: inspect calls/results directly
- Doubles Assertions: doubles-oriented assertion extensions plugged into the engine-owned assertion context
-
-
First Dogfooding And Publishing
- Publish
@overkill-dev/engine: first real package publication from the new layout once the engine flow is usable - Publish
@overkill-dev/reporter-line: first reporter publication once it works end to end - Publish
@overkill-dev/assert: publish reusable assertion-extension helpers once the extension model is usable - Publish
@overkill-dev/doubles: publish the doubles package once the core API is usable - Dogfood Through Engine API: use the published engine, assertion-helper, doubles, and reporter packages in Overkill’s own tests/tools directly
- Remove Internal-Only Test Wiring: stop depending on repo-internal execution paths where the published engine API is sufficient
- Publish
-
Minimal Runner
-
@overkill-dev/run: first orchestration package facade -
RunRequest,ResolvedRun, AndRunFactsAPI: typed request, resolution, and serializable run facts surface mirrored by the CLI - Reporter Completion Errors: decide how
run-end,onFinish, and final-result reporter failures are surfaced afterRunResultis created - Command-Line Business Logic API: parser, command intent model, config loading flow, command dispatch, fallback error reporting, and exit-code mapping are owned by
@overkill-dev/run - Command-Selected Module Loading: command modules load only for the selected command path so ordinary microtest startup does not load benchmark, baseline writer, coverage, or reporter-heavy modules
- Named Profile Model: support choosing the runner profile used by a run request, including the existing
microtestprofile - Reporter Sink Exit Codes: map configuration failures such as sink conflicts to stable CLI exit codes
- Managed Output Sink Coordination: validate raw vs managed
stdout/stderrreporter sinks, render managed line intents through the default plain renderer, and add@overkill-dev/reporter-briefplus@overkill-dev/output-renderer-github-actions - Resource Budget Resolution: carry Node-first heap, RSS, resident-growth, and active-resource budgets through grouped
resourceUsagepolicy in config,RunRequest, andRunFacts - Supervised Resource Exhaustion Attribution: enforce budgeted microtest profiles through
processModel: 'supervised-process', stream active test identity, kill wedged children on hard timeout or budget breach, and recordresource-exhaustedwith the active case - Run A Specific File Or Module: one complete public run flow for explicit inputs
- Internal Test Export Cleanup: migrate Overkill test modules from exported
testSuiteto the publictestNodeconvention - Full Config Loading: finish config discovery, import, schema validation, and any final decision on default vs named exports for
defineConfig - Runner Profile Naming Model: support project-owned runner profile names such as
backend-httporui-browserwithout deriving behavior from the name - Test Family Discriminator: require
profiles.<name>.testFamilyand use it as the discriminant for profile config - Microtest Profile Validation: validate the initial
testFamily: 'microtest'profile surface and reject fields that are not valid for microtests - Microtest Execution Model: model
processModelasin-processorsupervised-processfor microtests - Microtest Scheduling Model: model
concurrentandserialscheduling for microtests - Microtest Capability Restrictions: implement capability restrictions for
microtestprofiles only in the minimal runner; property profiles adopt the same restricted boundary later, while integration, browser, benchmark, and type-test profiles do not use this model - Cross-Process Custom Engine Boundary: design custom engine execution for supervised and other multi-process runs without parent-side user-module execution
- Profile Reporter Resolution: support profile reporter overrides with global reporter fallback
- CLI Fallback Diagnostics: return runner errors not sent to terminal-capable reporters, and map completed runs to the documented non-zero CLI exit status
- CLI Run Command Surface: expose the
@overkill-dev/runcommand implementation consumed by the@overkill-dev/testbinary wrapper, keepbin: overkillout of@overkill-dev/run, write returnedfallbackDiagnosticstostderr, apply the returned exit code, and parse textual run flags including--measure-resource-usageand--resource-budget <name=value>foractiveResourceCount,javaScriptEngineHeapBytes,residentSetBytes, andresidentSetGrowthBytesPerSecond
-
-
Runner Error Capture
- Execution-Mode Global Error Hooks: install run-scoped
unhandledRejectionanduncaughtExceptionobservers in every execution boundary, including in-process runs, supervised children, host processes, and worker-pool workers, correlate failures with the active test viaAsyncLocalStorage, and surface them as runner errors instead of raw process output - Permission Denial Runner Errors: classify
ERR_ACCESS_DENIEDthrows and Node permission diagnostics aspermissionrunner errors in every execution mode, deliver them through reporter events and CLI fallback diagnostics, and preserve useful fields such as permission kind and resource path
- Execution-Mode Global Error Hooks: install run-scoped
-
Runner Timing Diagnostics
- Run Timing Concept: document summary vs precise timing, timing ownership, debug/benchmark boundaries, and OpenTelemetry export stance
- Run Timing Data Model: add stable summary timings, precise spans, truncation, aggregate totals, and timing overhead
- Timing Request Surface: support
--timings,timings.collection, programmatic request overrides, and strategy-required precise collection - Default Reporter Timing Output: show compact timing summaries in line, brief, and dot reporters
- Detailed Runner Phase Timings: record config loading, file discovery, collection, coarse import windows, propagation, filtering, runtime expansion, sharding, ordering, resource lowering, placement, dynamic dispatch, duration-history placement, hedging, reporters, and cleanup
- Execution Boundary Timings: record host startup, supervised process spawn/ready/teardown, worker-pool start/shutdown, and individual worker create/ready/import/teardown timings
- Resource Lifecycle Timings: record per-resource acquire/dispose timings by name, scope, process/worker, status, and aggregate
- Timing Reporter Surface: expose top overhead offenders in human output and full timing data to machine-readable reporters
- OpenTelemetry Reporter: plan optional
@overkill-dev/reporter-opentelemetryas an exporter from Overkill timing data
-
Discovery, Test Data, And Selection
- Explicit Module Listing: list explicit test files/modules through the runner
- Profile File Discovery: implement
files.includeplusfiles.excludewith Node glob's separate exclude support; do not support negated include globs as a second exclusion mechanism - Named Profile File Sets: support
files.setsas non-overlapping named discovery buckets with unique names, shared glob validation, empty-set handling, and explicit-file validation - Test Data Model And Propagation: split authored annotations from execution controls, move family to engine-owned facts, keep artifacts as observed results, and resolve root/suite/table/case propagation
- Programmatic Selection API: typed filter helpers for IDEs, MCP servers, and code-based integrations
- CLI Filter Grammar: stable selection/filter syntax for the runner
- Selection And Filtering: select tests by file, title, id, tag, ownership, params, and suite path
- Ordering And Seeding: deterministic ordering plus seeded variation
- Sharding: stable identity-based partitioning of the filtered run set
-
Resources And Runtime Composition
-
@overkill-dev/resources: typed runtime/resource composition package - Typed Context Composition: compose runtimes/resources into test context explicitly, including dependency context for resources that acquire other resource handles
- Lifecycle Scopes And Teardown: setup/teardown, dependency ordering, and sharing/isolation rules
- Microtest Resource Boundary: enforce that resources and runtimes cannot attach to microtest profiles or microtest-authored tests
- Runner-Aware Runtime Descriptor Attachment: make
withRuntime(runtimeGraph, body),withResource(resource, body), andwithResources(resources, body)attach descriptors that collection can read before scheduling - Descriptor-Based Microtest Resource Rejection: replace the current boolean attachment marker with collected resource descriptor metadata so microtest profiles reject resources regardless of which wrapper or facade attached them
- Plural Scope Names: migrate injected runtime handles from
scope.runtimetoscope.runtimesand expose direct resource handles throughscope.resources - Direct Resource Binding: implement
withResource(...)andwithResources(...)as runner-aware direct resource wrappers - Composable Body Wrappers: normalize nested runtime and resource wrappers before planning, and reject duplicate public runtime or resource keys
- Resource Lifecycle Scopes: support per-run, per-file, per-suite, per-case, and shared-per-worker lifetimes
- Runtime Matrices: implement
defineRuntimeMatrix(...)with stable variant ids, shape checks, and planning-time expansion - Runtime Composition: implement
composeRuntimes(...)with runtime-name collision rejection and Cartesian matrix expansion - Local Service Resources: model deterministic app servers, registries, browser test servers, and other local services as explicit resources
- Simulation Resource Integration: model simulations through resource descriptors;
@overkill-dev/simulationdefines generic simulations, finite scenario catalogs, and simulated HTTP servers, while@overkill-dev/resourcesexports the resource adapter - Scenario Binding Propagation: forward planning-visible scenario bindings to owning resources through nested runtime graphs
- Scenario Cache Identity: include acquire-time scenario bindings in resource cache keys while request-routed scenarios reuse existing acquired handles
- Runtime Scenarios: support named scenario catalogs that affect runtime identity, artifacts, filtering, and replay metadata when declared as runner-visible descriptors
- Runtime Selection Filters: add runtime, variant, and dimension filters plus repeated
--runtimeCLI sugar - Interaction Transcripts: support generic transcript recording for calls, HTTP, browser requests, subscriptions, and custom protocols
- Runtime Attachments: allow fixtures and runtimes to attach structured artifacts such as screenshots, accessibility JSON, service logs, and transcripts
- Execution Requirements Contribution: collected runtimes/resources lower requirements into placement constraints without owning the runner
-
-
Default Test Authoring
-
@overkill-dev/testRoot Authoring Facade: add the high-level authoring surface on top of the existing binary package -
@overkill-dev/testBinary Wrapper: ship the only first-partybin: overkilland delegate command behavior to@overkill-dev/run - Output-Capture CLI Flags: decide and implement the minimal textual surface for capture mode, including whether
--no-captureshould exist or remain config-only - File-Level Test Data Authoring: use root authoring semantics for module-wide annotations and controls
- Standard Distribution Subpaths: expose
config,reporters,bench,assert,resources, andbaselinesthrough explicit subpaths while preserving leaf package ownership -
test(...)Andsuite(...): implement the first default authoring API on top of engine and export it from@overkill-dev/test -
table(...): implement the first parameterized authoring surface and export it from@overkill-dev/test -
defineMacro(...): implement the first preferred reuse mechanism for tests-as-values and export it from@overkill-dev/test -
createTestFacade(...): initial authoring-surface composition; no longer the desired registration or profile-selection mechanism - Profile-Neutral Root Authoring: keep
test(...),suite(...), and related helpers usable from@overkill-dev/testwhile the selected profile owns family policy - Facade Runtime Binding: update
createTestFacade(...)to bind one runtime graph, direct resources, andmapScope(...)without selecting the test family - Facade Scope Mapping Guardrails: prevent
mapScope(...)from overwriting core scope keys in types and at runtime -
runIfMain(...): supported direct-file execution companion for exported specs - already exists partially in engine, we should consider moving it to @overkill-dev/test package so we can also rely on default reporters and so on. But we probably also need to decide which testFamily and which profile is picked when using runIfMain - Lightweight Doubles Root Exports: expose lightweight doubles APIs and doubles assertion references from
@overkill-dev/test - Root Import Boundary: after root authoring APIs exist, verify the root export does not expose or load config, reporters, resources, benchmark, baseline, coverage, or command implementation APIs
-
throwingTest: decide whether throwable test authoring belongs in the root facade or a dedicated compatibility export, then implement the chosen surface -
skippedTest: implement the planned skipped-case helper after the concept settles; it should record a reasoned planned skip instead of mutating source selection -
defineHarness(...): implement the authoring helper after the root facade and coretest(...)/suite(...)APIs exist - Interaction Transcript Helpers: implement
recordCalls(...),recordSink(...), andrecordEvents(...)once transcript semantics are ready - Async-Control Helpers: implement
scope.microtasks(),scope.immediate(),scope.flushAsync(), andscope.inFlight(...)as one async-control authoring milestone - Stable Domain Aliases: project-local re-export pattern for domain helpers such as runtimes, resources, macros, and assertion references, not for profile selection
- Standard Distribution Documentation: normal install path is
npm install -D @overkill-dev/test
-
-
Coverage
- Microtest-Only Coverage Model: explicit, off-by-default coverage restricted to microtest profiles and activated per run through
--coverageorRunRequest.coverage - V8 Coverage Integration: runner-owned native coverage flow and artifact emission
- Coverage Config Surface: profile-scoped formats, include/exclude all-files policy, thresholds, and outputDir in microtest config
- Coverage Profile Validation: reject coverage config on non-microtest profiles
- Coverage Run Record Integration: persist the coverage request, resolved policy, output paths, and artifact references without changing the selected process model or scheduling
- Microtest-Only Coverage Model: explicit, off-by-default coverage restricted to microtest profiles and activated per run through
-
Replay, Records, Baselines, And Debug
- Run Record Persistence Policy: profile
runRecords.persist,--record, workflow-required persistence, start/final records, and persistence runner errors - Compact History Index: persisted run summaries, case outcome history,
WorkIdduration history, retention windows, and schema repair - History CLI:
overkill history list,prune,compact, andclear - Replay Picker:
overkill replaywithout an id opens a TTY picker over replayable records and errors outside TTY - Record Retention And Maintenance: bounded automatic maintenance after persisted runs plus explicit prune, compact, and clear commands
- Last-Failed Selection: resolve
--last-failedfrom compact history and persist the selected run - Witness File Format: stable witness schema for replayable failures
- Replay Commands: replay a recorded run or witness through public surfaces
-
@overkill-dev/baselines: shared baseline model and package facade - Baseline Verbs: update/apply/bootstrap/diff/list workflows owned by
@overkill-dev/run - Snapshot Workflow: checked-in baseline compare/update flow
- Failure Artifacts: structured logs, traces, snapshots, and artifact attribution rules
- Debug Mode: richer diagnostics and retained runtime state for investigation
- Run Record Persistence Policy: profile
-
Executable Test Families
- implement the public contracts in #698 and
documentation/desired-concept/architecture/test-families.mdin the order below, validating each step before the next. - Family Definitions And Identity: add
defineTestFamily, typed settings validation, lazy backends, features, and explicit module references; replace the closed family union with executable definitions and stable IDs, and add generic family variants to work, work-unit, and artifact identity while preserving neutral identity keys. - Engine Preparation And Evidence: implement
engine.prepareFamily, exactprepared.collect, branded collected runs, andengine.execute(collected, options); keep runtime bodies in the ordinary executor and route analysis events through the engine-owned sink, result validation, and reporter lifecycle. - Lifetime And Failure Contract: enforce single-use preparation/selection/execution, engine ownership, cancellation, incomplete-evidence errors, and exactly-once disposal before final delivery; retain cleanup failures and reject foreign, consumed, or disposed values.
- Typed Profiles And Built-In Migration: infer each imported family's settings independently, retain supplied built-in aliases and existing defaults/policy, reject bare optional-family strings, and route direct callers and runner profiles through the same engine path.
- Worker Resolution And Record Compatibility: resolve definitions inside explicit configuration/bootstrap boundaries, validate frozen identity/version/settings, reject unsupported placement for local-only definitions, and preserve historical inspection and versioned readers without loading family code.
- Family Contract Acceptance: cover heterogeneous profiles, invalid settings, empty/skipped selections, variant isolation, selected result order, bad or missing events, reporter failures, cancellation, abandoned preparations, and supervised runtime regressions; preserve property-derived policy/replay, benchmark sampling/budgets, and Stryker frozen policy, selection, coverage, and precedence during migration.
- implement the public contracts in #698 and
-
Shared Contract Extensions
- Prerequisites: Resources And Runtime Composition, assertion/result contracts, and Executable Test Families.
- Portable Implementation References: resolve body/resource/observer modules with validated inputs, realm/build/source-map identity, and no closure/handle serialization.
- Resource Check Lifecycle: implement
withResourceChecks, durable ledgers, partial-install cleanup, admission, close-time draining, bounded supervision, and exactly-once finalization/disposal. - Resource Check Outcomes And Evidence: retain owner/attempt/lifetime attribution, live/final evidence, standalone errors, and finalization before retry/verdict without changing body plans.
- Runtime-Aware Measurement Lifecycle: extend common collectors with async readiness/finalization, sync-path preservation, partial-start cleanup, target clocks/cooldown/GC, unavailable readings, and invocation identity.
- Executable Identity And Catalog Acceptance: carry
WorkIdthrough reporter/evidence events and records; validate variants and stale/duplicate completion without counting recollection as discovery. - Simulation Witness Replay Acceptance: validate owner/producer/payload compatibility, signed nullable seeds, restore-before-projection, failure disposal, and current HTTP-handler scenario replay.
- Explicit Simulation Identity: request-routed fixtures reject missing identity, including default; fixed-scenario servers bind identity at startup.
- Simulation HTML Host: offer safe HTML/config assembly; consumers own markup, assets, builds, and scenario-to-app mapping.
-
Sharded Result Merging
- Brief Final-Result Reporter: expose a final-result factory for compact merged summaries
- Shard Run Record Persistence: persist completed shard
RunRecordfiles whenmerge-resultsneeds durable inputs, without updating aggregate compact history before merge - Merge Results API And CLI: implement
mergeResults(...)andoverkill merge-results <paths...>over record files and directories - Merge Compatibility Validation: reject mismatched profiles, seeds, shard totals, selected identities, loader config, and Overkill package versions
- Incomplete Merge Results: emit a merged failure record for missing shards, duplicate work identities, unreadable records, or incompatible inputs
- Merged Run Record: write non-replayable
kind: 'merged'records with shard lineage and final-result reporter delivery
-
Additional Reporters
-
@overkill-dev/reporter-tap: TAP-compatible real-time output package, including source-location output for assertion failures -
@overkill-dev/reporter-json: canonical machine-readable final-result package -
@overkill-dev/reporter-html: generic artifact/failure HTML report package
-
-
Benchmarking
-
@overkill-dev/bench: first benchmark package facade over ordinary test-node authoring - Standard Distribution Integration: include
@overkill-dev/benchin the@overkill-dev/teststandard dependency set while keeping benchmark measurement outside the root authoring import and ordinary microtest startup path - Benchmark Command Namespace: implement
overkill bench runandoverkill bench list - Benchmark Config Namespace: load
benchmark.profiles.<name>from the project config without mixing benchmarks into runnerprofiles - Shared Benchmark Profile Registry: replace historical
benchmark.profileswithprofiles.<name>andtestFamily: 'benchmark'; preserveoverkill benchrouting, inference, direct-file behavior, and validation. - Benchmark Profile Selection: select benchmark profiles from benchmark commands without supporting
overkill run --profile benchmark - Benchmark Runner Integration: implement benchmark commands through regular runner planning, placement, execution, result, and reporter flow
- Benchmark Baseline Namespace: implement
overkill bench baseline update/apply/bootstrap/diff/listfor performance baselines - Workload Model: define benchmark inputs/workloads explicitly
- Benchmark Resource And Runtime Integration: reuse ordinary resource/runtime descriptors for benchmark fixtures, services, browsers, registries, temporary directories, and PTY needs
- No Benchmark Hook Model: keep benchmark lifecycle out of
setup,teardown,beforeEach, andafterEach, using resources, runtimes, cleanup, and cooldown instead - Measurement Strategies And Metric Collectors: collect wall-clock, CPU/runtime, memory/resource, process, and custom metrics through explicit measurement surfaces
- Benchmark HTML Reporter: implement
@overkill-dev/reporter-benchmark-htmlwith workload/metric tables, distributions, uncertainty, and budget/baseline comparisons - Host Calibration And Ambient Noise: record host capacity, load, memory pressure, runtime metadata, and available platform noise signals before measured samples
- Adaptive Benchmark Placement: infer safe concurrency from measurement strategy plus calibration, and reduce to serial or fewer lanes when needed
- Warmup, Cooldown, And Sample Cleanup: separate preparation, measured samples, benchmark-owned cleanup, and settling phases
- Explicit GC Policy: expose GC only in supervised or hosted benchmark processes and call it only when the benchmark or profile requests it
-
hostProcessExecution Shape: support an internal host process around worker-pool execution for Node/V8 flags, profiling, debugging, and benchmark isolation needs - Benchmark Budget Surface: express checked-in
budgetsanddiagnosticMetrics, generate ordinary assertion failures after aggregation, and reject missing or undeclared metric coverage - Calibration And Normalization: normalize benchmark results for comparability
- Process And PTY Benchmarks: external process and terminal-oriented benchmark support
- Benchmark Strategy Contract Acceptance: cover cold-module strategies, sample timelines, calibration commands, raw/normalized identity, cleanup, and generated budgets through public/direct and runner paths.
- Dogfooded Startup Guardrails: after benchmark commands are usable, add Overkill benchmark checks for root
@overkill-dev/testimport cost and ordinary microtest CLI startup
-
-
Integration Test Family
- Integration Profile Model: support
testFamily: 'integration'as a later runner profile family - Direct Integration
runIfMain()Execution: decide whether direct-file execution can support integration profiles without losing supervised isolation - Integration Runtime Environment: define temp directories, local services, process use, network use, and failure artifacts without applying microtest capability restrictions
- Integration Resource Defaults: model app servers, registries, HTTP/IPC workflows, and richer fixtures as explicit resources
- Integration Retry Policy: allow integration profiles to opt into retries with attribution-preserving artifacts
- Integration Failure Artifacts: support transcripts, logs, service output, and scenario witnesses for integration resources
- Integration Profile Model: support
-
Advanced Runner Execution
- Worker Pool Execution: support
processModel: 'worker-pool'as the bounded multi-worker substrate -
hostProcessExecution Shape: resolve a supervised host process around worker-pool execution when runs need Node/V8 arguments, profiling, debugging, or hosted benchmark isolation - Worker Lifecycle: support reusable workers (make it the default for integration tests) and fresh workers that exit after one assigned work unit
- Per-File Disposable Isolation: define process-per-file as worker pool plus file granularity plus fresh-worker-per-unit lifecycle
- Work Unit Planning: introduce
WorkId,WorkUnit,WorkUnitId,PlacementPlan, andPlacementTrace - Work Distribution Strategies: support file, case, and named group distribution
- Named Group Distribution: define file-set references, group granularity, per-group scheduling/order/lifecycle overrides, unmatched handling, and overlap errors
- Case-Count Balanced Placement: make worker-pool placement balance by selected case count with seeded tie-breaks
- Resource-Aware Placement: lower resources into serial keys, capacity weights, affinity keys, and fault domains
- Dynamic Scheduling Concepts: specify duration history, runtime reprioritization, pending-unit splitting, hedging, compatible batching, and warm-lane affinity as later execution concepts
- Duration History Index And Placement: persist compact
WorkIdduration samples underruntimeStateDirand feedduration-history-balancedplacement with stale/missing fallback - Dynamic Lease Dispatch: consume frozen placement through pending-work leases, runtime reprioritization, and hard-constraint-preserving reassignment
- Pending-Unit Splitting: split eligible not-started units along
WorkIdboundaries with derived trace identities linked to the parentWorkUnitId - Hedged Duplicate Execution: duplicate only explicitly idempotent or disposable isolated stragglers, cancel slower duplicates, and fail conflicting outcomes
- Compatible Batching: batch identical execution envelopes without merging
WorkUnit, result, artifact, or duration identities - Warm-Lane Affinity: prefer warmed reusable workers as a soft tie-breaker after hard constraints, capacity, spreading, and fairness
- Runtime And Executor Affinity: expand runtime/workload matrices before placement and assign only to compatible executors
- Worker Count And Capacity Controls: implement default sizing, host caps, profile caps, weighted capacity, and
--workers N - Supervised Process Execution: keep
supervised-processas a single child boundary, not a worker pool - Single-Worker Serial Execution: support one worker or process for benchmarks, exclusive resources, and constrained runs
- Frozen Placement Assignment: freeze work units, constraints, ordering, sharding, and initial executor assignment before execution
- Placement Trace And Duration Facts: record worker assignment, retries, recovery decisions, hedging decisions, and work-unit duration facts; boundary and worker timing details come from Runner Timing Diagnostics
- Execution Plan Resolution: reject incompatible process model, lifecycle, distribution, runtime, and resource combinations before execution
- Worker Pool Execution: support
-
Higher Test Layers
-
@overkill-dev/simulation: generic simulation package with finite scenario catalogs and simulated HTTP server definitions -
@overkill-dev/model: model-based testing package -
@overkill-dev/differential: differential testing package -
@overkill-dev/linearizability: concurrency/consistency package -
linearizability(...): first concurrency/consistency helper
-
-
Property-Based Testing
- Desired concept: Property-Based Testing Resolution; concept PR: #556
- Property Authoring And Distribution: implement
@overkill-dev/property, the@overkill-dev/test/propertysubpath, first-classproperty({ title, generator, test })values, and advanced nestedwithProperties(...)/scope.forall(...)authoring without runtime subtests or heavier root microtest imports - Property Family And Policy: implement
testFamily: 'property', microtest-like capability restrictions, collection-time policy validation, generation and shrink budgets, timeout/resource supervision, and no code coverage - Fluent Generators: expose primitive and bounded recursive generators with
map,flatMap, andfilter, using fast-check behind the independent Overkill API; custom generators compose these surfaces and inherit shrinking - Generator Validity And Reconstruction: preserve dependent domains, size/depth limits, deterministic context, and fresh mutable input graphs for every trial, replay, example, and confirmation
- Trial Phases And Accounting: execute checked-in corpus, local cache, explicit examples, bounded automatic edges, and new exploration sequentially within each property; keep phase counts separate and allow ordinary runner parallelism between properties
- Exhaustive Domains And Input Coverage: exhaust small known finite domains within policy limits; count required coverage over accepted new exploration or the complete exhaustive domain, with inconclusive results when requirements remain unmet
- Targeted Exploration: use finite labeled scores to guide bounded deterministic exploration without making score reachability a pass condition
- First-Failure Shrinking: stop generation at the first failure, preserve the first failed assertion or body error through automatic failure identity with an explicit stable-key override, and retain unrelated failures as diagnostics
- Failure Confirmation And Termination: report non-reproducible failures and stop shrinking; bound candidate production, rejection, reconstruction, duplicates, and evaluations while preserving confirmed evidence and native cleanup or supervision outcomes
- Versioned Graph Codecs: reuse devalue behind reviewable JSON codecs that preserve shared references, cycles, and supported value state; provide explicit custom codecs and reject lossy or unsupported inputs
- Decoder Compatibility And Migration: retain supported historical decoders, write the current format, validate compatibility fixtures, and migrate only through explicit operations
- Regression Corpora: save confirmed replayable failures in the bounded local cache and explicitly promote selected witnesses into a checked-in corpus; fail incompatible checked-in entries, diagnose and skip incompatible local entries, and replay regressions independently of current generation limits
- Corpus API And CLI: implement inspection, promotion, and migration through typed runner APIs and the
overkill corpusnamespace, with decoded previews, atomic writes, stable target identity, and no implicit repository changes - Witness Replay: replay first-class properties without generation or shrinking through normal lifecycle and supervision; replay nested properties by executing the enclosing case and substituting the saved input at an unambiguous target
- Property Results And Diagnostics: keep one engine case and boundary assertion per property; report phase counts, seeds, coverage, non-reproducibility, and shrink termination without claiming a globally minimal counterexample
- Generator Preview And Relation Checks: provide bounded
sampleGenerator(...)previews and metamorphic relation helpers through@overkill-dev/property - Property Integration Validation: cover dependent shrinking, recursive limits, non-injective mappings, impossible filters, mutation isolation, deterministic streams, exhaustive accounting, codec graph fidelity, historical fixtures, corpus policies, nested replay, and supervision on the pinned runtime
-
Property-Derived Contract Acceptance
- Prerequisites: Property-Based Testing and Executable Test Families; model precedes linearizability.
- Model Contract Acceptance: validate predicted traces before system creation, independent inputs, enabled-transition shrinking, and stored-trace replay beyond generation limits.
- Differential Contract Acceptance: validate sequential implementations, fresh inputs, returned/threw comparison, control-error propagation, and input replay.
- Linearizability Contract Acceptance: validate real-time order/pending operations, rejection versus inconclusive search exhaustion, bounded cancellation/draining, lane witnesses, and non-reproducibility.
-
Builder Layer
- Builder-Oriented Composition Surface: preserve an explicit layer for assembling higher-level testing APIs from lower-level contracts
- Third-Party DSL Support: allow specialized test APIs to be built without forking the runner or engine
-
Standard Distribution And Optional Packages
- Standard Distribution Model:
@overkill-dev/testis the one-package normal user install and standard distribution - Standard Stack Dependencies:
@overkill-dev/testdepends directly on the standard stack so users do not align package versions by hand - No Preset Packages: remove
@overkill-dev/micro,@overkill-dev/default,@overkill-dev/integration, and@overkill-dev/allfrom the concept - Optional Package Policy: property stays in the standard distribution; browser, model, differential, linearizability, type-test, mutation, ESLint, AWS CDK, contract, accessibility, and similar domain packages remain separate installs
- Explicit Optional Integration: optional packages integrate through typed profiles, runtime/resource factories, reporters, baseline adapters, or authoring helpers
- No Dynamic CLI Extension: installing an optional package does not mutate the
overkillCLI, trigger package-name discovery, or widen top-level config through module augmentation
- Standard Distribution Model:
-
Mutation Testing And Stryker
- Desired concept: Mutation Runner Contract; concept PR: #540
-
@overkill-dev/stryker-runner: publish the optional first-party StrykerTestRunnerplugin above the public@overkill-dev/runAPI; leave mutation planning, sandboxing, scores, thresholds, incremental reports, and mutant parallelism to Stryker - Microtest Profile Selection: reuse an existing Node microtest profile; infer it only when exactly one is eligible, require a name when several exist, and reject missing, unknown, non-microtest, browser, and mixed-family selections before collection
- Mutation Execution Policy: expose the serial scheduling override while preserving configured discovery, loaders, capabilities, resource policies, and process model; disable ordinary V8 coverage, Overkill retries, and baseline updates
- Frozen Campaign Policy: resolve authoritative discovery, loader, selection, permission, and resource facts outside the execution module graph; transport them to execution hosts and reject conflicting reevaluated policy
- Mutation Target Boundaries: exclude configuration, test definitions, and adapter code from mutation targets; normalize sandbox origins without executing original files outside the sandbox
- Public Runner Lifecycle: support initialization before user imports, execution of a collected plan without reimporting modules, and fresh scopes, observers, and result collection for each invocation without bypassing test policy
- Configured Process Models: run normal in-process profiles inside Stryker's reusable plugin worker and supervised profiles inside the normal restricted test child; keep case execution serial while Stryker parallelizes mutants
- Fresh-State Configuration: default
freshStatetofalse; when enabled, reset the complete transitive module graph and mutable state for every invocation through a fresh execution boundary; document reused-graph cache limitations - Environment Reload Contract: report
reloadEnvironment: falsefor reused native ESM execution andtruefor fresh-state or supervised execution; honor Stryker-requested reloads and static-mutant transitions - Instrumentation Activation: initialize Stryker's namespace in the executing process, activate static mutants before user imports, and activate runtime mutants after unmutated collection but before cases
- Instrumentation Lifecycle: reset counters, active mutation, test identity, and hit-limit state for each invocation, including error paths, while preserving namespace and counter-object identities retained by reused instrumented modules
- Case-Level Mutant Coverage: set and clear
currentTestIdat awaited local case boundaries through cleanup in both process models; retain outside-case reads as static coverage and transport complete counters to Stryker - Coverage Modes And Fallbacks: support Stryker
perTest,all, andoff; run the full eligible suite for static or uncertain attribution and never inferNoCoveragefrom missing transport data, missing tests, or V8 coverage - Stable Mutation Test IDs: encode structural
CaseId, parameter identity, canonical project-relative origins, and distinct validWorkIdvalues without depending on display names or source positions - Catalog And Exact Selection: establish the complete dry-run catalog, validate identities and selection metadata on every invocation, execute requested IDs exactly, and return errors for unknown or duplicate IDs, drift, unexpected skips, collection failures, or empty mutant selections
- Structured Dry-Run Results: report stable IDs, durations, successes, failures, and intentional skips; return timeout or infrastructure errors without producing a mutation score
- Mutant Result Classification: return
Killedonly for safely attributed eligible-case failures, including assertions, body errors, and policy violations; keep pre-case, unattributed, infrastructure, transport, reporter, persistence, and inconclusive failures asError - Result Precedence: enforce
Error > Timeout > Killed > Survived, including cleanup timeouts and continued execution after a kill; preserve earlier case failures in diagnostics - Bail And Completion: stop admitting cases after a confirmed kill unless Stryker sets
disableBail; finish admitted work, cleanup, observer shutdown, and result delivery, and report actual executed counts and killing IDs - Timeout Enforcement: honor both profile case timeouts and Stryker's complete invocation deadline, including startup and cleanup; classify exceeded hit limits as timeouts and terminate and reap adapter-owned hosts even for synchronous infinite loops
- Native Guard Ownership: retain console, permission, uncaught-error, rejection, and async-leak checks without monkey-patching; exclude Stryker logging, transport, and bootstrap from test-policy attribution and stop owned observations when each run ends
- Surviving Worker Reuse: close scopes, observers, and invocation bookkeeping after unsafe invocations without automatically retiring a surviving plugin worker or enabling fresh state; preserve safe attribution and leave dead-worker, deadline, and required-reload recovery to Stryker
- Campaign Reproducibility: accept or generate one campaign seed shared across dry runs, mutants, workers, and replacements; expose it for reproduction and derive case seeds from stable executable identity independently of selection and scheduling
- Reporting And Default Persistence: suppress all ordinary and custom Overkill reporters and disk writes for results, run records, duration history, artifacts, and reports; retain structured adapter results in memory and leave Stryker campaign output under Stryker configuration
- Opt-In Diagnostic Persistence: provide
off,failures, andallwithoffas default; persist only available, profile-permitted artifacts and transcripts with campaign, mutant, invocation, and case identity, retain existing limits, and classify delivery or write failures as infrastructure errors - Mutation Integration Validation: cover sandbox-relative and parameterized identities, runtime and static activation, exact selection, catalog drift, coverage transport, both process models, reused and fresh state, policy ownership, precedence, timeout cleanup, native early exit, abort, and termination without results on the pinned Node version
- Mutation Accuracy And Cost: compare optimized and full-suite results on repeatable suites and measure reuse versus fresh-state costs across startup, collection, and execution
- Mutation User Documentation: document installation, Stryker configuration, profile inference, execution overrides, coverage modes, seeds, result mapping, persistence, process guarantees, and reused-state limitations
-
Ecosystem Integrations
- Native Type-Test Support: implement
documentation/desired-concept/authoring/testing-models.mdafter the family contract above. Checked adapter milestones record the earlier implementation; the original-source contract supersedes its generated-input direction.- Native Assertion Contract: define inline type assertions using Overkill composition and assertion style, with one source file and one description per case; specify supported assertion references, generic bodies, tables, and macro semantics. Integration direction and API findings: Define the @overkill-dev/type-test concept #672.
- Best-Effort Generated Input: use the existing public TSTyche API with adapter-owned, dot-prefixed sibling modules preserving source extensions; map original and generated locations, repair unambiguous direct self-imports and translated-module imports, and document identity limits without waiting for upstream changes.
- Explicit Checker Configuration: derive configurations per original project and compiler version, preserve effective options and references, include generated inputs explicitly, and verify the expected project and compiler instead of accepting baseline fallback.
-
@overkill-dev/type-test: publish one optional package for native assertion references, typed profile wiring, source translation, and TSTyche execution; keep checker dependencies and lazy execution loading outside@overkill-dev/test, and provide direct checking of engine roots without requiring the high-level DSL. - Explicit Adapter Contract: add generic runner hooks for profile validation, collection, execution lowering, and frozen facts; register the optional adapter explicitly without global side effects or module augmentation, recognize assertion exports through import aliases, and reject unsupported profile options including coverage.
- Source Ownership And Translation: associate assertion-bearing bodies with stable source anchors across engine cases,
test, suites, macros, tables, and supported wrappers; translate supported inline references without changing lexical context, contextual typing, or native diagnostics, and retain original-source mappings. - Type-Test Execution: identify checker-backed cases during collection and lower their frozen plan into external checker work without invoking ordinary builder bodies; account explicitly for shared static evidence and reject unsupported forms or missing checks.
- Compiler Planning: freeze source fingerprints, case-to-assertion ownership, original project configuration, and exact TypeScript versions; put compiler variants in
WorkId, retain logicalCaseId, and record reproducible compiler and input facts. - Dependency And Compiler API Assessment: validate a supported public comparison interface or document the focused comparison work needed for the specified semantics; evaluate TSTyche reuse/extraction and supported stable TypeScript plus a pinned 7.1 prerelease without private imports or compiler patching.
- Public Package And Compiler Selection: expose
family,sameType,assignableTo, andassignableFromwith the documented expression/negation forms; resolve original projects and explicitly installed local/package compiler selections, preserve exact compiler identity, and remove automatic compiler substitution/downloads from checking. - Original-Source Relations: replace sibling modules, derived projects, import repair, and generated mappings with original-program assertion ownership and analysis; implement the documented equality/assignability rules, retain generic and table semantics, and return inconclusive for unsupported evidence without runtime execution or generated input.
- Exact Selection And Diagnostics: bind ordered selected work to its original assertion sites, preserve shared-body ownership and original locations, define no-assertion/skip/inconclusive outcomes, buffer passing results until input validation succeeds, and distinguish relation failures from project/checker errors. Dedicated JSX/decorator and precise expected-diagnostic assertions are outside the initial catalog.
- Compiler Lifecycle: implement preparation/check deadlines, cooperative cancellation, hard-deadline process ownership, disposal, repeated/watch invocations, and source/configuration/resolution drift invalidation without file emission or referenced-project builds.
- Persistence And Reproduction: retain family/compiler identity, original inputs, variant ownership, diagnostics, and outcomes in records and failure selection; validate compatible implementations and source inputs before reproduction while preserving earlier record semantics.
- Type-Test Acceptance: cover direct engine and runner parity, inference and overloads, the relation semantics table, source ownership across composition layers, matrices and variant isolation, exact selection, shared/static bodies, nominal identity/self-imports, missing evidence, cancellation/drift/disposal, and reporter/record delivery; assert no replacement checker inputs are created.
- User Documentation: explain the public family and assertion exports, installed compiler aliases, direct engine lifecycle, profiles, source requirements, supported forms, diagnostics, and reproduction; remove obsolete generated-input and full-matcher-parity instructions.
-
@overkill-dev/eslint-rule-test: ESLint rule-testing package -
@overkill-dev/eslint-plugin: implement the settled static authoring catalog indocumentation/desired-concept/authoring/higher-test-layers.md; users choose rule enablement and severity.- Binding Analysis: follow renamed imports, destructuring, aliases, local object properties, shadowing, and assignment rebinding using binding identity.
- Authoring-Layer Coverage: define supported forms per rule across
@overkill-dev/test, custom facades, assertion helpers, and direct engine usage; distinguish test entries from support modules. -
no-scope-escape: prevent forwarding or escaping scope, assertion/require facades, planning APIs, and extracted assertion methods to ordinary helpers. -
no-single-use-trivial-assertion: reject local, unexported, single-reference wrappers around one built-in check while preserving useful composites, narrowing, and diagnostics. -
no-self-comparison: reject equality checks comparing the same stable binding with itself. -
no-constant-actual-assert: detect likely reversed actual/expected operands. -
prefer-semantic-assertions: prefer informative built-in assertions without changing comparison or evaluation semantics. -
no-shared-mutable-test-state: detect shared mutation through bindings and object aliases while respecting declared resource lifetimes. -
no-reexported-test-node: reject forwarding another module's root unchanged while allowing local suite composition. -
consistent-run-if-main: enforce the configured direct-file execution policy. -
no-setup-in-error-assertion: prevent separate setup operations insidethrows/rejectscallbacks from satisfying the expected-error matcher. -
no-duplicate-assertions: detect adjacent identical checks on stable, side-effect-free operands. -
prefer-table: suggest tables for independent repeated cases while preserving stateful scenarios and intentional node composition. -
no-conditional-assertions: detect conditional checks in test bodies only; allow branches in custom assertion implementations and straight-line checks afterscope.require, which gates by throwing. - Static Authoring Acceptance: cover supported authoring layers, aliasing/rebinding, false positives, and semantic preservation; do not duplicate runtime/discovery checks.
-
@overkill-dev/aws-cdk: first package-specific assertion adapter wrapping@aws-cdk/assertions - Contract Adapters: other adapters for external assertion/testing ecosystems
- Stable Machine-Readable APIs: keep reporter/results/planning data usable for IDE and MCP integrations
- Native Type-Test Support: implement
-
Browser And Remote Expansion
- Prerequisites: Shared Contract Extensions, Executable Test Families, and resource/runner/benchmark contracts. Distributed merging also requires Sharded Result Merging.
- Execution And Session Foundations
- Browser Capability Proofs: prove public controller/provider paths for redirects/workers/sockets, cache/SW attempts, warm caching, native routes, and remote reachability; publish supported configurations before dependent guarantees.
- Browser Runtime Package: first optional browser-oriented runtime/package
- Browser-Compatible Engine Surface: separate browser-safe engine/assertion/authoring/resource imports from Node-only attribution, filesystem, reporting, and clocks through a platform adapter.
- Browser Execution Modes: define Browser-executed bodies inside a browser document and Host-driven browser automation bodies on an execution host, using shared provisioning and provider-native adapters.
- Browser Runtime Integration: browser support integrates through fixtures/resources and explicit project profiles, not a
browsertest family or CLI mutation - Browser Family Boundaries: require integration for browser correctness and benchmark for browser performance; reject browser runtimes and host-driven browser automation in microtest/property families regardless of provisioning.
- Native Integration Boundaries: expose provider-native actions/locators with explicit provider identity; Overkill owns lifecycle, guards, checks, measurements, and reporting without a universal action API.
- Modern Protocol Scope: exclude a first-party Classic WebDriver command adapter; assess public Playwright BiDi support before adding clients for demonstrated gaps, while permitting necessary provider/session setup.
- Browser Collection Policy: collect browser-free definitions by default; offer one explicitly configured collection browser for top-level browser imports, with matrix expansion remaining planning metadata.
- Mixed-Placement Browser Suites: permit compatible modes in the same suite or file within one family; partition execution by compatible placement and use explicit module boundaries for incompatible import graphs.
- Cross-Realm Resources: apply shared owner/consumer/observer references through one resource declaration, preserving wrappers and same-realm resources without duplicate profile bindings.
- Browser Harness Delivery: own module delivery, bootstrap/source maps, server and browser/result channel through replaceable tool integrations; require no prebuilt app and preserve direct/runner lifecycle.
- Browser Channel Fencing: validate control, harness/results, and app reachability separately; fence assignment/session/document/work/attempt messages and reject stale/lost evidence without action replay.
- Browser Attempt Isolation: create a fresh context/session per integration-test attempt while permitting browser-process reuse; keep backend reset and benchmark sample state explicit.
- Browser Body Lifetime: bind browser-executed bodies to their execution document; replacing it before completion fails the attempt, while host-driven tests handle app-navigation/reload journeys without resumable browser phases.
- Functional Capability Compatibility: fail selected ordinary-browser targets lacking required capabilities unless an intentional skip is authored.
- Browser-Local Native Input: provide bounded provider-specific keyboard/pointer assistance with target/document/attempt identity, capabilities, readiness, timeouts, and cancellation; no general controller RPC.
- Protected Sessions And Evidence
- Network Transport Capabilities: define standard HTTP(S)/WebSocket guard coverage; require explicit validated capabilities for other transports and reject unsupported requested configurations.
- Request Prevention And Guard Outcomes: prevent denied transmissions; unexpected violations fail resource checks without automatic retries; bounded expected denials stay blocked, only affected work stops, and lost observation is infrastructure failure.
- Forbidden Request Attempts: apply network guards to forbidden application request attempts even when cache or service workers supply responses without outbound traffic; require reliable observation alongside prevention.
- Fine-Grained HTTPS Prevention: support URL-path/method enforcement through validated capabilities; reject unsupported policies and require explicit configuration for TLS inspection.
- Guarded Service Workers And Caching: support service-worker journeys and warm-cache benchmarks with compatible prevention; reject incompatible configurations without silently disabling app behavior, caching, or protection.
- Default App Guards: guard first-party app/session resources including benchmarks; keep bare execution lightweight and honor loopback defaults, external grants, explicit deny precedence, and console/error rules.
- Expected Browser Diagnostics: provide bounded, count-checked expectations registered before actions; fail missing/excess events without granting network permissions or erasing failures.
- Standalone Guard Outcomes: closing standalone guarded sessions throws structured findings while preserving other errors; engine-managed sessions use ordinary failed checks.
- Browser Record And Reproduction Compatibility: retain placement, module/harness build, guard/scenario/startup and collector/reset facts; validate reproduction and versioned historical readers.
- Browser Extension Resources: provide validated browser/profile provisioning with protection before background startup, fresh profile/process isolation where needed, and declared benchmark state.
- Measurements And Quality Adapters
- Shared Browser Benchmark Authoring: retain ordinary authoring and one sampling/budget lifecycle; compose both body placements with independent collectors and explicitly browser-local timed operations.
- Browser Sample Attribution: bind collectors/guards to warmup/measured invocations, finalize before aggregation, and retain ambiguous/incomplete lifetime evidence.
- Browser Benchmark State: require declared retained/reset browser and app state per workload, with presets and state-aware comparison identity.
- Action-To-Result Latency: measure correlated browser input to an authored DOM condition or app mark; keep host command duration and rendering evidence separate, with no silent timing substitution.
- Browser Metric Collectors: adapt timing/marks, input, paint/navigation, requests, rendering/animation/scroll, CPU/memory, and named audits with provenance; incomplete required metrics cannot pass budgets.
- Diagnostic Benchmark Reruns: make heavier capture reruns opt-in, preserve original failures, and retain separate measurement/attempt identities.
- Page Audit Checkpoints: expose heavier page-state audits through explicit reusable assertions/macros while continuous app guards remain enabled by default.
- Interactive Accessibility: provide reusable native keyboard/focus assertions for tabbing, activation, modal containment/restoration, skip links, and route/form-error focus alongside scanner-backed semantic/ARIA checks.
- Accessibility Preferences And Geometry: adapt contrast/target geometry, light/dark/forced-color/reduced-motion, touch, text enlargement, and zoom/reflow with actual environment provenance.
- Screen-Reader Adapters: provide optional first-party actual reader automation for validated OS/browser/reader combinations, reusing existing libraries and owning required desktop focus/settings/exclusive access as resources.
- HTML Conformance Inputs: offer validation of the actual original response and rendered-DOM checkpoints separately, retaining input provenance without silently refetching another document.
- CSS Conformance Inputs: adapt validation of explicit external, inline, and generated stylesheet inputs, preserving source/coverage limitations separately from layout or accessibility correctness.
- Semantic Data Checks: adapt microformats and JSON-LD/Microdata/RDFa, schema.org/profile checks, and visible-app fact assertions; distinguish syntax, semantic processing, vocabulary, and provider rules.
- Discoverability And SEO Checks: assert status/redirects, source/rendered metadata, titles, language, canonical/hreflang, robots, and links without promising indexing, ranking, or rich results.
- Accessibility Attachments: retain bounded native reports/transcripts with explicit checkpoint, target/input scope, tool/rule versions, and coverage; distinguish violations, incomplete findings, unavailable capabilities, and infrastructure failures.
- Visual Baselines: checked-in visual comparison artifacts
- Remote Execution And Acceptance
- Remote Execution Planning: model remote/distributed execution in orchestration
- Distributed Result Merging: merge results from remote/distributed runs
- Dogfooded HTML Reporter Browser Tests: exercise benchmark HTML report loading, chart rendering, workload selection, zoom, tooltips, and offline behavior through the first-party browser runtime.
- Browser Testing Guidance: recommend unit tests without a real DOM and integration tests against the fully rendered app; provide no first-party component mounting API or component-testing DSL, while allowing user-supplied rendering setup.
- Browser Scope Guidance: document supported mobile browsers, authored worker use, consumer native-app integration, rejected standalone test export, and retained offline reports.
- Browser CLI Extension Deferred: use project-named profiles such as
ui-browserwith ordinaryoverkill run,overkill list, and baseline verbs until a dedicated command need is proven - Browser Contract Acceptance: verify both modes, mixed/split-module suites, document loss, cleanup/guards, standalone/direct-runner parity, local/remote evidence, cache/SW, audits/readers/extensions, and rejected boundaries.
-
Concept And Documentation Acceptance
- Public Contract Conformance: validate implementation against the owning desired-concept contracts, including public/direct/runner parity, rejected boundaries, records, baseline staleness, empty selections, cleanup, and supported platforms.
- User Guides And Reference: complete quick start, common/advanced workflows, package/CLI/API reference, and examples per the documentation strategy; reconcile walkthroughs, types, and loader guidance.
Activity
Sync concept docs after assertion split review
Keep desired-concept documentation independent of delivery status