Skip to content

Implementation milestones #194

Description

@lo1tuma

Implementation Milestones

Implement unchecked items in dependency order. Checked items preserve history; later corrections supersede older wording. The desired concept is specified in documentation/desired-concept/; this issue owns the project plan. Research inventories and rejected/deferred ideas are outside implementation scope.

A milestone includes its specified public/direct/runner/CLI behavior, types/exports, identity/records, failure/cancellation/cleanup paths, documentation, and acceptance checks. Platform-dependent features require a validated supported path and explicit compatibility rejection elsewhere; a facade or universal unsupported result does not complete a feature.

  • Monorepo Foundation

  • First Usable Engine

    • Public @overkill-dev/engine Entry Point: one supported import surface for defining suites/tests and executing them without repo-internal imports
    • Direct Engine Execution Flow: create suites/tests, execute them through the public API, and get a result back
    • Structured Outcomes And Run Result: stable Pass / Fail / Skip / Inconclusive protocol plus run summary/result shapes
    • Stable Identity And Run Counts: engine-owned test/case identity, discovered vs executed counts, and orphan detection
    • Reporter Event Stream Contract: engine-owned structured event surface for reporters and machine consumers
    • Engine Self-Tests Through Public API: rewrite engine tests so they exercise the public engine entry point, not internal source paths
  • First Human Output

    • Reporter Lifecycle And Sink Semantics: real-time vs final-result reporters plus sink declarations as required by the concept
      • Define RealTimeReporter and FinalResultReporter as separate lifecycles
      • Require declared reporter sinks and validate direct-engine conflicts before run-start
      • Document lifecycle and sink behavior on the engine package surface
    • @overkill-dev/reporter-line: first standalone human reporter package exposed through its own facade
    • Human Failure Rendering: useful terminal rendering for pass/fail/summary output
    • @overkill-dev/reporter-dot: minimal progress reporter for compact local/CI output
  • Structured Assertions

    • Engine-Owned Assertion Protocol: keep assertion semantics, injected case.assert / case.require, and assertion counting in @overkill-dev/engine
    • Built-In Assertion Catalog: first useful built-in assertion set
    • assert / require Semantics: define non-gating vs gating assertion behavior
    • done(), plan(n), And Zero-Assertion Detection: explicit assertion-count contract and default no-assertions failure
    • Custom Assertion Protocol: decide the public shape for custom nodes, composite assertion boundaries, and foreign assertion bridges
    • Assertion Source Locations: capture call-site locations for built-in, custom, composite, and foreign assertion failures
      • Shared Source Location Relativization: centralize project-root path rendering so every reporter benefits from one path policy
      • Adapt reporters to display source locations
      • Helper And Macro Location Forwarding: provide an explicit API for wrapped assertions to report the caller location
    • Failed Check And Diff Model: structured mismatch payloads and diff shapes
    • Bounded Assertion Serialization: cap depth, node count, entry count, string bytes, and serialized bytes before reporter delivery
    • Error Matcher Contract: explicit matcher shape for throws / rejects
    • @overkill-dev/assert: reusable assertion-extension helpers such as defineCompositeAssertion(...), foreign-assertion bridges, and packaged assertion adapters
  • Runtime Assertion Integrity

    • Durable Contract Failures: persist assertion-contract violations before throwing so catches, error assertions, and foreign bridges cannot erase them or turn engine control signals into expected application failures.
    • Assertion Context Lifetime: reject recording after body settlement or interruption, preserve originating attempt/trial identity, and deliver durable late-activity diagnostics without changing published verdicts.
    • Assertion-Time Operand Meaning: preserve injected builder checks and diagnostics against later operand mutation while retaining the public low-level lazy protocol contract.
    • Returned Builder Assertion Ownership: reject nodes from other cases, retries, or trials and prevent repeated recorded node identities from inflating assertion counts or satisfying plan(n).
    • Assertion Protocol Validation: validate node/child structure and reject malformed values or structural cycles as durable contract failures while allowing supported cyclic application operands.
    • Nested Assertion Context Ownership: keep property-trial checks and custom assertion diagnostics in their injected contexts, including supported asynchronous composition.
    • Test Entry Export Validation: reject runtime exports besides the selected engine's testNode during discovery; preserve support-module exports and erased type-only exports.
  • Doubles

    • @overkill-dev/doubles: first doubles package facade
    • testDouble(): primary public double creation API
    • Rule-Based Behaviors: configurable responses and behavior rules
    • Typed Pattern Matchers aka integrate ts-pattern
    • Call History Introspection: inspect calls/results directly
    • Doubles Assertions: doubles-oriented assertion extensions plugged into the engine-owned assertion context
  • First Dogfooding And Publishing

    • Publish @overkill-dev/engine: first real package publication from the new layout once the engine flow is usable
    • Publish @overkill-dev/reporter-line: first reporter publication once it works end to end
    • Publish @overkill-dev/assert: publish reusable assertion-extension helpers once the extension model is usable
    • Publish @overkill-dev/doubles: publish the doubles package once the core API is usable
    • Dogfood Through Engine API: use the published engine, assertion-helper, doubles, and reporter packages in Overkill’s own tests/tools directly
    • Remove Internal-Only Test Wiring: stop depending on repo-internal execution paths where the published engine API is sufficient
  • Minimal Runner

    • @overkill-dev/run: first orchestration package facade
    • RunRequest, ResolvedRun, And RunFacts API: typed request, resolution, and serializable run facts surface mirrored by the CLI
    • Reporter Completion Errors: decide how run-end, onFinish, and final-result reporter failures are surfaced after RunResult is created
    • Command-Line Business Logic API: parser, command intent model, config loading flow, command dispatch, fallback error reporting, and exit-code mapping are owned by @overkill-dev/run
    • Command-Selected Module Loading: command modules load only for the selected command path so ordinary microtest startup does not load benchmark, baseline writer, coverage, or reporter-heavy modules
    • Named Profile Model: support choosing the runner profile used by a run request, including the existing microtest profile
    • Reporter Sink Exit Codes: map configuration failures such as sink conflicts to stable CLI exit codes
    • Managed Output Sink Coordination: validate raw vs managed stdout / stderr reporter sinks, render managed line intents through the default plain renderer, and add @overkill-dev/reporter-brief plus @overkill-dev/output-renderer-github-actions
    • Resource Budget Resolution: carry Node-first heap, RSS, resident-growth, and active-resource budgets through grouped resourceUsage policy in config, RunRequest, and RunFacts
    • Supervised Resource Exhaustion Attribution: enforce budgeted microtest profiles through processModel: 'supervised-process', stream active test identity, kill wedged children on hard timeout or budget breach, and record resource-exhausted with the active case
    • Run A Specific File Or Module: one complete public run flow for explicit inputs
    • Internal Test Export Cleanup: migrate Overkill test modules from exported testSuite to the public testNode convention
    • Full Config Loading: finish config discovery, import, schema validation, and any final decision on default vs named exports for defineConfig
    • Runner Profile Naming Model: support project-owned runner profile names such as backend-http or ui-browser without deriving behavior from the name
    • Test Family Discriminator: require profiles.<name>.testFamily and use it as the discriminant for profile config
    • Microtest Profile Validation: validate the initial testFamily: 'microtest' profile surface and reject fields that are not valid for microtests
    • Microtest Execution Model: model processModel as in-process or supervised-process for microtests
    • Microtest Scheduling Model: model concurrent and serial scheduling for microtests
    • Microtest Capability Restrictions: implement capability restrictions for microtest profiles only in the minimal runner; property profiles adopt the same restricted boundary later, while integration, browser, benchmark, and type-test profiles do not use this model
    • Cross-Process Custom Engine Boundary: design custom engine execution for supervised and other multi-process runs without parent-side user-module execution
    • Profile Reporter Resolution: support profile reporter overrides with global reporter fallback
    • CLI Fallback Diagnostics: return runner errors not sent to terminal-capable reporters, and map completed runs to the documented non-zero CLI exit status
    • CLI Run Command Surface: expose the @overkill-dev/run command implementation consumed by the @overkill-dev/test binary wrapper, keep bin: overkill out of @overkill-dev/run, write returned fallbackDiagnostics to stderr, apply the returned exit code, and parse textual run flags including --measure-resource-usage and --resource-budget <name=value> for activeResourceCount, javaScriptEngineHeapBytes, residentSetBytes, and residentSetGrowthBytesPerSecond
  • Runner Error Capture

    • Execution-Mode Global Error Hooks: install run-scoped unhandledRejection and uncaughtException observers in every execution boundary, including in-process runs, supervised children, host processes, and worker-pool workers, correlate failures with the active test via AsyncLocalStorage, and surface them as runner errors instead of raw process output
    • Permission Denial Runner Errors: classify ERR_ACCESS_DENIED throws and Node permission diagnostics as permission runner errors in every execution mode, deliver them through reporter events and CLI fallback diagnostics, and preserve useful fields such as permission kind and resource path
  • Runner Timing Diagnostics

    • Run Timing Concept: document summary vs precise timing, timing ownership, debug/benchmark boundaries, and OpenTelemetry export stance
    • Run Timing Data Model: add stable summary timings, precise spans, truncation, aggregate totals, and timing overhead
    • Timing Request Surface: support --timings, timings.collection, programmatic request overrides, and strategy-required precise collection
    • Default Reporter Timing Output: show compact timing summaries in line, brief, and dot reporters
    • Detailed Runner Phase Timings: record config loading, file discovery, collection, coarse import windows, propagation, filtering, runtime expansion, sharding, ordering, resource lowering, placement, dynamic dispatch, duration-history placement, hedging, reporters, and cleanup
    • Execution Boundary Timings: record host startup, supervised process spawn/ready/teardown, worker-pool start/shutdown, and individual worker create/ready/import/teardown timings
    • Resource Lifecycle Timings: record per-resource acquire/dispose timings by name, scope, process/worker, status, and aggregate
    • Timing Reporter Surface: expose top overhead offenders in human output and full timing data to machine-readable reporters
    • OpenTelemetry Reporter: plan optional @overkill-dev/reporter-opentelemetry as an exporter from Overkill timing data
  • Discovery, Test Data, And Selection

    • Explicit Module Listing: list explicit test files/modules through the runner
    • Profile File Discovery: implement files.include plus files.exclude with Node glob's separate exclude support; do not support negated include globs as a second exclusion mechanism
    • Named Profile File Sets: support files.sets as non-overlapping named discovery buckets with unique names, shared glob validation, empty-set handling, and explicit-file validation
    • Test Data Model And Propagation: split authored annotations from execution controls, move family to engine-owned facts, keep artifacts as observed results, and resolve root/suite/table/case propagation
    • Programmatic Selection API: typed filter helpers for IDEs, MCP servers, and code-based integrations
    • CLI Filter Grammar: stable selection/filter syntax for the runner
    • Selection And Filtering: select tests by file, title, id, tag, ownership, params, and suite path
    • Ordering And Seeding: deterministic ordering plus seeded variation
    • Sharding: stable identity-based partitioning of the filtered run set
  • Resources And Runtime Composition

    • @overkill-dev/resources: typed runtime/resource composition package
    • Typed Context Composition: compose runtimes/resources into test context explicitly, including dependency context for resources that acquire other resource handles
    • Lifecycle Scopes And Teardown: setup/teardown, dependency ordering, and sharing/isolation rules
    • Microtest Resource Boundary: enforce that resources and runtimes cannot attach to microtest profiles or microtest-authored tests
    • Runner-Aware Runtime Descriptor Attachment: make withRuntime(runtimeGraph, body), withResource(resource, body), and withResources(resources, body) attach descriptors that collection can read before scheduling
    • Descriptor-Based Microtest Resource Rejection: replace the current boolean attachment marker with collected resource descriptor metadata so microtest profiles reject resources regardless of which wrapper or facade attached them
    • Plural Scope Names: migrate injected runtime handles from scope.runtime to scope.runtimes and expose direct resource handles through scope.resources
    • Direct Resource Binding: implement withResource(...) and withResources(...) as runner-aware direct resource wrappers
    • Composable Body Wrappers: normalize nested runtime and resource wrappers before planning, and reject duplicate public runtime or resource keys
    • Resource Lifecycle Scopes: support per-run, per-file, per-suite, per-case, and shared-per-worker lifetimes
    • Runtime Matrices: implement defineRuntimeMatrix(...) with stable variant ids, shape checks, and planning-time expansion
    • Runtime Composition: implement composeRuntimes(...) with runtime-name collision rejection and Cartesian matrix expansion
    • Local Service Resources: model deterministic app servers, registries, browser test servers, and other local services as explicit resources
    • Simulation Resource Integration: model simulations through resource descriptors; @overkill-dev/simulation defines generic simulations, finite scenario catalogs, and simulated HTTP servers, while @overkill-dev/resources exports the resource adapter
    • Scenario Binding Propagation: forward planning-visible scenario bindings to owning resources through nested runtime graphs
    • Scenario Cache Identity: include acquire-time scenario bindings in resource cache keys while request-routed scenarios reuse existing acquired handles
    • Runtime Scenarios: support named scenario catalogs that affect runtime identity, artifacts, filtering, and replay metadata when declared as runner-visible descriptors
    • Runtime Selection Filters: add runtime, variant, and dimension filters plus repeated --runtime CLI sugar
    • Interaction Transcripts: support generic transcript recording for calls, HTTP, browser requests, subscriptions, and custom protocols
    • Runtime Attachments: allow fixtures and runtimes to attach structured artifacts such as screenshots, accessibility JSON, service logs, and transcripts
    • Execution Requirements Contribution: collected runtimes/resources lower requirements into placement constraints without owning the runner
  • Default Test Authoring

    • @overkill-dev/test Root Authoring Facade: add the high-level authoring surface on top of the existing binary package
    • @overkill-dev/test Binary Wrapper: ship the only first-party bin: overkill and delegate command behavior to @overkill-dev/run
    • Output-Capture CLI Flags: decide and implement the minimal textual surface for capture mode, including whether --no-capture should exist or remain config-only
    • File-Level Test Data Authoring: use root authoring semantics for module-wide annotations and controls
    • Standard Distribution Subpaths: expose config, reporters, bench, assert, resources, and baselines through explicit subpaths while preserving leaf package ownership
    • test(...) And suite(...): implement the first default authoring API on top of engine and export it from @overkill-dev/test
    • table(...): implement the first parameterized authoring surface and export it from @overkill-dev/test
    • defineMacro(...): implement the first preferred reuse mechanism for tests-as-values and export it from @overkill-dev/test
    • createTestFacade(...): initial authoring-surface composition; no longer the desired registration or profile-selection mechanism
    • Profile-Neutral Root Authoring: keep test(...), suite(...), and related helpers usable from @overkill-dev/test while the selected profile owns family policy
    • Facade Runtime Binding: update createTestFacade(...) to bind one runtime graph, direct resources, and mapScope(...) without selecting the test family
    • Facade Scope Mapping Guardrails: prevent mapScope(...) from overwriting core scope keys in types and at runtime
    • runIfMain(...): supported direct-file execution companion for exported specs - already exists partially in engine, we should consider moving it to @overkill-dev/test package so we can also rely on default reporters and so on. But we probably also need to decide which testFamily and which profile is picked when using runIfMain
    • Lightweight Doubles Root Exports: expose lightweight doubles APIs and doubles assertion references from @overkill-dev/test
    • Root Import Boundary: after root authoring APIs exist, verify the root export does not expose or load config, reporters, resources, benchmark, baseline, coverage, or command implementation APIs
    • throwingTest: decide whether throwable test authoring belongs in the root facade or a dedicated compatibility export, then implement the chosen surface
    • skippedTest: implement the planned skipped-case helper after the concept settles; it should record a reasoned planned skip instead of mutating source selection
    • defineHarness(...): implement the authoring helper after the root facade and core test(...) / suite(...) APIs exist
    • Interaction Transcript Helpers: implement recordCalls(...), recordSink(...), and recordEvents(...) once transcript semantics are ready
    • Async-Control Helpers: implement scope.microtasks(), scope.immediate(), scope.flushAsync(), and scope.inFlight(...) as one async-control authoring milestone
    • Stable Domain Aliases: project-local re-export pattern for domain helpers such as runtimes, resources, macros, and assertion references, not for profile selection
    • Standard Distribution Documentation: normal install path is npm install -D @overkill-dev/test
  • Coverage

    • Microtest-Only Coverage Model: explicit, off-by-default coverage restricted to microtest profiles and activated per run through --coverage or RunRequest.coverage
    • V8 Coverage Integration: runner-owned native coverage flow and artifact emission
    • Coverage Config Surface: profile-scoped formats, include/exclude all-files policy, thresholds, and outputDir in microtest config
    • Coverage Profile Validation: reject coverage config on non-microtest profiles
    • Coverage Run Record Integration: persist the coverage request, resolved policy, output paths, and artifact references without changing the selected process model or scheduling
  • Replay, Records, Baselines, And Debug

    • Run Record Persistence Policy: profile runRecords.persist, --record, workflow-required persistence, start/final records, and persistence runner errors
    • Compact History Index: persisted run summaries, case outcome history, WorkId duration history, retention windows, and schema repair
    • History CLI: overkill history list, prune, compact, and clear
    • Replay Picker: overkill replay without an id opens a TTY picker over replayable records and errors outside TTY
    • Record Retention And Maintenance: bounded automatic maintenance after persisted runs plus explicit prune, compact, and clear commands
    • Last-Failed Selection: resolve --last-failed from compact history and persist the selected run
    • Witness File Format: stable witness schema for replayable failures
    • Replay Commands: replay a recorded run or witness through public surfaces
    • @overkill-dev/baselines: shared baseline model and package facade
    • Baseline Verbs: update/apply/bootstrap/diff/list workflows owned by @overkill-dev/run
    • Snapshot Workflow: checked-in baseline compare/update flow
    • Failure Artifacts: structured logs, traces, snapshots, and artifact attribution rules
    • Debug Mode: richer diagnostics and retained runtime state for investigation
  • Executable Test Families

    • implement the public contracts in #698 and documentation/desired-concept/architecture/test-families.md in the order below, validating each step before the next.
    • Family Definitions And Identity: add defineTestFamily, typed settings validation, lazy backends, features, and explicit module references; replace the closed family union with executable definitions and stable IDs, and add generic family variants to work, work-unit, and artifact identity while preserving neutral identity keys.
    • Engine Preparation And Evidence: implement engine.prepareFamily, exact prepared.collect, branded collected runs, and engine.execute(collected, options); keep runtime bodies in the ordinary executor and route analysis events through the engine-owned sink, result validation, and reporter lifecycle.
    • Lifetime And Failure Contract: enforce single-use preparation/selection/execution, engine ownership, cancellation, incomplete-evidence errors, and exactly-once disposal before final delivery; retain cleanup failures and reject foreign, consumed, or disposed values.
    • Typed Profiles And Built-In Migration: infer each imported family's settings independently, retain supplied built-in aliases and existing defaults/policy, reject bare optional-family strings, and route direct callers and runner profiles through the same engine path.
    • Worker Resolution And Record Compatibility: resolve definitions inside explicit configuration/bootstrap boundaries, validate frozen identity/version/settings, reject unsupported placement for local-only definitions, and preserve historical inspection and versioned readers without loading family code.
    • Family Contract Acceptance: cover heterogeneous profiles, invalid settings, empty/skipped selections, variant isolation, selected result order, bad or missing events, reporter failures, cancellation, abandoned preparations, and supervised runtime regressions; preserve property-derived policy/replay, benchmark sampling/budgets, and Stryker frozen policy, selection, coverage, and precedence during migration.
  • Shared Contract Extensions

    • Prerequisites: Resources And Runtime Composition, assertion/result contracts, and Executable Test Families.
    • Portable Implementation References: resolve body/resource/observer modules with validated inputs, realm/build/source-map identity, and no closure/handle serialization.
    • Resource Check Lifecycle: implement withResourceChecks, durable ledgers, partial-install cleanup, admission, close-time draining, bounded supervision, and exactly-once finalization/disposal.
    • Resource Check Outcomes And Evidence: retain owner/attempt/lifetime attribution, live/final evidence, standalone errors, and finalization before retry/verdict without changing body plans.
    • Runtime-Aware Measurement Lifecycle: extend common collectors with async readiness/finalization, sync-path preservation, partial-start cleanup, target clocks/cooldown/GC, unavailable readings, and invocation identity.
    • Executable Identity And Catalog Acceptance: carry WorkId through reporter/evidence events and records; validate variants and stale/duplicate completion without counting recollection as discovery.
    • Simulation Witness Replay Acceptance: validate owner/producer/payload compatibility, signed nullable seeds, restore-before-projection, failure disposal, and current HTTP-handler scenario replay.
    • Explicit Simulation Identity: request-routed fixtures reject missing identity, including default; fixed-scenario servers bind identity at startup.
    • Simulation HTML Host: offer safe HTML/config assembly; consumers own markup, assets, builds, and scenario-to-app mapping.
  • Sharded Result Merging

    • Brief Final-Result Reporter: expose a final-result factory for compact merged summaries
    • Shard Run Record Persistence: persist completed shard RunRecord files when merge-results needs durable inputs, without updating aggregate compact history before merge
    • Merge Results API And CLI: implement mergeResults(...) and overkill merge-results <paths...> over record files and directories
    • Merge Compatibility Validation: reject mismatched profiles, seeds, shard totals, selected identities, loader config, and Overkill package versions
    • Incomplete Merge Results: emit a merged failure record for missing shards, duplicate work identities, unreadable records, or incompatible inputs
    • Merged Run Record: write non-replayable kind: 'merged' records with shard lineage and final-result reporter delivery
  • Additional Reporters

    • @overkill-dev/reporter-tap: TAP-compatible real-time output package, including source-location output for assertion failures
    • @overkill-dev/reporter-json: canonical machine-readable final-result package
    • @overkill-dev/reporter-html: generic artifact/failure HTML report package
  • Benchmarking

    • @overkill-dev/bench: first benchmark package facade over ordinary test-node authoring
    • Standard Distribution Integration: include @overkill-dev/bench in the @overkill-dev/test standard dependency set while keeping benchmark measurement outside the root authoring import and ordinary microtest startup path
    • Benchmark Command Namespace: implement overkill bench run and overkill bench list
    • Benchmark Config Namespace: load benchmark.profiles.<name> from the project config without mixing benchmarks into runner profiles
    • Shared Benchmark Profile Registry: replace historical benchmark.profiles with profiles.<name> and testFamily: 'benchmark'; preserve overkill bench routing, inference, direct-file behavior, and validation.
    • Benchmark Profile Selection: select benchmark profiles from benchmark commands without supporting overkill run --profile benchmark
    • Benchmark Runner Integration: implement benchmark commands through regular runner planning, placement, execution, result, and reporter flow
    • Benchmark Baseline Namespace: implement overkill bench baseline update/apply/bootstrap/diff/list for performance baselines
    • Workload Model: define benchmark inputs/workloads explicitly
    • Benchmark Resource And Runtime Integration: reuse ordinary resource/runtime descriptors for benchmark fixtures, services, browsers, registries, temporary directories, and PTY needs
    • No Benchmark Hook Model: keep benchmark lifecycle out of setup, teardown, beforeEach, and afterEach, using resources, runtimes, cleanup, and cooldown instead
    • Measurement Strategies And Metric Collectors: collect wall-clock, CPU/runtime, memory/resource, process, and custom metrics through explicit measurement surfaces
    • Benchmark HTML Reporter: implement @overkill-dev/reporter-benchmark-html with workload/metric tables, distributions, uncertainty, and budget/baseline comparisons
    • Host Calibration And Ambient Noise: record host capacity, load, memory pressure, runtime metadata, and available platform noise signals before measured samples
    • Adaptive Benchmark Placement: infer safe concurrency from measurement strategy plus calibration, and reduce to serial or fewer lanes when needed
    • Warmup, Cooldown, And Sample Cleanup: separate preparation, measured samples, benchmark-owned cleanup, and settling phases
    • Explicit GC Policy: expose GC only in supervised or hosted benchmark processes and call it only when the benchmark or profile requests it
    • hostProcess Execution Shape: support an internal host process around worker-pool execution for Node/V8 flags, profiling, debugging, and benchmark isolation needs
    • Benchmark Budget Surface: express checked-in budgets and diagnosticMetrics, generate ordinary assertion failures after aggregation, and reject missing or undeclared metric coverage
    • Calibration And Normalization: normalize benchmark results for comparability
    • Process And PTY Benchmarks: external process and terminal-oriented benchmark support
    • Benchmark Strategy Contract Acceptance: cover cold-module strategies, sample timelines, calibration commands, raw/normalized identity, cleanup, and generated budgets through public/direct and runner paths.
    • Dogfooded Startup Guardrails: after benchmark commands are usable, add Overkill benchmark checks for root @overkill-dev/test import cost and ordinary microtest CLI startup
  • Integration Test Family

    • Integration Profile Model: support testFamily: 'integration' as a later runner profile family
    • Direct Integration runIfMain() Execution: decide whether direct-file execution can support integration profiles without losing supervised isolation
    • Integration Runtime Environment: define temp directories, local services, process use, network use, and failure artifacts without applying microtest capability restrictions
    • Integration Resource Defaults: model app servers, registries, HTTP/IPC workflows, and richer fixtures as explicit resources
    • Integration Retry Policy: allow integration profiles to opt into retries with attribution-preserving artifacts
    • Integration Failure Artifacts: support transcripts, logs, service output, and scenario witnesses for integration resources
  • Advanced Runner Execution

    • Worker Pool Execution: support processModel: 'worker-pool' as the bounded multi-worker substrate
    • hostProcess Execution Shape: resolve a supervised host process around worker-pool execution when runs need Node/V8 arguments, profiling, debugging, or hosted benchmark isolation
    • Worker Lifecycle: support reusable workers (make it the default for integration tests) and fresh workers that exit after one assigned work unit
    • Per-File Disposable Isolation: define process-per-file as worker pool plus file granularity plus fresh-worker-per-unit lifecycle
    • Work Unit Planning: introduce WorkId, WorkUnit, WorkUnitId, PlacementPlan, and PlacementTrace
    • Work Distribution Strategies: support file, case, and named group distribution
    • Named Group Distribution: define file-set references, group granularity, per-group scheduling/order/lifecycle overrides, unmatched handling, and overlap errors
    • Case-Count Balanced Placement: make worker-pool placement balance by selected case count with seeded tie-breaks
    • Resource-Aware Placement: lower resources into serial keys, capacity weights, affinity keys, and fault domains
    • Dynamic Scheduling Concepts: specify duration history, runtime reprioritization, pending-unit splitting, hedging, compatible batching, and warm-lane affinity as later execution concepts
    • Duration History Index And Placement: persist compact WorkId duration samples under runtimeStateDir and feed duration-history-balanced placement with stale/missing fallback
    • Dynamic Lease Dispatch: consume frozen placement through pending-work leases, runtime reprioritization, and hard-constraint-preserving reassignment
    • Pending-Unit Splitting: split eligible not-started units along WorkId boundaries with derived trace identities linked to the parent WorkUnitId
    • Hedged Duplicate Execution: duplicate only explicitly idempotent or disposable isolated stragglers, cancel slower duplicates, and fail conflicting outcomes
    • Compatible Batching: batch identical execution envelopes without merging WorkUnit, result, artifact, or duration identities
    • Warm-Lane Affinity: prefer warmed reusable workers as a soft tie-breaker after hard constraints, capacity, spreading, and fairness
    • Runtime And Executor Affinity: expand runtime/workload matrices before placement and assign only to compatible executors
    • Worker Count And Capacity Controls: implement default sizing, host caps, profile caps, weighted capacity, and --workers N
    • Supervised Process Execution: keep supervised-process as a single child boundary, not a worker pool
    • Single-Worker Serial Execution: support one worker or process for benchmarks, exclusive resources, and constrained runs
    • Frozen Placement Assignment: freeze work units, constraints, ordering, sharding, and initial executor assignment before execution
    • Placement Trace And Duration Facts: record worker assignment, retries, recovery decisions, hedging decisions, and work-unit duration facts; boundary and worker timing details come from Runner Timing Diagnostics
    • Execution Plan Resolution: reject incompatible process model, lifecycle, distribution, runtime, and resource combinations before execution
  • Higher Test Layers

    • @overkill-dev/simulation: generic simulation package with finite scenario catalogs and simulated HTTP server definitions
    • @overkill-dev/model: model-based testing package
    • @overkill-dev/differential: differential testing package
    • @overkill-dev/linearizability: concurrency/consistency package
    • linearizability(...): first concurrency/consistency helper
  • Property-Based Testing

    • Desired concept: Property-Based Testing Resolution; concept PR: #556
    • Property Authoring And Distribution: implement @overkill-dev/property, the @overkill-dev/test/property subpath, first-class property({ title, generator, test }) values, and advanced nested withProperties(...) / scope.forall(...) authoring without runtime subtests or heavier root microtest imports
    • Property Family And Policy: implement testFamily: 'property', microtest-like capability restrictions, collection-time policy validation, generation and shrink budgets, timeout/resource supervision, and no code coverage
    • Fluent Generators: expose primitive and bounded recursive generators with map, flatMap, and filter, using fast-check behind the independent Overkill API; custom generators compose these surfaces and inherit shrinking
    • Generator Validity And Reconstruction: preserve dependent domains, size/depth limits, deterministic context, and fresh mutable input graphs for every trial, replay, example, and confirmation
    • Trial Phases And Accounting: execute checked-in corpus, local cache, explicit examples, bounded automatic edges, and new exploration sequentially within each property; keep phase counts separate and allow ordinary runner parallelism between properties
    • Exhaustive Domains And Input Coverage: exhaust small known finite domains within policy limits; count required coverage over accepted new exploration or the complete exhaustive domain, with inconclusive results when requirements remain unmet
    • Targeted Exploration: use finite labeled scores to guide bounded deterministic exploration without making score reachability a pass condition
    • First-Failure Shrinking: stop generation at the first failure, preserve the first failed assertion or body error through automatic failure identity with an explicit stable-key override, and retain unrelated failures as diagnostics
    • Failure Confirmation And Termination: report non-reproducible failures and stop shrinking; bound candidate production, rejection, reconstruction, duplicates, and evaluations while preserving confirmed evidence and native cleanup or supervision outcomes
    • Versioned Graph Codecs: reuse devalue behind reviewable JSON codecs that preserve shared references, cycles, and supported value state; provide explicit custom codecs and reject lossy or unsupported inputs
    • Decoder Compatibility And Migration: retain supported historical decoders, write the current format, validate compatibility fixtures, and migrate only through explicit operations
    • Regression Corpora: save confirmed replayable failures in the bounded local cache and explicitly promote selected witnesses into a checked-in corpus; fail incompatible checked-in entries, diagnose and skip incompatible local entries, and replay regressions independently of current generation limits
    • Corpus API And CLI: implement inspection, promotion, and migration through typed runner APIs and the overkill corpus namespace, with decoded previews, atomic writes, stable target identity, and no implicit repository changes
    • Witness Replay: replay first-class properties without generation or shrinking through normal lifecycle and supervision; replay nested properties by executing the enclosing case and substituting the saved input at an unambiguous target
    • Property Results And Diagnostics: keep one engine case and boundary assertion per property; report phase counts, seeds, coverage, non-reproducibility, and shrink termination without claiming a globally minimal counterexample
    • Generator Preview And Relation Checks: provide bounded sampleGenerator(...) previews and metamorphic relation helpers through @overkill-dev/property
    • Property Integration Validation: cover dependent shrinking, recursive limits, non-injective mappings, impossible filters, mutation isolation, deterministic streams, exhaustive accounting, codec graph fidelity, historical fixtures, corpus policies, nested replay, and supervision on the pinned runtime
  • Property-Derived Contract Acceptance

    • Prerequisites: Property-Based Testing and Executable Test Families; model precedes linearizability.
    • Model Contract Acceptance: validate predicted traces before system creation, independent inputs, enabled-transition shrinking, and stored-trace replay beyond generation limits.
    • Differential Contract Acceptance: validate sequential implementations, fresh inputs, returned/threw comparison, control-error propagation, and input replay.
    • Linearizability Contract Acceptance: validate real-time order/pending operations, rejection versus inconclusive search exhaustion, bounded cancellation/draining, lane witnesses, and non-reproducibility.
  • Builder Layer

    • Builder-Oriented Composition Surface: preserve an explicit layer for assembling higher-level testing APIs from lower-level contracts
    • Third-Party DSL Support: allow specialized test APIs to be built without forking the runner or engine
  • Standard Distribution And Optional Packages

    • Standard Distribution Model: @overkill-dev/test is the one-package normal user install and standard distribution
    • Standard Stack Dependencies: @overkill-dev/test depends directly on the standard stack so users do not align package versions by hand
    • No Preset Packages: remove @overkill-dev/micro, @overkill-dev/default, @overkill-dev/integration, and @overkill-dev/all from the concept
    • Optional Package Policy: property stays in the standard distribution; browser, model, differential, linearizability, type-test, mutation, ESLint, AWS CDK, contract, accessibility, and similar domain packages remain separate installs
    • Explicit Optional Integration: optional packages integrate through typed profiles, runtime/resource factories, reporters, baseline adapters, or authoring helpers
    • No Dynamic CLI Extension: installing an optional package does not mutate the overkill CLI, trigger package-name discovery, or widen top-level config through module augmentation
  • Mutation Testing And Stryker

    • Desired concept: Mutation Runner Contract; concept PR: #540
    • @overkill-dev/stryker-runner: publish the optional first-party Stryker TestRunner plugin above the public @overkill-dev/run API; leave mutation planning, sandboxing, scores, thresholds, incremental reports, and mutant parallelism to Stryker
    • Microtest Profile Selection: reuse an existing Node microtest profile; infer it only when exactly one is eligible, require a name when several exist, and reject missing, unknown, non-microtest, browser, and mixed-family selections before collection
    • Mutation Execution Policy: expose the serial scheduling override while preserving configured discovery, loaders, capabilities, resource policies, and process model; disable ordinary V8 coverage, Overkill retries, and baseline updates
    • Frozen Campaign Policy: resolve authoritative discovery, loader, selection, permission, and resource facts outside the execution module graph; transport them to execution hosts and reject conflicting reevaluated policy
    • Mutation Target Boundaries: exclude configuration, test definitions, and adapter code from mutation targets; normalize sandbox origins without executing original files outside the sandbox
    • Public Runner Lifecycle: support initialization before user imports, execution of a collected plan without reimporting modules, and fresh scopes, observers, and result collection for each invocation without bypassing test policy
    • Configured Process Models: run normal in-process profiles inside Stryker's reusable plugin worker and supervised profiles inside the normal restricted test child; keep case execution serial while Stryker parallelizes mutants
    • Fresh-State Configuration: default freshState to false; when enabled, reset the complete transitive module graph and mutable state for every invocation through a fresh execution boundary; document reused-graph cache limitations
    • Environment Reload Contract: report reloadEnvironment: false for reused native ESM execution and true for fresh-state or supervised execution; honor Stryker-requested reloads and static-mutant transitions
    • Instrumentation Activation: initialize Stryker's namespace in the executing process, activate static mutants before user imports, and activate runtime mutants after unmutated collection but before cases
    • Instrumentation Lifecycle: reset counters, active mutation, test identity, and hit-limit state for each invocation, including error paths, while preserving namespace and counter-object identities retained by reused instrumented modules
    • Case-Level Mutant Coverage: set and clear currentTestId at awaited local case boundaries through cleanup in both process models; retain outside-case reads as static coverage and transport complete counters to Stryker
    • Coverage Modes And Fallbacks: support Stryker perTest, all, and off; run the full eligible suite for static or uncertain attribution and never infer NoCoverage from missing transport data, missing tests, or V8 coverage
    • Stable Mutation Test IDs: encode structural CaseId, parameter identity, canonical project-relative origins, and distinct valid WorkId values without depending on display names or source positions
    • Catalog And Exact Selection: establish the complete dry-run catalog, validate identities and selection metadata on every invocation, execute requested IDs exactly, and return errors for unknown or duplicate IDs, drift, unexpected skips, collection failures, or empty mutant selections
    • Structured Dry-Run Results: report stable IDs, durations, successes, failures, and intentional skips; return timeout or infrastructure errors without producing a mutation score
    • Mutant Result Classification: return Killed only for safely attributed eligible-case failures, including assertions, body errors, and policy violations; keep pre-case, unattributed, infrastructure, transport, reporter, persistence, and inconclusive failures as Error
    • Result Precedence: enforce Error > Timeout > Killed > Survived, including cleanup timeouts and continued execution after a kill; preserve earlier case failures in diagnostics
    • Bail And Completion: stop admitting cases after a confirmed kill unless Stryker sets disableBail; finish admitted work, cleanup, observer shutdown, and result delivery, and report actual executed counts and killing IDs
    • Timeout Enforcement: honor both profile case timeouts and Stryker's complete invocation deadline, including startup and cleanup; classify exceeded hit limits as timeouts and terminate and reap adapter-owned hosts even for synchronous infinite loops
    • Native Guard Ownership: retain console, permission, uncaught-error, rejection, and async-leak checks without monkey-patching; exclude Stryker logging, transport, and bootstrap from test-policy attribution and stop owned observations when each run ends
    • Surviving Worker Reuse: close scopes, observers, and invocation bookkeeping after unsafe invocations without automatically retiring a surviving plugin worker or enabling fresh state; preserve safe attribution and leave dead-worker, deadline, and required-reload recovery to Stryker
    • Campaign Reproducibility: accept or generate one campaign seed shared across dry runs, mutants, workers, and replacements; expose it for reproduction and derive case seeds from stable executable identity independently of selection and scheduling
    • Reporting And Default Persistence: suppress all ordinary and custom Overkill reporters and disk writes for results, run records, duration history, artifacts, and reports; retain structured adapter results in memory and leave Stryker campaign output under Stryker configuration
    • Opt-In Diagnostic Persistence: provide off, failures, and all with off as default; persist only available, profile-permitted artifacts and transcripts with campaign, mutant, invocation, and case identity, retain existing limits, and classify delivery or write failures as infrastructure errors
    • Mutation Integration Validation: cover sandbox-relative and parameterized identities, runtime and static activation, exact selection, catalog drift, coverage transport, both process models, reused and fresh state, policy ownership, precedence, timeout cleanup, native early exit, abort, and termination without results on the pinned Node version
    • Mutation Accuracy And Cost: compare optimized and full-suite results on repeatable suites and measure reuse versus fresh-state costs across startup, collection, and execution
    • Mutation User Documentation: document installation, Stryker configuration, profile inference, execution overrides, coverage modes, seeds, result mapping, persistence, process guarantees, and reused-state limitations
  • Ecosystem Integrations

    • Native Type-Test Support: implement documentation/desired-concept/authoring/testing-models.md after the family contract above. Checked adapter milestones record the earlier implementation; the original-source contract supersedes its generated-input direction.
      • Native Assertion Contract: define inline type assertions using Overkill composition and assertion style, with one source file and one description per case; specify supported assertion references, generic bodies, tables, and macro semantics. Integration direction and API findings: Define the @overkill-dev/type-test concept #672.
      • Best-Effort Generated Input: use the existing public TSTyche API with adapter-owned, dot-prefixed sibling modules preserving source extensions; map original and generated locations, repair unambiguous direct self-imports and translated-module imports, and document identity limits without waiting for upstream changes.
      • Explicit Checker Configuration: derive configurations per original project and compiler version, preserve effective options and references, include generated inputs explicitly, and verify the expected project and compiler instead of accepting baseline fallback.
      • @overkill-dev/type-test: publish one optional package for native assertion references, typed profile wiring, source translation, and TSTyche execution; keep checker dependencies and lazy execution loading outside @overkill-dev/test, and provide direct checking of engine roots without requiring the high-level DSL.
      • Explicit Adapter Contract: add generic runner hooks for profile validation, collection, execution lowering, and frozen facts; register the optional adapter explicitly without global side effects or module augmentation, recognize assertion exports through import aliases, and reject unsupported profile options including coverage.
      • Source Ownership And Translation: associate assertion-bearing bodies with stable source anchors across engine cases, test, suites, macros, tables, and supported wrappers; translate supported inline references without changing lexical context, contextual typing, or native diagnostics, and retain original-source mappings.
      • Type-Test Execution: identify checker-backed cases during collection and lower their frozen plan into external checker work without invoking ordinary builder bodies; account explicitly for shared static evidence and reject unsupported forms or missing checks.
      • Compiler Planning: freeze source fingerprints, case-to-assertion ownership, original project configuration, and exact TypeScript versions; put compiler variants in WorkId, retain logical CaseId, and record reproducible compiler and input facts.
      • Dependency And Compiler API Assessment: validate a supported public comparison interface or document the focused comparison work needed for the specified semantics; evaluate TSTyche reuse/extraction and supported stable TypeScript plus a pinned 7.1 prerelease without private imports or compiler patching.
      • Public Package And Compiler Selection: expose family, sameType, assignableTo, and assignableFrom with the documented expression/negation forms; resolve original projects and explicitly installed local/package compiler selections, preserve exact compiler identity, and remove automatic compiler substitution/downloads from checking.
      • Original-Source Relations: replace sibling modules, derived projects, import repair, and generated mappings with original-program assertion ownership and analysis; implement the documented equality/assignability rules, retain generic and table semantics, and return inconclusive for unsupported evidence without runtime execution or generated input.
      • Exact Selection And Diagnostics: bind ordered selected work to its original assertion sites, preserve shared-body ownership and original locations, define no-assertion/skip/inconclusive outcomes, buffer passing results until input validation succeeds, and distinguish relation failures from project/checker errors. Dedicated JSX/decorator and precise expected-diagnostic assertions are outside the initial catalog.
      • Compiler Lifecycle: implement preparation/check deadlines, cooperative cancellation, hard-deadline process ownership, disposal, repeated/watch invocations, and source/configuration/resolution drift invalidation without file emission or referenced-project builds.
      • Persistence And Reproduction: retain family/compiler identity, original inputs, variant ownership, diagnostics, and outcomes in records and failure selection; validate compatible implementations and source inputs before reproduction while preserving earlier record semantics.
      • Type-Test Acceptance: cover direct engine and runner parity, inference and overloads, the relation semantics table, source ownership across composition layers, matrices and variant isolation, exact selection, shared/static bodies, nominal identity/self-imports, missing evidence, cancellation/drift/disposal, and reporter/record delivery; assert no replacement checker inputs are created.
      • User Documentation: explain the public family and assertion exports, installed compiler aliases, direct engine lifecycle, profiles, source requirements, supported forms, diagnostics, and reproduction; remove obsolete generated-input and full-matcher-parity instructions.
    • @overkill-dev/eslint-rule-test: ESLint rule-testing package
    • @overkill-dev/eslint-plugin: implement the settled static authoring catalog in documentation/desired-concept/authoring/higher-test-layers.md; users choose rule enablement and severity.
      • Binding Analysis: follow renamed imports, destructuring, aliases, local object properties, shadowing, and assignment rebinding using binding identity.
      • Authoring-Layer Coverage: define supported forms per rule across @overkill-dev/test, custom facades, assertion helpers, and direct engine usage; distinguish test entries from support modules.
      • no-scope-escape: prevent forwarding or escaping scope, assertion/require facades, planning APIs, and extracted assertion methods to ordinary helpers.
      • no-single-use-trivial-assertion: reject local, unexported, single-reference wrappers around one built-in check while preserving useful composites, narrowing, and diagnostics.
      • no-self-comparison: reject equality checks comparing the same stable binding with itself.
      • no-constant-actual-assert: detect likely reversed actual/expected operands.
      • prefer-semantic-assertions: prefer informative built-in assertions without changing comparison or evaluation semantics.
      • no-shared-mutable-test-state: detect shared mutation through bindings and object aliases while respecting declared resource lifetimes.
      • no-reexported-test-node: reject forwarding another module's root unchanged while allowing local suite composition.
      • consistent-run-if-main: enforce the configured direct-file execution policy.
      • no-setup-in-error-assertion: prevent separate setup operations inside throws/rejects callbacks from satisfying the expected-error matcher.
      • no-duplicate-assertions: detect adjacent identical checks on stable, side-effect-free operands.
      • prefer-table: suggest tables for independent repeated cases while preserving stateful scenarios and intentional node composition.
      • no-conditional-assertions: detect conditional checks in test bodies only; allow branches in custom assertion implementations and straight-line checks after scope.require, which gates by throwing.
      • Static Authoring Acceptance: cover supported authoring layers, aliasing/rebinding, false positives, and semantic preservation; do not duplicate runtime/discovery checks.
    • @overkill-dev/aws-cdk: first package-specific assertion adapter wrapping @aws-cdk/assertions
    • Contract Adapters: other adapters for external assertion/testing ecosystems
    • Stable Machine-Readable APIs: keep reporter/results/planning data usable for IDE and MCP integrations
  • Browser And Remote Expansion

    • Prerequisites: Shared Contract Extensions, Executable Test Families, and resource/runner/benchmark contracts. Distributed merging also requires Sharded Result Merging.
    • Execution And Session Foundations
    • Browser Capability Proofs: prove public controller/provider paths for redirects/workers/sockets, cache/SW attempts, warm caching, native routes, and remote reachability; publish supported configurations before dependent guarantees.
    • Browser Runtime Package: first optional browser-oriented runtime/package
    • Browser-Compatible Engine Surface: separate browser-safe engine/assertion/authoring/resource imports from Node-only attribution, filesystem, reporting, and clocks through a platform adapter.
    • Browser Execution Modes: define Browser-executed bodies inside a browser document and Host-driven browser automation bodies on an execution host, using shared provisioning and provider-native adapters.
    • Browser Runtime Integration: browser support integrates through fixtures/resources and explicit project profiles, not a browser test family or CLI mutation
    • Browser Family Boundaries: require integration for browser correctness and benchmark for browser performance; reject browser runtimes and host-driven browser automation in microtest/property families regardless of provisioning.
    • Native Integration Boundaries: expose provider-native actions/locators with explicit provider identity; Overkill owns lifecycle, guards, checks, measurements, and reporting without a universal action API.
    • Modern Protocol Scope: exclude a first-party Classic WebDriver command adapter; assess public Playwright BiDi support before adding clients for demonstrated gaps, while permitting necessary provider/session setup.
    • Browser Collection Policy: collect browser-free definitions by default; offer one explicitly configured collection browser for top-level browser imports, with matrix expansion remaining planning metadata.
    • Mixed-Placement Browser Suites: permit compatible modes in the same suite or file within one family; partition execution by compatible placement and use explicit module boundaries for incompatible import graphs.
    • Cross-Realm Resources: apply shared owner/consumer/observer references through one resource declaration, preserving wrappers and same-realm resources without duplicate profile bindings.
    • Browser Harness Delivery: own module delivery, bootstrap/source maps, server and browser/result channel through replaceable tool integrations; require no prebuilt app and preserve direct/runner lifecycle.
    • Browser Channel Fencing: validate control, harness/results, and app reachability separately; fence assignment/session/document/work/attempt messages and reject stale/lost evidence without action replay.
    • Browser Attempt Isolation: create a fresh context/session per integration-test attempt while permitting browser-process reuse; keep backend reset and benchmark sample state explicit.
    • Browser Body Lifetime: bind browser-executed bodies to their execution document; replacing it before completion fails the attempt, while host-driven tests handle app-navigation/reload journeys without resumable browser phases.
    • Functional Capability Compatibility: fail selected ordinary-browser targets lacking required capabilities unless an intentional skip is authored.
    • Browser-Local Native Input: provide bounded provider-specific keyboard/pointer assistance with target/document/attempt identity, capabilities, readiness, timeouts, and cancellation; no general controller RPC.
    • Protected Sessions And Evidence
    • Network Transport Capabilities: define standard HTTP(S)/WebSocket guard coverage; require explicit validated capabilities for other transports and reject unsupported requested configurations.
    • Request Prevention And Guard Outcomes: prevent denied transmissions; unexpected violations fail resource checks without automatic retries; bounded expected denials stay blocked, only affected work stops, and lost observation is infrastructure failure.
    • Forbidden Request Attempts: apply network guards to forbidden application request attempts even when cache or service workers supply responses without outbound traffic; require reliable observation alongside prevention.
    • Fine-Grained HTTPS Prevention: support URL-path/method enforcement through validated capabilities; reject unsupported policies and require explicit configuration for TLS inspection.
    • Guarded Service Workers And Caching: support service-worker journeys and warm-cache benchmarks with compatible prevention; reject incompatible configurations without silently disabling app behavior, caching, or protection.
    • Default App Guards: guard first-party app/session resources including benchmarks; keep bare execution lightweight and honor loopback defaults, external grants, explicit deny precedence, and console/error rules.
    • Expected Browser Diagnostics: provide bounded, count-checked expectations registered before actions; fail missing/excess events without granting network permissions or erasing failures.
    • Standalone Guard Outcomes: closing standalone guarded sessions throws structured findings while preserving other errors; engine-managed sessions use ordinary failed checks.
    • Browser Record And Reproduction Compatibility: retain placement, module/harness build, guard/scenario/startup and collector/reset facts; validate reproduction and versioned historical readers.
    • Browser Extension Resources: provide validated browser/profile provisioning with protection before background startup, fresh profile/process isolation where needed, and declared benchmark state.
    • Measurements And Quality Adapters
    • Shared Browser Benchmark Authoring: retain ordinary authoring and one sampling/budget lifecycle; compose both body placements with independent collectors and explicitly browser-local timed operations.
    • Browser Sample Attribution: bind collectors/guards to warmup/measured invocations, finalize before aggregation, and retain ambiguous/incomplete lifetime evidence.
    • Browser Benchmark State: require declared retained/reset browser and app state per workload, with presets and state-aware comparison identity.
    • Action-To-Result Latency: measure correlated browser input to an authored DOM condition or app mark; keep host command duration and rendering evidence separate, with no silent timing substitution.
    • Browser Metric Collectors: adapt timing/marks, input, paint/navigation, requests, rendering/animation/scroll, CPU/memory, and named audits with provenance; incomplete required metrics cannot pass budgets.
    • Diagnostic Benchmark Reruns: make heavier capture reruns opt-in, preserve original failures, and retain separate measurement/attempt identities.
    • Page Audit Checkpoints: expose heavier page-state audits through explicit reusable assertions/macros while continuous app guards remain enabled by default.
    • Interactive Accessibility: provide reusable native keyboard/focus assertions for tabbing, activation, modal containment/restoration, skip links, and route/form-error focus alongside scanner-backed semantic/ARIA checks.
    • Accessibility Preferences And Geometry: adapt contrast/target geometry, light/dark/forced-color/reduced-motion, touch, text enlargement, and zoom/reflow with actual environment provenance.
    • Screen-Reader Adapters: provide optional first-party actual reader automation for validated OS/browser/reader combinations, reusing existing libraries and owning required desktop focus/settings/exclusive access as resources.
    • HTML Conformance Inputs: offer validation of the actual original response and rendered-DOM checkpoints separately, retaining input provenance without silently refetching another document.
    • CSS Conformance Inputs: adapt validation of explicit external, inline, and generated stylesheet inputs, preserving source/coverage limitations separately from layout or accessibility correctness.
    • Semantic Data Checks: adapt microformats and JSON-LD/Microdata/RDFa, schema.org/profile checks, and visible-app fact assertions; distinguish syntax, semantic processing, vocabulary, and provider rules.
    • Discoverability And SEO Checks: assert status/redirects, source/rendered metadata, titles, language, canonical/hreflang, robots, and links without promising indexing, ranking, or rich results.
    • Accessibility Attachments: retain bounded native reports/transcripts with explicit checkpoint, target/input scope, tool/rule versions, and coverage; distinguish violations, incomplete findings, unavailable capabilities, and infrastructure failures.
    • Visual Baselines: checked-in visual comparison artifacts
    • Remote Execution And Acceptance
    • Remote Execution Planning: model remote/distributed execution in orchestration
    • Distributed Result Merging: merge results from remote/distributed runs
    • Dogfooded HTML Reporter Browser Tests: exercise benchmark HTML report loading, chart rendering, workload selection, zoom, tooltips, and offline behavior through the first-party browser runtime.
    • Browser Testing Guidance: recommend unit tests without a real DOM and integration tests against the fully rendered app; provide no first-party component mounting API or component-testing DSL, while allowing user-supplied rendering setup.
    • Browser Scope Guidance: document supported mobile browsers, authored worker use, consumer native-app integration, rejected standalone test export, and retained offline reports.
    • Browser CLI Extension Deferred: use project-named profiles such as ui-browser with ordinary overkill run, overkill list, and baseline verbs until a dedicated command need is proven
    • Browser Contract Acceptance: verify both modes, mixed/split-module suites, document loss, cleanup/guards, standalone/direct-runner parity, local/remote evidence, cache/SW, audits/readers/extensions, and rejected boundaries.
  • Concept And Documentation Acceptance

    • Public Contract Conformance: validate implementation against the owning desired-concept contracts, including public/direct/runner parity, rejected boundaries, records, baseline staleness, empty selections, cleanup, and supported platforms.
    • User Guides And Reference: complete quick start, common/advanced workflows, package/CLI/API reference, and examples per the documentation strategy; reconcile walkthroughs, types, and loader guidance.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions