Skip to content

Add a feature registry and round-trip tests for collection URL params - #4209

Closed
andguy95 wants to merge 10 commits into
an-product-in-collection-standard-pathnamefrom
an-fast-check-url-roundtrip
Closed

andguy95 wants to merge 10 commits into
an-product-in-collection-standard-pathnamefrom
an-fast-check-url-roundtrip

Conversation

@andguy95

@andguy95 andguy95 commented Oct 19, 2026 •

Copy link
Copy Markdown
Collaborator

Warning

Prototyping not meant for production. Exploring property based testing.

TL;DR: packages/hydrogen/features.yaml records what Hydrogen promises, which tests prove each promise, which mutants those tests are known to catch, and which gaps are knowingly unproven. pnpm lint enforces the registry's shape, pnpm test runs every listed proof, and --falsify applies each listed mutant and fails if a proof stays green under it. The first entry is the collection URL round-trip, proven by property tests over serializeCollectionParams and parseCollectionParams and falsified by ten mutants of url.ts.

Humans are good at stating what a system does and what must hold true for it. Agents are good at building toward a stated target. The registry is the ledger between the two. A feature's promise is the human statement, verified_by names the tests that make it checkable, falsified_by names the bugs those tests are proven to catch, and known_gaps is where a counterexample lands until a human marks it accepted with a reason or issue with a link. The check turns that ledger into something CI defends: a promise cannot be shipped without a passing proof, a property test cannot exist without a promise that claims it, and a proof that cannot fail is caught by its mutants.

The collection URL params are the first entry because serializeCollectionParams and parseCollectionParams are two hand-written halves of one contract, and nothing checked that they agree beyond a handful of fixed examples. It is the smallest place in Hydrogen where two functions must be exact inverses.

Before

Nothing failed if a property test was orphaned, skipped, or empty, and the only record of what url.ts promises was its comments.

After

$ node scripts/check-features.ts --proofs
 ✓ src/core/collection/__tests__/url.property.test.ts (2 tests)
features.yaml: all checks pass

With every it in the proof file changed to it.skip:

$ node scripts/check-features.ts --proofs
      Tests  2 skipped (2)
features.yaml: 1 problem
  collection.url-params: src/core/collection/__tests__/url.property.test.ts has no passing tests

Mutants of url.ts that the proof must catch:

$ node scripts/check-features.ts --falsify
  killed collection.url-params: mutant "the price minimum is written to the lte param"
  killed collection.url-params: mutant "out-of-stock availability is read as in stock"
  killed collection.url-params: mutant "descending sort_by values parse as ascending"
  killed collection.url-params: mutant "created-descending maps to the PRICE sort key"
  killed collection.url-params: mutant "filter key parts are not percent-decoded"
  killed collection.url-params: mutant "a product metafield namespace may swallow dots"
  killed collection.url-params: mutant "the manual sort key is never written to the URL"
  killed collection.url-params: mutant "ascending and descending suffixes are swapped when writing sort_by"
  killed collection.url-params: mutant "tags are read from the wrong param"
  killed collection.url-params: mutant "the ID sort key loses its direction suffix"
features.yaml: all checks pass

Falsifying found two weaknesses in the first version of the proof. It compared filters with filterEquals and derived the expected reverse from isDirectionalSortKey, both from url.ts, so a mutant of either moved the expectation along with the result and survived. The proof now sorts both sides by their JSON and compares with toEqual, and lists the four directional keys itself.

What this changes

  • packages/hydrogen/features.yaml: one entry, collection.url-params, with its promise, entry point, proof, ten mutants, e2e group, and six accepted gaps.
  • scripts/check-features.ts: ids are unique and shaped area.feature-name, every listed path exists, shipped features name at least one proof, every *.property.test.ts under packages/hydrogen/src is claimed by a feature, depends_on points at registered ids with no cycles, every known gap has exactly one of accepted or issue, and every mutant targets an entry point. With --proofs it runs vitest on the union of verified_by paths and fails a path that was not collected, has failing tests, or has no passing tests. With --falsify it also applies each mutant's find and replace (the fragment must occur exactly once), reruns that feature's proofs, restores the file, and fails a mutant that survived or that broke the run instead of failing it.
  • Wiring: pnpm lint runs the static check. pnpm test runs it with --proofs. yaml is added as a root devDependency to parse the registry.
  • src/core/collection/__tests__/url.property.test.ts with two properties, plus fast-check as a devDependency of packages/hydrogen.
    • parseCollectionParams(serializeCollectionParams(p)) equals p for every canonical CollectionParams. Filters compare as sorted arrays under toEqual, because the parser emits them in a fixed kind order and that order is not part of the contract.
    • parseSortByValue(getSortByValue(k, r)) returns k for all eight ProductCollectionSortKeys, keeping r only for the four directional keys the test lists. A compile-time check fails if a sort key is added to the Storefront API type but not to this list.
    • Strings are any grapheme, with the characters the URL scheme splits on drawn half the time. Price bounds are any double except NaN and -0.

Developer impact

No changeset. The registry, check, and tests are repo-only, and no runtime code in @shopify/hydrogen changes. The changesets reminder bot will comment, and that comment can be ignored.

Out of scope

  • verified_by proves a file passes, not that it tests the stated promise. The promise text and the test are linked by name only. falsified_by narrows that: the proof is known to catch the listed bugs.
  • --falsify is not in pnpm test. It runs each mutant as a separate vitest process, about eight seconds for ten mutants today. Adding it to CI is a separate decision.
  • Status transitions are not gated beyond "shipped needs a proof". Nothing yet reads planned entries and does work.
  • The six known_gaps are places where serialize and parse disagree. None is reachable through the collection store, so each is accepted with its reason in the yaml. Treating any as a bug in url.ts is a separate change.
  • parseLocalizedNumber reads window.Shopify.locale while the serializer always writes String(n). In a storefront whose locale groups with ., a 12.5 bound parses back as 125. The proof cannot reach that branch from node. It is tracked outside this PR and will land in known_gaps as an issue once it has a link.

Risk

  • The proofs run twice in CI, once in the hydrogen vitest run and once under --proofs. About one second today.
  • The check reads vitest's JSON reporter. A format change in a vitest upgrade would report every proof as "was not collected by vitest", which is loud rather than silent.
  • The round-trip property runs 500 random cases per run. A failure in CI prints the seed and the shrunk counterexample, so it is reproducible, but a run may surface an input that earlier runs did not.
  • Mutants are textual. A reformat of url.ts that touches a find fragment makes --falsify fail with "does not occur exactly once", which is loud. The mutation runs in the working tree and restores the file in a finally, so only a hard kill mid-run leaves url.ts mutated.

How to Test

  1. Run pnpm install.
  2. Run node scripts/check-features.ts --proofs and confirm it ends with features.yaml: all checks pass.
  3. Run node scripts/check-features.ts --falsify and confirm ten killed lines and all checks pass.
  4. In packages/hydrogen/src/core/collection/__tests__/url.property.test.ts, change the second it( to it.skip(. Rerun step 3 and confirm mutant "the ID sort key loses its direction suffix" survived every proof. Revert the file.
  5. Change each it( to it.skip(. Rerun step 2 and confirm the check names collection.url-params and the file with has no passing tests. Revert the file.
  6. Create an empty packages/hydrogen/src/core/cart/__tests__/cart.property.test.ts. Run node scripts/check-features.ts and confirm it reports the file as not claimed by any feature. Delete the file.

@shopify

shopify Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Oxygen deployed a preview of your an-fast-check-url-roundtrip branch. Details:

Storefront Status Preview link Deployment details Last update (UTC)
Skeleton (skeleton.hydrogen.shop) ✅ Successful (Logs) Preview deployment Inspect deployment October 21, 2026 1:05 AM

Learn more about Hydrogen's GitHub integration.

@andguy95 andguy95 changed the title Add round-trip property tests for collection URL params Add a feature registry and round-trip tests for collection URL params Oct 20, 2026
@andguy95
andguy95 force-pushed the an-fast-check-url-roundtrip branch from c5aa2b5 to 3c1dd34 Compare October 20, 2026 22:38
@andguy95
andguy95 changed the base branch from main to an-price-filter-locale-parse October 20, 2026 22:38
@andguy95
andguy95 added this pull request to stack #4224 October 20, 2026 22:39
@andguy95
andguy95 force-pushed the an-fast-check-url-roundtrip branch from 3c1dd34 to cb69881 Compare October 21, 2026 00:56
@andguy95
andguy95 removed this pull request from stack #4224 October 21, 2026 00:57
@andguy95
andguy95 changed the base branch from an-price-filter-locale-parse to an-product-in-collection-standard-pathname October 21, 2026 00:57
@andguy95
andguy95 added this pull request to stack #4232 October 21, 2026 00:57
packages/hydrogen/features.yaml pairs each promise Hydrogen makes with the
tests that prove it and the gaps that are knowingly unproven. The check
enforces the registry's shape in `pnpm lint` and, with --proofs in
`pnpm test`, runs every verified_by file and fails when one is missing,
failing, or collects no passing test.
A proof that cannot fail proves nothing. Each feature in features.yaml
now lists falsified_by mutants: one-line find/replace edits to an entry
point that a proof must catch. `check-features.ts --falsify` applies
each mutant, reruns that feature's proofs, restores the file, and fails
on a mutant that survived or broke the run. It only runs once the static
check and the proofs are green, and it is not wired into pnpm test.

Running it against collection.url-params found two places where the
proof read its expectation from the code under test. Filters were
compared with filterEquals and the expected reverse came from
isDirectionalSortKey, so mutating either moved both sides. The proof
now sorts filters by JSON and compares with toEqual, and lists the four
directional keys itself. All ten mutants die.

Widening the generators to any grapheme and any finite double surfaced
three more serialize/parse disagreements, recorded as accepted gaps:
dotted metafield namespaces and empty keys, the three non-Liquid sort
keys, and a -0 price bound.
…filter order

The property compares filters after canonicalOrder, so the promise and the
test title now say so. Each exclusion the generators make, plus the locale
the node run pins to en, has a test that passes while the gap exists.
Fixing a gap fails its test, which forces the registry to change with it.

canonicalOrder compares code units instead of localeCompare so canonically
equivalent strings no longer tie. Taxonomy values may be empty; they
round-trip.
parseRegistry rejects absolute paths, dash-prefixed strings, and any
segment that is empty or .., so a verified_by entry can no longer reach
vitest as a flag and a mutant can no longer overwrite a file outside the
package. The static check now runs before anything is spawned.

--falsify copies packages/hydrogen (minus node_modules, which is linked
back) into a temp directory and points vitest at it with --root. The real
tree is never written, so SIGINT, a thrown error, or a failed restore
leaves nothing behind, and two runs or a watcher never see each other's
mutants.
verified_by now names a file and the it titles or describe paths in it.
--proofs and --falsify judge only those tests, so a green test elsewhere
in the file no longer stands in for the property. url.test.ts joins
verified_by as the anchor that pins both sides to literal Liquid URLs.

A known gap may carry shown_by, a test in the same shape that must pass
while the gap exists. Every gap in the registry has one, including the
locale gap the node run hides: a de storefront reads a serialized 12.5
as 125.

accepted now needs accepted_by, a @handle the check matches against
.github/CODEOWNERS. The handle names the reviewer GitHub will require
before the file merges, so an agent cannot close a gap on its own.
A team handle would let anyone on the team, or an agent writing the
team's name, close a gap. accepted_by now has to be an individual
@handle, and that person has to appear on the CODEOWNERS line for
packages/hydrogen/features.yaml itself. The * fallback and other lines
do not count. registryOwners is exported and tested on its own.

The existing acceptances are attributed to @andguy95, who is added to
that CODEOWNERS line. Confirm before merging.
Rebased onto an-price-filter-locale-parse, which makes the serializer's
own price output survive every shop locale. The gap's shown_by test
started failing, as the registry intends when a gap closes, so the gap
and its test go. The fix's locale tests in url.test.ts join verified_by
in their place.
Ctrl-C used to leave the report dir and the package copy under tmpdir.
Node died inside spawnSync before any finally ran. The parent now keeps
a handler so it survives the signal, and runProofs exits with the
shell's code for the signal once vitest has died from it, after removing
every live temp dir. A signal sent to the parent alone is still only
seen when the run ends; spawnSync gives no earlier point to see it.
@andguy95
andguy95 force-pushed the an-fast-check-url-roundtrip branch from cb69881 to 7312f47 Compare October 21, 2026 01:02
@andguy95 andguy95 closed this Oct 21, 2026
An error occurred while trying to automatically change base from an-product-in-collection-standard-pathname to main October 21, 2026 20:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant