Repository navigation
Conversation
Contributor
|
Oxygen deployed a preview of your
Learn more about Hydrogen's GitHub integration. |
andguy95
force-pushed
the
an-fast-check-url-roundtrip
branch
from
October 20, 2026 22:38
c5aa2b5 to
3c1dd34
Compare
andguy95
force-pushed
the
an-fast-check-url-roundtrip
branch
from
October 21, 2026 00:56
3c1dd34 to
cb69881
Compare
andguy95
changed the base branch from
an-price-filter-locale-parse
to
an-product-in-collection-standard-pathname
October 21, 2026 00:57
packages/hydrogen/features.yaml pairs each promise Hydrogen makes with the tests that prove it and the gaps that are knowingly unproven. The check enforces the registry's shape in `pnpm lint` and, with --proofs in `pnpm test`, runs every verified_by file and fails when one is missing, failing, or collects no passing test.
A proof that cannot fail proves nothing. Each feature in features.yaml now lists falsified_by mutants: one-line find/replace edits to an entry point that a proof must catch. `check-features.ts --falsify` applies each mutant, reruns that feature's proofs, restores the file, and fails on a mutant that survived or broke the run. It only runs once the static check and the proofs are green, and it is not wired into pnpm test. Running it against collection.url-params found two places where the proof read its expectation from the code under test. Filters were compared with filterEquals and the expected reverse came from isDirectionalSortKey, so mutating either moved both sides. The proof now sorts filters by JSON and compares with toEqual, and lists the four directional keys itself. All ten mutants die. Widening the generators to any grapheme and any finite double surfaced three more serialize/parse disagreements, recorded as accepted gaps: dotted metafield namespaces and empty keys, the three non-Liquid sort keys, and a -0 price bound.
…filter order The property compares filters after canonicalOrder, so the promise and the test title now say so. Each exclusion the generators make, plus the locale the node run pins to en, has a test that passes while the gap exists. Fixing a gap fails its test, which forces the registry to change with it. canonicalOrder compares code units instead of localeCompare so canonically equivalent strings no longer tie. Taxonomy values may be empty; they round-trip.
parseRegistry rejects absolute paths, dash-prefixed strings, and any segment that is empty or .., so a verified_by entry can no longer reach vitest as a flag and a mutant can no longer overwrite a file outside the package. The static check now runs before anything is spawned. --falsify copies packages/hydrogen (minus node_modules, which is linked back) into a temp directory and points vitest at it with --root. The real tree is never written, so SIGINT, a thrown error, or a failed restore leaves nothing behind, and two runs or a watcher never see each other's mutants.
verified_by now names a file and the it titles or describe paths in it. --proofs and --falsify judge only those tests, so a green test elsewhere in the file no longer stands in for the property. url.test.ts joins verified_by as the anchor that pins both sides to literal Liquid URLs. A known gap may carry shown_by, a test in the same shape that must pass while the gap exists. Every gap in the registry has one, including the locale gap the node run hides: a de storefront reads a serialized 12.5 as 125. accepted now needs accepted_by, a @handle the check matches against .github/CODEOWNERS. The handle names the reviewer GitHub will require before the file merges, so an agent cannot close a gap on its own.
A team handle would let anyone on the team, or an agent writing the team's name, close a gap. accepted_by now has to be an individual @handle, and that person has to appear on the CODEOWNERS line for packages/hydrogen/features.yaml itself. The * fallback and other lines do not count. registryOwners is exported and tested on its own. The existing acceptances are attributed to @andguy95, who is added to that CODEOWNERS line. Confirm before merging.
Rebased onto an-price-filter-locale-parse, which makes the serializer's own price output survive every shop locale. The gap's shown_by test started failing, as the registry intends when a gap closes, so the gap and its test go. The fix's locale tests in url.test.ts join verified_by in their place.
Ctrl-C used to leave the report dir and the package copy under tmpdir. Node died inside spawnSync before any finally ran. The parent now keeps a handler so it survives the signal, and runProofs exits with the shell's code for the signal once vitest has died from it, after removing every live temp dir. A signal sent to the parent alone is still only seen when the run ends; spawnSync gives no earlier point to see it.
andguy95
force-pushed
the
an-fast-check-url-roundtrip
branch
from
October 21, 2026 01:02
cb69881 to
7312f47
Compare
An error occurred while trying to automatically change base from
an-product-in-collection-standard-pathname
to
main
October 21, 2026 20:17
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Warning
Prototyping not meant for production. Exploring property based testing.
TL;DR:
packages/hydrogen/features.yamlrecords what Hydrogen promises, which tests prove each promise, which mutants those tests are known to catch, and which gaps are knowingly unproven.pnpm lintenforces the registry's shape,pnpm testruns every listed proof, and--falsifyapplies each listed mutant and fails if a proof stays green under it. The first entry is the collection URL round-trip, proven by property tests overserializeCollectionParamsandparseCollectionParamsand falsified by ten mutants ofurl.ts.Humans are good at stating what a system does and what must hold true for it. Agents are good at building toward a stated target. The registry is the ledger between the two. A feature's
promiseis the human statement,verified_bynames the tests that make it checkable,falsified_bynames the bugs those tests are proven to catch, andknown_gapsis where a counterexample lands until a human marks itacceptedwith a reason orissuewith a link. The check turns that ledger into something CI defends: a promise cannot beshippedwithout a passing proof, a property test cannot exist without a promise that claims it, and a proof that cannot fail is caught by its mutants.The collection URL params are the first entry because
serializeCollectionParamsandparseCollectionParamsare two hand-written halves of one contract, and nothing checked that they agree beyond a handful of fixed examples. It is the smallest place in Hydrogen where two functions must be exact inverses.Before
Nothing failed if a property test was orphaned, skipped, or empty, and the only record of what
url.tspromises was its comments.After
With every
itin the proof file changed toit.skip:Mutants of
url.tsthat the proof must catch:Falsifying found two weaknesses in the first version of the proof. It compared filters with
filterEqualsand derived the expectedreversefromisDirectionalSortKey, both fromurl.ts, so a mutant of either moved the expectation along with the result and survived. The proof now sorts both sides by their JSON and compares withtoEqual, and lists the four directional keys itself.What this changes
packages/hydrogen/features.yaml: one entry,collection.url-params, with its promise, entry point, proof, ten mutants, e2e group, and six accepted gaps.scripts/check-features.ts: ids are unique and shapedarea.feature-name, every listed path exists,shippedfeatures name at least one proof, every*.property.test.tsunderpackages/hydrogen/srcis claimed by a feature,depends_onpoints at registered ids with no cycles, every known gap has exactly one ofacceptedorissue, and every mutant targets an entry point. With--proofsit runs vitest on the union ofverified_bypaths and fails a path that was not collected, has failing tests, or has no passing tests. With--falsifyit also applies each mutant'sfindandreplace(the fragment must occur exactly once), reruns that feature's proofs, restores the file, and fails a mutant that survived or that broke the run instead of failing it.pnpm lintruns the static check.pnpm testruns it with--proofs.yamlis added as a root devDependency to parse the registry.src/core/collection/__tests__/url.property.test.tswith two properties, plusfast-checkas a devDependency ofpackages/hydrogen.parseCollectionParams(serializeCollectionParams(p))equalspfor every canonicalCollectionParams. Filters compare as sorted arrays undertoEqual, because the parser emits them in a fixed kind order and that order is not part of the contract.parseSortByValue(getSortByValue(k, r))returnskfor all eightProductCollectionSortKeys, keepingronly for the four directional keys the test lists. A compile-time check fails if a sort key is added to the Storefront API type but not to this list.Developer impact
No changeset. The registry, check, and tests are repo-only, and no runtime code in
@shopify/hydrogenchanges. The changesets reminder bot will comment, and that comment can be ignored.Out of scope
verified_byproves a file passes, not that it tests the stated promise. The promise text and the test are linked by name only.falsified_bynarrows that: the proof is known to catch the listed bugs.--falsifyis not inpnpm test. It runs each mutant as a separate vitest process, about eight seconds for ten mutants today. Adding it to CI is a separate decision.plannedentries and does work.known_gapsare places where serialize and parse disagree. None is reachable through the collection store, so each isacceptedwith its reason in the yaml. Treating any as a bug inurl.tsis a separate change.parseLocalizedNumberreadswindow.Shopify.localewhile the serializer always writesString(n). In a storefront whose locale groups with., a12.5bound parses back as125. The proof cannot reach that branch from node. It is tracked outside this PR and will land inknown_gapsas anissueonce it has a link.Risk
--proofs. About one second today.url.tsthat touches afindfragment makes--falsifyfail with "does not occur exactly once", which is loud. The mutation runs in the working tree and restores the file in afinally, so only a hard kill mid-run leavesurl.tsmutated.How to Test
pnpm install.node scripts/check-features.ts --proofsand confirm it ends withfeatures.yaml: all checks pass.node scripts/check-features.ts --falsifyand confirm tenkilledlines andall checks pass.packages/hydrogen/src/core/collection/__tests__/url.property.test.ts, change the secondit(toit.skip(. Rerun step 3 and confirmmutant "the ID sort key loses its direction suffix" survived every proof. Revert the file.it(toit.skip(. Rerun step 2 and confirm the check namescollection.url-paramsand the file withhas no passing tests. Revert the file.packages/hydrogen/src/core/cart/__tests__/cart.property.test.ts. Runnode scripts/check-features.tsand confirm it reports the file as not claimed by any feature. Delete the file.