Skip to content

test(hydrogen): add property tests with fast-check - #4354

Draft
fredericoo wants to merge 2 commits into
mainfrom
fb-fast-check
Draft

fredericoo wants to merge 2 commits into
mainfrom
fb-fast-check

Conversation

@fredericoo

@fredericoo fredericoo commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

TL;DR: Hydrogen's tests check the inputs and the response orders someone thought of. This PR adds fast-check, which generates them, and 13 property tests for @shopify/hydrogen. Seven pass. Six fail on main and stay red in this PR on purpose. Each one points at a bug or a rule that needs a decision, and the fixes come in a PR stacked on this one.

A property test states a rule, such as "a settled cart matches Shopify's cart". fast-check then searches for an input, or an order of responses, that breaks the rule. When it finds one, it shrinks it to a small failing case and prints a seed that replays it.

Before

it("rounds a float to the nearest integer", () => {
  expect(sanitizeQuantity(3.7)).toBe(4);
});

One hand-picked input per test. Nothing tried sanitizeQuantity("1e309").

After

it("returns a whole number, at least the default minimum, for any input", () => {
  fc.assert(
    fc.property(rawQuantity, (raw) => {
      const quantity = sanitizeQuantity(raw);

      expect(quantity).toSatisfy(Number.isInteger);
      expect(quantity).toBeGreaterThanOrEqual(1);
    }),
  );
});
Error: Property failed after 8 tests
{ seed: 534071695, path: "7:1:0:0:0:0:3:1:1:1:1:1:1:1", endOnFailure: true }
Counterexample: ["1e309"]
Shrunk 13 time(s)

What this changes

  • Adds fast-check as a devDependency of @shopify/hydrogen, with no configuration. Seeds stay random on every run, as fast-check recommends.
  • Adds six *.property.test.ts files next to the code they test:
Area Rule On main
Input handling sanitizeQuantity returns a whole number of at least 1 for any input Fails
It stays within explicit bounds for any input Passes
It keeps a quantity that is already within the bounds, however it is written Passes
Security rules getSameOriginPath only returns a location on the origin, and only for a target on the origin Passes
It keeps a link to a page of the site Passes
createRunWithCache serves cached data to the key that stored it, and to no other key Fails
Round trips and consistency The cart cookie reads back the cart ID it stored Passes
formatMoney formats any price with any valid display options Fails
It shows a range from its lowest to its highest price, whatever the order of the prices Passes
Async races The cart store shows Shopify's cart whenever nothing is pending Passes
It shows Shopify's price on every line after a change reprices the cart Fails
It returns to Shopify's cart after a reset() Fails
Action sequences The cart store shows, and sends Shopify, what the shopper asked for Fails
  • The cart tests run the real store against a small fake of Shopify and its Standard Actions script. fc.scheduler() decides when each response reaches the store, so every run tries a different order.
  • Each cart property tries up to 1,000 sessions and has a 30-second timeout. The one that passes runs them all, which took 2.4 seconds on CI, about half of Vitest's 5-second default.
  • Adds two lines on property tests to the Testing section of AGENTS.md.

The six failures

  1. Quantity overflow. sanitizeQuantity("1e309") returns Infinity, and so does "Infinity". When the cart store gets such a quantity in a set submission, the cart shows Infinity and sends quantity: Infinity, which the request to Hydrogen's cart route serialises as null. Hydrogen's own quantity field accepts digits only, so through that field it takes a number of at least 309 digits. A custom field or a script gets there with 1e309. A cart fragment that selects quantityAvailable caps the quantity and avoids this.
  2. Cache key collision. String keys that differ only in unpaired surrogates, such as "\ud800" and "\ud801", share one cache entry, and share it with the same key holding U+FFFD in their place. TextEncoder turns each unpaired surrogate into U+FFFD before hashing. Array keys are not affected. Low severity: such strings come from input like JSON.parse('"\\ud800"').
  3. formatMoney throws. It throws RangeError: maximumFractionDigits value is out of range when the output has no currency symbol and only one fraction-digit limit is set: maximumFractionDigits below 2, or minimumFractionDigits above 2. The output has no currency symbol with withoutCurrency: true, with codes that are not three letters such as USDC, and in the withoutTrailingZerosAndCurrency field. The exception is a whole amount whose trailing zeros are stripped, which formats.
  4. Stale line prices. When a change to one line makes Shopify reprice the others, as an automatic discount does, the store updates the changed line and the cart total. The other lines keep their old price, and nothing marks them as pending.
  5. Reset during a refresh. A reset() while a background refresh is in flight, for example straight after refresh(), leaves the cart empty with loading: true and never loads it again. The existing test "reset ignores an in-flight authoritative revalidation" expects exactly that, so this one needs a decision before a fix.
  6. Two changes to one line, answered out of order. Change a line's quantity and add the same variant, in either order. If Shopify answers the second request first, the cart shows the wrong quantity until both answers and the refresh that follows have arrived. A click on that line in the meantime sends Shopify a quantity based on the wrong number. The same property passes when responses arrive in the order the requests were sent.

Developer impact

No changeset. The diff adds tests and a devDependency, and nothing in the published package changes.

Out of scope

  • Fixes for the six failures. They come in a stacked PR.
  • Cart behaviour that the fake does not model: error responses, updateCart calls from code, cart attributes, the first cart load, and destroy().
  • The collection URL round-trip properties prototyped in Add a feature registry and round-trip tests for collection URL params #4209.
  • Negative zero. formatMoney shows an amount of -0 as -$0.00, so a range of 0 and -0 reads differently in each order. The money generator leaves -0 out.

Risk

  • CI is red on this PR by design, so it cannot merge on its own.
  • Seeds are random, so a property can fail on an unrelated PR. The failure prints the seed that replays it.
  • The cart tests are only as right as the fake. It assumes Shopify applies requests in the order the browser sent them, including requests the store aborted, and that responses arrive in any order.

Adds fast-check as a devDependency and 13 property tests across input
handling, security rules, round trips, async races and action sequences.

Six properties fail on main and are left red on purpose. Each one points
at a bug or a rule that needs a decision; the fixes come in a stacked PR.
@shopify

shopify Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Oxygen deployed a preview of your fb-fast-check branch. Details:

Storefront Status Preview link Deployment details Last update (UTC)
Skeleton (skeleton.hydrogen.shop) ✅ Successful (Logs) Preview deployment Inspect deployment November 10, 2026 7:55 PM

Learn more about Hydrogen's GitHub integration.

A cart property that holds runs 1,000 sessions. On CI that took 2.4
seconds, about half of Vitest's 5-second default, and three more
properties will run in full once their fixes land.
@andguy95

Copy link
Copy Markdown
Collaborator

@fredericoo,

Overall i think the PR makes sense adding the fast-check. PR mentions that we will stack fixes.

Perhaps additionally we should scope domains worth adding fast checks to. When I did this initially, I caught some bugs on these too:

  • Collection state
  • ShopifyScript args
  • i18n config
  • Proxy headers

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants