Skip to content

Add JSONTestSuite parser tests - #231

Open
DicksonWu654 wants to merge 2 commits into
uhop:masterfrom
DicksonWu654:jsontestsuite-tests
Open

DicksonWu654 wants to merge 2 commits into
uhop:masterfrom
DicksonWu654:jsontestsuite-tests

Conversation

@DicksonWu654

Copy link
Copy Markdown

This adds JSONTestSuite as a regression test. tests/node/test-json-test-suite.js streams each raw file through the parser. The 318 test_parsing cases have to match their y_/n_ labels, and i_ cases can go either way. For the 22 test_transform cases, it checks that anything accepted comes back stable through the stringer.

  • The files are vendored from commit 1ef36fa01286573e846ac449e8683f8833c5b26a under tests/data/json-test-suite/ with the MIT license. That's about 355 KB, no network needed
  • Nothing is skipped, and npm test passes (2,457 assertions)
  • The new file takes about 0.26s

Happy to move the fixtures or trim anything if you'd like it organized differently

@uhop uhop self-assigned this Nov 16, 2026
@uhop

uhop commented Nov 16, 2026

Copy link
Copy Markdown
Owner

Thank you for the PR. JSONTestSuite is a good fit for stream-json, and the vendored files match upstream 1ef36fa byte for byte. Before merging, I'd like the tests to check more of what the suite offers.

Test both parsers and both verifiers

stream-json has four components that read JSON text: src/parser.js, src/utils/verifier.js, src/jsonc/parser.js, and src/jsonc/verifier.js. The test in this PR sends test_parsing through the JSON verifier only, so the JSON parser sees just the 22 test_transform files, and the JSONC components see none.

Split the test into four files, one per component. Near-identical copies are fine, and each file must run on its own with node tests/node/<file>:

  • tests/node/test-json-test-suite-parser.js
  • tests/node/test-json-test-suite-verifier.js
  • tests/node/test-jsonc-test-suite-parser.js
  • tests/node/test-jsonc-test-suite-verifier.js

What each file checks

Feed every test_parsing file twice: as one chunk, and as one byte per chunk. The second run puts a chunk boundary between every pair of bytes, and both runs must give the same result.

  • y_: accepted. In the parser files, collect the values with streamValues() and compare them with t.deepEqual to JSON.parse(new TextDecoder().decode(buffer)).
  • n_: rejected. In the verifier files, also check that the error carries integer line, pos, and offset. Parser errors are plain Error objects without them, so isParserError doesn't fit the parser files.
  • i_: pin the current outcome instead of allowing either. All 35 are accepted except three, which are rejected: i_string_UTF-16LE_with_BOM.json, i_string_utf16BE_no_BOM.json, and i_string_utf16LE_no_BOM.json. A change in behavior then shows up as a failing test.

The JSONC files expect the same outcomes, except that seven n_ files are accepted, because comments and trailing commas are valid JSONC:

  • n_array_extra_comma.json
  • n_array_number_and_comma.json
  • n_object_lone_continuation_byte_in_key_and_trailing_comma.json
  • n_object_trailing_comma.json
  • n_object_trailing_comment.json
  • n_object_trailing_comment_slash_open.json
  • n_structure_object_with_comment.json

The two parser files also keep the test_transform check: parse, stringify, and confirm that a second pass gives the same text. The JSONC file uses src/jsonc/stringer.js. All 22 inputs are accepted today, so a rejection there counts as a failure.

I ran these checks locally against master under Node, Bun, and Deno: every expectation holds on all three, and the checks for the two JSON components, or the two JSONC ones, run in about 0.3 seconds.

README

Expand tests/data/json-test-suite/README.md with:

  • what the suite is: the appendix to Nicolas Seriot's article Parsing JSON is a Minefield;
  • the y_, n_, and i_ prefixes, which the tests depend on;
  • that test_transform has no expected answers, and what the tests check instead;
  • that the files are byte-exact and many are malformed on purpose (invalid UTF-8, NUL bytes, UTF-16), so they must not be formatted or re-encoded, which is why .prettierignore lists them;
  • what was left out of upstream: parsers/, results/, run_tests.py, and article/;
  • how to refresh the files from a newer upstream commit.

@uhop uhop left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the comment.

Expand JSONTestSuite coverage across the JSON and JSONC parsers and verifiers.
@DicksonWu654

Copy link
Copy Markdown
Author

Thanks so much for the detailed review! Pushed 96939ca with the four standalone files, each feeding every test_parsing file as one chunk and one byte per chunk, the pinned i_ outcomes and the seven JSONC exceptions. The README now covers everything on your list.

All four pass on their own with node tests/node/<file> (about 0.5s each) and npm test passes. I only ran them under Node tho, so Bun and Deno haven't been checked on my side

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants