Skip to content

fix(js): don't split surrogate pairs - #1403

Merged
tushuhei merged 1 commit into
google:mainfrom
kwy404:fix-js-surrogate-pairs
Sep 29, 2026
Merged

tushuhei merged 1 commit into
google:mainfrom
kwy404:fix-js-surrogate-pairs

Conversation

@kwy404

@kwy404 kwy404 commented Sep 28, 2026

Copy link
Copy Markdown
Contributor

Parser.parseBoundaries walks the sentence by UTF-16 code units, so it can return a boundary between the two halves of a surrogate pair, which cuts a character outside the BMP (emoji, or CJK Extension B characters like 𠮷) in two. With the default models, loadDefaultSimplifiedChineseParser().parse('我爱\u{1F363}') returns ['我', '爱\ud83c', '\udf63'], and loadDefaultTraditionalChineseParser().translateHTMLString('吉野家の𠮷') puts a ZWSP between \ud842 and \udfb7, so the browser renders two replacement characters instead of 𠮷. The Python version does not have this problem because it iterates by code points. This is related to the UTF-16 discussion in #504.

This skips the positions that fall inside a surrogate pair, detected with codePointAt(i - 1) > 0xffff. Text without surrogate pairs gets exactly the same boundaries as before, and parse and HTMLProcessor are both fixed since they use parseBoundaries.

Added a test to the Parser.parse tests with a model that separates everywhere: parse('𠮷野家') returns ['\ud842', '\udfb7', '野', '家'] before the change and ['𠮷', '野', '家'] after.

Tested with npm test (Node and Chromium) and npm run lint.

@tushuhei
tushuhei merged commit 5b77a86 into google:main Sep 29, 2026
24 checks passed
kwy404 added a commit to kwy404/kwy404 that referenced this pull request Sep 29, 2026
tushuhei pushed a commit that referenced this pull request Oct 5, 2026
Parser.parse walks the sentence by UTF-16 code units, so it can put a
boundary between the two halves of a surrogate pair and cut a character
outside the BMP (emoji, or CJK Extension B characters like 𠮷) in two.
This skips those positions, like the JavaScript parser does since #1403.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants