Skip to content

fix(java): don't split surrogate pairs - #1411

Merged
tushuhei merged 1 commit into
google:mainfrom
kwy404:fix-java-surrogate-pairs
Oct 5, 2026
Merged

tushuhei merged 1 commit into
google:mainfrom
kwy404:fix-java-surrogate-pairs

Conversation

@kwy404

@kwy404 kwy404 commented Oct 4, 2026

Copy link
Copy Markdown
Contributor

Parser.parse walks the sentence by UTF-16 code units, so it can put a boundary between the two halves of a surrogate pair, which cuts a character outside the BMP (emoji, or CJK Extension B characters like 𠮷) in two. With the default models, Parser.loadDefaultSimplifiedChineseParser().parse("我爱\uD83C\uDF63") returns ["我", "爱\ud83c", "\udf63"], and Parser.loadDefaultTraditionalChineseParser().translateHTMLString("吉野家の𠮷") puts a ZWSP between \ud842 and \udfb7. This is the Java side of #1403, which fixed the same problem in the JavaScript parser (see also #504).

This skips the positions that fall inside a surrogate pair, checked with Character.isSurrogatePair(sentence.charAt(i - 1), sentence.charAt(i)). Text without surrogate pairs gets exactly the same phrases as before, and translateHTMLString is fixed too since it uses parse.

Added testParseSurrogatePair to ParserTest with the same model as the JavaScript test, which separates everywhere: parse("𠮷野家") returns ["\ud842", "\udfb7", "野", "家"] before the change and ["𠮷", "野", "家"] after.

Tested with mvn package (JDK 17) and google-java-format.

Parser.parse walks the sentence by UTF-16 code units, so it can put a
boundary between the two halves of a surrogate pair and cut a character
outside the BMP (emoji, or CJK Extension B characters like 𠮷) in two.
This skips those positions, like the JavaScript parser does since google#1403.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants