Repository navigation
Conversation
NormalizePrefix used an empty Memory to mean "no rule matched", so a rule whose replacement is empty (nmt_nfkc deletes U+0001-U+0008, U+000B, U+000E-U+001F, U+007F, U+008F and U+009F) fell back to the original bytes and the character was encoded as <unk>. Return the bytes to write directly so an empty replacement deletes the input, like SentencePiece and Hugging Face do. Input that normalizes to nothing (whitespace only, or only deleted characters) no longer trips the Debug.Assert in the Unigram Encode. Fixes dotnet#7908
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #7920 +/- ##
=======================================
Coverage 70.85% 70.86%
=======================================
Files 1420 1420
Lines 279991 280029 +38
Branches 27594 27595 +1
=======================================
+ Hits 198394 198433 +39
+ Misses 73448 73446 -2
- Partials 8149 8150 +1
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
This was referenced Nov 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #7908
NormalizePrefixused an emptyMemory<byte>to mean "no rule matched", and the callers checkedLength == 0for that. A charsmap entry whose replacement is "" also has length 0, so the original bytes were written instead and the char was encoded as<unk>. nmt_nfkc deletes 30 code points this way (U+0001-U+0008, U+000B, U+000E-U+001F, U+007F, U+008F, U+009F).Change:
NormalizePrefixnow returns the bytes to write as aReadOnlySpan<byte>(the input slice, the replacement char, or the charsmap value, which can be empty). The leading whitespace loop stops on an empty replacement, same as SentencePiece'sp.first != " "check. This covers both the.modelpath and thetokenizer.jsonPrecompiled step, since that one goes through the same normalizer.Two related changes:
"\u001f"expected[▁, <unk>], which was the bug. It now expects an empty result, matching SentencePiece and HF.Debug.Assert(normalized.Length > 0)inSentencePieceUnigramModel.Encode. That already happens on main for whitespace-only input like" "in Debug builds. Release was fine becauseNormalizeTextreturns an empty span for that case andEncodehandles size 0. I removed the assert and added a" "row.Not touched: the tie-breaking difference in #7909.
Tests: 30 code points x (
.model,tokenizer.json) checking thata{c}b,{c}hello,hello{c}and so on encode the same as without the char and never produce<unk>, plus a check that tab is still mapped to a space.Microsoft.ML.Tokenizers.Testson net8.0: 511 passed. Release build has no warnings. I couldn't run net48/net9.0 locally.