Skip to content

Apply SentencePiece charsmap entries that map to an empty string - #7920

Open
ShmueLior wants to merge 1 commit into
dotnet:mainfrom
ShmueLior:fix/7908-charsmap-empty-replacement
Open

ShmueLior wants to merge 1 commit into
dotnet:mainfrom
ShmueLior:fix/7908-charsmap-empty-replacement

Conversation

@ShmueLior

Copy link
Copy Markdown

Fixes #7908

NormalizePrefix used an empty Memory<byte> to mean "no rule matched", and the callers checked Length == 0 for that. A charsmap entry whose replacement is "" also has length 0, so the original bytes were written instead and the char was encoded as <unk>. nmt_nfkc deletes 30 code points this way (U+0001-U+0008, U+000B, U+000E-U+001F, U+007F, U+008F, U+009F).

Change: NormalizePrefix now returns the bytes to write as a ReadOnlySpan<byte> (the input slice, the replacement char, or the charsmap value, which can be empty). The leading whitespace loop stops on an empty replacement, same as SentencePiece's p.first != " " check. This covers both the .model path and the tokenizer.json Precompiled step, since that one goes through the same normalizer.

Two related changes:

  • The existing Unigram test row for "\u001f" expected [▁, <unk>], which was the bug. It now expects an empty result, matching SentencePiece and HF.
  • Input that normalizes to nothing hit Debug.Assert(normalized.Length > 0) in SentencePieceUnigramModel.Encode. That already happens on main for whitespace-only input like " " in Debug builds. Release was fine because NormalizeText returns an empty span for that case and Encode handles size 0. I removed the assert and added a " " row.

Not touched: the tie-breaking difference in #7909.

Tests: 30 code points x (.model, tokenizer.json) checking that a{c}b, {c}hello, hello{c} and so on encode the same as without the char and never produce <unk>, plus a check that tab is still mapped to a space. Microsoft.ML.Tokenizers.Tests on net8.0: 511 passed. Release build has no warnings. I couldn't run net48/net9.0 locally.

NormalizePrefix used an empty Memory to mean "no rule matched", so a rule
whose replacement is empty (nmt_nfkc deletes U+0001-U+0008, U+000B,
U+000E-U+001F, U+007F, U+008F and U+009F) fell back to the original bytes
and the character was encoded as <unk>. Return the bytes to write directly
so an empty replacement deletes the input, like SentencePiece and Hugging
Face do.

Input that normalizes to nothing (whitespace only, or only deleted
characters) no longer trips the Debug.Assert in the Unigram Encode.

Fixes dotnet#7908
@dotnet-policy-service dotnet-policy-service Bot added the community-contribution Pull requests submitted by community contributors label Nov 17, 2026
@codecov

codecov Bot commented Nov 17, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 98.11321% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 70.86%. Comparing base (663276f) to head (d621b11).

Files with missing lines Patch % Lines
...L.Tokenizers/Normalizer/SentencePieceNormalizer.cs 85.71% 1 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##             main    #7920   +/-   ##
=======================================
  Coverage   70.85%   70.86%           
=======================================
  Files        1420     1420           
  Lines      279991   280029   +38     
  Branches    27594    27595    +1     
=======================================
+ Hits       198394   198433   +39     
+ Misses      73448    73446    -2     
- Partials     8149     8150    +1     
Flag Coverage Δ
Debug 70.86% <98.11%> (+<0.01%) ⬆️
production 64.30% <85.71%> (-0.01%) ⬇️
test 90.73% <100.00%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
...t.ML.Tokenizers/Model/SentencePieceUnigramModel.cs 67.11% <ø> (+0.15%) ⬆️
test/Microsoft.ML.Tokenizers.Tests/UnigramTests.cs 97.60% <100.00%> (+0.06%) ⬆️
...L.Tokenizers/Normalizer/SentencePieceNormalizer.cs 76.05% <85.71%> (+0.92%) ⬆️

... and 5 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

community-contribution Pull requests submitted by community contributors

Projects

None yet

Development

Successfully merging this pull request may close these issues.

SentencePieceTokenizer: charsmap entries that map to an empty string are not applied, so control characters become <unk>

1 participant