You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Fix non-Latin text encoding for RoBERTa trainers - #7911
EnglishRobertaTokenizer currently drops input characters that do not map directly to a single byte. That silently removes non-Latin text before the TorchSharp RoBERTa/NAS-BERT trainers see it.
This change encodes characters above Latin-1 as UTF-8 bytes before byte-level BPE, while retaining the existing single-byte mapping for Latin-1 characters so pretrained model token IDs remain stable. Token offsets and partial token-count results continue to refer to UTF-16 positions in the original input.
❌ Patch coverage is 92.92929% with 7 lines in your changes missing coverage. Please review.
✅ Project coverage is 70.85%. Comparing base (663276f) to head (0223499). ⚠️ Report is 4 commits behind head on main.
Thanks for the coverage report. The focused EnglishRobertaTests passed locally earlier (10/10), including the non-Latin encoding and offset cases. I added a long Japanese input to exercise the UTF-8 buffer growth path and removed an unreachable empty-token guard; the update is commit 25c569c. I can’t rerun tests locally because the .NET SDK is unavailable in this environment. I also inspected the failed Windows .NET Framework 4.8 Debug job: Helix reports a BadExit (exit 255), but its console log has no individual failing test or assertion. Could someone rerun the job or share the full Helix diagnostics so I can address the cause?
Update: I inspected the failing Helix console. The Microsoft.ML.CodeAnalyzer.Tests process segfaulted (signal 11) on Ubuntu 22.04 arm64 before it produced any test results; there is no failing assertion in the tokenizer or this PR. I added coverage for unpaired high and low UTF-16 surrogates and now match Encoding.UTF8 replacement behavior (commit 0223499). git diff --check passes; I could not run the .NET tests locally because the SDK is unavailable. Please rerun the arm64 Helix job to confirm the infrastructure crash is transient.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #7858
Summary
EnglishRobertaTokenizercurrently drops input characters that do not map directly to a single byte. That silently removes non-Latin text before the TorchSharp RoBERTa/NAS-BERT trainers see it.This change encodes characters above Latin-1 as UTF-8 bytes before byte-level BPE, while retaining the existing single-byte mapping for Latin-1 characters so pretrained model token IDs remain stable. Token offsets and partial token-count results continue to refer to UTF-16 positions in the original input.
Regression coverage exercises Korean, Cyrillic, Japanese, supplementary-plane emoji, Latin-1 compatibility, cache reuse, and forward/reverse token limits.
Validation
dotnet test test/Microsoft.ML.Tokenizers.Tests/Microsoft.ML.Tokenizers.Tests.csproj --framework net8.0 --no-restore— 451 passed.EnglishRobertaTestsrun — 10 passed.dotnet build src/Microsoft.ML.Tokenizers/Microsoft.ML.Tokenizers.csproj --framework netstandard2.0 --no-restore— succeeded with 0 warnings and 0 errors.