Skip to content

Fix non-Latin text encoding for RoBERTa trainers - #7911

Open
nikitabhagat2307-prog wants to merge 4 commits into
dotnet:mainfrom
nikitabhagat2307-prog:fix/roberta-utf8-non-latin-encoding
Open

nikitabhagat2307-prog wants to merge 4 commits into
dotnet:mainfrom
nikitabhagat2307-prog:fix/roberta-utf8-non-latin-encoding

Conversation

@nikitabhagat2307-prog

Copy link
Copy Markdown

Fixes #7858

Summary

EnglishRobertaTokenizer currently drops input characters that do not map directly to a single byte. That silently removes non-Latin text before the TorchSharp RoBERTa/NAS-BERT trainers see it.

This change encodes characters above Latin-1 as UTF-8 bytes before byte-level BPE, while retaining the existing single-byte mapping for Latin-1 characters so pretrained model token IDs remain stable. Token offsets and partial token-count results continue to refer to UTF-16 positions in the original input.

Regression coverage exercises Korean, Cyrillic, Japanese, supplementary-plane emoji, Latin-1 compatibility, cache reuse, and forward/reverse token limits.

Validation

  • dotnet test test/Microsoft.ML.Tokenizers.Tests/Microsoft.ML.Tokenizers.Tests.csproj --framework net8.0 --no-restore — 451 passed.
  • Focused EnglishRobertaTests run — 10 passed.
  • dotnet build src/Microsoft.ML.Tokenizers/Microsoft.ML.Tokenizers.csproj --framework netstandard2.0 --no-restore — succeeded with 0 warnings and 0 errors.

@dotnet-policy-service dotnet-policy-service Bot added the community-contribution Pull requests submitted by community contributors label Nov 12, 2026
@nikitabhagat2307-prog

Copy link
Copy Markdown
Author

@dotnet-policy-service agree

@codecov

codecov Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 92.92929% with 7 lines in your changes missing coverage. Please review.
✅ Project coverage is 70.85%. Comparing base (663276f) to head (0223499).
⚠️ Report is 4 commits behind head on main.

Files with missing lines Patch % Lines
...oft.ML.Tokenizers/Model/EnglishRobertaTokenizer.cs 87.50% 6 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #7911      +/-   ##
==========================================
- Coverage   70.85%   70.85%   -0.01%     
==========================================
  Files        1420     1420              
  Lines      279991   279966      -25     
  Branches    27594    27587       -7     
==========================================
- Hits       198394   198376      -18     
+ Misses      73448    73442       -6     
+ Partials     8149     8148       -1     
Flag Coverage Δ
Debug 70.85% <92.92%> (-0.01%) ⬇️
production 64.30% <87.50%> (-0.01%) ⬇️
test 90.73% <100.00%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
...crosoft.ML.Tokenizers.Tests/EnglishRobertaTests.cs 100.00% <100.00%> (ø)
...oft.ML.Tokenizers/Model/EnglishRobertaTokenizer.cs 77.01% <87.50%> (+3.27%) ⬆️

... and 9 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@nikitabhagat2307-prog

nikitabhagat2307-prog commented Nov 12, 2026 •

Copy link
Copy Markdown
Author

Thanks for the coverage report. The focused EnglishRobertaTests passed locally earlier (10/10), including the non-Latin encoding and offset cases. I added a long Japanese input to exercise the UTF-8 buffer growth path and removed an unreachable empty-token guard; the update is commit 25c569c. I can’t rerun tests locally because the .NET SDK is unavailable in this environment. I also inspected the failed Windows .NET Framework 4.8 Debug job: Helix reports a BadExit (exit 255), but its console log has no individual failing test or assertion. Could someone rerun the job or share the full Helix diagnostics so I can address the cause?

@nikitabhagat2307-prog

Copy link
Copy Markdown
Author

Update: I inspected the failing Helix console. The Microsoft.ML.CodeAnalyzer.Tests process segfaulted (signal 11) on Ubuntu 22.04 arm64 before it produced any test results; there is no failing assertion in the tokenizer or this PR. I added coverage for unpaired high and low UTF-16 surrogates and now match Encoding.UTF8 replacement behavior (commit 0223499). git diff --check passes; I could not run the .NET tests locally because the SDK is unavailable. Please rerun the arm64 Helix job to confirm the infrastructure crash is transient.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

community-contribution Pull requests submitted by community contributors

Projects

None yet

1 participant