You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Narrow URL-like protection so PostScript glyph names inside numeric ranges are cleaned #100
During the 2026-04-28 wave-3 tokenizer review, the C1 added vocab had one residual PostScript-name token: /hyphenminus.
Corpus spot checks showed contexts such as:
4.600/hyphenminus5.600 KDa
0.3/hyphenminus 100%
50.000/hyphenminus250.000 κατοίκους
The cleaner strips standalone foo /hyphenminus bar, and correctly preserves URL/path contexts such as https://example.org/a/hyphenminus/b, but it currently preserves numeric slash ranges because URL_LIKE_TOKEN_REGEX treats tokens like 4.600/hyphenminus5.600 as URL-like host/path tokens.
Suggested later fix:
keep protecting real URLs, www. links, and host/path tokens with alphabetic host components
do not classify purely numeric dotted ranges followed by slash as URL-like
add tests for numeric /hyphenminus stripping and real URL preservation
This is a small cleanup follow-up, not a blocker for the completed tokenizer wave.
I looked into picking this up, but it looks already handled on development. Commit 2940cea added hyphenminus_span_is_inside_true_url, and glyph_span_is_protected now lets /hyphenminus through in numeric tokens while keeping real URLs protected. The test core_clean_text_normalizes_hyphenminus_ranges_but_keeps_urls covers 4.600/hyphenminus5.600 and a real URL.
The fix isn't on master yet, which may be why this is still open. It only applies to /hyphenminus, and I think that's right: it has a - replacement, while other glyphs would just be removed, so 4.600/period5.600 would become 4.6005.600.
This is from reading the code. I couldn't build the crate on my Windows setup, so I haven't run it. This may be safe to close once development lands on master, unless you'd like something more done here.
Activity
ishita73 commentedon Jul 12, 2026
Hello. I'd like to work on this issue . Please let me know if there are any specific expectations before I start.
arshil42 commentedon Nov 2, 2026
I looked into picking this up, but it looks already handled on
development. Commit 2940cea addedhyphenminus_span_is_inside_true_url, andglyph_span_is_protectednow lets/hyphenminusthrough in numeric tokens while keeping real URLs protected. The testcore_clean_text_normalizes_hyphenminus_ranges_but_keeps_urlscovers4.600/hyphenminus5.600and a real URL.The fix isn't on
masteryet, which may be why this is still open. It only applies to/hyphenminus, and I think that's right: it has a-replacement, while other glyphs would just be removed, so4.600/period5.600would become4.6005.600.This is from reading the code. I couldn't build the crate on my Windows setup, so I haven't run it. This may be safe to close once
developmentlands onmaster, unless you'd like something more done here.