Skip to content

Narrow URL-like protection so PostScript glyph names inside numeric ranges are cleaned #100

Open

Description

@fffoivos

During the 2026-04-28 wave-3 tokenizer review, the C1 added vocab had one residual PostScript-name token: /hyphenminus.

Corpus spot checks showed contexts such as:

  • 4.600/hyphenminus5.600 KDa
  • 0.3/hyphenminus 100%
  • 50.000/hyphenminus250.000 κατοίκους

The cleaner strips standalone foo /hyphenminus bar, and correctly preserves URL/path contexts such as https://example.org/a/hyphenminus/b, but it currently preserves numeric slash ranges because URL_LIKE_TOKEN_REGEX treats tokens like 4.600/hyphenminus5.600 as URL-like host/path tokens.

Suggested later fix:

  • keep protecting real URLs, www. links, and host/path tokens with alphabetic host components
  • do not classify purely numeric dotted ranges followed by slash as URL-like
  • add tests for numeric /hyphenminus stripping and real URL preservation

This is a small cleanup follow-up, not a blocker for the completed tokenizer wave.

Activity

  1. ishita73 commented on Jul 12, 2026

    @ishita73

    Hello. I'd like to work on this issue . Please let me know if there are any specific expectations before I start.

  2. arshil42 commented on Nov 2, 2026

    @arshil42

    I looked into picking this up, but it looks already handled on development. Commit 2940cea added hyphenminus_span_is_inside_true_url, and glyph_span_is_protected now lets /hyphenminus through in numeric tokens while keeping real URLs protected. The test core_clean_text_normalizes_hyphenminus_ranges_but_keeps_urls covers 4.600/hyphenminus5.600 and a real URL.

    The fix isn't on master yet, which may be why this is still open. It only applies to /hyphenminus, and I think that's right: it has a - replacement, while other glyphs would just be removed, so 4.600/period5.600 would become 4.6005.600.

    This is from reading the code. I couldn't build the crate on my Windows setup, so I haven't run it. This may be safe to close once development lands on master, unless you'd like something more done here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions