Skip to content

fix: open generated/config files with explicit utf-8 encoding - #179

Open
xX-its-amit-Xx wants to merge 1 commit into
bilayer-containers:mainfrom
xX-its-amit-Xx:fix/utf-8-encoding
Open

xX-its-amit-Xx wants to merge 1 commit into
bilayer-containers:mainfrom
xX-its-amit-Xx:fix/utf-8-encoding

Conversation

@xX-its-amit-Xx

Copy link
Copy Markdown

🤖 AI disclosure: This PR body and the code changes were drafted by Claude (Anthropic's AI), then reviewed, manually tested on Windows, and validated against a cross-platform CI replay by me before submission. Commits carry a Co-Authored-By: Claude trailer. Splitting this out from #178 per @gnodar01's review.

Summary

Pass encoding="utf-8" everywhere the codebase reads or writes a generated artifact, config YAML, or LinkML schema. Fixes a reproducible Windows-only UnicodeEncodeError / UnicodeDecodeError in bilayers_cli generate (Jupyter interface) and in tests/regression/test_generate_regression.py. Latent on Gradio / Streamlit / CellProfiler-plugin writes too — they happen to produce ASCII-only output today.

Repro

On Windows (Python 3.14, fresh venv with pip install -e ".[test,interfaces,plugins,linkml]"):

$ bilayers_cli generate tests/fixtures/classical_segmentation/config.yaml
Parsing config...
Running generate for cellprofiler_plugin...
CellProfiler plugin generated successfully!!
Running generate for gradio...
app.py generated successfully!!
Running generate for jupyter...
Error occurred while generating for jupyter: 'charmap' codec can't encode characters in position 23795-23796: character maps to <undefined>
Running generate for streamlit...
streamlit_app.py generated successfully!!
Finished generating all interfaces

pytest tests/regression/test_generate_regression.py then fails mirror-imaged with UnicodeDecodeError: 'charmap' codec can't decode byte 0x8f in position 23800 reading the golden notebook.

Root cause

Every open(...) site involved uses Python's locale-dependent default encoding:

  • interfaces/{gradio,jupyter,streamlit,cellprofiler_plugin}/generate.py — generator writes
  • src/bilayers/parse.py::load_config — config YAML read
  • src/bilayers/schema.py — LinkML schema YAML load at import time
  • tests/regression/test_generate_regression.py — normalize_notebook JSON read and read_text() golden comparison

On Linux/macOS the locale default is UTF-8; on Windows it's cp1252. nbformat emits non-cp1252 bytes (e.g. 0x8f) in some notebook fields, which breaks the round-trip on Windows. The bug is silent on Linux and fails closed on Windows.

Fix

Pass encoding="utf-8" at every read/write of a file that is canonically UTF-8 (.py, .ipynb, .yaml). Eight sites; the change is mechanical.

Test plan

  • pytest tests/ on Windows (Python 3.14, venv): 17 / 17 pass. Previously the regression test failed on this platform.
  • bilayers_cli generate tests/fixtures/classical_segmentation/config.yaml on Windows: all four interfaces generate cleanly. Previously Jupyter failed.
  • bilayers_cli generate --cli ... emits python -m classical_segmentation --folder /bilayers/input_images --threshold_method otsu --min_size 5 --max_size 20 --save_dir /bilayers/output_images.
  • ruff check src tests noxfile.py → All checks passed!.
  • CI replay on my fork (this same test.yml workflow on ubuntu-latest + Python 3.9): green — https://github.com/xX-its-amit-Xx/bilayers/actions/runs/26635076654. The Linux side is unchanged by this patch (default encoding was already UTF-8 there), which is the intended outcome.

Open question for maintainers

noxfile.py reads / writes a handful of tmp_path("…").txt files without encoding=. They currently only hold ASCII (platform tags, docker image names, algorithm folder names) so the same bug doesn't bite there. Happy to normalize them here for consistency or leave for a separate noxfile cleanup — your call.

bilayers_cli generate fails on Windows with UnicodeEncodeError
('charmap' codec can't encode ...) when the Jupyter notebook contains
the non-ASCII bytes nbformat normally emits, because the writer calls
open(..., "w") and Windows defaults to cp1252. The same bug latently
affects the gradio, streamlit and cellprofiler_plugin generators, plus
load_config / schema.yaml reads in src/bilayers/, and the regression
test's normalize_notebook() / read_text() comparisons.

This patch passes encoding="utf-8" everywhere the codebase writes a
generated artifact or reads a YAML / golden file, so behavior is the
same across Linux, macOS and Windows. No format / public-API changes.

Repro before: pytest tests/regression/test_generate_regression.py
fails on Windows with UnicodeDecodeError at byte 0x8f.
Repro after: 17/17 tests pass on both Windows and the Linux CI.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@gnodar01

gnodar01 commented Jun 1, 2026

Copy link
Copy Markdown
Contributor

lgtm, @rajavishah let me know if you have other thoughts, else please approve the workflow and accept/merge the pr.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants