Repository navigation
fix: open generated/config files with explicit utf-8 encoding - #179
Open
xX-its-amit-Xx wants to merge 1 commit into
Open
xX-its-amit-Xx wants to merge 1 commit into
xX-its-amit-Xx wants to merge 1 commit into
Conversation
bilayers_cli generate fails on Windows with UnicodeEncodeError
('charmap' codec can't encode ...) when the Jupyter notebook contains
the non-ASCII bytes nbformat normally emits, because the writer calls
open(..., "w") and Windows defaults to cp1252. The same bug latently
affects the gradio, streamlit and cellprofiler_plugin generators, plus
load_config / schema.yaml reads in src/bilayers/, and the regression
test's normalize_notebook() / read_text() comparisons.
This patch passes encoding="utf-8" everywhere the codebase writes a
generated artifact or reads a YAML / golden file, so behavior is the
same across Linux, macOS and Windows. No format / public-API changes.
Repro before: pytest tests/regression/test_generate_regression.py
fails on Windows with UnicodeDecodeError at byte 0x8f.
Repro after: 17/17 tests pass on both Windows and the Linux CI.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
5 tasks done
Contributor
|
lgtm, @rajavishah let me know if you have other thoughts, else please approve the workflow and accept/merge the pr. |
rajavishah
approved these changes
Jun 2, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Pass
encoding="utf-8"everywhere the codebase reads or writes a generated artifact, config YAML, or LinkML schema. Fixes a reproducible Windows-onlyUnicodeEncodeError/UnicodeDecodeErrorinbilayers_cli generate(Jupyter interface) and intests/regression/test_generate_regression.py. Latent on Gradio / Streamlit / CellProfiler-plugin writes too — they happen to produce ASCII-only output today.Repro
On Windows (Python 3.14, fresh venv with
pip install -e ".[test,interfaces,plugins,linkml]"):pytest tests/regression/test_generate_regression.pythen fails mirror-imaged withUnicodeDecodeError: 'charmap' codec can't decode byte 0x8f in position 23800reading the golden notebook.Root cause
Every
open(...)site involved uses Python's locale-dependent default encoding:interfaces/{gradio,jupyter,streamlit,cellprofiler_plugin}/generate.py— generator writessrc/bilayers/parse.py::load_config— config YAML readsrc/bilayers/schema.py— LinkML schema YAML load at import timetests/regression/test_generate_regression.py—normalize_notebookJSON read andread_text()golden comparisonOn Linux/macOS the locale default is UTF-8; on Windows it's cp1252. nbformat emits non-cp1252 bytes (e.g.
0x8f) in some notebook fields, which breaks the round-trip on Windows. The bug is silent on Linux and fails closed on Windows.Fix
Pass
encoding="utf-8"at every read/write of a file that is canonically UTF-8 (.py,.ipynb,.yaml). Eight sites; the change is mechanical.Test plan
pytest tests/on Windows (Python 3.14, venv): 17 / 17 pass. Previously the regression test failed on this platform.bilayers_cli generate tests/fixtures/classical_segmentation/config.yamlon Windows: all four interfaces generate cleanly. Previously Jupyter failed.bilayers_cli generate --cli ...emitspython -m classical_segmentation --folder /bilayers/input_images --threshold_method otsu --min_size 5 --max_size 20 --save_dir /bilayers/output_images.ruff check src tests noxfile.py→All checks passed!.test.ymlworkflow onubuntu-latest+ Python 3.9): green — https://github.com/xX-its-amit-Xx/bilayers/actions/runs/26635076654. The Linux side is unchanged by this patch (default encoding was already UTF-8 there), which is the intended outcome.Open question for maintainers
noxfile.pyreads / writes a handful oftmp_path("…").txtfiles withoutencoding=. They currently only hold ASCII (platform tags, docker image names, algorithm folder names) so the same bug doesn't bite there. Happy to normalize them here for consistency or leave for a separate noxfile cleanup — your call.