Repository navigation
Conversation
|
Hi! Responding to your Jinja comparison gist, please just read the basic documentation: https://jinja.palletsprojects.com/en/stable/templates/ Few of the things that you say are “Structurally prevented” are actually prevented in any way, e.g.
I think it’s still not clear to you what MarkupSafe does. Or you know it but your LLM picks up obsolete text and you accept its output unquestioningly. In any case, custom HTML is very much possible, please stop going back to pretending it isn’t.
Not at all, we’re very free to design blocks however we like, e.g. very trivially you could extend a chain at the start (there are also other solution such as a nested block in the
{% block thing %}
{% if thing == "foo" %}
foo
{% elif thing == "bar" %}
bar
{% else %}
baz
{% endif %}
{% endblock %}
{% extends "parent.j2" %}
{% block thing %}
{% if thing == "spam" %}
spam!
{% else %}
{{ super() }}
{% endif %}
{% endblock %}something else: why do you paste in the javascript in multiple places? Shouldn’t it be loaded once and reused? |
|
Hi @flying-sheep, appreciate the engagement. On MarkupSafe: On custom HTML: my concern covers both bespoke visualizations and, more importantly, pre-existing HTML representations. Third-party scientific packages that define custom types typically already have To make the middle-ground concrete, I put up a small POC on our fork: settylab/anndata#8. It routes the top-level repr through one autoescape-enabled Jinja template and wraps existing formatter-produced fragments in Stepping back, the architectural choice hinges on requirements we haven't explicitly agreed on:
On JavaScript composition: you're right that The Jinja question is forward-looking (reducing future XSS risk through default-safe idioms, type-level trust discipline) rather than correcting a present defect, so the trade-off depends on which requirements above weigh heaviest. Open to iterating on the POC or the current design. |
I disagree. I might have been unclear in why I brought up jinja but using it won’t and can’t make anything “structurally impossible”. To clarify: I simply consider jinja a better replacement for functions that are desigened to turn structured data into HTML, compared to simple functions that use Python string manipulation APIs. The approach “mark pieces of data as markup safe before passing it into a rendering black box” works better than having a bunch of parameters to a function, some of which contain valid markup while others contain data-derived raw strings that need to be escaped in order to not accidentally breaking page layout because there’s a stray “<”. Safety against malice isn’t the primary concern, it’s more about robustness, where safety is achieved along the way.
That’s a realistic pattern, but I think we’re not one one page about what “extending” means in the context of this PR. I mean by it that there are other packages building on anndata, which might want to both hook into the existing structure (e.g. add an attribute or so next to the others, but otherwise reuse the rest as-is) and also use the existing render machinery for stylistically integrated rendering of the new parts. You’re talking about integrating 3rd party HTML representations while extending.
Ah I saw it being called in multiple places and assumed it’d be included multiple times per repr. I think we can’t do better than one inclusion per repr. We can however improve runtime behavior – it could create a global API and reuse it if it finds it instead of creating a copy of the whole API. |
Two related robustness fixes to the inlined repr JS:
1. `repr.js`: replace the last remaining `innerHTML = \`<h3 id="${modalTitleId}">…\`` in the README modal with plain DOM construction (`createElement`/`textContent`/`appendChild`). The literal `id="${modalTitleId}"` substring inside the inlined JS source was being regex-matched by the Jupyter-compatibility tests as a duplicate HTML ID attribute across cells. Using DOM APIs removes the problematic substring entirely and matches the surrounding code style.
2. `javascript.py`: wrap the per-container initialisation body in an `install-once` guard. Every cell still ships the full source so any cell stays self-sufficient across deletion, reorder, or notebook reopen, but only the first to execute actually installs `window.anndataRepr`; subsequent cells reuse the installed `init(container)`. Addresses the runtime-redundancy concern without giving up per-cell portability.
…hover Replace iter_outer as the section-iteration source in the repr, and derive the canonical section list from get_literal_members(AnnDataElem). Why not iter_outer: iter_outer yields (name, getattr(adata, name)) pairs, and propagates the first exception it hits — so a single broken section (corrupt aligned mapping, subclass with a crashing property, etc.) terminates the generator mid-iteration. _repr_html_'s top-level except catches that, the repr returns None, and the whole cell output disappears. The adversarial "Evil AnnData" case hit this. Iterating get_literal_members(AnnDataElem) ourselves and doing the getattr inside each section's try/except isolates the failure: a broken section renders as an error placeholder and the remaining sections still appear. iter_outer stays for callers that want strict semantics (AnnData.__str__, to_memory, _reduce, I/O). Also: use the same Literal as the single source of truth wherever the repr previously derived a set of section names from iter_outer (_detect_unknown_sections, _get_custom_sections_by_position). Those helpers now compute the set locally and no longer need the caller to thread it through. CSS fix: .anndata-entry:hover .anndata-entry__copy propagated the :hover state to every ancestor entry, revealing every ancestor's copy button when a deeply-nested row was hovered. Scope the trigger to the entry's own row: the entry itself for plain div rows (which never contain nested entries) and the <summary> for expandable rows (nested children live in .anndata-entry__nested-content, outside the summary).
|
On JS composition: taken. Just pushed an On extensibility and the Styler analogy: useful pointer, and it maps cleanly onto the case you described. Packages that hook into anndata's structure (adding an attribute or section next to the others) and want stylistically integrated rendering of the new parts. The current design is already shaped similarly: On Jinja / the POC: the reframe is useful. Landing on robustness (one trust boundary + autoescape vs. threading mixed markup/data through ad-hoc |
# Conflicts: # .vscode/settings.json # src/anndata/_settings.pyi # tests/test_repr.py
Upstream migrated `settings` to `scverse_misc` (scverse#2446), replacing the hand-rolled `SettingsManager` with a pydantic `scverse_misc.Settings` subclass and deleting `_settings.pyi` and `tests/test_settings.py`. Conflicts resolved in: * `src/anndata/_settings.py` -- port the eleven `repr_html_*` options from `settings.register(...)` calls to pydantic field declarations on the new `Settings` class, keeping their defaults (from `_repr_constants`) and descriptions unchanged. `validate_int`/`validate_bool` were plain type checks, so the fields are declared as bare `int`/`bool` and pydantic now performs the same validation. Environment-variable configuration (`ANNDATA_REPR_HTML_*`) is preserved by `scverse_misc`'s `anndata_` env prefix, so `get_from_env` plumbing is no longer needed. * `src/anndata/_settings.pyi` -- accept upstream's deletion. The stub typed `SettingsManager`, which no longer exists; pydantic supplies the field types natively. `tests/test_settings.py` is likewise removed by upstream; it contained no HTML-repr coverage. The repr settings remain exercised by `tests/repr/`.
…keys Upstream switched the `disallow_forward_slash_in_h5ad` default to `True` (scverse#2436), so a `/` in a key now aborts an h5ad write rather than emitting a deprecation warning. `validate_key()` still hardcoded the old severity, which made the repr advertise a slash key as merely deprecated while the corresponding write raised `ValueError`. Grade the key against the live setting instead of a constant, so the badge the repr shows matches what a write would actually do, and keep the softer "(deprecated)" wording for users who opt back out. The branch carried a canary test asserting the old behaviour; replace it with coverage of both settings states.
# Conflicts: # .vscode/settings.json # biome.jsonc # pyproject.toml # tests/test_repr.py
Also satisfy the stricter lint/typing checks from main (ISC004, ANN401, mypy over src and tests): make `get_setting` generic, type `register_formatter` overloads, and narrow duck-typed helpers.
Stability / performance - Render X first; validate repr settings (pydantic ranges) and read them directly (fixes the undeclared `repr_html_max_string_length`) - Read category colors once per column (lazy: 1 dask compute instead of 3) - Bound unique counts per obs/var table (20M scanned values) - Vectorized serializability check for long lists in uns - Cap column names listed for DataFrames in obsm/varm - Unknown-attribute detection no longer computes pending cached properties - CSS follows the page color scheme instead of the OS (no dark repr on light, unthemed pages); JS install keyed by anndata version; inlined CSS/JS stripped of comments/indentation (~47 KB -> ~29 KB per output) Correctness - `error` takes precedence over previews (as documented) - Nested AnnData: isinstance (subclasses), inherits explicit limits - Raw: unique container ids, shared index preview, consistent "vars" label - Hide memory estimate for lazy AnnData, clarify it for backed - isinstance checks via anndata.abc/compat types; has_xp_base for array API - Sparse labels from storage format (views read e.g. `csr_matrix`) Maintainability - Registry: copy-on-write, re-registering a class replaces it, decorator returns the class, unregister by class or section name - One <details> section builder, one header-badge implementation, one lazy-column navigation helper, shared section tooltips, pluralized counts - FallbackFormatter rewritten around safe-access helpers (same guarantees) - Remove dead code (unused helpers and constants, get_setting) - Tests: repr-only warning filter, registry isolation fixture, regression tests for all of the above
- numpy scalars in uns are scalars again (not array-API arrays)
- nan/inf previews no longer crash the float formatter
- warn for object columns with non-string values (write_h5ad fails)
- a failing SectionFormatter renders an error section instead of vanishing;
broken X shows the error message
- uns entries get `section="uns"`, so `sections=("uns",)` formatters apply
- colors stored as arrays (from disk) or dask arrays (read_lazy) render as
swatches, reading only the preview and nothing in metadata-only mode
- lazy DataFrames in obsm/varm (Dataset2D) render from metadata instead of
being flagged as unknown type
- dask arrays always show their shape and mark sparse chunks
- themes: VS Code high contrast and "auto" themes (e.g. Furo) follow the OS
- X row aligned with the entry columns; type cells ellipsize when narrow
- subclass badge no longer repeats the class name; pluralized raw labels
…ted cases Cases are small decorated functions grouped into seven categories with a table of contents, expectations per case, coverage tags, per-case isolation (tracebacks render in the page), optional-dependency skips, CLI filtering (--list, --only, -o, --strict) and headless-Chrome theme probes (--browser-check). Adds cases for previously uncovered sections, dtypes, storage states, scale limits, themes and failure modes.
- probing unrecognized attributes no longer surfaces warnings from
deprecated properties (e.g. MuData's mod_names FutureWarning)
- the subclass badge is only shown for actual AnnData subclasses, not for
AnnData-like containers such as MuData
- pluralized column counts ("1 column")
…label as documented
|
Picking up the trust-boundary thread: both variants are now built as stacked PRs on our fork, kept in sync with this branch, and ready to merge into it on demand. Maintainers have write access to settylab/anndata, so either one can be merged straight into Option A, MarkupSafe only: settylab#10 Every internal fragment is built as Option B, Jinja + MarkupSafe: settylab#9 The same trust boundary, plus templates ( Both are backward compatible: the
One behavior change applies to both: the public My preference is A. It gives the robustness you described with a third of B's runtime cost, and the rendering logic stays in one language. I'd also understand picking B for the template structure you outlined. Either way, I'd suggest landing it as a follow-up (or merging it here) once you're happy with the rest, to keep this diff reviewable. Your call. |
Rich HTML representation for AnnData
Summary
Implements rich HTML representation (
_repr_html_) for AnnData objects in Jupyter notebooks. Builds on previous draft PRs (#784, #694, #521, #346) with a complete, production-ready implementation.Live Demo (61 cases, updated Oct 6, 2026) | Reviewer's Guide (technical details, design decisions, extensibility examples)
Screenshot
Features
Interactive Display
.rawsection showing unprocessed data (Reportn_varsof.rawin__repr__#349)Visual Indicators
unspalettes (e.g.,cell_type_colors)read_lazy) objects and subclassesunsvalues, color swatches for*_colorsuns["README"])Serialization Warnings
Proactively warns about data that won't serialize:
write_h5adfails)/(severity followssettings.disallow_forward_slash_in_h5ad)Compatibility
.anndata-reprprevents style conflictsread_lazy()(categories, colors), strict metadata-only mode, zero reads for backed h5adExtensibility
Three extension mechanisms for ecosystem packages (MuData, SpatialData, TreeData):
obst/vart,mod)See the Reviewer's Guide for examples and API documentation.
Testing
python tests/visual_inspect_repr_html.py(--list,--only,--browser-check)Related
main(anndata 0.13.x):Xaslayers[None](feat!: Unify X and layers #1707), backed sparse without scipy inheritance (feat: allow gpu io insparse_datasetby removingscipyinheritance #1927), array-API acceptance viahas_xp_base(fix: Accept array-API arrays that cannot export via DLPack #2541), validated pydantic settingsAcknowledgments
Thanks to @selmanozleyen (#784), @gtca (#694), @VolkerH (#521), @ivirshup (#346, #675), and @Zethson (#675) for prior work and discussions.
Technical Notes and Edits
Lazy Loading
Constants are in
_repr_constants.py(outside_repr/) to prevent loading ~6K lines onimport anndata. The full module loads only when_repr_html_()is called.Config Changes
pyproject.toml: Addedvartto codespell ignore list (TreeData section name).Edit (Dec 27, 2025)
To simplify review and reduce the diff, I've merged settylab/anndata#3 into this PR. That PR was originally created as a follow-up to explore additional features based on the discussion with @Zethson about SpatialData/MuData extensibility.
What changed:
.rawsection - Expandable row showing unprocessed data (Reportn_varsof.rawin__repr__#349)Edit (Jan 4, 2026)
Moved detailed implementation documentation (architecture, design decisions, extensibility examples, configuration reference) to the Reviewer's Guide to keep this PR description focused on features.
Code refactoring:
html.pyinto focused modules for maintainabilitycomponents.py(badges, buttons, icons)sections.py(obs/var, mapping, uns, raw)core.py(avoids circular imports)utils.pyFormatterContextconsolidates all 6 rendering settings (read once at entry, propagated via context)html.pyreduced from ~2100 to ~740 lines, clean import hierarchyNew features:
read_lazy()AnnData objects (experimental) - indicates when obs/var are xarray-backed(lazy)indicator on columnsBug fixes:
adata-text-mutedclass for uniform appearanceRelated issue discovered:
read_lazy()returns index values as byte-representation strings (e.g.,"b'cell_0'"instead of"cell_0") - seeISSUE_READ_LAZY_INDEX.mdEdit (Jan 6, 2026)
Smart partial loading for
read_lazy()AnnData:Previously, lazy AnnData showed no category previews to avoid disk I/O. Now we do minimal, configurable loading to get richer visualization cheaply: only the first N category labels and their colors are read from storage (not the full column data). New setting
repr_html_max_lazy_categories(default: 100, set to 0 for metadata-only mode).Visual tests reorganized: 8 (Dask), 8b (lazy categories), 8c (metadata-only), 9 (backed).
Edit (Jan 6, 2026 - continued)
FormattedOutput API and architecture:
Clean separation between formatters and renderers - formatters inspect data and produce complete
FormattedOutput, renderers only receiveFormattedOutput(never the original data).The
FormattedOutputdataclass fields were renamed to be self-documenting:meta_contentpreview(text) orpreview_html(HTML)html_content+is_expandable=Trueexpanded_htmlhtml_content+is_expandable=Falsepreview_htmlis_expandableexpanded_html is not Nonetype_htmltype_namevisually)Naming convention:
*_htmlsuffix indicates raw HTML (caller responsible for escaping), plain text fields are auto-escaped.UI/UX improvements:
▼/▲arrows instead of⋯/▲for consistencyEdit (Jan 7, 2026)
Test architecture overhaul:
Tests reorganized from a single file into 10 focused modules for maintainability and parallel execution:
test_repr_core.pytest_repr_sections.pytest_repr_formatters.pytest_repr_ui.pytest_repr_warnings.pytest_repr_registry.pytest_repr_lazy.pytest_html_validator.pyHTMLValidator class (
conftest.py) provides structured HTML assertions:Key features: regex-based (no dependencies), section-aware matching, exact attribute matching to avoid "obs" matching "obsm".
Optional strict validation when dependencies available:
validate_html5()- W3C HTML5 + ARIA (requiresvnu)validate_js()- JavaScript syntax (requiresesprima)Jupyter Notebook/Lab compatibility tests (13 new tests in
TestJupyterNotebookCompatibility):Validates CSS scoping, JavaScript isolation, unique IDs across multiple cells, and Jupyter dark mode support.
Bug fix:
readme-modal-titleID is now unique per container to prevent ID collisions when multiple AnnData objects are displayed in the same notebook.Edit (Jan 8, 2026)
Maintainability improvements:
_render_entry_rowandrender_formatted_entryto eliminate duplicationget_formatter_for()andlist_formatters()methods to FormatterRegistry__init__.pystatic/directorytests/repr/html_validator.pymodule (conftest.py: 960→270 lines)_repr_constants.pyrender_entry_type_cell()signaturelazy.pymodulestatic/css_colors.txtfor easy updatesFile structure changes:
API simplifications:
render_entry_type_cell()now acceptsTypeCellConfigdataclass instead of 10 individual parametersis_lazy_adata(),is_lazy_column(),get_lazy_categories(),get_lazy_categorical_info()importlib.resources.files()(Python 3.9+)Edit (Jan 9, 2026)
Robustness & escaping coverage testing:
Added 108 tests in
test_repr_robustness.pyacross 14 test classes:html.escape()is called at every user-data insertion point using a<b>MARKER</b>probe__repr__,__len__,__sizeof__, properties)Escaping tests trust
html.escape()(stdlib) and only verify it's called at every insertion point, rather than exercising the escaping mechanism itself with attack vectors.Test cleanup:
Removed redundant and overly-specific tests to focus on meaningful coverage. Tests now verify behavior that matters (e.g., XSS escaped, errors visible, truncation applied) rather than testing identical code paths multiple times.
Visual inspection: Consolidated to 26 scenarios with single comprehensive "Evil AnnData" test combining all adversarial patterns.
Fixes:
repr_html_max_readme_sizeto_settings.pyitype stubspytest.warnsfor expected warnings)Updated stats:
Edit (Jan 16, 2026)
Error handling consolidation:
Refactored error handling to use a single
errorfield inFormattedOutputinstead of separateis_hard_errorparameters scattered across the codebase.Key changes:
FormattedOutputerror: str | Nonefield with documented precedence overpreview/preview_htmlFallbackFormatterFormatterRegistry.format_value()render_formatted_entry()is_hard_errorparam, now detects viaoutput.error_validate_key_and_collect_warnings()(key_warnings, is_key_not_serializable)- key issues mark as not serializable, preserving previewError vs Warning separation:
output.error: Hard rendering failure - row highlighted red, error message replaces previewoutput.is_serializable=False: Serialization warning - red background, but preview preservedNew behavior when formatters fail:
This prevents long error messages from appearing in HTML while preserving full details in warnings for debugging. Serialization issues (like non-string keys, lambdas, custom objects) preserve the value preview while showing the reason in the tooltip.
Updated stats:
Edit (Jan 26, 2026)
Review response changes (addressing @flying-sheep's review):
Typing:
Any→objectReplaced all ~95 uses of
Anyacross 7 files. Formatter method signatures now useobj: objectsince AnnData'sunsaccepts genuinely arbitrary objects and formatters handle AnnData-like objects (e.g., MuData) via duck typing.dict[str, Any]with known structure replaced with precise union types.CSS: Native nesting + dark mode + variable dedup
repr.cssto native CSS nesting (&). Selector repetitions of.anndata-reprreduced from 173 to 13. File length unchanged (~1164 lines) because the feature surface is genuinely large (~68 component blocks, 14 dtype colors, copy button, README styling, state variants), not because of repetition.[data-theme="dark"]for Furo/sphinx-book-theme) alongside existing Jupyter/VS Code detection.@media (prefers-color-scheme: dark)block and theme-selector block.&--variant) produce invalid CSS at nesting depth 2+ (browser treats&as:is(parent child), so&--viewbecomes:is(.anndata-repr .anndata-badge)--view). 7 modifier rules flattened to sibling selectors.Security tests simplified
Replaced ~34 attack-vector-heavy tests with 12 focused escaping-coverage tests. Each test puts a
<b>MARKER</b>probe at one user-data insertion point and verifies it appears escaped. RemovedTestCSSAttacks,TestEncodingAttacks; trimmedTestBadColorArrays,TestEvilReadme; consolidatedTestUltimateEvilAnnDatato 1 test. Total: 108 tests (14 classes), down from 123 (16 classes).Other:
FormatterContext.column_namerenamed toFormatterContext.keyFormatterRegistry.format_value()Edit (Oct 6, 2026)
Synced with
main(anndata 0.13.x) and addressed a full code review. No features were removed.Upstream compatibility
Xnow lives inlayers[None](feat!: Unify X and layers #1707): it is shown once, as the first row, and no longer as aNonelayerANN401/ISC004/PD, mypy oversrcandtests)anndata.abc.CSRDataset/CSCDataset,compat.DaskArray/AwkArray/CupySparseMatrix,Dataset2D,isinstance(…, AnnData)(subclasses included),has_xp_basefor array-API arraysStability and performance
max_depth=0or negative limits are rejected) and read directly;repr_html_max_string_lengthis now a declared setting (12 settings)Dataset2D) render from metadata, uns colors render as swatchescached_propertyscolor-schemeinstead of the OS (no dark repr on light, unthemed pages); added VS Code high contrast anddata-theme="auto"(Furo)Correctness
errortakes precedence over previews (as documented); failing custom sections and.Xshow their error instead of vanishingsection="uns", sosections=("uns",)formatters applynan/inf, colors stored as arrays, dask arrays with sparse chunks, sparse views (csr_matrixinstead of the view class) render correctlyMaintainability
@register_formatterreturns the class, unregister by class or section name<details>section builder, one header-badge implementation, one lazy-column helper;FallbackFormatterrewritten around two safe-access helpers with the same guarantees; dead code removed--browser-checktheme probes (18/18 pass)Future-Proofing: Related PRs and Issues
Already Handled
_reprXstored aslayers[None]Xrendered once (first row), hidden from layersisinstanceonanndata.abc.CSRDataset/CSCDataset, labels from.formatArrayAPIFormatterviacompat.has_xp_base, duck-typed second tier for tensorsMay Require Updates When Merged
LazyCategoricalDtypeAPIlazy.pyreadsCategoricalArray._categories/_orderedthrough one helper; switch ton_categories/head_categories(n)obs_check_series_serializability)Internal API Usage Inventory
lazy.py:_get_backing_array()col.variable._data.array(xarray internals)isinstance(dtype, LazyCategoricalDtype)lazy.py:_category_values()CategoricalArray._categories(+_ordered)dtype.n_categories,dtype.head_categories(n)ArrayAPIFormattertier 2shape/dtype/ndim