Skip to content

feat: Add HTML representation - #2236

Open
katosh wants to merge 223 commits into
scverse:mainfrom
settylab:html_rep
Open

katosh wants to merge 223 commits into
scverse:mainfrom
settylab:html_rep

Conversation

@katosh

@katosh katosh commented Nov 29, 2025 •

Copy link
Copy Markdown
Contributor

Rich HTML representation for AnnData

Summary

Implements rich HTML representation (_repr_html_) for AnnData objects in Jupyter notebooks. Builds on previous draft PRs (#784, #694, #521, #346) with a complete, production-ready implementation.

Live Demo (61 cases, updated Oct 6, 2026) | Reviewer's Guide (technical details, design decisions, extensibility examples)

Screenshot

screenshot2

Features

Interactive Display

  • Foldable sections with auto-collapse for large datasets
  • Search/filter with regex and case-sensitive toggles
  • Copy-to-clipboard for field names
  • Nested AnnData expansion with configurable depth
  • .raw section showing unprocessed data (Report n_vars of .raw in __repr__ #349)

Visual Indicators

  • Category colors from uns palettes (e.g., cell_type_colors)
  • Type badges for views, backed mode, lazy (read_lazy) objects and subclasses
  • Serialization warnings for data that won't write to H5AD/Zarr
  • Value previews for simple uns values, color swatches for *_colors
  • README support via modal (plain text from uns["README"])
  • Memory info in footer (omitted for lazy objects)

Serialization Warnings

Proactively warns about data that won't serialize:

Level Issue Related
🔴 Error datetime64/timedelta64 in obs/var #455, #2238
🔴 Error Non-string keys #321
🔴 Error Object columns with dicts/lists/custom objects #1923, #567, #636
🔴 Error Non-serializable types in uns
🟡 Warning Object columns with non-string values (write_h5ad fails)
🟡 Warning Keys with / (severity follows settings.disallow_forward_slash_in_h5ad) #1447, #2099
🟡 Warning String→categorical auto-conversion #534, #926

Compatibility

  • Dark mode follows the page: JupyterLab, VS Code (incl. high contrast), Furo (incl. auto), pydata/sphinx-book-theme; light on unthemed pages
  • No-JS / no-CSS fallback with graceful degradation
  • JupyterLab safe - CSS scoped to .anndata-repr prevents style conflicts
  • Lazy-loading safe - bounded partial reads for read_lazy() (categories, colors), strict metadata-only mode, zero reads for backed h5ad
  • Bounded cost - unique counts, column lists and serializability checks are capped for very large objects
  • Zero dependencies added

Extensibility

Three extension mechanisms for ecosystem packages (MuData, SpatialData, TreeData):

  1. TypeFormatter - Custom visualization for value types
  2. SectionFormatter - Add new sections (e.g., obst/vart, mod)
  3. Building blocks - CSS/JS/helpers for packages needing full control

See the Reviewer's Guide for examples and API documentation.

Testing

  • 673 unit tests organized by responsibility (core, sections, formatters, UI, warnings, registry, lazy, robustness, Jupyter compatibility)
  • 111 escaping/robustness tests covering escaping at every user-data insertion point, broken objects, size bombs, visible failures
  • HTMLValidator for structured HTML assertions (section-aware, no external dependencies)
  • 61 visual inspection cases in 7 categories with per-case expectations and headless-Chrome theme probes: python tests/visual_inspect_repr_html.py (--list, --only, --browser-check)

Related

Acknowledgments

Thanks to @selmanozleyen (#784), @gtca (#694), @VolkerH (#521), @ivirshup (#346, #675), and @Zethson (#675) for prior work and discussions.


Technical Notes and Edits

Lazy Loading

Constants are in _repr_constants.py (outside _repr/) to prevent loading ~6K lines on import anndata. The full module loads only when _repr_html_() is called.

Config Changes

pyproject.toml: Added vart to codespell ignore list (TreeData section name).


Edit (Dec 27, 2025)

To simplify review and reduce the diff, I've merged settylab/anndata#3 into this PR. That PR was originally created as a follow-up to explore additional features based on the discussion with @Zethson about SpatialData/MuData extensibility.

What changed:

  • Exported building blocks - CSS, JavaScript, and rendering helpers for external packages to build custom reprs while reusing anndata's styling
  • .raw section - Expandable row showing unprocessed data (Report n_vars of .raw in __repr__ #349)
  • Enhanced serialization warnings - Extended to cover datetime64, non-string keys, slashes in keys, and all sections
  • Regex search - Case-sensitive and regex toggles for filtering
  • Robust error handling - Failed sections show visible error indicators instead of being silently hidden

Edit (Jan 4, 2026)

Moved detailed implementation documentation (architecture, design decisions, extensibility examples, configuration reference) to the Reviewer's Guide to keep this PR description focused on features.

Code refactoring:

  • Split html.py into focused modules for maintainability
  • UI components extracted to components.py (badges, buttons, icons)
  • Section renderers moved to sections.py (obs/var, mapping, uns, raw)
  • Shared rendering primitives extracted to core.py (avoids circular imports)
  • Preview utilities moved to utils.py
  • FormatterContext consolidates all 6 rendering settings (read once at entry, propagated via context)
  • Result: html.py reduced from ~2100 to ~740 lines, clean import hierarchy

New features:

  • "Lazy" badge for read_lazy() AnnData objects (experimental) - indicates when obs/var are xarray-backed
  • Visual test for lazy AnnData (9b) - demonstrates lazy loading with (lazy) indicator on columns

Bug fixes:

  • Consistent meta column styling - all meta column text now uses adata-text-muted class for uniform appearance
  • Bytes index decoding - properly decode bytes values in index previews

Related issue discovered:

  • read_lazy() returns index values as byte-representation strings (e.g., "b'cell_0'" instead of "cell_0") - see ISSUE_READ_LAZY_INDEX.md

Edit (Jan 6, 2026)

Smart partial loading for read_lazy() AnnData:

Previously, lazy AnnData showed no category previews to avoid disk I/O. Now we do minimal, configurable loading to get richer visualization cheaply: only the first N category labels and their colors are read from storage (not the full column data). New setting repr_html_max_lazy_categories (default: 100, set to 0 for metadata-only mode).

Visual tests reorganized: 8 (Dask), 8b (lazy categories), 8c (metadata-only), 9 (backed).


Edit (Jan 6, 2026 - continued)

FormattedOutput API and architecture:

Clean separation between formatters and renderers - formatters inspect data and produce complete FormattedOutput, renderers only receive FormattedOutput (never the original data).

The FormattedOutput dataclass fields were renamed to be self-documenting:

Old Field New Field Purpose
meta_content preview (text) or preview_html (HTML) Preview column content
html_content + is_expandable=True expanded_html Collapsible content below row
html_content + is_expandable=False preview_html Inline preview in preview column
is_expandable Removed Use expanded_html is not None
(new) type_html Custom HTML for type column (replaces type_name visually)

Naming convention: *_html suffix indicates raw HTML (caller responsible for escaping), plain text fields are auto-escaped.

UI/UX improvements:

  • Zebra striping for section rows (alternating background colors)
  • Expand buttons now use ▼/▲ arrows instead of ⋯/▲ for consistency
  • No borders between entries within sections (cleaner look)
  • Fixed button alignment - Expand and wrap buttons now align properly
  • Category list styling - explicit muted color ensures consistent appearance in nested contexts

Edit (Jan 7, 2026)

Test architecture overhaul:

Tests reorganized from a single file into 10 focused modules for maintainability and parallel execution:

File Focus
test_repr_core.py HTML validation, settings, badges
test_repr_sections.py Section rendering (obs, var, uns, etc.)
test_repr_formatters.py Type-specific formatters
test_repr_ui.py Folding, colors, search, clipboard
test_repr_warnings.py Serialization warnings
test_repr_registry.py Plugin registry
test_repr_lazy.py Lazy AnnData support
test_html_validator.py HTMLValidator tests + Jupyter compatibility

HTMLValidator class (conftest.py) provides structured HTML assertions:

v = validate_html(html)
v.assert_section_exists("obs")
v.assert_section_contains_entry("obs", "batch")
v.assert_section_initially_collapsed("obs")  # or _not_initially_collapsed

Key features: regex-based (no dependencies), section-aware matching, exact attribute matching to avoid "obs" matching "obsm".

Optional strict validation when dependencies available:

  • validate_html5() - W3C HTML5 + ARIA (requires vnu)
  • validate_js() - JavaScript syntax (requires esprima)

Jupyter Notebook/Lab compatibility tests (13 new tests in TestJupyterNotebookCompatibility):

Validates CSS scoping, JavaScript isolation, unique IDs across multiple cells, and Jupyter dark mode support.

Bug fix: readme-modal-title ID is now unique per container to prevent ID collisions when multiple AnnData objects are displayed in the same notebook.


Edit (Jan 8, 2026)

Maintainability improvements:

Fix Description
Entry rendering Consolidated _render_entry_row and render_formatted_entry to eliminate duplication
Debug logging Added get_formatter_for() and list_formatters() methods to FormatterRegistry
Import hierarchy Documented module dependency tree at top of __init__.py
Static assets Moved CSS (~1060 lines), JS (~380 lines), markdown parser (~150 lines) to static/ directory
FormattedOutput docs Enhanced field documentation with precedence rules and CSS class reference
HTMLValidator Moved to separate tests/repr/html_validator.py module (conftest.py: 960→270 lines)
Magic strings Extracted CSS classes and section names to _repr_constants.py
TypeCellConfig Added dataclass to simplify render_entry_type_cell() signature
Lazy module Consolidated lazy loading utilities to new lazy.py module
CSS colors Moved 148 CSS color names to static/css_colors.txt for easy updates

File structure changes:

src/anndata/_repr/
├── static/                  # NEW: Static assets directory
│   ├── __init__.py
│   ├── repr.css             # CSS template (~1060 lines)
│   ├── repr.js              # JavaScript (~380 lines)
│   ├── markdown-parser.js   # Markdown parser (~150 lines)
│   └── css_colors.txt       # CSS named colors (148 colors)
├── lazy.py                  # NEW: Lazy loading utilities
└── ...

API simplifications:

  • render_entry_type_cell() now accepts TypeCellConfig dataclass instead of 10 individual parameters
  • Lazy utilities consolidated: is_lazy_adata(), is_lazy_column(), get_lazy_categories(), get_lazy_categorical_info()
  • Static assets loaded via importlib.resources.files() (Python 3.9+)

Edit (Jan 9, 2026)

Robustness & escaping coverage testing:

Added 108 tests in test_repr_robustness.py across 14 test classes:

  • Escaping coverage (12 tests): verifies html.escape() is called at every user-data insertion point using a <b>MARKER</b> probe
  • Unicode edge cases (emoji, CJK, RTL override, zero-width chars)
  • Broken objects (crashing __repr__, __len__, __sizeof__, properties)
  • Size handling (huge strings, many categories, deep nesting)
  • Color array robustness (too many/few, invalid formats, empty)
  • Thread safety (concurrent repr generation)

Escaping tests trust html.escape() (stdlib) and only verify it's called at every insertion point, rather than exercising the escaping mechanism itself with attack vectors.

Test cleanup:

Removed redundant and overly-specific tests to focus on meaningful coverage. Tests now verify behavior that matters (e.g., XSS escaped, errors visible, truncation applied) rather than testing identical code paths multiple times.

Visual inspection: Consolidated to 26 scenarios with single comprehensive "Evil AnnData" test combining all adversarial patterns.

Fixes:

  • Added repr_html_max_readme_size to _settings.pyi type stubs
  • Fixed strict warnings compatibility (pytest.warns for expected warnings)
  • Section error truncation now shows "..." indicator when message exceeds limit

Updated stats:

Metric Value
Total tests 601
Robustness tests 108 (14 test classes)
Visual scenarios 26
Settings 11

Edit (Jan 16, 2026)

Error handling consolidation:

Refactored error handling to use a single error field in FormattedOutput instead of separate is_hard_error parameters scattered across the codebase.

Key changes:

Component Change
FormattedOutput Added error: str | None field with documented precedence over preview/preview_html
FallbackFormatter Made bulletproof - wraps every attribute access in try/except, checks serializability and includes reason in warnings
FormatterRegistry.format_value() Accumulates failed formatters instead of stopping at first failure
render_formatted_entry() Removed is_hard_error param, now detects via output.error
_validate_key_and_collect_warnings() Returns (key_warnings, is_key_not_serializable) - key issues mark as not serializable, preserving preview

Error vs Warning separation:

  • output.error: Hard rendering failure - row highlighted red, error message replaces preview
  • output.is_serializable=False: Serialization warning - red background, but preview preserved
  • Tooltip format: "Not serializable to H5AD/Zarr: {reason}" uses ":" to connect to reason, ";" separates independent warnings

New behavior when formatters fail:

  1. Registry tries all matching formatters in priority order
  2. Failed formatters are accumulated (full message for warnings, type-only for HTML)
  3. If a later formatter succeeds: warnings emitted about earlier failures
  4. If all fail: accumulated errors passed to fallback formatter

This prevents long error messages from appearing in HTML while preserving full details in warnings for debugging. Serialization issues (like non-string keys, lambdas, custom objects) preserve the value preview while showing the reason in the tooltip.

Updated stats:

Metric Value
Total tests 601
Robustness tests 108 (14 test classes)
Source lines ~6,500 Python + ~2,130 static assets
Test lines ~10,450 (13 files)

Edit (Jan 26, 2026)

Review response changes (addressing @flying-sheep's review):

Typing: Any → object

Replaced all ~95 uses of Any across 7 files. Formatter method signatures now use obj: object since AnnData's uns accepts genuinely arbitrary objects and formatters handle AnnData-like objects (e.g., MuData) via duck typing. dict[str, Any] with known structure replaced with precise union types.

CSS: Native nesting + dark mode + variable dedup

  • Full conversion of repr.css to native CSS nesting (&). Selector repetitions of .anndata-repr reduced from 173 to 13. File length unchanged (~1164 lines) because the feature surface is genuinely large (~68 component blocks, 14 dtype colors, copy button, README styling, state variants), not because of repetition.
  • Added Sphinx theme dark mode selectors ([data-theme="dark"] for Furo/sphinx-book-theme) alongside existing Jupyter/VS Code detection.
  • Dark mode variables (~35 declarations) deduplicated: defined once in Python and substituted into both the @media (prefers-color-scheme: dark) block and theme-selector block.
  • Limitation: BEM modifiers (&--variant) produce invalid CSS at nesting depth 2+ (browser treats & as :is(parent child), so &--view becomes :is(.anndata-repr .anndata-badge)--view). 7 modifier rules flattened to sibling selectors.

Security tests simplified

Replaced ~34 attack-vector-heavy tests with 12 focused escaping-coverage tests. Each test puts a <b>MARKER</b> probe at one user-data insertion point and verifies it appears escaped. Removed TestCSSAttacks, TestEncodingAttacks; trimmed TestBadColorArrays, TestEvilReadme; consolidated TestUltimateEvilAnnData to 1 test. Total: 108 tests (14 classes), down from 123 (16 classes).

Other:

  • FormatterContext.column_name renamed to FormatterContext.key
  • Key validation moved into FormatterRegistry.format_value()
  • HTML validator tests updated for native CSS nesting (vnu doesn't support nesting syntax yet, so CSS parse errors are filtered)

Edit (Oct 6, 2026)

Synced with main (anndata 0.13.x) and addressed a full code review. No features were removed.

Upstream compatibility

  • X now lives in layers[None] (feat!: Unify X and layers #1707): it is shown once, as the first row, and no longer as a None layer
  • Passes the stricter checks now run in CI (ruff ANN401/ISC004/PD, mypy over src and tests)
  • Type checks use upstream types instead of class-name strings: anndata.abc.CSRDataset/CSCDataset, compat.DaskArray/AwkArray/CupySparseMatrix, Dataset2D, isinstance(…, AnnData) (subclasses included), has_xp_base for array-API arrays

Stability and performance

  • Settings are validated (e.g. max_depth=0 or negative limits are rejected) and read directly; repr_html_max_string_length is now a declared setting (12 settings)
  • Lazy AnnData: colors are read once per column (was 3 dask computes), metadata-only mode reads no colors, lazy obsm DataFrames (Dataset2D) render from metadata, uns colors render as swatches
  • Unique counts are capped per table (20M scanned values), long uns lists use a vectorized serializability check (0.78 s → ~35 ms for 2M items), obsm DataFrame column lists are capped at 100 names
  • Detecting unrecognized attributes no longer computes pending cached_propertys
  • Inlined CSS/JS ~47 KB → ~29 KB per output; JS init is keyed by anndata version
  • Dark mode follows the page's color-scheme instead of the OS (no dark repr on light, unthemed pages); added VS Code high contrast and data-theme="auto" (Furo)

Correctness

  • error takes precedence over previews (as documented); failing custom sections and .X show their error instead of vanishing
  • uns entries carry section="uns", so sections=("uns",) formatters apply
  • numpy scalars, nan/inf, colors stored as arrays, dask arrays with sparse chunks, sparse views (csr_matrix instead of the view class) render correctly
  • Nested AnnData inherits explicit limits; Raw uses unique ids; layout fixes for the X row and narrow outputs

Maintainability

  • Registry: re-registering a formatter class replaces it (no duplicates when re-running cells), @register_formatter returns the class, unregister by class or section name
  • One <details> section builder, one header-badge implementation, one lazy-column helper; FallbackFormatter rewritten around two safe-access helpers with the same guarantees; dead code removed
  • Tests: registry isolation and repr-scoped warning filters as fixtures; regression tests for all of the above (673 total)
  • Visual inspection reorganized into 61 isolated cases in 7 categories, with a table of contents, expectations per case and --browser-check theme probes (18/18 pass)
Future-Proofing: Related PRs and Issues

Already Handled

PR/Issue Description Status in _repr
#1707 X stored as layers[None] ✅ X rendered once (first row), hidden from layers
#1927 Backed sparse without scipy inheritance ✅ isinstance on anndata.abc.CSRDataset/CSCDataset, labels from .format
#2063, #2541 Array-API support ✅ ArrayAPIFormatter via compat.has_xp_base, duck-typed second tier for tensors

May Require Updates When Merged

PR/Issue Description Current Handling
#2288 LazyCategoricalDtype API lazy.py reads CategoricalArray._categories/_ordered through one helper; switch to n_categories/head_categories(n)
#1923 List data types in obs Marked not serializable (_check_series_serializability)

Internal API Usage Inventory

Location Current Pattern Replacement Opportunity
lazy.py:_get_backing_array() col.variable._data.array (xarray internals) Post-#2288: isinstance(dtype, LazyCategoricalDtype)
lazy.py:_category_values() CategoricalArray._categories (+ _ordered) Post-#2288: dtype.n_categories, dtype.head_categories(n)
ArrayAPIFormatter tier 2 Duck typing on shape/dtype/ndim Keep for tensors outside the array-API protocol

@scverse scverse deleted a comment from azure-pipelines Bot Apr 20, 2026
@flying-sheep

flying-sheep commented Apr 20, 2026 •

Copy link
Copy Markdown
Member

Hi! Responding to your Jinja comparison gist, please just read the basic documentation: https://jinja.palletsprojects.com/en/stable/templates/

Few of the things that you say are “Structurally prevented” are actually prevented in any way, e.g.

4. Ecosystem Custom HTML Structurally impossible without |safe.

I think it’s still not clear to you what MarkupSafe does. Or you know it but your LLM picks up obsolete text and you accept its output unquestioningly. In any case, custom HTML is very much possible, please stop going back to pretending it isn’t.

5. Per-Type Dispatch […] Jinja template blocks override entire sections […] Ecosystem packages would then need to extend{% elif %} chains, reimplementing the dispatch logic in Jinja syntax.

Not at all, we’re very free to design blocks however we like, e.g. very trivially you could extend a chain at the start (there are also other solution such as a nested block in the else branch):

parent.j2

{% block thing %}
{% if thing == "foo" %}
  foo
{% elif thing == "bar" %}
  bar
{% else %}
  baz
{% endif %}
{% endblock %}

child.j2

{% extends "parent.j2" %}
{% block thing %}
{% if thing == "spam" %}
  spam!
{% else %}
  {{ super() }}
{% endif %}
{% endblock %}

something else: why do you paste in the javascript in multiple places? Shouldn’t it be loaded once and reused?

@katosh

katosh commented Apr 20, 2026

Copy link
Copy Markdown
Contributor Author

Hi @flying-sheep, appreciate the engagement.

On MarkupSafe: Markup is the primary MarkupSafe mechanism; the gist was loose in calling |safe "the only escape hatch." That's a fair precision correction and the substantive argument below doesn't depend on it.

On custom HTML: my concern covers both bespoke visualizations and, more importantly, pre-existing HTML representations. Third-party scientific packages that define custom types typically already have _repr_html_ implementations (pandas, ete3, plotly, pygments, matplotlib, bokeh, bionty term viewers, TreeData tree renderers, and domain-specific packages written by individual scientists). The realistic extension pattern is Markup(obj._repr_html_()) at the boundary — reimplementing these visualizations as Jinja templates is a big ask for maintainers whose primary goal is their own scientific package, not integration with a specific template engine. Under that pattern, Jinja's safety benefit for third-party formatters reduces to auditable Markup assertions at integration boundaries rather than autoescape-by-default, and I haven't yet seen a counter-argument that doesn't reduce to the structural-impossibility framing we've already revised past.

To make the middle-ground concrete, I put up a small POC on our fork: settylab/anndata#8. It routes the top-level repr through one autoescape-enabled Jinja template and wraps existing formatter-produced fragments in markupsafe.Markup at the boundary.

Stepping back, the architectural choice hinges on requirements we haven't explicitly agreed on:

  1. Safety against malicious .h5ad files at every user-data insertion point.
  2. Extensibility for third-party packages with minimal authoring burden, ideally reusing existing _repr_html_.
  3. Maintainability without cross-language contracts that can drift.
  4. Dependency cost justified by concrete gains?
  5. Contributor onboarding for Python developers without prior template-engine experience.

On JavaScript composition: you're right that repr.js is currently inlined per top-level repr in javascript.py, so N reprs on a page means N copies of the definition. The current design makes each cell self-sufficient because in Jupyter I can't rely on cell execution order, cells can be deleted or rerun, and on notebook reopen each cell's <script> re-runs independently — so any "load once, init many" pattern risks initialisation failures for reprs that are displayed before whichever cell happens to carry the definition. I considered alternatives (idempotency guard, data-URI <script src>, IPython.display.Javascript, build-time minification) but each regresses on at least one axis — file size, runtime cost, cross-context portability (static HTML export, untrusted notebooks, kernel restart), or added complexity — so I landed on the inlined approach. Is there a pattern you have in mind that avoids those trade-offs?

The Jinja question is forward-looking (reducing future XSS risk through default-safe idioms, type-level trust discipline) rather than correcting a present defect, so the trade-off depends on which requirements above weigh heaviest. Open to iterating on the POC or the current design.

@flying-sheep

flying-sheep commented Apr 21, 2026 •

Copy link
Copy Markdown
Member

[…] calling |safe "the only escape hatch." That's a fair precision correction and the substantive argument below doesn't depend on it.

I disagree. I might have been unclear in why I brought up jinja but using it won’t and can’t make anything “structurally impossible”. To clarify: I simply consider jinja a better replacement for functions that are desigened to turn structured data into HTML, compared to simple functions that use Python string manipulation APIs. The approach “mark pieces of data as markup safe before passing it into a rendering black box” works better than having a bunch of parameters to a function, some of which contain valid markup while others contain data-derived raw strings that need to be escaped in order to not accidentally breaking page layout because there’s a stray “<”. Safety against malice isn’t the primary concern, it’s more about robustness, where safety is achieved along the way.

The realistic extension pattern is Markup(obj._repr_html_()) at the boundary

That’s a realistic pattern, but I think we’re not one one page about what “extending” means in the context of this PR. I mean by it that there are other packages building on anndata, which might want to both hook into the existing structure (e.g. add an attribute or so next to the others, but otherwise reuse the rest as-is) and also use the existing render machinery for stylistically integrated rendering of the new parts.

You’re talking about integrating 3rd party HTML representations while extending. _repr_html_/_repr_mimebundle_ isn’t built to be extensible at all, but many packages export only that one as public API to access a HTML representation, so using it while writing an extension for anndata’s HTML repr will probably be a common pattern. One example of prior art to make a public HTML repr api customizable is pandas’ “styler” object.

so N reprs on a page means N copies of the definition

Ah I saw it being called in multiple places and assumed it’d be included multiple times per repr. I think we can’t do better than one inclusion per repr. We can however improve runtime behavior – it could create a global API and reuse it if it finds it instead of creating a copy of the whole API.

Two related robustness fixes to the inlined repr JS:

1. `repr.js`: replace the last remaining `innerHTML = \`<h3 id="${modalTitleId}">…\`` in the README modal with plain DOM construction (`createElement`/`textContent`/`appendChild`). The literal `id="${modalTitleId}"` substring inside the inlined JS source was being regex-matched by the Jupyter-compatibility tests as a duplicate HTML ID attribute across cells. Using DOM APIs removes the problematic substring entirely and matches the surrounding code style.

2. `javascript.py`: wrap the per-container initialisation body in an `install-once` guard. Every cell still ships the full source so any cell stays self-sufficient across deletion, reorder, or notebook reopen, but only the first to execute actually installs `window.anndataRepr`; subsequent cells reuse the installed `init(container)`. Addresses the runtime-redundancy concern without giving up per-cell portability.
…hover

Replace iter_outer as the section-iteration source in the repr, and
derive the canonical section list from get_literal_members(AnnDataElem).

Why not iter_outer:
  iter_outer yields (name, getattr(adata, name)) pairs, and propagates
  the first exception it hits — so a single broken section (corrupt
  aligned mapping, subclass with a crashing property, etc.) terminates
  the generator mid-iteration. _repr_html_'s top-level except catches
  that, the repr returns None, and the whole cell output disappears.
  The adversarial "Evil AnnData" case hit this.

  Iterating get_literal_members(AnnDataElem) ourselves and doing the
  getattr inside each section's try/except isolates the failure: a
  broken section renders as an error placeholder and the remaining
  sections still appear. iter_outer stays for callers that want strict
  semantics (AnnData.__str__, to_memory, _reduce, I/O).

Also: use the same Literal as the single source of truth wherever the
repr previously derived a set of section names from iter_outer
(_detect_unknown_sections, _get_custom_sections_by_position). Those
helpers now compute the set locally and no longer need the caller to
thread it through.

CSS fix: .anndata-entry:hover .anndata-entry__copy propagated the
:hover state to every ancestor entry, revealing every ancestor's
copy button when a deeply-nested row was hovered. Scope the trigger
to the entry's own row: the entry itself for plain div rows (which
never contain nested entries) and the <summary> for expandable rows
(nested children live in .anndata-entry__nested-content, outside
the summary).
@katosh

katosh commented Apr 21, 2026

Copy link
Copy Markdown
Contributor Author

On JS composition: taken. Just pushed an install-once guard.

On extensibility and the Styler analogy: useful pointer, and it maps cleanly onto the case you described. Packages that hook into anndata's structure (adding an attribute or section next to the others) and want stylistically integrated rendering of the new parts. The current design is already shaped similarly: TypeFormatter[T] plays Styler.format's role per-value, SectionFormatter is the table-level override, and FormattedOutput's facet fields (preview, summary, readme, type badge) stand in for apply()'s CSS-returning callables. Two intentional departures from Styler: registration is package-scoped via @register_formatter at import time (so bionty, treedata, spatialdata register once and every AnnData holding their types renders consistently) rather than instance-scoped df.style; and formatters return structured facets rather than CSS strings, closer to an nbconvert mime-bundle, so an extension opts into specific facets without owning cell layout.

On Jinja / the POC: the reframe is useful. Landing on robustness (one trust boundary + autoescape vs. threading mixed markup/data through ad-hoc escape_html()), with safety as a side effect, narrows the question. POC is up: settylab/anndata#9 · rendered preview. Minimal user-facing change. What building it surfaced: ~+500 lines net, templates split from the Python that produces their context, composition via Markup("\n").join(...) / macros through get_macros(), two new runtime deps. On the benefit side, autoescape-by-default is a structural guarantee where escape_html is reviewer discipline; with TestEscapingCoverage already asserting escaping at every user-data insertion point, the margin this adds sits narrow against the maintenance surface. The call is yours; the POC is there if you'd rather move forward with it.

katosh added 2 commits April 28, 2026 14:38
# Conflicts:
#	.vscode/settings.json
#	src/anndata/_settings.pyi
#	tests/test_repr.py
@ilan-gold ilan-gold modified the milestones: 0.13.0, 0.14.0 Jul 8, 2026
katosh and others added 11 commits July 9, 2026 10:46
Upstream migrated `settings` to `scverse_misc` (scverse#2446), replacing the
hand-rolled `SettingsManager` with a pydantic `scverse_misc.Settings`
subclass and deleting `_settings.pyi` and `tests/test_settings.py`.

Conflicts resolved in:

* `src/anndata/_settings.py` -- port the eleven `repr_html_*` options from
  `settings.register(...)` calls to pydantic field declarations on the new
  `Settings` class, keeping their defaults (from `_repr_constants`) and
  descriptions unchanged. `validate_int`/`validate_bool` were plain type
  checks, so the fields are declared as bare `int`/`bool` and pydantic now
  performs the same validation. Environment-variable configuration
  (`ANNDATA_REPR_HTML_*`) is preserved by `scverse_misc`'s `anndata_` env
  prefix, so `get_from_env` plumbing is no longer needed.
* `src/anndata/_settings.pyi` -- accept upstream's deletion. The stub typed
  `SettingsManager`, which no longer exists; pydantic supplies the field
  types natively.

`tests/test_settings.py` is likewise removed by upstream; it contained no
HTML-repr coverage. The repr settings remain exercised by `tests/repr/`.
…keys

Upstream switched the `disallow_forward_slash_in_h5ad` default to `True`
(scverse#2436), so a `/` in a key now aborts an h5ad write rather than emitting a
deprecation warning. `validate_key()` still hardcoded the old severity, which
made the repr advertise a slash key as merely deprecated while the
corresponding write raised `ValueError`.

Grade the key against the live setting instead of a constant, so the badge the
repr shows matches what a write would actually do, and keep the softer
"(deprecated)" wording for users who opt back out.

The branch carried a canary test asserting the old behaviour; replace it with
coverage of both settings states.
# Conflicts:
#	.vscode/settings.json
#	biome.jsonc
#	pyproject.toml
#	tests/test_repr.py
Also satisfy the stricter lint/typing checks from main (ISC004, ANN401,
mypy over src and tests): make `get_setting` generic, type
`register_formatter` overloads, and narrow duck-typed helpers.
Stability / performance
- Render X first; validate repr settings (pydantic ranges) and read them
  directly (fixes the undeclared `repr_html_max_string_length`)
- Read category colors once per column (lazy: 1 dask compute instead of 3)
- Bound unique counts per obs/var table (20M scanned values)
- Vectorized serializability check for long lists in uns
- Cap column names listed for DataFrames in obsm/varm
- Unknown-attribute detection no longer computes pending cached properties
- CSS follows the page color scheme instead of the OS (no dark repr on
  light, unthemed pages); JS install keyed by anndata version; inlined CSS/JS
  stripped of comments/indentation (~47 KB -> ~29 KB per output)

Correctness
- `error` takes precedence over previews (as documented)
- Nested AnnData: isinstance (subclasses), inherits explicit limits
- Raw: unique container ids, shared index preview, consistent "vars" label
- Hide memory estimate for lazy AnnData, clarify it for backed
- isinstance checks via anndata.abc/compat types; has_xp_base for array API
- Sparse labels from storage format (views read e.g. `csr_matrix`)

Maintainability
- Registry: copy-on-write, re-registering a class replaces it, decorator
  returns the class, unregister by class or section name
- One <details> section builder, one header-badge implementation, one
  lazy-column navigation helper, shared section tooltips, pluralized counts
- FallbackFormatter rewritten around safe-access helpers (same guarantees)
- Remove dead code (unused helpers and constants, get_setting)
- Tests: repr-only warning filter, registry isolation fixture, regression
  tests for all of the above
- numpy scalars in uns are scalars again (not array-API arrays)
- nan/inf previews no longer crash the float formatter
- warn for object columns with non-string values (write_h5ad fails)
- a failing SectionFormatter renders an error section instead of vanishing;
  broken X shows the error message
- uns entries get `section="uns"`, so `sections=("uns",)` formatters apply
- colors stored as arrays (from disk) or dask arrays (read_lazy) render as
  swatches, reading only the preview and nothing in metadata-only mode
- lazy DataFrames in obsm/varm (Dataset2D) render from metadata instead of
  being flagged as unknown type
- dask arrays always show their shape and mark sparse chunks
- themes: VS Code high contrast and "auto" themes (e.g. Furo) follow the OS
- X row aligned with the entry columns; type cells ellipsize when narrow
- subclass badge no longer repeats the class name; pluralized raw labels
…ted cases

Cases are small decorated functions grouped into seven categories with a
table of contents, expectations per case, coverage tags, per-case isolation
(tracebacks render in the page), optional-dependency skips, CLI filtering
(--list, --only, -o, --strict) and headless-Chrome theme probes
(--browser-check). Adds cases for previously uncovered sections, dtypes,
storage states, scale limits, themes and failure modes.
- probing unrecognized attributes no longer surfaces warnings from
  deprecated properties (e.g. MuData's mod_names FutureWarning)
- the subclass badge is only shown for actual AnnData subclasses, not for
  AnnData-like containers such as MuData
- pluralized column counts ("1 column")
@katosh

katosh commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor Author

Picking up the trust-boundary thread: both variants are now built as stacked PRs on our fork, kept in sync with this branch, and ready to merge into it on demand. Maintainers have write access to settylab/anndata, so either one can be merged straight into html_rep from its PR page.

Option A, MarkupSafe only: settylab#10

Every internal fragment is built as Markup (Markup.format / % / escape), so data-derived strings are escaped by type rather than by discipline. A correction to my earlier comment: there is a linter for the f-string pitfall. Ruff's S704 flags Markup(f"...") and Markup(var) while allowing Markup("<td>{}</td>").format(x). It is enabled for the repr, and the six remaining trust assertions (extension *_html fields, pandas' _repr_html_, the packaged CSS/JS, the escape and join helpers) each carry a noqa with a reason.

Option B, Jinja + MarkupSafe: settylab#9

The same trust boundary, plus templates (_macros.j2, entry.j2, section.j2, …) for the structure, StrictUndefined, and S704.

Both are backward compatible: the FormattedOutput field names are unchanged, a plain str in a *_html field is still trusted (wrapped in Markup at one documented function), and plain-text fields are escaped. Neither changes rendered output, so the existing preview applies to both; on the 61-case visual page the only differences are row order in the lazy cases, which also varies between two runs of this branch.

this PR A: MarkupSafe B: Jinja
_repr lines (py + j2) 5,891 6,176 6,262
render time (50k×2k, 30 obs cols, 10 obsm, 30 uns incl. nested AnnData) baseline +5 to 9% +25%
first repr in a fresh process ~25 ms ~same ~45 ms
new runtime deps none markupsafe jinja2, markupsafe

One behavior change applies to both: the public render_* helpers now return Markup, so render_badge(...) + "<hr>" escapes the plain-string side. That fails safe and is documented, with the fixes.

My preference is A. It gives the robustness you described with a third of B's runtime cost, and the rendering logic stays in one language. I'd also understand picking B for the template structure you outlined. Either way, I'd suggest landing it as a follow-up (or merging it here) once you're happy with the rest, to keep this diff reviewable. Your call.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

HTML Repr

4 participants