Skip to content

feat: add experimental read_elem_async for zarr - #2748

Draft
kaloster wants to merge 15 commits into
scverse:mainfrom
kaloster:feat/async-read-elem
Draft

kaloster wants to merge 15 commits into
scverse:mainfrom
kaloster:feat/async-read-elem

Conversation

@kaloster

@kaloster kaloster commented Nov 16, 2026 •

Copy link
Copy Markdown
  • Closes #
  • Tests included or not required because:
  • Release notes not necessary because:
  • LLM or coding agent used: Claude Opus 5.5 via Claude Code wrote the first version. Claude Fable 5.2 via Claude Code rebased it onto main and wrote the later fix commits. GPT (Codex CLI) and Claude Code review passes reviewed it.
  • If one was used, I, a human, have reviewed, understood and tested every changed line

Part of #1897, opened as a draft to get your take. I tried the direction from the 2025 comments there: a separate async path on zarr's event loop, with the sync readers behaving as before.

What it adds

anndata.experimental.read_elem_async(elem): a coroutine that returns what read_elem returns, zarr only.

  • Group readers gather their children, and you gather several calls to read several stores. A failed child cancels its siblings.
  • Own registry in _io/specs/async_methods.py, sharing the construction step with the sync readers.
  • An element with no async reader is an error that points to read_elem and fails the whole call (e.g. a node without encoding metadata, a dataframe 0.1.0, or a third-party encoding registered for the sync reader only).
  • Sync zarr nodes are read on zarr's loop (also from Jupyter), async nodes on the running loop.

Not in here: callbacks (so no read_dispatched problem), read_elem_lazy, writes, h5py.

What we measured

All of obs from 103 AnnData-Zarr stores on S3 (13.3M cells, in region, obstore, zarr 3.4.2).

Read Thread pool, 32 threads read_elem_async, max 64 requests in flight, one client per bucket
All obs columns (2 runs per arm) 13.8 s 13.1 s
Without the two string columns (medians of 5) 6.7 s 5.3-5.5 s
  • With no request limit, async took 23-26 s and some runs failed (about 1,700 requests in flight, connection timeouts). I capped it with a semaphore in a store wrapper, outside anndata.
  • The string columns hide the gain: about half the loop thread's time is string work. Without them async is 18% faster and uses 19% less CPU.

Known limits

  • No limit on store requests in flight (question 2).
  • The result (pandas, scipy, AnnData) is built on the loop thread: zarr's shared I/O loop for sync nodes, the caller's loop for async nodes.
  • It needs private zarr names (zarr.core.sync._get_loop, Group._async_group, Array._async_array before 3.1.4), kept in two anndata.compat helpers.

Questions

  1. Is a separate registry the shape you want, or would you rather go another way for async internally for read_elem and write_elem #1897?
  2. The request limit looks like it belongs in zarr's store layer, not in anndata. Agree?
  3. Is experimental the right home for this?

Add `anndata.experimental.read_elem_async`, a coroutine that reads an
element of a zarr store. Group readers gather their children, so the
columns of a dataframe, the buffers of a sparse matrix, and the elements
of an AnnData are read concurrently. Callers gather calls to read many
elements or stores at once.

The async readers live in a separate registry keyed on zarr.AsyncGroup
and zarr.AsyncArray (`_io/specs/async_methods.py`), next to the lazy
registry. They use only zarr's async API. The sync path is unchanged;
the dataframe, nullable-string, AnnData, and sparse readers share their
construction step with the async readers through small helpers.

Loop rule: zarr.Group and zarr.Array are read on zarr's event loop, the
loop that opened them, also when awaited from another loop.
zarr.AsyncGroup and zarr.AsyncArray are read on the running loop.
Encodings without an async reader fall back to the sync reader on a
dedicated thread pool, so blocked fallbacks cannot starve the executor
that zarr's I/O uses.

Refs scverse#1897
@codecov

codecov Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 98.03922% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 86.34%. Comparing base (7c3b0f8) to head (f31a751).
⚠️ Report is 1 commits behind head on main.
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
src/anndata/_io/specs/async_methods.py 96.87% 3 Missing ⚠️
src/anndata/_core/sparse_dataset.py 96.29% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2748      +/-   ##
==========================================
- Coverage   87.92%   86.34%   -1.58%     
==========================================
  Files          51       52       +1     
  Lines        8445     8608     +163     
==========================================
+ Hits         7425     7433       +8     
- Misses       1020     1175     +155     
Files with missing lines Coverage Δ
src/anndata/_io/specs/__init__.py 100.00% <100.00%> (ø)
src/anndata/_io/specs/methods.py 92.44% <100.00%> (-0.17%) ⬇️
src/anndata/_io/specs/registry.py 95.73% <100.00%> (+1.04%) ⬆️
src/anndata/_io/utils.py 79.04% <100.00%> (+0.20%) ⬆️
src/anndata/_types.py 100.00% <100.00%> (ø)
src/anndata/compat/__init__.py 85.54% <100.00%> (-3.67%) ⬇️
src/anndata/experimental/__init__.py 88.23% <100.00%> (ø)
src/anndata/experimental/backed/_io.py 93.22% <100.00%> (-0.12%) ⬇️
src/anndata/_core/sparse_dataset.py 89.39% <96.29%> (-0.87%) ⬇️
src/anndata/_io/specs/async_methods.py 96.87% <96.87%> (ø)

... and 7 files with indirect coverage changes

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant