Skip to content

feat: DataFrame API for custom obs and var - #2328

Draft
ilan-gold wants to merge 12 commits into
mainfrom
ig/dataframe_api
Draft

ilan-gold wants to merge 12 commits into
mainfrom
ig/dataframe_api

Conversation

@ilan-gold

@ilan-gold ilan-gold commented Feb 4, 2026 •

Copy link
Copy Markdown
Contributor

cc @srivarra @Intron7

possible TODOs:

  • concat and merge API (or disallow and keep pandas/xarray special-cased, would require equal implementation as well)
  • adopt __dataframe__ protocol in a meaningful way ideally for writing, but potentially as a solution to the above to cases (i.e., drop into pandas and leave the data there): https://data-apis.org/dataframe-protocol/latest/
  • device-specific handling/indexing (see feat: support array-api #2071 ) for handling cuDF or GPU-based polars
  • Remove _gen_dataframe special casing
  • Determine if we need a to_numpy method

This PR is somewhat self-testing because I will have in theory removed all references to both pandas.Dataframe and XDataset/Dataset2D everywhere except concatenation internally.

One downside of this approach is that it puts us somewhat on the hook for pandas.DataFrame API but the flipside is that if we choose to switch to another library, we could do that relatively cleanly now/more easily

@codecov

codecov Bot commented Feb 4, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 93.87755% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.50%. Comparing base (26478f0) to head (fc13bf6).
⚠️ Report is 123 commits behind head on main.

Files with missing lines Patch % Lines
src/anndata/tests/helpers.py 50.00% 2 Missing ⚠️
src/anndata/_core/index.py 75.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2328      +/-   ##
==========================================
- Coverage   86.72%   82.50%   -4.23%     
==========================================
  Files          46       46              
  Lines        7165     7167       +2     
==========================================
- Hits         6214     5913     -301     
- Misses        951     1254     +303     
Files with missing lines Coverage Δ
src/anndata/_core/aligned_df.py 96.29% <100.00%> (-0.38%) ⬇️
src/anndata/_core/aligned_mapping.py 95.02% <100.00%> (-0.05%) ⬇️
src/anndata/_core/anndata.py 83.22% <100.00%> (-0.13%) ⬇️
src/anndata/_core/merge.py 84.62% <100.00%> (-10.63%) ⬇️
src/anndata/_core/storage.py 100.00% <100.00%> (+2.43%) ⬆️
src/anndata/_core/views.py 85.53% <100.00%> (-5.96%) ⬇️
src/anndata/_core/xarray.py 63.00% <100.00%> (-34.02%) ⬇️
src/anndata/_types.py 100.00% <ø> (ø)
src/anndata/experimental/__init__.py 88.88% <100.00%> (+0.65%) ⬆️
src/anndata/types.py 100.00% <100.00%> (ø)
... and 2 more

... and 13 files with indirect coverage changes

@ilan-gold ilan-gold modified the milestones: 0.12.10, 0.13.0 Feb 4, 2026


class Dataset2D(Mapping[Hashable, XDataArray | Self]):
class Dataset2D:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

adopt __dataframe__ protocol in a meaningful way ideally for writing, but potentially as a solution to the above to cases

So will Dataset2D eventually support the __dataframe__ protocol?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So will Dataset2D eventually support the dataframe protocol?

Mmmm depends if I actually implement its use here unless you have a use-case (you probably do?). In theory, I don't see why not but I haven't actually investigated what would go into that. There hasn't even been activity in the repo that hosts the protocols for like 2 years but it is documented on the arrow website: https://arrow.apache.org/docs/python/interchange_protocol.html

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like in DataFrame land people have been using PyCapsule for an interchange as of late, here it is in pandas

Specifically Pandas seem to be fully onboard with the Arrow PyCapsule interface.

For new development, we highly recommend using the Arrow C Data Interface alongside the Arrow PyCapsule Interface instead of the interchange protocol. From pandas 3.0 onwards, from_dataframe uses the PyCapsule Interface, only falling back to the interchange protocol if that fails.

From pandas 4.0 onwards, that fallback will no longer be available and only the PyCapsule Interface will be used.

I guess with this now, do you think __dataframe__ is still worth your time and effort, if you'll have to add some backwards compatibility or special handling come pandas 4?

data-apis/dataframe-api#363

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess with this now, do you think dataframe is still worth your time and effort, if you'll have to add some backwards compatibility or special handling come pandas 4?

I'm definitely not tied to the idea, so if that is the better interchange, we should use that instead. Thanks for pointing this out!

@srivarra

Copy link
Copy Markdown
Contributor

@ilan-gold Could I try out the Arrow PyCapsule interchange here?

@ilan-gold

Copy link
Copy Markdown
Contributor Author

@ilan-gold Could I try out the Arrow PyCapsule interchange here?

Go for it!

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

DataFrame API for obs and var keys via runtime-checkable Protocol Native support for cuPy/cuDF backed Anndata

2 participants