You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
An Xarray accessor to make xr.Datasets comply with the Python dataframe interchange protocol #86
I had a helpful discussion about this with @TomNicholas today in the Distributed Arrays Working Group meeting. Here are the high level notes:
What if in the long term, we made xr.DataArray and xr.Dataset compliant with the Python Dataframe Protocol?
In this formulation, a xr.Dataset whose variables had consistent dimensions could be thought of as a table, where xr.Variables are treated as chunked columns in the dataframe spec.
xarray would be a sort of "holder" of array references that could be lazily evaluated by a dataframe library that adheres to the Data API, where the iterating and computing would be part of that dataframe library's implementation (and likely, corresponding logical plan for evaluating df operations).
Since this would be a big change to Xarray, and further, it's likely not to work, how could this library prototype this feature ahead of time? Ideally, it would be nice if upon import xarray_sql, all xr.Datasets available to the user suddenly had the dunder methods that made them compliant to the dataframe data API.
Tom recommended playing around with Custom accessors as an Xarray-native way to achieve this sort of functionality.
If an xr.Dataset (with consistent dimensions) was also a __dataframe__, then it would be really easy to integrate it into DataFusion/SQL.
PyArrow specifically provides a __dataframe__ interchange method, from_dataframe
It is an open question if this type of integration is better suited to be in Python on top of Xarray or in Rust on top of the raw data format. In both Tom and my view, it seems like we should do both and figure out a good integration point between them.
Couldn't most of the value of Zarr --> Rust --> DataFusion be achieved with xarray_sql + Xarray backed by the zarrs-python IO layer? We'll have to evaluate this empirically.
There will be a large swath of users who will need the performance gains provided by a direct integration of DataFusion with (virtual) Zarr in Rust with SQL/Python wrappers on top. At the same time, there will be a large swath of users who need the interoperability ("joinability") of melding any data loadable in Xarray via a SQL interface.
Edit: TIL The Dataframe API and the Dataframe protocol are related, but distinct things! That explains my confusion. Just updated the link with the right term + docs.
all xr.Datasets available to the user suddenly had the dunder methods that made them compliant to the dataframe data API.
That specific idea is a non-starter. A cursory look at the DataFrame API specification shows methods which won't be able to be made compatible with xarray. To pick just one example DataFrame.max() has signature
changed the title [-]An Xarray accessor to make `xr.Dataset`s comply with the Python dataframe API upon import[/-][+]An Xarray accessor to make `xr.Dataset`s comply with the Python dataframe interchange protocol[/+]on Nov 11, 2025
I think that the Xarray data model fits in fairly well with this interchange protocol: chunked variables are morally equivalent to this interchange protocol's "chunked column" spec.
I asked a question in the datafusion discord, and a developer there (Tim Saucer) pointed out that to achieve proper laziness, we may be better off directly using the PyCapsule interface to integrate with ArrayStreams:
It seems like from Python (likely, via Cython), we could wrap Xarray chunk queries around a get_next PyArrow call. The docs point out that this is a more appropriate fit than the dataframe interchange protocol since it better supports datafusion tables.
Activity
TomNicholas commentedon Nov 4, 2025
That specific idea is a non-starter. A cursory look at the DataFrame API specification shows methods which won't be able to be made compatible with xarray. To pick just one example
DataFrame.max()has signaturewhereas
xarray.Dataset.max()has signatureIt makes sense for one to have
dimand the other not to - it seems very difficult/unlikely for these to converge.Instead I was suggesting that you just make some lightweight method for converting, e.g. something like
you can achieve this method-like syntax (
ds.xr_sql.to_arrow_dataframe()) using xarray's accessor registration mechanism.alxmrs commentedon Nov 4, 2025
Thanks, I like your alternative. You're right that what I want is a dataframe interchange, not a dataframe API surface for Datasets.
[-]An Xarray accessor to make `xr.Dataset`s comply with the Python dataframe API upon import[/-][+]An Xarray accessor to make `xr.Dataset`s comply with the Python dataframe interchange protocol[/+]alxmrs commentedon Nov 11, 2025
From the dataframe interchange protocol docs (see https://data-apis.org/dataframe-protocol/latest/design_requirements.html):
I think that the Xarray data model fits in fairly well with this interchange protocol: chunked variables are morally equivalent to this interchange protocol's "chunked column" spec.
alxmrs commentedon Dec 24, 2025
I asked a question in the datafusion discord, and a developer there (Tim Saucer) pointed out that to achieve proper laziness, we may be better off directly using the PyCapsule interface to integrate with ArrayStreams:
It seems like from Python (likely, via Cython), we could wrap Xarray chunk queries around a
get_nextPyArrow call. The docs point out that this is a more appropriate fit than the dataframe interchange protocol since it better supports datafusion tables.alxmrs commentedon Dec 25, 2025
I found this section of the Cython docs particularly useful: it describes how to write a Cython callback for a C ABI.
https://cython.readthedocs.io/en/latest/src/tutorial/clibraries.html#callbacks