Skip to content

Filtering when reading a Grid file, avoid time leaking into the Grid - #1667

Open
dylannelson wants to merge 18 commits into
mainfrom
dylannelson/time-filtering
Open

dylannelson wants to merge 18 commits into
mainfrom
dylannelson/time-filtering

Conversation

@dylannelson

@dylannelson dylannelson commented Aug 11, 2026 •

Copy link
Copy Markdown
Member

Closes #1444 Closes #1720 Closes #1339

Overview

  1. time was shown to exist in a grid from a user reading in a grid and data from a single file
  2. Later when the user tried to use .subset there was a crash related to time being different on the grid vs the data
  3. time shouldn't realistically be on the grid to start with, so this was a concern, and was causing downstream issues not addressed when reading in the data
  4. There was 2 main ideas of where to place filtering to avoid this, Grid.__init__ or _read_ugrid, both were tested extensively
    • Grid.__init__ is run in multiple scenarios, including when a file is read from disk, but
      • in other scenarios like _slice_from_grid it can cause issues.
      • This solution also wasn't properly removing the time because it wasn't a dimension
    • the path to _read_ugrid occurs primarily when a file is read from disk
      • _read_ugrid(ds) only solved the the issue for that file type
    • We went back to Grid.__init__ as it solved the issue in all data types, but required a different approach
  5. This has been tested in multiple ways and is shown to:
    • Scalar time gone from grid coords and grid variables
    • node_lon, node_lat, face_node_connectivity preserved
    • Grid non-empty (n_node, n_face > 0)
    • bounding_box runs, no error
    • Data time preserved
    • Subset returned faces
    • Sliced grid keeps subgrid_face_indices
    • sel(time=...) still works
    • Dimensional time axis + coord stripped
    • Leaked data variable stripped
  6. Following further requests made in Sam's comment below, we added further scope to account for nearly any kind of other data that could come in that we may not want, even scenarios we haven't seen
  7. Handled a few edge cases with the new rules above, which required a new list to be made to allow models with unique data that can't/wouldn't normally be created within UXarray
  8. Added an optional argument in the API to turn of this cleaning if there is ever data we didn't account for, but the user wants/needs for whatever reason. I expect this to be very unlikely to be used by a typical user, as most users aren't accessing data directly on the grid, but this could give a workaround while we handle a patch or to allow them to keep the data on the grid when they export (may be useful for if they want to compare grids before/after changes without worrying about the cleaning affecting results). This seemed potentially more revevant as I saw more and more instances of model specific data being needed for it's particular IO, and removing it could result in downstream issues. There was a similar case with the CRS coordinate spatial_ref, which wasn't seen in any of the test data, but if it exists, needs to be read by _is_projected_grid in coordinates.py

Changes Made:

  1. Added test for _drop_non_grid_data
  2. Added function _drop_non_grid_data for cleaning stray coordinates
  3. Called _drop_non_grid_data in Grid.__init__
  4. Connected the parts of the API where this function can occur, or where it needs info from when/if it was run. It occurs in 4 places:
    • Grid.__init__
    • Grid.from_dataset
    • open_grid
    • slice.py

We are effectively following these rules now:

  1. Drop any data variable that has a dim not in DIM_NAMES.

  2. Drop any coordinate that has a dim not in DIM_NAMES.

  3. Drop any scalar coordinate.

  4. Drop no scalar data variables

  5. Never drop a variable on our defined lists/conventions (new)

    • Grid coordinate names (SPHERICAL_COORD_NAMES, CARTESIAN_COORD_NAMES): node_lon, face_x, etc
    • Connectivity names (CONNECTIVITY_NAMES): face_node_connectivity, face_edge_connectivity, etc
    • Descriptor names (DESCRIPTOR_NAMES): face_areas, n_nodes_per_face, etc
    • CRS variables: any variable named in another variable’s grid_mapping attribute

Expected Usage

This cleaning will occur during any and all grid creations

PR Checklist

General

  • An issue is created and linked
  • Added appropriate labels (if your uxarray repo permissions allow it)
  • Filled out Overview and Expected Usage (if applicable) sections

Testing & Benchmarking

  • Adequate tests are created if there is new functionality
  • Tests are not too basic (such as simply calling a function and nothing else)
  • Tests cover all major paths in your new functions
  • If this PR could affect performance, ran ASV benchmarks and confirmed they show expected behavior (add a new benchmark if necessary)

Documentation

  • [N/A] Docstrings have been added to all new functions
  • [N/A] Docstrings have been updated with any function changes
  • [N/A] User (public) functions have been added to docs/api.rst
  • Internal (private) function names start with an underscore (_)

AI Disclosure

AI Usage:
Claude Code

  • I take responsibility for all AI-generated content in my PR.
  • I have tested all AI-generated content in my PR.

Time was getting into grid data, causing issues when subsetting down the line. This should clean the data in a way that shouldn't happen in the future
pre-commit run --files uxarray/io/_ugrid.py
Seemed to turn up a few issues with spaces and line lengths

@erogluorhan erogluorhan left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe too early for a review, but I'd like to share some thoughts:

  1. time was shown to exist in a grid from a user reading in a grid and data from a single file

    A little more clarity could be helpful here. IIRC, time is not shown directly in the Grid object but in the Grid._ds.

  2. That said, I'd like to know what actual purpose we had with that object attribute, i.e. _ds. If it was to keep complete track of the original file content through xarray.Dataset, I'd say this dropping at the level of _ds might break that.

  3. Maybe we'd want to fix it at the Grid itself rather than altering _ds, but would it be possible?

  4. Finally, ugrid (i.e. _ugrid.py) is only one of the several formats we support. What about other formats in our I/O, e.g. _mpas.py etc. Their grid files can come in a similar way. Maybe we will need to do this fix at the Grid level rather than particular I/O modules?

@dylannelson

Copy link
Copy Markdown
Member Author

#1444 — Solution history

This is a bit about each version of the idea, why it's located where it is, and what the final code looks like.


Version 1 — .values guard in _slice_from_grid

# uxarray/core/dataarray.py — _slice_from_grid()
n_face=sliced_grid._ds["subgrid_face_indices"].values   # was: [...["subgrid_face_indices"]]
  • Location: within _slice_from_grid in core/dataarray.py.
  • Idea: Pass the face indexer as .values (a plain array) so xarray does positional indexing with no coordinate alignment. It fixes the crash for every subset method, but only guards the symptom — the stray time stays on the grid, so it never addresses why the grid carries a time.

Version 2 — Name-based clean build in _read_ugrid

# uxarray/io/_ugrid.py — end of _read_ugrid()
ds = _keep_only_grid_vars(ds)   # keep only recognized grid vars/coords, drop the rest
  • Location: at the end of _read_ugrid in io/_ugrid.py.
  • Idea: Filter the parsed dataset down to recognized grid variables, dropping extras where the grid is built. This removes the stray time at its source (the leaky reader), so nothing downstream can collide. It's essentially the final answer — a simpler-looking alternative was proposed next, which prompted the comparison below.

Version 3 — drop_dims in Grid.__init__ (team's idea)

# uxarray/grid/grid.py — Grid.__init__
extra_dims = set(grid_ds.dims) - set(DIM_NAMES)
grid_ds = grid_ds.drop_dims(extra_dims)
  • Location: at the top of Grid.__init__ in grid/grid.py.
  • Idea: Drop any dimension not in DIM_NAMES, centrally for every reader, to remove stray non-grid data by axis. It captured a dimensional time, but the leaked time here is a scalar (0-d) coord with no dimension — so drop_dims finds nothing to remove and the bug survives.

Version 4 — "Best of both" (drop_dims + 0-d/name drop)

grid_ds = grid_ds.drop_dims(set(grid_ds.dims) - set(DIM_NAMES))              # dimensional extras
drop = [n for n, v in grid_ds.variables.items() if n not in KEEP and v.ndim == 0]
grid_ds = grid_ds.drop_vars(drop, errors="ignore")                          # scalar extras
  • Location: at the top of Grid.__init__ in grid/grid.py.
  • Idea: Keep V3's drop_dims and add a second pass dropping 0-d non-grid vars, catching the scalar time it missed. It works, but uses two mechanisms for what one can do — V5's name-based keep-list covers scalar, dimensional, and data-var leaks in a single pass.

Version 5 — Single name-based keep-list in _read_ugrid (final)

# uxarray/io/_ugrid.py — end of _read_ugrid()
def _keep_only_grid_vars(ds):
    keep = {"grid_topology", *SPHERICAL_COORD_NAMES, *CARTESIAN_COORD_NAMES,
            *CONNECTIVITY_NAMES, *DESCRIPTOR_NAMES}
    return ds.drop_vars([n for n in ds.variables if n not in keep], errors="ignore")
  • Location: at the end of _read_ugrid in io/_ugrid.py — different location (see next section).
  • Idea: One name-based keep-list that drops every kind of non-grid extra (scalar, dimensional, data vars) in a single pass, collapsing V2–V4 into the simplest complete fix. Could be paired with the .values option, but didn't seem necessary.

Why V5 lives in _read_ugrid, not Grid.__init__

Two code paths build a Grid. They share Grid.__init__ but not _read_ugrid, so why is it in _read_ugrid?

Path A — opening a grid from a file (where the time leaks in):

ux.open_grid(...)                   core/api.py
  └─ _open_dataset_with_fallback()  # full xr.Dataset — stray `time` included
  └─ Grid.from_dataset(...)         grid.py
       └─ no source_grid_spec
       └─ _parse_grid_type()
       └─ _read_ugrid(...)          io/_ugrid.py   ← renames in place, returns EVERYTHING (leak enters here)
  └─ Grid.__init__(...)             grid.py   # self._ds = grid_ds

Path B — building a sliced grid during a subset (trying not to disturb this path):

uxda.subset.bounding_box(...)
  └─ Grid.isel(...)                    grid.py
       └─ _slice_face_indices(...)     grid/slice.py
            ds = grid._ds.isel(...)    # copy of cleaned grid
            ds["subgrid_face_indices"] = ...
       └─ Grid.from_dataset(...)       slice.py
            └─ grid_ds = ds            (_read_ugrid is skipped)
  └─ Grid.__init__(...)                grid.py   # self._ds holds subgrid_*_indices
Through _read_ugrid? Through Grid.__init__? _ds holds subgrid_*_indices?
Path A (file open) Yes Yes No
Path B (subset) No Yes Yes (needed by _slice_from_grid)
  • Filter in _read_ugrid (V5): Path A only. time is dropped as the grid is read; Path B never enters _read_ugrid, so its subgrid_*_indices survive, and its sliced grid (copied from the already-clean grid._ds) comes out clean for free.
  • Filter in Grid.__init__: both paths. It would strip subgrid_face_indices from Path B's sliced grid → _slice_from_grid hits a KeyError. Fixes the leak but breaks subsetting.

Final code preview

Basically came down to two edits, both in uxarray/io/_ugrid.py.

Edit 1 — add one import at the top of the file (ugrid is already imported; only DESCRIPTOR_NAMES is new):

import numpy as np
import xarray as xr

import uxarray.conventions.ugrid as ugrid
from uxarray.constants import INT_DTYPE, INT_FILL_VALUE
from uxarray.grid.connectivity import _replace_fill_values
from uxarray.conventions.descriptors import DESCRIPTOR_NAMES  # <---- this line

Edit 2 — filter at the end of _read_ugrid, then define the helper directly after it:

def _read_ugrid(ds):
    """Parses an unstructured grid dataset and encodes it in the UGRID
    conventions."""

    # ... (topology parse, coord/connectivity renames, dim swaps — unchanged) ...

    dim_dict[ds["face_node_connectivity"].dims[1]] = ugrid.N_MAX_FACE_NODES_DIM

    ds = ds.swap_dims(dim_dict)

    # ===== NEW =====
    # Strip non-grid extras (e.g. a stray scalar `time` coordinate, or unrelated data variables)
    ds = _keep_only_grid_vars(ds)
    # ===== end NEW =====

    return ds, dim_dict


# ===== NEW helper — place directly after _read_ugrid =====
def _keep_only_grid_vars(ds):
    """Return ``ds`` with only recognized UGRID grid variables/coordinates.

    Anything else on the dataset (a stray scalar ``time`` coordinate, unrelated
    data variables, etc.) is dropped so it cannot leak onto ``grid._ds``.

    Runs on the file-read path only.
    """
    # uxarray's own canonical grid-variable names (the same lists Grid filters against)
    keep = {"grid_topology"}
    keep.update(ugrid.SPHERICAL_COORD_NAMES)   # node/edge/face lon-lat
    keep.update(ugrid.CARTESIAN_COORD_NAMES)   # node/edge/face x-y-z
    keep.update(ugrid.CONNECTIVITY_NAMES)      # face_node_connectivity, edge_node_connectivity, ...
    keep.update(DESCRIPTOR_NAMES)              # n_nodes_per_face, face_areas, boundary_*_indices, ...

    # drop_vars removes variables/coords by name (never bare dimensions), so grid
    # dims survive with their variables; errors="ignore" tolerates absent names.
    drop = [name for name in ds.variables if name not in keep]
    return ds.drop_vars(drop, errors="ignore")
# ===== end NEW helper =====


# ===== existing code =====
def _encode_ugrid(ds):
    """Encodes an unstructured grid represented under a ``Grid`` object as a
    ``xr.Dataset`` with an updated grid topology variable."""

    if "grid_topology" in ds:
        ds = ds.drop_vars(["grid_topology"])
    # ... (rest of _encode_ugrid, unchanged) ...

@dylannelson

dylannelson commented Aug 11, 2026 •

Copy link
Copy Markdown
Member Author

@erogluorhan

4. Finally, ugrid (i.e. `_ugrid.py`) is only one of the several formats we support. What about other formats in our I/O, e.g. _mpas.py etc. Their grid files can come in a similar way. Maybe we will need to do this fix at the Grid level rather than particular I/O modules?

Sorry saw this after I was typing all the above content.

Edit: Here's an update, more focused on # 4 to start

This is a great question and I wish I had realized sooner. Trying to find a solution that was very high level, like in Grid.__innit__ seemed like a it made the most sense but kept causing problems downstream. Looking to something less high up made it easier, but I totally missed that it would only work for certain file types

So I had to look at 2 questions:

  1. Does this leak happen for other file types?
    • If no, we are good, and it was a ugrid specific issue
    • If yes, and it's every other file, then we may have to find some function at a higher level that can handle all the leaks the same
  2. Is there somewhere up, that can solve this problem by interacting with all the different file types in the same way, without disturbing other functionality
    • I've looked at this before and was struggling to find a place

Looking at # 1, I performed a similar check like before, read in a demo dataset, add time, save, reopen, check if there's time, and see if it crashes when there's a subset. Here's what I got

Reader Demo grid tested (test/meshfiles/…) How grid vars are populated Leaks?
MPAS mpas/QU/mesh.QU.1920km.151026.nc DataArrays derived from in_ds yes
ESMF esmf/ne30/ne30pg3.grid.nc xr.DataArray(<DataArray>, …) yes
ICON icon/R02B04/icon_grid_0010_R02B04_G.nc in_ds["…"].T - 1 (DataArray) yes
Exodus exodus/outCSne8/outCSne8.g DataArrays from the source yes
SCRIP scrip/ne30pg2/grid.nc xr.DataArray(<numpy>, …) no
GEOS-CS geos-cs/c12/test-c12.native.nc4 in_ds["…"].values.ravel() no
UGRID ugrid/quad-hexagon/grid.nc renames the input dataset in place no
Structured / FESOM2 structured/outCSne30_vortex_structured.nc (FESOM2 untested) crashes when re-opening yes but worse

So basically from what I'm seeing right now, we have 3 different outcomes

  • it's fine as is (good, 3 file types)
  • it's the same problem as ugrid (bad)
  • it introduces a problem that crashes when reading (worse than what happened with ugrid)

I'll keep looking for a solution for # 2, but worried at it's impact, especially since every file type seems to have a unique way to handle the data when opening.

@erogluorhan

Copy link
Copy Markdown
Member

The table in your latest comment showing no leaks for UGRID confused me a bit (From your other words, it sounds like UGRID has the time leaking). Could you clarify?

@dylannelson

Copy link
Copy Markdown
Member Author

@erogluorhan
For # 1 and # 2 and # 3, I think from what I'm seeing, _ds is used for a lot of functionality, and the added time being in there causes it to leak out into anything that can call _ds (which is a lot, shown below). The "Grid" and "_ds" can't be thought of as separate, because they are so heavily intertwined.

"the Grid" is essentially _ds

Almost every Grid property is a thin read of self._ds. Straight from uxarray/grid/grid.py (trimmed to the returns):

@property
def n_face(self):                 return self._ds.sizes["n_face"]          # :878
@property
def face_node_connectivity(self): return self._ds["face_node_connectivity"]  # :1261
@property
def attrs(self):                  return self._ds.attrs                    # :836
@property
def coordinates(self):            # only the NAMES the conventions recognize:
    return {c for c in SPHERICAL_COORD_NAMES if c in self._ds} | \
           {c for c in CARTESIAN_COORD_NAMES if c in self._ds}            # :799
@property
def dims(self):
    return {d for d in DIM_NAMES if d in self._ds.dims}                   # :787

Line references (pinned to 17c20f91):

The same pattern appears to be true for others as well, like: n_node, n_edge, node_lon, every *_connectivity,
face_areas, bounds, … — all return self._ds[...].

This can be a problem because time can leak onto other calls, like what we saw with subset. When the subset pulls _ds["subgrid_face_indices"] out as an index array, time rides along and collides with the data's time.

@dylannelson

dylannelson commented Aug 12, 2026 •

Copy link
Copy Markdown
Member Author

@erogluorhan

The table in your latest comment showing no leaks for UGRID confused me a bit (From your other words, it sounds like UGRID has the time leaking). Could you clarify?

It's no leak for UGRID in my current version, which fixed it. main would instead show that yes, there is a leak in UGRID

@dylannelson

dylannelson commented Aug 12, 2026 •

Copy link
Copy Markdown
Member Author

and for

fixing at the Grid without altering _ds,

that's effectively what I had originally, via .values. (Version 1 in the big comment above)

The issue with this is that it fixes a symptom, without addressing the cause. We discussed this early on and it was turned down when we originally discussed it. It's basically a patch in the .isel indexer (_slice_from_grid) so that when the data is on the way out, we strip any extra data that may cause a clash. This works for our subset incident, but it won't permanently remove the leaked time and it won't guarantee it won't cause issues in other functions,

@erogluorhan

Copy link
Copy Markdown
Member

and for

fixing at the Grid without altering _ds,

that's effectively what I had originally, via .values. (Version 1 in the big comment above)

The issue with this is that it fixes a symptom, without addressing the cause. We discussed this early on and it was turned down when we originally discussed it. It's basically a patch in the .isel indexer (_slice_from_grid) so that when the data is on the way out, we strip any extra data that may cause a clash. This works for our subset incident, but it won't permanently remove the leaked time and it won't guarantee it won't cause issues in other functions,

You're right; "without altering _ds" doesn't seem feasible or ideal, but I think we should discuss if the fix should be in the individual I/O modules (like _mpas etc.) or right before the Grid object creation, i.e. after the specific grid format reading. Thougts?

@rajeeja

rajeeja commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

Other readers like MPAS, ESMF, Exodus, ICON, and FESOM2 still leave time on _ds — MPAS reproduces the same IndexError from #1444. Could the filter move to the end of the auto-detect branch in Grid.from_dataset, before the return cls(...) at grid.py:323, so it covers all formats at once? The keep-list also drops bounds and max_face_radius when re-reading a uxarray-written grid.

@dylannelson

Copy link
Copy Markdown
Member Author

This new push should have everything we have wanted from the last few discussions
Early on we had the idea of moving the solution to grid.py in Grid.__init__ and doing the filtering there. The filtering would look at the dimensions and remove any that didn't belong. Unfortunately this didn't work, while also introducing downstream issues. It looked like this:

extra_dims = set(grid_ds.dims) - set(DIM_NAMES)
grid_ds = grid_ds.drop_dims(extra_dims)

The time in #1444 is a scalar (0-d) coordinate — it has no dimension, so it never appears in grid_ds.dims, extra_dims comes back empty, and drop_dims doesn't change it. I moved on from looking in grid.py at this point but now I'm back to it, and it seems to be working.

This solution is different in that it is dropping non-grid coordinates by name, which catches the scalar one that we've been having trouble with. Here's what this solution looks like instead:

grid_coord_names = set(ugrid.SPHERICAL_COORD_NAMES) | set(ugrid.CARTESIAN_COORD_NAMES)
stray = [c for c in ds.coords if c not in grid_coord_names]
return ds.drop_vars(stray, errors="ignore")

Testing

Before:
image
After:
image

Notebook Preview:
18-new-fix-demo.html

Downstream impact

Performed a few downstream tests with claude code to test what affects it may have, here's the results of what it looked at:

__init__ runs on every construction path — from_dataset (all formats), from_topology/from_healpix/etc., the subset's sliced grids, and Grid.copy(). Static sweep:

Area Finding Verdict
Grid properties (n_face, node_lon, …) read by name; recognized coords kept safe
Reads of stray coords off grid._ds grep '_ds.coords' / _ds["time"] → none safe
Grid↔data linking (open_dataset) by dimensions (_map_dims_to_ugrid), not coords safe
Subset bookkeeping subgrid_face/edge/node_indices are data vars safe
Grid.copy() re-runs __init__ on a clean _ds safe (idempotent)
Encoders / to_xarray no .coords use safe
Every .coords read in the package on the data objects / shapely — never grid._ds safe
Tests none assert coords on grid._ds safe

@dylannelson dylannelson added run-benchmark Run ASV benchmark workflow extra dims Handling of non-grid dimensions, such as time or altitude labels Aug 19, 2026
@github-actions

github-actions Bot commented Aug 19, 2026 •

Copy link
Copy Markdown

ASV Benchmarking

Benchmark Comparison Results

Benchmarks that have improved:

Change Before [ce8690d] After [8f889e0] Ratio Benchmark (Parameter)
- 755k 676k 0.9 mpas_ocean.NeighborhoodDask.track_peakmem_mean('480km', 'grid_chunks')
- 15.2±0.1ms 13.8±0.3ms 0.91 mpas_ocean.RemapUpsample.time_inverse_distance_weighted_remapping

Benchmarks that have stayed the same:

Change Before [ce8690d] After [8f889e0] Ratio Benchmark (Parameter)
710±10μs 736±10μs 1.04 connectivity.Connectivity.time_edge_face('120km')
247±7μs 256±7μs 1.03 connectivity.Connectivity.time_edge_face('480km')
3.57±0.06ms 3.53±0.06ms 0.99 connectivity.Connectivity.time_edge_node('120km')
599±20μs 599±20μs 1.00 connectivity.Connectivity.time_edge_node('480km')
41.3±5μs 42.9±6μs 1.04 connectivity.Connectivity.time_face_edge('120km')
30.3±3μs 34.1±4μs ~1.13 connectivity.Connectivity.time_face_edge('480km')
444±5μs 441±20μs 1.00 connectivity.Connectivity.time_face_face('120km')
246±10μs 244±8μs 0.99 connectivity.Connectivity.time_face_face('480km')
46.0±8μs 51.2±7μs ~1.11 connectivity.Connectivity.time_face_node('120km')
31.8±3μs 35.8±9μs ~1.13 connectivity.Connectivity.time_face_node('480km')
355±8μs 391±9μs ~1.10 connectivity.Connectivity.time_n_nodes_per_face('120km')
208±9μs 214±30μs 1.03 connectivity.Connectivity.time_n_nodes_per_face('480km')
702±20μs 687±20μs 0.98 connectivity.Connectivity.time_node_edge('120km')
240±6μs 252±10μs 1.05 connectivity.Connectivity.time_node_edge('480km')
45.5±0.7ms 47.0±0.9ms 1.03 connectivity.Connectivity.time_node_face('120km')
2.79±0.02ms 2.71±0.2ms 0.97 connectivity.Connectivity.time_node_face('480km')
1.42M 1.42M 1.00 connectivity.ConnectivityTracemalloc.track_peakmem_edge_face('120km')
105k 105k 1.00 connectivity.ConnectivityTracemalloc.track_peakmem_edge_face('480km')
6.48M 6.48M 1.00 connectivity.ConnectivityTracemalloc.track_peakmem_edge_node('120km')
419k 419k 1.00 connectivity.ConnectivityTracemalloc.track_peakmem_edge_node('480km')
2.05k 2.05k 1.00 connectivity.ConnectivityTracemalloc.track_peakmem_face_edge('120km')
2.05k 2.05k 1.00 connectivity.ConnectivityTracemalloc.track_peakmem_face_edge('480km')
1.6M 1.6M 1.00 connectivity.ConnectivityTracemalloc.track_peakmem_face_face('120km')
101k 101k 1.00 connectivity.ConnectivityTracemalloc.track_peakmem_face_face('480km')
2.18k 2.18k 1.00 connectivity.ConnectivityTracemalloc.track_peakmem_face_node('120km')
2.18k 2.18k 1.00 connectivity.ConnectivityTracemalloc.track_peakmem_face_node('480km')
240k 240k 1.00 connectivity.ConnectivityTracemalloc.track_peakmem_n_nodes_per_face('120km')
26k 26k 1.00 connectivity.ConnectivityTracemalloc.track_peakmem_n_nodes_per_face('480km')
1.9M 1.9M 1.00 connectivity.ConnectivityTracemalloc.track_peakmem_node_edge('120km')
127k 127k 1.00 connectivity.ConnectivityTracemalloc.track_peakmem_node_edge('480km')
11.9M 11.9M 1.00 connectivity.ConnectivityTracemalloc.track_peakmem_node_face('120km')
747k 747k 1.00 connectivity.ConnectivityTracemalloc.track_peakmem_node_face('480km')
7.02±0.06ms 7.12±0.08ms 1.01 face_bounds.FaceBounds.time_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/mpas/QU/oQU480.231010.nc'))
2.45±0.09ms 2.37±0.08ms 0.97 face_bounds.FaceBounds.time_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/scrip/outCSne8/outCSne8.nc'))
7.79±1ms 7.80±7ms 1.00 face_bounds.FaceBounds.time_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/ugrid/geoflow-small/grid.nc'))
1.46±0.03ms 1.46±0.04ms 0.99 face_bounds.FaceBounds.time_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/ugrid/quad-hexagon/grid.nc'))
57.3k 57.3k 1.00 face_bounds.FaceBounds.track_nbytes_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/mpas/QU/oQU480.231010.nc'))
12.3k 12.3k 1.00 face_bounds.FaceBounds.track_nbytes_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/scrip/outCSne8/outCSne8.nc'))
123k 123k 1.00 face_bounds.FaceBounds.track_nbytes_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/ugrid/geoflow-small/grid.nc'))
128 128 1.00 face_bounds.FaceBounds.track_nbytes_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/ugrid/quad-hexagon/grid.nc'))
1.27M 1.27M 1.00 face_bounds.FaceBounds.track_nbytes_grid_with_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/mpas/QU/oQU480.231010.nc'))
50.1k 50.1k 1.00 face_bounds.FaceBounds.track_nbytes_grid_with_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/scrip/outCSne8/outCSne8.nc'))
516k 516k 1.00 face_bounds.FaceBounds.track_nbytes_grid_with_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/ugrid/geoflow-small/grid.nc'))
712 712 1.00 face_bounds.FaceBounds.track_nbytes_grid_with_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/ugrid/quad-hexagon/grid.nc'))
1.77M 1.77M 1.00 face_bounds.FaceBounds.track_peakmem_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/mpas/QU/oQU480.231010.nc'))
1.75M 1.75M 1.00 face_bounds.FaceBounds.track_peakmem_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/scrip/outCSne8/outCSne8.nc'))
1.91M 1.91M 1.00 face_bounds.FaceBounds.track_peakmem_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/ugrid/geoflow-small/grid.nc'))
35.3k 35.3k 1.00 face_bounds.FaceBounds.track_peakmem_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/ugrid/quad-hexagon/grid.nc'))
876±20ns 857±30ns 0.98 geometry_kernels.AccucrossKernels.time_accucross
1.96±0.02μs 1.91±0.02μs 0.97 geometry_kernels.AccucrossKernels.time_accucross_pair
266±9ns 256±9ns 0.96 geometry_kernels.EFTPrimitives.time_acc_sqrt_re
265±20ns 246±10ns 0.93 geometry_kernels.EFTPrimitives.time_diff_of_products
225±10ns 216±10ns 0.96 geometry_kernels.EFTPrimitives.time_two_prod
235±9ns 215±20ns 0.91 geometry_kernels.EFTPrimitives.time_two_sum
722±70ns 661±20ns 0.92 geometry_kernels.GCAConstLatIntersection.time_accux_constlat_kernel
791±30ns 747±30ns 0.94 geometry_kernels.GCAConstLatIntersection.time_gca_const_lat_intersection
811±20ns 852±30ns 1.05 geometry_kernels.GCAConstLatIntersection.time_try_gca_const_lat_intersection
551±5ns 566±10ns 1.03 geometry_kernels.GCAGCAIntersection.time_accux_gca_kernel
636±10ns 651±20ns 1.02 geometry_kernels.GCAGCAIntersection.time_gca_gca_intersection
756±20ns 741±20ns 0.98 geometry_kernels.GCAGCAIntersection.time_try_gca_gca_intersection
21.7±0.2μs 20.5±0.2μs 0.95 geometry_kernels.OrientPredicates.time_on_minor_arc
21.5±0.2μs 20.7±0.4μs 0.96 geometry_kernels.OrientPredicates.time_orient3d_on_sphere
2.39±0ms 2.39±0.01ms 1.00 geometry_samebody.SameBodyConstLat.time_accux_dispatch
1.04±0.04ms 1.02±0ms 0.98 geometry_samebody.SameBodyConstLat.time_accux_kernel
1.63±0.01ms 1.64±0ms 1.01 geometry_samebody.SameBodyConstLat.time_fp64_dispatch
121±0.9μs 121±0.2μs 1.00 geometry_samebody.SameBodyConstLat.time_fp64_kernel
23.2±0.1ms 24.7±0.04ms 1.06 geometry_samebody_gcagca.SameBodyGcaGca.time_accux_dispatch
5.54±0.01ms 5.53±0.01ms 1.00 geometry_samebody_gcagca.SameBodyGcaGca.time_accux_kernel
18.0±0.3ms 19.3±0.04ms 1.07 geometry_samebody_gcagca.SameBodyGcaGca.time_fp64_dispatch
699±4μs 702±7μs 1.00 geometry_samebody_gcagca.SameBodyGcaGca.time_fp64_kernel
962±10ms 993±8ms 1.03 import.Imports.timeraw_import_uxarray
1 1 1.00 lazy_grid_construction.LazyGridConstruction.track_computes_isel
2 2 1.00 lazy_grid_construction.LazyGridConstruction.track_computes_open_grid_chunked
68.3±1ms 66.1±1ms 0.97 lazy_grid_construction.OpenGridChunked.time_open_grid('120km')
95.0±1ms 92.9±0.9ms 0.98 lazy_grid_construction.OpenGridChunked.time_open_grid('480km')
1.99M 2.01M 1.01 lazy_grid_construction.OpenGridChunked.track_peakmem_open_grid('120km')
2.23M 2.11M 0.94 lazy_grid_construction.OpenGridChunked.track_peakmem_open_grid('480km')
2.38±0.02ms 2.44±0.05ms 1.03 mpas_ocean.CheckNorm.time_check_norm('120km')
1.91±0.02ms 1.88±0.01ms 0.98 mpas_ocean.CheckNorm.time_check_norm('480km')
593±20μs 611±3μs 1.03 mpas_ocean.ConnectivityConstruction.time_face_face_connectivity('120km')
286±6μs 284±2μs 0.99 mpas_ocean.ConnectivityConstruction.time_face_face_connectivity('480km')
454±8μs 446±20μs 0.98 mpas_ocean.ConnectivityConstruction.time_n_nodes_per_face('120km')
318±6μs 317±3μs 1.00 mpas_ocean.ConnectivityConstruction.time_n_nodes_per_face('480km')
3.28±0.01ms 3.35±0.06ms 1.02 mpas_ocean.ConstructFaceLatLon.time_cartesian_averaging('120km')
2.42±0ms 2.42±0.03ms 1.00 mpas_ocean.ConstructFaceLatLon.time_cartesian_averaging('480km')
82.2±0.8ms 81.6±0.3ms 0.99 mpas_ocean.ConstructFaceLatLon.time_welzl('120km')
7.32±0.05ms 7.38±0.1ms 1.01 mpas_ocean.ConstructFaceLatLon.time_welzl('480km')
13.5±0.09ms 13.3±0.04ms 0.99 mpas_ocean.ConstructTreeStructures.time_ball_tree('120km')
648±8μs 616±30μs 0.95 mpas_ocean.ConstructTreeStructures.time_ball_tree('480km')
6.86±0.2ms 6.71±0.03ms 0.98 mpas_ocean.ConstructTreeStructures.time_kd_tree('120km')
394±20μs 341±8μs ~0.87 mpas_ocean.ConstructTreeStructures.time_kd_tree('480km')
637±8ms 643±2ms 1.01 mpas_ocean.CrossSections.time_const_lat('120km', 1)
315±1ms 325±2ms 1.03 mpas_ocean.CrossSections.time_const_lat('120km', 2)
164±3ms 166±0.7ms 1.01 mpas_ocean.CrossSections.time_const_lat('120km', 4)
576±6ms 583±5ms 1.01 mpas_ocean.CrossSections.time_const_lat('480km', 1)
288±3ms 291±3ms 1.01 mpas_ocean.CrossSections.time_const_lat('480km', 2)
147±6ms 150±2ms 1.02 mpas_ocean.CrossSections.time_const_lat('480km', 4)
28.1±0.1ms 28.3±0.2ms 1.01 mpas_ocean.DualMesh.time_dual_mesh_construction('120km')
3.34±0.02ms 3.46±0.06ms 1.04 mpas_ocean.DualMesh.time_dual_mesh_construction('480km')
11.4±0.8ms 11.8±1ms 1.04 mpas_ocean.FaceAreas.time_face_areas('120km')
3.97±0.1ms 3.89±0.1ms 0.98 mpas_ocean.FaceAreas.time_face_areas('480km')
229k 229k 1.00 mpas_ocean.FaceAreas.track_nbytes_face_areas('120km')
14.3k 14.3k 1.00 mpas_ocean.FaceAreas.track_nbytes_face_areas('480km')
2.12M 2.12M 1.00 mpas_ocean.FaceAreas.track_peakmem_face_areas('120km')
674k 674k 1.00 mpas_ocean.FaceAreas.track_peakmem_face_areas('480km')
52.1±0.5ms 51.8±0.9ms 0.99 mpas_ocean.GeoDataFrame.time_to_geodataframe('120km', 'exclude')
158±1ms 153±3ms 0.97 mpas_ocean.GeoDataFrame.time_to_geodataframe('120km', 'split')
4.80±0.2ms 4.75±0.1ms 0.99 mpas_ocean.GeoDataFrame.time_to_geodataframe('480km', 'exclude')
29.2±0.2ms 30.5±0.7ms 1.05 mpas_ocean.GeoDataFrame.time_to_geodataframe('480km', 'split')
8.82±0.7ms 9.09±0.4ms 1.03 mpas_ocean.Gradient.time_gradient('120km')
1.41±0.03ms 1.38±0.01ms 0.98 mpas_ocean.Gradient.time_gradient('480km')
457k 457k 1.00 mpas_ocean.Gradient.track_nbytes_gradient('120km')
28.7k 28.7k 1.00 mpas_ocean.Gradient.track_nbytes_gradient('480km')
3.44M 3.44M 1.00 mpas_ocean.Gradient.track_peakmem_gradient('120km')
226k 226k 1.00 mpas_ocean.Gradient.track_peakmem_gradient('480km')
362±30μs 377±10μs 1.04 mpas_ocean.HoleEdgeIndices.time_construct_hole_edge_indices('120km')
168±9μs 162±7μs 0.96 mpas_ocean.HoleEdgeIndices.time_construct_hole_edge_indices('480km')
180±3μs 207±20μs ~1.15 mpas_ocean.Integrate.time_integrate('120km')
161±9μs 164±0.7μs 1.02 mpas_ocean.Integrate.time_integrate('480km')
18.4M 18.4M 1.00 mpas_ocean.Integrate.track_nbytes_integrate('120km')
1.2M 1.2M 1.00 mpas_ocean.Integrate.track_nbytes_integrate('480km')
79.0±1ms 77.8±0.6ms 0.98 mpas_ocean.MatplotlibConversion.time_dataarray_to_polycollection('120km', 'exclude')
78.6±2ms 78.5±0.5ms 1.00 mpas_ocean.MatplotlibConversion.time_dataarray_to_polycollection('120km', 'include')
80.9±1ms 78.0±0.7ms 0.96 mpas_ocean.MatplotlibConversion.time_dataarray_to_polycollection('120km', 'split')
5.68±0.05ms 5.39±0.03ms 0.95 mpas_ocean.MatplotlibConversion.time_dataarray_to_polycollection('480km', 'exclude')
5.77±0.04ms 5.71±0.3ms 0.99 mpas_ocean.MatplotlibConversion.time_dataarray_to_polycollection('480km', 'include')
5.73±0.1ms 5.50±0.1ms 0.96 mpas_ocean.MatplotlibConversion.time_dataarray_to_polycollection('480km', 'split')
170±0.4ms 169±1ms 0.99 mpas_ocean.NeighborhoodBuild.time_build('120km', 1.0)
357±2ms 351±2ms 0.98 mpas_ocean.NeighborhoodBuild.time_build('120km', 5.0)
9.34±0.2ms 9.10±0.04ms 0.97 mpas_ocean.NeighborhoodBuild.time_build('480km', 1.0)
11.6±0.01ms 11.3±0.2ms 0.97 mpas_ocean.NeighborhoodBuild.time_build('480km', 5.0)
167±0.3ms 165±0.9ms 0.99 mpas_ocean.NeighborhoodBuild.time_query_radius('120km', 1.0)
358±1ms 350±1ms 0.98 mpas_ocean.NeighborhoodBuild.time_query_radius('120km', 5.0)
9.04±0.03ms 8.91±0.05ms 0.99 mpas_ocean.NeighborhoodBuild.time_query_radius('480km', 1.0)
11.3±0.04ms 11.1±0.06ms 0.98 mpas_ocean.NeighborhoodBuild.time_query_radius('480km', 5.0)
612.76 612.76 1.00 mpas_ocean.NeighborhoodBuild.track_mean_neighbors('120km', 15.0)
74.17 74.17 1.00 mpas_ocean.NeighborhoodBuild.track_mean_neighbors('120km', 5.0)
37.29 37.29 1.00 mpas_ocean.NeighborhoodBuild.track_mean_neighbors('480km', 15.0)
6.57 6.57 1.00 mpas_ocean.NeighborhoodBuild.track_mean_neighbors('480km', 5.0)
141M 141M 1.00 mpas_ocean.NeighborhoodBuild.track_nbytes_neighbors('120km', 15.0)
17.4M 17.4M 1.00 mpas_ocean.NeighborhoodBuild.track_nbytes_neighbors('120km', 5.0)
563k 563k 1.00 mpas_ocean.NeighborhoodBuild.track_nbytes_neighbors('480km', 15.0)
123k 123k 1.00 mpas_ocean.NeighborhoodBuild.track_nbytes_neighbors('480km', 5.0)
145M 145M 1.00 mpas_ocean.NeighborhoodBuild.track_peakmem_build('120km', 15.0)
21.5M 21.5M 1.00 mpas_ocean.NeighborhoodBuild.track_peakmem_build('120km', 5.0)
824k 824k 1.00 mpas_ocean.NeighborhoodBuild.track_peakmem_build('480km', 15.0)
384k 384k 1.00 mpas_ocean.NeighborhoodBuild.track_peakmem_build('480km', 5.0)
26.3±0.6ms 25.7±0.8ms 0.98 mpas_ocean.NeighborhoodDask.time_mean('120km', 'grid_chunks')
11.0±0.08ms 10.9±0.2ms 0.98 mpas_ocean.NeighborhoodDask.time_mean('120km', 'numpy')
24.1±0.8ms 23.5±0.8ms 0.98 mpas_ocean.NeighborhoodDask.time_mean('120km', 'time_chunks')
5.72±0.04ms 5.69±0.2ms 0.99 mpas_ocean.NeighborhoodDask.time_mean('480km', 'grid_chunks')
335±9μs 322±6μs 0.96 mpas_ocean.NeighborhoodDask.time_mean('480km', 'numpy')
3.89±0.03ms 3.85±0.02ms 0.99 mpas_ocean.NeighborhoodDask.time_mean('480km', 'time_chunks')
5.9M 5.82M 0.99 mpas_ocean.NeighborhoodDask.track_peakmem_mean('120km', 'grid_chunks')
2.75M 2.75M 1.00 mpas_ocean.NeighborhoodDask.track_peakmem_mean('120km', 'numpy')
5.67M 5.68M 1.00 mpas_ocean.NeighborhoodDask.track_peakmem_mean('120km', 'time_chunks')
177k 177k 1.00 mpas_ocean.NeighborhoodDask.track_peakmem_mean('480km', 'numpy')
521k 528k 1.01 mpas_ocean.NeighborhoodDask.track_peakmem_mean('480km', 'time_chunks')
3.81±0.04s 3.78±0.01s 0.99 mpas_ocean.NeighborhoodReduce.time_dataset_reduce('120km', 'mean')
3.90±0.01s 3.89±0.01s 1.00 mpas_ocean.NeighborhoodReduce.time_dataset_reduce('120km', 'median')
97.9±0.6ms 98.8±0.3ms 1.01 mpas_ocean.NeighborhoodReduce.time_dataset_reduce('480km', 'mean')
99.2±0.06ms 99.3±0.1ms 1.00 mpas_ocean.NeighborhoodReduce.time_dataset_reduce('480km', 'median')
427±2ms 421±0.9ms 0.99 mpas_ocean.NeighborhoodReduce.time_neighborhood_reduce('120km', 'mean')
454±2ms 451±1ms 0.99 mpas_ocean.NeighborhoodReduce.time_neighborhood_reduce('120km', 'median')
13.6±0.02ms 13.8±0.02ms 1.02 mpas_ocean.NeighborhoodReduce.time_neighborhood_reduce('480km', 'mean')
14.0±0.09ms 13.9±0.05ms 0.99 mpas_ocean.NeighborhoodReduce.time_neighborhood_reduce('480km', 'median')
3.79±0.01ms 3.77±0.01ms 0.99 mpas_ocean.NeighborhoodReduce.time_reduce('120km', 'mean')
32.0±0.1ms 31.8±0.1ms 0.99 mpas_ocean.NeighborhoodReduce.time_reduce('120km', 'median')
121±8μs 121±2μs 1.00 mpas_ocean.NeighborhoodReduce.time_reduce('480km', 'mean')
335±4μs 358±9μs 1.07 mpas_ocean.NeighborhoodReduce.time_reduce('480km', 'median')
234k 234k 1.00 mpas_ocean.NeighborhoodReduce.track_peakmem_reduce('120km', 'mean')
235k 235k 1.00 mpas_ocean.NeighborhoodReduce.track_peakmem_reduce('120km', 'median')
19.5k 19.5k 1.00 mpas_ocean.NeighborhoodReduce.track_peakmem_reduce('480km', 'mean')
19.6k 19.6k 1.00 mpas_ocean.NeighborhoodReduce.track_peakmem_reduce('480km', 'median')
401±10μs 399±10μs 1.00 mpas_ocean.PointInPolygon.time_face_search_lonlat('120km')
435±20μs 380±10μs ~0.87 mpas_ocean.PointInPolygon.time_face_search_lonlat('480km')
366±8μs 372±20μs 1.02 mpas_ocean.PointInPolygon.time_face_search_xyz('120km')
351±5μs 331±10μs 0.94 mpas_ocean.PointInPolygon.time_face_search_xyz('480km')
100±1ms 101±0.6ms 1.00 mpas_ocean.RemapDownsample.time_bilinear_remapping
16.3±0.2ms 16.3±0.3ms 1.00 mpas_ocean.RemapDownsample.time_inverse_distance_weighted_remapping
14.7±0.2ms 14.7±0.1ms 1.00 mpas_ocean.RemapDownsample.time_nearest_neighbor_remapping
436±3ms 427±0.7ms 0.98 mpas_ocean.RemapUpsample.time_bilinear_remapping
6.19±0.2ms 6.11±0.1ms 0.99 mpas_ocean.RemapUpsample.time_nearest_neighbor_remapping
4.41±0.2ms 4.19±0.3ms 0.95 mpas_ocean.ZonalAverage.time_zonal_average('120km')
2.36±0.05ms 2.35±0.2ms 0.99 mpas_ocean.ZonalAverage.time_zonal_average('480km')
1.0407520744898715 1.0348878609791856 0.99 nogil_scaling.GILScaling.track_gil_scaling
1.35±0.02ms 1.44±0.09ms 1.07 open_dataset.OpenDataset.time_open_dataset('dataset', 'dataset')
2.38±0.06ms 2.54±0.09ms 1.07 open_dataset.OpenDataset.time_open_dataset('dataset', 'file')
5.05±0.06ms 5.93±0.2ms ~1.18 open_dataset.OpenDataset.time_open_dataset('file', 'dataset')
6.19±0.07ms 7.55±0.4ms ~1.22 open_dataset.OpenDataset.time_open_dataset('file', 'file')
190±7μs 201±20μs 1.06 open_dataset.OpenDataset.time_open_dataset('grid', 'dataset')
1.22±0.03ms 1.26±0.03ms 1.03 open_dataset.OpenDataset.time_open_dataset('grid', 'file')
1.14±0.04s 1.09±0.02s 0.95 plots.PlotContour.time_contour_projected
305±3ms 311±4ms 1.02 plots.PlotHealpix.time_default_plot(6)
2.72±0s 2.72±0s 1.00 plots.PlotHealpix.time_default_plot(8)
28M 28M 1.00 plots.PlotHealpix.track_peakmem_default_plot(6)
443M 443M 1.00 plots.PlotHealpix.track_peakmem_default_plot(8)
8.71±0.2ms 8.87±1ms 1.02 quad_hexagon.QuadHexagon.time_open_dataset
7.26±0.07ms 7.67±0.3ms 1.06 quad_hexagon.QuadHexagon.time_open_grid
408 408 1.00 quad_hexagon.QuadHexagon.track_nbytes_open_dataset
392 392 1.00 quad_hexagon.QuadHexagon.track_nbytes_open_grid
71.3k 71.3k 1.00 quad_hexagon.QuadHexagon.track_peakmem_open_dataset
71.1k 71.1k 1.00 quad_hexagon.QuadHexagon.track_peakmem_open_grid
10.2±0.2ms 10.1±0.2ms 0.99 subset.BoundingBox.time_bounding_box('120km')
5.94±0.1ms 5.99±0.2ms 1.01 subset.BoundingBox.time_bounding_box('480km')
2.38M 2.38M 1.00 subset.BoundingBox.track_peakmem_bounding_box('120km')
203k 203k 1.00 subset.BoundingBox.track_peakmem_bounding_box('480km')
3.97±0.04ms 4.11±8ms 1.03 subset.BoundingBoxHealpix.time_bounding_box(6)
14.9±0.4ms 15.3±0.5ms 1.03 subset.BoundingBoxHealpix.time_bounding_box(8)
2.18M 2.18M 1.00 subset.BoundingBoxHealpix.track_peakmem_bounding_box(6)
13.2M 13.2M 1.00 subset.BoundingBoxHealpix.track_peakmem_bounding_box(8)
731±6ms 744±3ms 1.02 to_raster.ToRaster.time_to_raster((10000.0, 10.0))
1.53±0s 1.54±0.01s 1.01 to_raster.ToRaster.time_to_raster((10000.0, 100.0))
1.78±0.01s 1.79±0.01s 1.00 to_raster.ToRaster.time_to_raster((100000.0, 100.0))
270M 270M 1.00 to_raster.ToRaster.track_peakmem((10000.0, 10.0))
680M 680M 1.00 to_raster.ToRaster.track_peakmem((10000.0, 100.0))
765M 765M 1.00 to_raster.ToRaster.track_peakmem((100000.0, 100.0))

Benchmarks that have got worse:

Change Before [ce8690d] After [8f889e0] Ratio Benchmark (Parameter)
+ 305M 390M 1.28 face_bounds.FaceBoundsColdStartRss.track_peakmem_open_and_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/mpas/QU/oQU480.231010.nc'))
+ 306M 390M 1.28 face_bounds.FaceBoundsColdStartRss.track_peakmem_open_and_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/scrip/outCSne8/outCSne8.nc'))
+ 308M 390M 1.27 face_bounds.FaceBoundsColdStartRss.track_peakmem_open_and_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/ugrid/geoflow-small/grid.nc'))
+ 306M 390M 1.27 face_bounds.FaceBoundsColdStartRss.track_peakmem_open_and_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/ugrid/quad-hexagon/grid.nc'))
+ 278M 389M 1.4 import.Imports.track_peakmem_import_uxarray
+ 327M 390M 1.19 mpas_ocean.CrossSectionsPeakMem.track_peakmem_const_lat('120km', 1)
+ 327M 390M 1.19 mpas_ocean.CrossSectionsPeakMem.track_peakmem_const_lat('120km', 2)
+ 327M 390M 1.19 mpas_ocean.CrossSectionsPeakMem.track_peakmem_const_lat('120km', 4)
+ 307M 390M 1.27 mpas_ocean.CrossSectionsPeakMem.track_peakmem_const_lat('480km', 1)
+ 308M 390M 1.27 mpas_ocean.CrossSectionsPeakMem.track_peakmem_const_lat('480km', 2)
+ 308M 390M 1.27 mpas_ocean.CrossSectionsPeakMem.track_peakmem_const_lat('480km', 4)
+ 327M 390M 1.19 mpas_ocean.GradientColdStartRss.track_peakmem_gradient('120km')
+ 301M 390M 1.29 mpas_ocean.GradientColdStartRss.track_peakmem_gradient('480km')
+ 328M 390M 1.19 mpas_ocean.ZonalAveragePeakMem.track_peakmem_zonal_average('120km')
+ 309M 390M 1.26 mpas_ocean.ZonalAveragePeakMem.track_peakmem_zonal_average('480km')

@dylannelson
dylannelson marked this pull request as ready for review August 19, 2026 19:34

@Sevans711 Sevans711 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks like a relatively clean fix overall, with lots of tests run by hand to confirm things seem to be working!

I left a few minor comments inline, and I have a couple larger comments here:

  1. I had a suspicion that extra dimensions (not just extra coordinates) on input datasets would be an issue, but I tested a few things locally and found that it wasn't a problem (for some reason…). For example, if you had a time dimension on the input dataset, instead of just a time coordinate, it doesn't seem to lead to a crash later (although, it does leak into Grid._ds). It isn't really obvious to me why it wouldn't crash…. Can you clarify why this might be the case, and/or add a test to ensure subsetting still works even after adding a time dimension to the grid?
  2. Would you also be able to create a file with a time coordinate somewhere in the test suite (in meshfiles folder), and test that, too, instead of only having a test where the time coordinate gets assigned after loading the dataset? That would help me to be more convinced that this fully closes the original issue. It's a bit tricky to know for certain because the original issue report doesn't contain a complete verifiable example. But, the workflow there seemed to be "load from file, then subset (which crashes)" so ideally a test here could include that complete workflow as well.

Comment thread test/test_subset.py Resolved
Comment thread uxarray/grid/grid.py Resolved
@erogluorhan

Copy link
Copy Markdown
Member
  1. I had a suspicion that extra dimensions (not just extra coordinates) on input datasets would be an issue, but I tested a few things locally and found that it wasn't a problem (for some reason…). For example, if you had a time dimension on the input dataset, instead of just a time coordinate, it doesn't seem to lead to a crash later (although, it does leak into Grid._ds). It isn't really obvious to me why it wouldn't crash…. Can you clarify why this might be the case, and/or add a test to ensure subsetting still works even after adding a time dimension to the grid?

I agree with this, I think we shouldn't let extra dimensions either into the Grid regardless of whether they lead to crash or not.

  1. Would you also be able to create a file with a time coordinate somewhere in the test suite (in meshfiles folder), and test that, too, instead of only having a test where the time coordinate gets assigned after loading the dataset? That would help me to be more convinced that this fully closes the original issue. It's a bit tricky to know for certain because the original issue report doesn't contain a complete verifiable example. But, the workflow there seemed to be "load from file, then subset (which crashes)" so ideally a test here could include that complete workflow as well.

I don't think we should expose a grid file with time coord/dim into our testing suite as it is uncommon to have such a presence and it would be misleading in the future.

@erogluorhan erogluorhan left a comment •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks very much for this fix! @Sevans711 has already made great points inline and overall ( see (1) here, but I believe (2) should be avoided), so I am approving it to make way once they're addressed.

@Sevans711

Copy link
Copy Markdown
Collaborator

Thanks very much for this fix! @Sevans711 has already made great points inline and overall ( see (1) here, but I believe (2) should be avoided), so I am approving it to make way once they're addressed.

Ah, for (2), I was going based on the file at gridpath("mpas", "QU", "oQU480.231010.nc") which is treated as dataset data and grid data during test_open_dataset_single_combined_xarray_dataset in test_api.py (related: #1479). That file already has a Time dimension in it. So, the testing suite does already include a "grid file" with time dimension, though it isn't really a simple grid file, rather it is a single file grid + data situation.

That was leading me to think it would be reasonable enough to include a second version of this file which is extremely similar but just has a scalar time coordinate instead of a Time dimension, and then call ux.open_dataset on it. What would your thoughts be on something like that, @erogluorhan?

Either way, I do still want to see evidence of the full original workflow being reproduced somewhere before I am ready to approve this as fully closing the original issue. Ideally this would involve an example of "load from file, then subset" crashing on main but not crashing on this branch. I spent a little time trying to reproduce something like that on my end, but I'm not succeeding quickly and I want to avoid duplicating work if you've already run such a test. @dylannelson my apologies if you have already shared test results like this - would you be able to point me to them if this test was already performed somewhere?

@dylannelson

Copy link
Copy Markdown
Member Author

@Sevans711
For 1, yeah the weirdness of dimensions vs coordinates here is what has taken me a while to understand and work around. It seems to have to do with how dimensions are used during .isel indexing. When it's a scalar coordinate it clashes with other existing times, whereas when its a dimension, it only attaches to variables that share a time with it, so time doesn't end up on the index at all.

  • this may be why other issues related to this one claim that subsetting didn't work, maybe because it had a time but didn't share the time so it did nothing or something wrong? (really hard to tell without seeing the data) But I'm not sure, I haven't explore this case much as it was never a dimension, and we don't know if it can be naturally.
  • note the time dimension will still be in the Grid._ds, it just won't raise conflicts. We can forcefully make it a dim, but it doesn't get carried into subsetting functionality at all.

The summary of a path deep dive through many files and functions basically came down to:

xarray attaches a coordinate to a variable only when the coordinate's dimensions are a subset of the variable's dimensions.

When it's scalar time → dims = (). And () is a subset of everything, so it glues onto the n_face index array.
Dimensional time → dims = (time,). (time,) is not a subset of (n_face,), so it stays off the index array.

Which happens about here it xarray: xarray/core/coordinates.py#L1214

Filtering dimensions seemed to be distinct from this problem (filtering coordinates) in my mind. Like I don't think we saw anything that indicates a time dimension could appear naturally like this. The downstream affects of filtering dimensions is different than coordinates too. If my understanding seems wrong, and a dimension filter here seems good, maybe we can scope this out over a call because there's lots of threads that have felt tangled while exploring this, and I don't want to overcompensate in this solution because it might introduce new problems.

Maybe of note, that uxarray also won't report any Dims that it has that aren't in DIM_NAMES - which is a static list (in grid.py):

    @property
    def dims(self) -> set:
        """Names of all unstructured grid dimensions."""
        from uxarray.conventions.ugrid import DIM_NAMES

        return set([dim for dim in DIM_NAMES if dim in self._ds.dims])

so it shouldn't ever really report anything outside of:

DIM_NAMES = [
    "n_node",
    "n_edge",
    "n_face",
    "n_max_face_nodes",
    "n_max_face_edges",
    "n_max_face_faces",
    "n_max_edge_nodes",
    "n_max_edge_edges",
    "n_max_edge_faces",
    "n_max_node_faces",
    "n_max_node_edges",
    "two",
]
  1. Was about to respond why we didn't add local files, but @erogluorhan got there first
    Basically all my local tests read a file, made a alternate version with time, saved, reopened, check if it has time, and run subset to see if it crashes. This effectively does the same because the same Grid.__init__ should be called either way. If you're interested in seeing that we could hop on a call and I can demo. I have a few notebooks dedicated to this.

@Sevans711 Sevans711 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the thorough investigation and updates here! The tests look reasonable by-eye, and I ran some of them locally against main to confirm they fail on main but are fixed by this PR.

I would prefer to just always drop non grid data during Grid.__init__, instead of introducing an opt-out option which gets added to the public API and woven throughout various parts of the codebase. My understanding of uxarray developer team preferences in general is that there is usually a hesitance to add new options to the public API (see e.g. the discussion in #1566).

Additionally, could you clarify a bit further what is the motivation for dropping scalar data variables? My understanding was that they weren't getting in the way of anything. Dropping them also led to needing to whitelist a few in particular: "grid_topology", "max_face_radius", plus a helper function for CRS variables (are these always scalars too? I might be misunderstanding the CRS variables case). The whitelist would be an ongoing maintenance burden whenever any new features are added which cache a scalar result on the grid. If the scalar data variables really aren't getting in the way of anything, my preference would be to just keep all of them, instead of building and maintaining a whitelist for them.

Comment thread uxarray/grid/utils.py
Comment on lines +36 to +39
set(ugrid.SPHERICAL_COORD_NAMES)
| set(ugrid.CARTESIAN_COORD_NAMES)
| set(ugrid.CONNECTIVITY_NAMES)
| set(DESCRIPTOR_NAMES)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the point of whitelisting these explicitly? Aren't they all being kept automatically, because they all correspond to variables with only grid dimensions?

If it is possible to claim the simpler invariant of "only keep variables with only grid dims, plus a few exceptions" by dropping these from keep, instead of needing to have code like "only keep [long list of whitelisted variables], plus variables with only grid dims, plus a few exceptions", I think that would be preferable.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DESCRIPTOR_NAMES has to stay: MPAS’s boundary_node_indices is on n_boundary_nodes.

The names only matter when a reader leaves one on a non-standard dim, which is the #1717 n2 shape problem. There's no value in removing them, but as Orhan said they can stay for now and be removed later if you want

Comment thread uxarray/grid/utils.py Outdated
Comment on lines +42 to +44
"bounds",
"face_bounds_lon",
"face_bounds_lat",

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suspect that these three ("bounds", "face_bounds_lon", "face_bounds_lat") would all be kept automatically if "min_max" and "lat_lon" were added to the list of dimensions being kept. (Locally, something like grid_dims = grid_dims + {"min_max", "lat_lon"}.) To me this would make more sense and represent a lower ongoing maintenance burden than explicitly whitelisting these three variables.

E.g., maybe a new feature like "face_bounds_x" might be added some day, with a "min_max" dimension. If the rule here is to consider "min_max" dimension as something to keep, there would be no need to touch this part of the code, but if it's just an explicit whitelist then "face_bounds_x" would need to be added here too.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1. I checked it: with lat_lon and min_max counted as grid dims, re-reading a grid that already has bounds and face_bounds_lon keeps them without listing names.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

they are kept now and the list is gone as we are no longer making a whitelist

Comment thread uxarray/grid/utils.py Outdated
"bounds",
"face_bounds_lon",
"face_bounds_lat",
"max_face_radius",

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For scalar vars like "grid_topology" and "max_face_radius" which need to be whitelisted, I would recommend defining the list somewhere in the conventions folder instead of here, and import it, similarly to ugrid.DIM_NAMES.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually… this, and the exceptions with CRS variables below, is only relevant if the decision is to drop scalar data variables. Wouldn't it be easier to reason through and maintain if we just keep all scalar data variables? (Still drop scalar coordinates, though.) Leaving a note in my overall review about this to clarify.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1. The test I suggested on test_non_grid_data.py:30 shows subsetting works with a scalar time data variable, even on main. See my review summary.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The lists have been removed and are no longer needed as we are keeping all scalar vars

@dylannelson

dylannelson commented Nov 11, 2026 •

Copy link
Copy Markdown
Member Author

@Sevans711 I see a lot of the comments above are about whitelisting and how it seems like a burden or a pain to manage, and while I agree, I don't think we can make such broad and strict rules like "Remove all of __" without making exceptions, due to the wide variance of data and models coming in, and how they are used at various points in the code. Since the beginning of issue, I've expressed concerns with putting it in Grid.__init__ because Grid.__init__ isn't used just when a file is read from disk, but at other times like subsetting as well, and data has been added and changed in between. The scope of this started with a single case, and expanded a lot broader since then. Along the way, I haven't found any single solution that's just easily works without edge cases we must account for. At this point I don't know what else to add or explain without once again coming up with another solution that will also likely have weird edge cases we have to account for.

Additionally, could you clarify a bit further what is the motivation for dropping scalar data variables?

This one I don't understand though, because I thought this has always been a part of the problem with time. Whether a stray scalar time reads back as a coordinate or a data variable depends only on how the file was written. It’s a coordinate if another variable’s coordinates attribute lists it, and a variable otherwise. If you want to remove this rule and it's own whitelist that's fine, it's just going to let in more data, which is part of what we're trying to avoid.

The API concerns I get too, but I think it was a very rare but potentially critical safety net for if the whitelist method doesn't let something through and the user says "Hey this variable/dim didn't get through, but I need it right now" and whatever model they're using can quickly suggest turning off the cleaning so it gets through, without having to wait for us to add it to a model specific exception and make a release. Just like how this entire fix is only for weird unexpected scenarios, I also thought it could be also be good to have a backdoor for other weird and unexpected scenarios. We can cut it out, but I think it could result in a user being unable to proceed because this growing list of filters stopped something they needed to get through. I was told early on we shouldn't even have thing like time on the grid in the first place, but we did, along with a lot of other data, and getting rid of it needs to be done carefully as data on the grid can't be just wiped out. Many other parts of the code have been developed to add and interact with data on the grid. As we discussed earlier "without altering _ds" doesn't seem feasible or ideal

If you want to take a shot at making a solution that could work with all edge cases and without a whitelist, send me what that could look like, but I haven't really found or thought of anything that works like that. I could respond to each of the questions above, but don't want to continue to do so if the whole idea is going to be scrapped and rescoped again.

@cmdupuis3 cmdupuis3 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. A grid opted out with a stray time coordinate raises IndexError on isel. Also, a grid coordinate on n_face (lev) shows up in data subsets.

in dataarray.py:_slice_from_grid, add .data:

         if self._face_centered():
             da_sliced = self.isel(
-                n_face=sliced_grid._ds["_subgrid_face_indices"], ignore_grid=True
+                n_face=sliced_grid._ds["_subgrid_face_indices"].data, ignore_grid=True
             )

         elif self._edge_centered():
             da_sliced = self.isel(
-                n_edge=sliced_grid._ds["_subgrid_edge_indices"], ignore_grid=True
+                n_edge=sliced_grid._ds["_subgrid_edge_indices"].data, ignore_grid=True
             )

         elif self._node_centered():
             da_sliced = self.isel(
-                n_node=sliced_grid._ds["_subgrid_node_indices"], ignore_grid=True
+                n_node=sliced_grid._ds["_subgrid_node_indices"].data, ignore_grid=True
             )
  1. geoflow-small loses mesh_depth from to_xarray(), and no warning is raised.
import warnings, uxarray as ux

path = "test/meshfiles/ugrid/geoflow-small/grid.nc"
with warnings.catch_warnings(record=True) as w:
    warnings.simplefilter("always")
    new = ux.open_grid(path)                         # branch default
old = ux.open_grid(path, drop_non_grid_data=False)   # == main

sorted(set(old.to_xarray().variables) - set(new.to_xarray().variables))  # ['mesh_depth']
[str(x.message) for x in w if "drop" in str(x.message).lower()]          # []

Recommend defaulting drop_non_grid_data to false:

# uxarray/core/api.py  open_grid(...)
-    drop_non_grid_data: bool = True,
+    drop_non_grid_data: bool = False,
# uxarray/grid/grid.py  Grid.__init__(...)
-        drop_non_grid_data: bool = True,
+        drop_non_grid_data: bool = False,
# uxarray/grid/grid.py  Grid.from_dataset(...)
-        drop_non_grid_data = kwargs.get("drop_non_grid_data", True)
+        drop_non_grid_data = kwargs.get("drop_non_grid_data", False)
  1. Speed issue with ds[name] lookups in _drop_non_grid_data, because every call materializes a new dataarray; recommend iterating through ds.variables instead.

@erogluorhan

Copy link
Copy Markdown
Member

@dylannelson, the back-and-forth on this PR is mostly on me. I approved on Aug 20 and reversed myself the next morning. I pushed back on the ndim == 0 narrowing that @rajeeja asked for and you had just implemented. Your Aug 25 question got a partial answer and then sat for ten weeks. That's four different scopes in three months.

This wasn't a normal PR: we were learning the problem while growing the fix, and you and then @Sevans711's consistent review carried most of that.

So instead of adding another opinion, I'll make the decision that unblocks the rest (review below). You've asked for a call a few times. Until reading below, if you still need that, let's do 30 minutes this week and close it out.

@erogluorhan erogluorhan left a comment •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

EDIT (11/18/2026): After internal discussion with @dylannelson and @Sevans711 , I realized I confused myself about the first item below in my review. See the updated one there.

Here is my call on the open design questions. With these three changes I think this is ready:

  1. EDIT (11/18/2026): The original suggestion below (i.e. move to Grid.from_dataset) was wrong. Filtering should hold for every Grid, and it is possible if the filter is kept in Grid.__init__.

    Filter once, on read. Call _drop_non_grid_data at the end of the auto-detect branch of Grid.from_dataset, not in Grid.__init__. That branch runs after every _read_*, so it is still one format-agnostic place (my Aug 11 point 4). It's also what @rajeeja suggested on Aug 17 and @Sevans711 on Aug 26. Copies and subsets no longer re-filter, so the bounds / face_bounds_* / max_face_radius entries and the slice.py / copy() plumbing go away.

  2. Keep scalar data variables, drop only scalar coordinates (@Sevans711's rule). Only coordinates are attached to the subset indexer, so a scalar time data variable can't cause #1444; the test suggested inline checks this. Adding @Sevans711's lat_lon / min_max dims means grid_topology and max_face_radius don't need listing either.

  3. No drop_non_grid_data option. We build the grid, and users shouldn't need to know or choose what we keep from the source file. The option is also only partly wired: from_topology, from_points and the other constructors ignore it, and 4 of its 9 hand-offs are untested. With 1 and 2 in place, there is little left for it to guard against.

I prototyped this (diff below). The filter becomes "grid dims plus lat_lon/min_max". It still has to keep descriptors (MPAS boundary_node_indices) and the grid_mapping CRS variable (needed for rioxarray's scalar spatial_ref coordinate). On every grid in test/meshfiles it keeps exactly what this branch keeps. The #1444 tests still pass, and in the full suite only the opt-out test and the scalar_data_var case fail, as expected.

Housekeeping:

  • Merge main; the branch has conflicts now.
  • In the description: remove the stray ( after "Closes #1444" and add "Closes #1720". Also note that Grid._ds / to_xarray() no longer carry non-grid variables (e.g. geoflow mesh_depth, FESOM2 nod_area), so it gets into the compatibility section of the release notes.
  • @rajeeja, you were right about ICON/FESOM2. clon/clat/vlon/... are kept now, and test_grid_files checks it. Could you withdraw or update your Aug 20 change request?
Prototype diff (filter placement and rule only; removing the option, its plumbing and the test updates are not included)
--- a/uxarray/grid/grid.py
+++ b/uxarray/grid/grid.py
@@ -203,7 +203,7 @@
         # drop variables that don't belong on the grid (e.g. a `time` carried in from a
         # file holding both grid and data), so they can't collide during subsetting.
         self._drop_non_grid_data = drop_non_grid_data
-        self._ds = _drop_non_grid_data(grid_ds) if drop_non_grid_data else grid_ds
+        self._ds = grid_ds
 
         # source grid specification (i.e. UGRID, MPAS, SCRIP, etc.)
         self.source_grid_spec = source_grid_spec
@@ -325,6 +325,7 @@
                     raise GridInvalidError(
                         f"Unsupported source_grid_spec={source_grid_spec!r}, in {cls.__name__}.from_dataset()"
                     )
+                grid_ds = _drop_non_grid_data(grid_ds)
             else:
                 # custom source grid spec is provided
                 source_grid_spec = kwargs.get("source_grid_spec", None)
--- a/uxarray/grid/utils.py
+++ b/uxarray/grid/utils.py
@@ -17,46 +17,22 @@
 
 
 def _drop_non_grid_data(ds):
-    """Drop variables that don't belong on a ``Grid`` (e.g. a ``time`` carried in
-    from a file that holds both the grid and its data).
-
-    A variable or coordinate is dropped when it has no dimensions, or when any of
-    its dimensions is not a UGRID grid dimension (``ugrid.DIM_NAMES``).
-
-    Variables uxarray reads or caches on the grid are always kept, whatever their
-    dimensions: grid coordinates, connectivity, descriptors (including MPAS
-    ``boundary_node_indices``), the cached ``bounds``, ``face_bounds_lon``,
-    ``face_bounds_lat`` and ``max_face_radius``, ``grid_topology``, and any CRS
-    variable named by a ``grid_mapping`` attribute (read by projected-grid
-    detection).
-    """
-    grid_dims = set(ugrid.DIM_NAMES)
-
-    keep = (
-        set(ugrid.SPHERICAL_COORD_NAMES)
-        | set(ugrid.CARTESIAN_COORD_NAMES)
-        | set(ugrid.CONNECTIVITY_NAMES)
-        | set(DESCRIPTOR_NAMES)
-        | {
-            "grid_topology",
-            "bounds",
-            "face_bounds_lon",
-            "face_bounds_lat",
-            "max_face_radius",
-        }
-    )
-    # CRS variables, referenced by name from a ``grid_mapping`` attribute
-    # (CF also allows the extended form "crs: x y")
-    for name in ds.variables:
-        grid_mapping = ds[name].attrs.get("grid_mapping")
-        if isinstance(grid_mapping, str):
-            keep.update(token.rstrip(":") for token in grid_mapping.split())
-
+    """Drop variables and coordinates with a non-grid dimension, and scalar coordinates."""
+    grid_dims = set(ugrid.DIM_NAMES) | {"lat_lon", "min_max"}
+    # descriptors (MPAS boundary_node_indices) and CRS variables named by grid_mapping
+    keep = set(DESCRIPTOR_NAMES) | {
+        var.attrs["grid_mapping"].split(":")[0]
+        for var in ds.variables.values()
+        if isinstance(var.attrs.get("grid_mapping"), str)
+    }
     drop = [
         name
-        for name in ds.variables
+        for name, var in ds.variables.items()
         if name not in keep
-        and (ds[name].ndim == 0 or any(dim not in grid_dims for dim in ds[name].dims))
+        and (
+            any(dim not in grid_dims for dim in var.dims)
+            or (var.ndim == 0 and name in ds.coords)
+        )
     ]
     return ds.drop_vars(drop)
 

Comment thread uxarray/grid/grid.py Resolved
Comment thread test/grid/grid/test_non_grid_data.py Resolved
Comment thread test/grid/grid/test_non_grid_data.py Resolved
@dylannelson

Copy link
Copy Markdown
Member Author

@cmdupuis3

  1. A grid opted out with a stray time coordinate raises IndexError on isel. Also, a grid coordinate on n_face (lev) shows up in data subsets.

The .data change is in _slice_from_grid now.

  1. geoflow-small loses mesh_depth from to_xarray(), and no warning is raised.
    Recommend defaulting drop_non_grid_data to false:

The team decided to remove the API option and remove the whitelist so it seems this can't be addressed at the moment

  1. Speed issue with ds[name] lookups in _drop_non_grid_data, because every call materializes a new dataarray; recommend iterating through ds.variables instead.

Good idea, it has been changed and fixed. _drop_non_grid_data loops over ds.variables now and never builds a DataArray per name.

@erogluorhan erogluorhan left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All look good to me now; thanks for pushing it forward throughout all the scoping! Approving, but please do the following:

1Update PR's description to reflect the current state of the work. For instance, the opt-out is gone now, so item (8) is no longer valid. Also in "Changes Made", item 4 should only have Grid._init_ now. Maybe there is more, please check.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

extra dims Handling of non-grid dimensions, such as time or altitude run-benchmark Run ASV benchmark workflow

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Improved cleaning of grids objects to avoid leaked data .subset with da that has time dimension Filter non-spatial variables from UGRID datasets

5 participants