Skip to content

Read derived-variable constituents as whole fields - #496

Open
bitterbark wants to merge 2 commits into
NCAR:mainfrom
bitterbark:derive_chunking
Open

bitterbark wants to merge 2 commits into
NCAR:mainfrom
bitterbark:derive_chunking

Conversation

@bitterbark

Copy link
Copy Markdown
Collaborator

When using ADF to concatenate high-frequency derived variables, one of my processes hung for a full 24 hours. Claude looked into this and didn't find a specific fix but did identify that using a different chunking method would speed up the process significantly. In this example, 20 years of 3-hourly PRECC+PRECL output on a 1-degree grid can be done in 2 minutes whereas it used to take 10.

Previous method: derive_variable() opened its constituents with xarray's default chunks, which follow the files' on-disk chunks. For 20 years of 3-hourly output on a 1-degree grid stored in 27 x 160 x 239 chunks, that is 8652 dask chunks per constituent, and deriving PRECT from PRECC + PRECL took about 10 minutes.

The new constit_read_chunks() reads each constituent as whole fields, with as many time steps per chunk as fit in dask's "array.chunk-size" (rounded down to a whole number of on-disk time chunks). The same derivation now takes 2.0 minutes (3 runs) and matches an NCO-made PRECT exactly. AdfData.load_dataset() gains a "chunks" argument for this; its default leaves every other caller unchanged.

derive_variable() opened its constituents with xarray's default chunks,
which follow the files' on-disk chunks.  For 20 years of 3-hourly output on
a 1-degree grid stored in 27 x 160 x 239 chunks, that is 8652 dask chunks
per constituent, and deriving PRECT from PRECC + PRECL took about 10
minutes.

The new constit_read_chunks() reads each constituent as whole fields, with
as many time steps per chunk as fit in dask's "array.chunk-size" (rounded
down to a whole number of on-disk time chunks).  The same derivation now
takes 2.0 minutes (3 runs) and matches an NCO-made PRECT exactly.
AdfData.load_dataset() gains a "chunks" argument for this; its default
leaves every other caller unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@richling-ucar
richling-ucar self-requested a review October 8, 2026 20:35
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant