Repository navigation
Read derived-variable constituents as whole fields - #496
Open
bitterbark wants to merge 2 commits into
Open
bitterbark wants to merge 2 commits into
bitterbark wants to merge 2 commits into
Conversation
derive_variable() opened its constituents with xarray's default chunks, which follow the files' on-disk chunks. For 20 years of 3-hourly output on a 1-degree grid stored in 27 x 160 x 239 chunks, that is 8652 dask chunks per constituent, and deriving PRECT from PRECC + PRECL took about 10 minutes. The new constit_read_chunks() reads each constituent as whole fields, with as many time steps per chunk as fit in dask's "array.chunk-size" (rounded down to a whole number of on-disk time chunks). The same derivation now takes 2.0 minutes (3 runs) and matches an NCO-made PRECT exactly. AdfData.load_dataset() gains a "chunks" argument for this; its default leaves every other caller unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
richling-ucar
self-requested a review
October 8, 2026 20:35
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When using ADF to concatenate high-frequency derived variables, one of my processes hung for a full 24 hours. Claude looked into this and didn't find a specific fix but did identify that using a different chunking method would speed up the process significantly. In this example, 20 years of 3-hourly PRECC+PRECL output on a 1-degree grid can be done in 2 minutes whereas it used to take 10.
Previous method: derive_variable() opened its constituents with xarray's default chunks, which follow the files' on-disk chunks. For 20 years of 3-hourly output on a 1-degree grid stored in 27 x 160 x 239 chunks, that is 8652 dask chunks per constituent, and deriving PRECT from PRECC + PRECL took about 10 minutes.
The new constit_read_chunks() reads each constituent as whole fields, with as many time steps per chunk as fit in dask's "array.chunk-size" (rounded down to a whole number of on-disk time chunks). The same derivation now takes 2.0 minutes (3 runs) and matches an NCO-made PRECT exactly. AdfData.load_dataset() gains a "chunks" argument for this; its default leaves every other caller unchanged.