Skip to content

Bring in Land and Native Grid - #502

Draft
richling-ucar wants to merge 536 commits into
mainfrom
ldf-native-grid-merge-main
Draft

richling-ucar wants to merge 536 commits into
mainfrom
ldf-native-grid-merge-main

Conversation

@richling-ucar

@richling-ucar richling-ucar commented Oct 26, 2026 •

Copy link
Copy Markdown
Collaborator

Native-grid and CLM land diagnostics, merged with main

This PR brings native-grid (ne30pg3) support and the CLM land diagnostics (from ww-clm-diags) onto current main, with the bug fixes found by a 64-run regression suite on Casper. The full results, plots and per-file comparisons are in the LDF Merge Test Log.

What this PR does

  • Native grids. Native ne30pg3 history becomes native time series and climatologies. They are then gridded to the baseline's lat/lon grid with xESMF and a weights file (bilinear by default, conservative on request), or plotted on the mesh with UXarray (unstructured_plotting). Native and lat/lon cases, and native cases and observations, can be compared either way: in UXarray mode the lat/lon side is remapped onto the native side's mesh (ESMF conservative weights, uxarray bilinear fallback).
  • CLM land diagnostics. Land runs are recognised from clm2 in hist_str. Area × landfrac weighting, separate time-series and climatology years, regional climatologies and time series from regions_lnd.yaml, the LMWG table and website.
  • Pipeline entry points. Each case can start from history, time series or climatology. cam_ts_done is now calc_cam_ts (old key still accepted). Existing time series are reused per variable.
  • Plot whatever data there is. When one case lacks a file, plots show the cases that have it instead of being skipped.
  • Meridional means over 5°S–5°N. meridional_mean now averages over the latbounds band (default 5°S–5°N), which main sets but ignores, averaging over all latitudes. All meridional-mean plots change.
  • LENS2 per variable. The CESM2 LENS2 range behind the global-mean time series comes from each variable's lens2_file in the variable defaults, found in the config's lens2_data_loc (as obs_file and obs_data_loc), instead of a hard-coded directory.
  • Climatology years in file names and metadata. Climo and regridded files are named after their climatology years (…_climo_0011-0013.nc), so a run over other years never reuses a file made for the old ones. Time series files record ts_yrs, climo and regridded files climo_yrs. Older names are still read.
  • One set of loaders for every file type (closes Clean up data set and data array calls from adf_dataset.py #424). AdfData has get_…, load_…_dataset and load_…_da for time series, climatologies, regridded files, raw history files (by stream) and TEM files, for the test cases and the reference. The reference loaders take only the variable. Scripts that opened files themselves (QBO, regional maps, TEM, tape recorder) now go through them and get the unit conversions.
  • TEM. TEM files carry the model's zonal-mean pressure (PZM) for their own time steps, so THZM becomes temperature with the right pressure; TEM fields get their unit conversions (WTEM in cm/s, UTENDEPFD in mm/s²).
  • Plain-text units. new_unit is plain text in all four variable-defaults files (W/m2, not W m$^{-2}$); the LaTeX moved to mpl: colorbar: label, so colorbars look the same.
  • One statistics table. amwg_table is now statistics_table, which also makes the land table; lmwg_table is gone. Its standard error is now σ/√N with the sample standard deviation.
  • main merged in: eba1f07, keeping main's plotting features (NCL colormaps, per-plot settings, colormap_2d, surface wind stress), then 2fcf147 (PR Add full python unit testing suite to Github Actions workflow #508), whose CI runs every unit test in the full ADF environment.
  • A case compared to itself over other years. The climo step picks the baseline by its place in the case list, not by name, so the test case reads its own time series (on main it reads the baseline's); regional legends add the climo years. From wwieder (PR additional updates #512), with TOTECOSYSC and TOTSOMC in the land defaults.
  • Finer grid onto the coarser one. Fields are compared on the coarser of the two grids, by mean cell area (within 10% counts as the same). When a test case is clearly coarser than the reference, the reference (baseline or observations) is regridded onto the case's grid, conservatively, keeping a cell only where at least half of it has data; the case keeps its own grid. Two grids that are the same are no longer regridded at all. On f09 this puts the 1° and 0.25° observations (CERES, MODIS, HadISST, ERA5, …) onto f09; GPCP and ERA-Interim are coarser and are unchanged. UXarray mode and native gridding warn when they interpolate a coarser grid onto a finer one.
  • Environment adf_v1.0.6: uxarray 2026.10.0 (native maps need at least 2025.6.0), plus pylint 4.1.3, astroid 4.3.4 and pytest 9.1.1 as developer tools.

Testing

Check Result Notes
Suite 1 · pipeline entry points (S00–S10) 11 of 11 No unexpected differences from main. Only the meridional band and the new one-side-only time-series plots differ.
Suite 2 · native grid, remapping, UXarray (N01–N11) 12 of 12 Gridded native climatologies match ncremap to about 1e-7; AMWG tables identical (N01–N03).
Suite 3 · land (L01–L14) 14 of 14 f09, gridded native and UXarray native; native table means within 0.9% of f09.
Suite 4 · against observations (S00b, S11–S13, N12, N13, with N08, N09a) 8 of 8 Every value identical to main in obs mode (S11–S13 against S00b); only the meridional band and the one-sided global-mean plots differ. Native ts and climo entries match N08 to 1.8e-6. Atmosphere only: no land observations to test against.
Suite 5 · every native scenario gridded and on the mesh (N14–N24, L15–L23) 20 of 20 Each native pairing, from every entry point and in both directions, plotted both ways. Against the run covering the same grids from another entry point, every shared plot is pixel-identical and every regridded value identical. Mirrors agree (N09b/N14 TS mean difference 0.57/0.56).
Meridional means on the mesh (N04, N09a, N09b, L07) 4 of 4 Native vs native matches the gridded runs; native vs obs or f09 plots the test case alone; land on the mesh plots both cases over 5°S–5°N.
Land reference: L01 against the land scientists' code (L00, ww-clm-diags c6db27e) matches Time series, climatologies, land-table numbers and every plotted curve identical. Differences are ours by design: coastal cells kept, plain-text units, colour-bar arrows, case legend. The land branch crashes in polar maps and needs a mesh for f09 regional plots.
Extended run (years 1–3, then 1–5, same folders) pass Only the new years' time series made; new year-tagged climo and regridded files; climatology and table equal the 1–5 run's.
TEM and QBO (TEM-2026-PZM) pass TEM files through the history loaders, with the model pressure PZM; WTEM and UTENDEPFD now converted; THZM as temperature from the TEM file's own pressure.
Regional maps, merged climo loaders (REGMAP, CLIMODS) 2 of 2 regional_map_multicase on the loaders (PSL now in hPa); regridding plus 80 lat/lon and zonal plots on load_climo_dataset.
S03, S13, N09a rerun on 8359365 3 of 3 match Reference loaders without case and plain-text units: every regridded value identical to the suite's runs, the same plots made; only the units strings change (mm d$^{-1}$ → mm/d).
statistics_table under all three names pass amwg_table and lmwg_table print a note and run it; standard error checked by hand.
Finer grid onto the coarser one (S03, S13, C01 on 92b01f2) 3 of 3 Same plots and warnings as before. S03 (same f09 grid): baseline files identical, test files change only in the two rows next to the poles, which the old f09→f09 regrid altered (up to 1.95 m/s in U). S13 and C01 (f09 against observations): 15 observation fields now on f09, the model on its own grid; area means within about 1e-4 of before. Coastal cells: with the half-coverage rule, OAFlux latent heat lands on 439 f09 land cells instead of 1,205.
PR #510 merged in (N01 three ways) 3 of 3 .092 vs .091 ne30pg3, gridded. Before the merge, after it with the new settings off, and with conservative_weights_file and gridded_keep_precision: same 1,014 plots and warnings each time. With the settings off every time series and gridded file holds the same values, and native time series are 0.69× the size; with them on, fluxes use the conservative map (TAUY global mean 0.4% → 0.005% off the native mean) and gridded files are half the size.
Unit tests (lib/test/unit_tests) 394 passed None skipped, on 14fb022 (380 on da9b821, 375 on 92b01f2, 363 on 72f4f7e, after main's #508 removed some tests). GitHub CI runs them all in the adf_v1.0.6 environment and passes, as does its canary on the latest Python and packages.
CI lint script (pylint, every modified lib/ file) pass Threshold 9.5. CI's lint job last ran on 14fb022 and passed; locally adf_dataset.py 9.97, adf_diag.py 9.81, adf_info.py 9.90, adf_utils.py 9.97, the new test files 9.74–10.00.
black 25.11.0 on lib/ clean 54 files unchanged (CI's pinned black).

Note

The latest clean rerun (all 38 runs, on 1a9fbb7) started from wiped output folders, in an env built from env/conda_environment.yaml. S00 ran on main in its own env; S00 and S01 agree in every value, so the env change moves no result. The full suite has not been rerun since 92b1105, so it predates wwieder's PRs #504, #506 and #509, the data-access work, the merge of main 2fcf147, 72f4f7e and the adf_v1.0.6 environment.

Bugs fixed

All 30 fixes (14 also affect main); click to open
# Bug Fix On main?
1 Gridded native time series read five years as about 75 days (time re-encoded in hours, bounds left in days). grid_to_latlon, three save_to_nc, use_time_bounds_midpoint partly
2 Every UXarray map failed: uxarray 2025.03.0 has no to_raster. uxarray 2025.6.0 env
3 Native test vs non-native reference crashed the map, vector and polar scripts. warn-and-skip checks (now remapped onto the mesh instead) no
4 The same case crashed zonal_mean. missing-reference checks no
5 Polar maps crashed without plot_press_levels. defaults to an empty list yes
6 Global-mean time series crashed when the LENS2 file could not be read. LENS2 left out with a warning yes
7 Regional plots failed on native lndgrid climo. _region_native no
8 UXarray runs with no regrid step made no maps. AdfData._native_climo_fallback no
9 Regional plots used the baseline's mesh for both cases. each case on its own grid no
10 A plot was dropped when one case lacked data. plot the cases that have it behaviour
11 get_regrid_file raised for variables with no reference. returns an empty list no
12 Native maps crashed on all-zero fields. “No Valid Data Points” panel no
13 Regional legends hid the curves. one shared case legend no
14 main's create_clean_grid fails on bounds with a time dimension. fixed here; also needs a PR to main main only
15 Single-case line plots: the title overlapped the case name, and the note said “no baseline data” for a baseline on another grid. constrained layout; “no comparable baseline data” no
16 Derived ASA (albedo, %) kept the W/m² units of FSR, so plots labelled it “W/m^2”. a formula that changes units sets them (“%”, “permil”) partly
17 Polar maps on the mesh drew missing cells as 0% and printed a nan mean. missing cells left blank; NaN-safe means no
18 Polar maps against observations crashed the run when an obs variable is on its own dimensions (land MODIS albedo on lsmlat/lsmlon). skip with a warning, as the lat/lon map does no
19 Maps on the mesh against a remapped lat/lon side crashed: cell_weights dropped the uxarray grid with isel(time=0). grid taken first; reference carries the test case's area/landfrac no
20 Obs on the mesh: the copied area kept the model's time labels and aligned to NaN against a one-step obs file. area, landfrac written time-independent no
21 Vector maps: an early return skipped the remaining pairs; crashes on the mesh without pressure levels or plot_press_levels. restructured like global_latlon_map; 30 plots byte-identical partly
22 Taylor diagram against a lat/lon side on the mesh: out of memory (native test) or KeyError: 'lat' (lat/lon test vs native reference). reference loaded per case on the right grid no
23 Native (UXarray) lat/lon maps drew missing cells as 0 in the test, baseline and difference panels (only the % panel left them blank): the ocean of land fields, and below-ground cells of atmosphere fields on pressure levels (N04's T at 850 hPa: mean 268.77 K instead of 281.10 K, RMSE 4.56 instead of 0.52). prep_ux_for_plot no longer fills NaN with 0 no
24 TEM fields were never converted (they are not in diag_var_list): model WTEM and UTENDEPFD plotted in raw units, while observed UTENDEPFD was scaled. TEM loaders read the converters from the variable defaults yes
25 TEM plots crashed with KeyError: 'label' after the plotting merge from main. colorbar labels passed explicitly yes
26 THZM was converted with pressure averaged over the time-series years, not the TEM file's years. PZM written into the TEM file yes
27 regional_map_multicase skipped the unit conversion (PSL in Pa) and the time-bounds correction. reads through the loaders yes
28 Tables divided by N, not √N, for the standard error, with the population standard deviation. σ/√N, sample standard deviation yes
29 A model baseline without cam_climo_loc crashed at start-up. guard in get_reference_climo_file yes
30 Two cases on the same grid were still regridded: f09 to f09 conservative regridding altered the rows next to the poles (up to 1.95 m/s in U), and a coarse case was interpolated up to a finer reference's grid (f09 to 0.25° ERA5). same grid not regridded; finer grid onto the coarser one (92b01f2) yes

Note

Two more were in the test setup (a config that read its own output, stale files between runs). Details for each are in the test log.

Release notes

  • cam_ts_done is now calc_cam_ts, with the opposite meaning, matching calc_cam_climo: calc_cam_ts: true makes time series from the history files, as cam_ts_done: false did. Replace cam_ts_done: X with calc_cam_ts: not X. The old key still works and prints a deprecation note; giving both with contradicting values stops the run.
  • Existing time series are reused per variable. With calc_cam_ts: true, files already in cam_ts_loc are kept and only the years they do not cover are made, each stretch in a file of its own.
  • Runs on pre-made climatologies. With calc_cam_ts: false and calc_cam_climo: false a case needs no time series; scripts that need them leave that case out.
  • New optional config keys, documented in config_cam_baseline_example.yaml. In diag_basic_info: unstructured_plotting, regions_file, lens2_data_loc. Per case: climo_start_year/climo_end_year (with ts_start_year/ts_end_year as other names for start_year/end_year), and for native-grid cases mesh_file, weights_file, regrid_method and, for CLM, latlon_file. At the top level: region_list and regional_var_list.
  • LENS2 needs lens2_data_loc. A config that runs global_mean_timeseries and wants the LENS2 range must set lens2_data_loc (on Derecho/Casper: /glade/campaign/cgd/cas/islas/CESM_DATA/LENS2/global_means/annualmeans). New lens2_file entries cover ACTREL, AODVIS, CDNUMC, CLDHGH, CLDLOW, CLDMED, CLDTOT, FSDS, FSDSC, FSNT, FSNTC, SWCF, TGCLDIWP, TGCLDLWP and TS. The aerosol burdens (BURDEN*dn) have LENS2 files but no variable-defaults entry, so they no longer get the range.
  • AMWG tables use the time-series years, as the land tables already did. With climo_start_year/climo_end_year narrower than the time series, the table still covers all the time-series years; the climatology years no longer steer its file choice or its one-year check.
  • Climo and regridded file names carry the climatology years ({stem}_YYYY-YYYY.nc). Existing files with the older names are still read, but are never overwritten or reused by a run over other years: that run makes new year-tagged files. Time series files now record ts_yrs.
  • Plots that change for main users: meridional means (5°S–5°N band), and single-case time-series, QBO, tape-recorder, zonal and meridional plots for runs where one case has no data, which used to be skipped.
  • Renamed data-access functions, for scripts outside the repo: load_timeseries_dataset now takes (case, field) (the old file-list form is load_timeseries_files); load_climo_ds and load_reference_climo_ds are now load_climo_dataset and load_reference_climo_dataset; load_climo_file is gone; the reference loaders take only the variable (load_reference_regrid_da(var)), and an old two-argument call stops with a TypeError.
  • amwg_table and lmwg_table are now statistics_table. Configs listing an old name still work and print a note. The output files keep their names (amwg_table_<case>.csv, lmwg_table_<case>.csv). The standard deviation, standard error and 95% CI columns change (σ/√N, sample standard deviation).
  • new_unit is plain text. For LaTeX on a plot set mpl: colorbar: label. Time-series axis labels now show the plain units.
  • TEM files need remaking (overwrite_tem: true, once) to carry the model pressure; older files still plot, converting THZM with the hybrid coordinate and a warning. WTEM and UTENDEPFD plots change (now converted), as do regional maps (unit conversions).
  • Environment adf_v1.0.6: rebuild with conda env create -f env/conda_environment.yaml. Native-grid maps need uxarray 2025.6.0 or later (the env has 2026.10.0).
  • Comparisons are on the coarser grid. When observations or a baseline are on a finer grid than the test case, they are now regridded onto the case's grid, written as {ref}_{case}_{var}_reference_on_case_grid_YYYY-YYYY.nc, instead of the case being interpolated up to theirs. For an f09 case this applies to every 1° and 0.25° observation file, so their lat/lon, polar, zonal, meridional and Taylor plots, and the numbers in the plot headers, change slightly. Cases on the same grid are no longer regridded.
  • Land defaults: TOTECOSYSC_1m is now TOTECOSYSC, and TOTSOMC is new (both summed to PgC in the tables). A config listing TOTECOSYSC_1m needs the new name.

Before this PR is ready

  • Commit this round's work. The fixes above, the new unit tests and the env change are in five commits, 8e60e34 to c1e6bd2, pushed to the branch. The two root-level CLM configs below were left out.
  • Commit fix 15. The single-case plot fix in lib/plotting_functions.py: committed and pushed as c711eb7.
  • Commit the year-tagged file names and fixes 16 and 17, with their unit tests: pushed as f45d03d, b2a660d and e8a7c65.
  • Rerun the full suite (now 38 runs with L00) on the year-tagged names: done on e8a7c65, all 38 pass.
  • Skip regridding when the grids are identical. f09 to f09 conservative regridding changes the rows next to the poles (288 cells each, up to 1.95 m/s in U). Done in 92b01f2, with the finer-onto-coarser rule: a field already on the target grid is passed through (S03).
  • Fix two unit strings in ldf_variable_defaults.yaml: ELAI m${-2}$ m${-2}$ and QBOT kg kg{-1}$ come out garbled in tables: fixed in 1a9fbb7.
  • Decide on the root-level CLM configs. Kept, and updated to the new config keys (start_year/end_year, calc_cam_ts) in eb0ccfb.
  • Document the new config keys in config_cam_baseline_example.yaml. Added, commented out: unstructured_plotting, regions_file, the per-case climo_start_year/climo_end_year, mesh_file, weights_file, regrid_method and latlon_file, and region_list/regional_var_list; lens2_data_loc is set.
  • Sign-off on the uxarray pin. Moving to 2026.10.0 (adf_v1.0.6, from PR Minor corrections #504) changes main's environment for every user.
  • CI does not run the science tests. CI installed only PyYAML and pytest, so 141 tests skipped, including every native-grid and land test. Settled by main's PR Add full python unit testing suite to Github Actions workflow #508, merged in as 2c08c5b: CI now runs all 363 tests in the full ADF environment and fails on a skip. scripts/ is still not linted.
  • New plots and website entries on main. Runs that used to skip time-series, zonal and meridional plots now make single-case versions.
  • Size. 69 files and about 22,750 added lines. Consider separate PRs for the fixes that also affect main (1, 5, 6), native grid, land, and the plot-whatever-there-is rule.
  • Open the small PR to main for fix 14 (create_clean_grid with time-dimensioned bounds).
  • Merge main again and rerun Suite 1. main 2fcf147 (PR Add full python unit testing suite to Github Actions workflow #508) is merged in as 2c08c5b, and CI runs and passes every unit test. Suite 1 has not been rerun yet.
  • Rerun the full suite on the current tip and adf_v1.0.6. The 64 runs predate PRs Minor corrections #504, Improve lat-lon plots #506 and Zero in middle of diff plots #509, the data-access work, the merge of main 2fcf147 and 72f4f7e, and ran on adf_v1.0.5; the data-access work was checked with seven targeted runs.
  • Commit the statistics_table rename and the standard-error fix (fix 28), with the twelve configs and the README that now name it: committed in 0946a4d.
  • Hard-coded LENS2 path. Replaced by a per-variable lens2_file in the variable defaults, found in the config's lens2_data_loc. Land variables get none (the old search matched atmosphere files for them).
  • Standard error in the tables. Divided by N instead of √N, so the standard error and 95% columns were too small. Fixed here (fix 28, 0946a4d): σ/√N with the sample standard deviation, in statistics_table.
  • Unit conversions only for listed variables. get_value_converters only knows variables in diag_var_list, so the time-series and history loaders do not convert anything else. The TEM loaders work around it (fix 24); a general fix belongs in a follow-up.
  • Multiple test cases. The land regional, global-mean and table scripts use only the first test case.
  • GenTS back end does not reuse existing time series per variable, as the ADF back end now does.
  • Gridded file metadata. Gridded files lack lat_bnds/lon_bnds and store area in km² where the f09 files use steradians.
  • Release notes for cam_ts_done → calc_cam_ts, the new env, and the new config keys: written (section above), ready for the GitHub release.
  • Update the wiki and LAND-DIAGS_README.md with the release-note changes.
  • Choice of remap onto the mesh. In UXarray mode the lat/lon side is now remapped onto the native mesh (fdb06e7, 4868272; Suite 5), but the method is fixed in the code: ESMF conservative weights (made once per grid pair, cached in regrid_weights/), uxarray bilinear only when ESMF cannot run, and inverse-distance weighting if bilinear fails (GPCP 2.5°). Gridded mode lets each case choose its weights_file and regrid_method; mesh mode has no setting. Conservative keeps area means (fluxes, precipitation, global and regional means) and is the right default; bilinear is smoother (winds, temperature) and matches the bilinear gridded runs. Add a setting such as mesh_regrid_method: conserve | bilinear, one per run or per case, or keep conservative only? PR #510 (bitterbark) suggests a third answer: its per-variable regrid_conservative: true flag in the variable defaults could choose the method here too, conservative for the flagged fluxes and bilinear for the rest, with no new config key. Make native-grid time series and gridding work at scale, using conservative interpolation when appropriate #510 itself only covers gridded mode (native onto lat/lon); see the next item.
  • Review and merge PR Make native-grid time series and gridding work at scale, using conservative interpolation when appropriate #510 (bitterbark, native-grid-v2 into this branch, opened 5 Nov). It makes native time series and gridding work at scale, and every new setting is off by default. Time series are chunked one time step per chunk (--cnk_map=rd1), which shrinks ne30pg3 files that were about 1.6× their data. date and datesec are now copied into time series files. grid_native_time_series grids in time chunks of about 256 MB and writes through a temporary file, so a killed job leaves no partial file. Runs split by variable can share a time series directory, each gridding only its own variables. An optional per-case conservative_weights_file is used for variables marked regrid_conservative: true in both defaults files (precipitation, surface and top-of-model fluxes, cloud forcing, RESTOM, wind stress). An optional gridded_keep_precision keeps 32-bit fields 32-bit after gridding. Merged on 10 Nov as 14fb022 (three conflicts, in the defaults files and grid_native_time_series); GitHub shows Make native-grid time series and gridding work at scale, using conservative interpolation when appropriate #510 as merged. 394 unit tests pass. Three N01 runs: with the new settings off, every time series and gridded file holds the same values as before, and native time series are 0.69× the size; with conservative_weights_file and gridded_keep_precision, the fluxes use the conservative map (TAUY global mean 0.4% → 0.005% off the native mean) and gridded files are half the size. The regrid_conservative flag does not pick the method in the remap onto the mesh or in 92b01f2's coarsening, which stay conservative for every variable. Details in the test log (PR Make native-grid time series and gridding work at scale, using conservative interpolation when appropriate #510 merged in).
  • Land masking in the remap onto the mesh. A land field remapped from f09 onto the mesh averages the f09 cells each mesh cell overlaps, leaving out the missing ocean cells and renormalising the rest; neither grid's land fraction is used. Along coasts a mesh cell can take full weight from a sliver of f09 land, a mesh land cell whose f09 neighbours are all ocean comes out empty, and f09 land can land in cells the model counts as mostly ocean. Inland values and the averages (the on-mesh file carries the test case's area and landfrac, so both sides are weighted over the same cells) are unaffected. Weight the source by its land fraction and mask the result with the mesh's, as the gridded path does with latlon_file? Same coastal issue as the land question on coastlines.
  • Misleading mesh warning. Gone: with the reference remapped onto the mesh, “declare mesh_file” is no longer printed (N09a, N19).
  • Ocean on native land maps. Native (UXarray) lat/lon maps of land fields showed the ocean as 0 in the test and baseline panels and as “no change” in the difference, because prep_ux_for_plot filled missing cells with 0. Fixed (bug 23, 92b1105): missing cells stay blank in all four panels. Redrawn for N04, N09a, L07, L14 and L15: every land map changes (land means unchanged), and so do the atmosphere maps at 850 hPa, where below-ground cells had been drawn as 0 and pulled the means and RMSE far off.
  • ±inf in “% Diff” panels. Seen in a run for the first time (C01, LHFLX against OAFlux): the % Diff panel prints Mean: inf, on both the old and the new code. utils.array_diff(percent=True) without rel_tol no longer masks a near-zero reference, as main did. Pass rel_tol for the atmosphere maps too, or mask non-finite values before the statistics?
  • Reproducibility for reviewers. The test configs and data live on glade, outside the repo. Add the configs (or a short how-to) somewhere reviewers can see them.

Science questions · native grid

  • Global means. UXarray runs weight by native cell area; gridded runs use cos(latitude) on the gridded field. Tables from native time series differ from f09 by at most 0.0035 in the mean. Which number goes in tables, and should a native run always compute its table on the native grid?
  • Polar rows. Should winds in the rows next to the poles be trusted after bilinear remapping (up to 1.9 m/s difference in V between remaps), or masked in polar plots?
  • Native zonal and meridional means. Are 2° bins right for ne30pg3 (about 1°), and should finer meshes get finer bins?
  • Native vs observations. In UXarray mode obs are now remapped onto the mesh (N09a, N19, N22). Is that wanted, or should native-vs-obs always be gridded (N08, N12, N13)? Answer (10 Nov): the finer grid should always go onto the coarser one. Done for lat/lon grids in 92b01f2; UXarray mode keeps the mesh and warns when the obs are coarser than it.
  • Mixed pairs in UXarray mode. The lat/lon case is now remapped onto the mesh automatically, in either direction (Suite 5). Is that right as the default? The comparison is at native resolution, so plot-header numbers differ from the gridded runs'.
  • Meridional band. The branch averages meridional means over 5°S–5°N; main ignores its own latbounds and averages globally. Which is wanted?
  • TAUX/TAUY sign. scale_factor: -1 is never applied because new_unit makes the file look converted, so maps show CAM's sign. Which sign is wanted?
  • Gridded file metadata. Does any downstream user need lat_bnds/lon_bnds or area in steradians in gridded files?

Science questions · land

  • Land masking when gridding. The ADF masks with the source and target land masks; ncremap -P clm weights by the sub-grid mask. Table means differ by at most 0.88% (SOILWATER_10CM). Which should the ADF match?
  • Regions across grids. A cell counts as inside a region when its centre is in the box, so two grids take in different land at the edges. TOTECOSYSC over CONUS is 1.72% higher on the native grid than on f09 for the same case, while global means are identical. Across grids (L14) that offset is about 17 times the real case difference. Grid both cases first, weight cells by overlap, use mask regions, or note it on the plots?
  • Antarctica. ELAI, GPP, ET and QRUNOFF_TO_COUPLER are all missing there, so their panels are skipped. Show them as zero instead?
  • Standard error. The land table used std/N instead of std/√N. Fixed here for both tables (statistics_table, σ/√N with the sample standard deviation).
  • AREA_BURNED units. Its new_unit is km⁻² d⁻¹ (kept as 1/km2/d in the plain-text change). Area burned should presumably be km² d⁻¹. Which is meant?
  • ENERGY_BALANCE land fraction. The derived ENERGY_BALANCE scales by a hard-coded cpl_landfrac = 0.29251054, which only fits one grid. Should it come from the case's own land fraction instead?
  • Carbon isotope formulas. C13/C14_TOTSOMC_pm and C13/C14_TOTLITC_pm used to get the plain sum of their constituents; the merge computes them with the per-mil isotope formula from the NCL land diagnostics. Is that right, and do the formulas match Jain et al. 1996?
  • Coastlines. The land scientists' code drops coastal cells when regridding lat/lon CLM (1,500 of 23,360 on f09); this branch keeps them. Is that the behaviour the LMWG wants, and is a land-aware regrid needed for other grid pairs?

Software questions

  • Regrid step for UXarray runs. Both setups now work: with the regrid step (native files passed through) and without it (native climo read directly). Keep both, or require one? Answer (6 Nov): keep both. Without a regrid step, the run now says once per variable what it cannot do: a lat/lon side (a lat/lon case or observations) is not put onto the mesh and is skipped, with a warning that names regridding_scripts, and a 3-D field is plotted on its native levels with a warning that its pressure-level plots are skipped. Before, these failed with misleading “no available mesh file” and “plot_press_levels value not present” messages. With the regrid step nothing changes: N09a made the same 920 plots. Done in da9b821.
  • Implicit gridded/ folders. The ADF reads gridded/ whenever it exists, so pre-made native time series must come from a run that never gridded them. Make it an explicit setting?
  • Writing next to inputs. The regrid stage writes gridded/ beside pre-made native climo, inside a folder the user supplied. Should the ADF only write under its own output folders?
  • Mesh configuration. Meshes are given per test case and once for the baseline. Is that the shape we want, and what about obs on a mesh?
  • Shared atm/land scripts. Keep is_lnd branches, or split into separate land scripts on shared helpers? CAM time-series plots also get no website category, unlike land.

Full test log: https://claude.ai/artifact/31HPszLwAiSNJqV5orBeYy

bitterbark and others added 8 commits November 3, 2026 16:31
They are variables of their own on the time dimension, so testing
whether they are dimensions of the variable never matched and they were
never copied.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…emp file

grid_native_time_series read each time series whole, which cannot work
for multi-year hourly 3-D fields (TBs).  It now opens each file in dask
chunks of ~256 MB of time steps (adf_utils.ts_time_chunk), so the
gridding and the write go a block of time steps at a time.

The write uses the new write_netcdf_synchronous: dask's synchronous
scheduler (threaded to_netcdf writes of large dask datasets hung
intermittently) and a temporary file renamed into place only on
success, so a job killed mid-write leaves no partial gridded file for a
rerun to take as finished.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Jobs split by variable can write into the same time series directory at
the same time.  Gridding now only touches the files of the variables
the run makes or uses (requested ones, derived variables' constituents
and the pressure field), so one job no longer grids another's files,
possibly mid-write.  The plot directory is made with exist_ok, so two
jobs starting together do not race on it.  The GenTS back end keeps the
old behavior (all files of the case and stream).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A native-grid case can now give a 'conservative_weights_file' next to its
'weights_file'.  Variables marked 'regrid_conservative: true' in the
variable defaults are gridded with it (method conservative); all other
variables, and all variables of a case without one, use 'weights_file'
and 'regrid_method' as before.  So e.g. a bilinear map can be used for
the state fields while fluxes keep their area means.

The main fluxes are marked in both variable defaults files:
precipitation, QFLX, LHFLX, SHFLX, and the top-of-model and surface
radiative fluxes (and SWCF, LWCF, RESTOM), and the wind stress
(TAUX, TAUY).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
xesmf returns 64-bit floats, so gridded copies of 32-bit model output
are twice the size of their data.  The new, optional
'gridded_keep_precision' basic setting (default false, i.e. unchanged
behavior) casts the gridded fields back to their native float type, via
a new keep_dtype keyword of adf_utils.grid_to_latlon.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Zero in middle of diff plots
@richling-ucar

Copy link
Copy Markdown
Collaborator Author

Closes draft PR #427

richling-ucar and others added 21 commits November 6, 2026 11:38
Conflicts:
- env/conda_environment.yaml: keep adf_v1.0.6 and uxarray=2026.10.0,
  take main's alphabetical order and markdown pin.
- test_reference_support_vars.py: keep this branch's
  load_reference_climo_dataset stand-in (main's load_reference_climo_ds
  does not exist here).
- test_adf_file_utils.py: take main's formatting; keep this branch's
  ts_years_covered/missing_year_ranges/years_tag/year_tagged_names tests.
Also point README/AGENTS at adf_v1.0.6.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fields are now compared on the coarser of the two grids, so neither is
interpolated to a resolution it does not have.  Resolution is the mean
cell area (adf_utils.mean_cell_area); grids within 10% count as the same
(adf_utils.coarser_grid), and then the test case goes onto the
reference's grid as before.

- regrid_and_vert_interp: when a test case's lat/lon grid is clearly
  coarser, the reference (baseline file or obs file) is regridded onto
  the case's grid with normalised conservative weights, skipping missing
  cells but keeping a coarse cell only if at least half of it has data
  (COARSEN_NA_THRES), into {ref}_{case}_{var}_reference_on_case_grid_
  YYYY-YYYY.nc; the case keeps its own grid.  A field already on the
  target grid is no longer regridded (f09 to f09 altered the polar rows).
- UXarray mode still remaps onto the mesh, and native gridding still uses
  the case's weights_file, but both warn when that interpolates a coarser
  grid onto a finer one.
- AdfData.get_ref_on_case_grid_file; _reference_files prefers it for a
  test_case.  The lat/lon map, vector map, polar, zonal, meridional and
  Taylor scripts load the reference per case when it exists.
- Tests: test_regrid_to_coarser.py, two new cases in
  test_native_climo_fallback.py.

Checked: 375 unit tests, black, pylint (9.80-10.00); S03 (same grid:
only the polar rows of the test files change), S13 and a new C01 run
(f09 vs HadISST and OAFlux) exit 0 with the same plots as before, the
obs now on f09 and area means within about 1e-4.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

analysis Related to data analysis and statistics bug-fix Fixes a particular bug (or set of bugs) code clean-up Made code simpler and/or easier to read. config files Concerns the ADF config (YAML) files enhancement New feature or request env-config Relating to ADF environment configuration high priority This needs to be done ASAP LDF Specific request for land diagnostics plotting Related to plot generation regridding Related to regridding (vertical or horizontal)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Clean up data set and data array calls from adf_dataset.py

5 participants