Repository navigation
Bring in Land and Native Grid - #502
Draft
richling-ucar wants to merge 536 commits into
Draft
richling-ucar wants to merge 536 commits into
richling-ucar wants to merge 536 commits into
Conversation
They are variables of their own on the time dimension, so testing whether they are dimensions of the variable never matched and they were never copied. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…emp file grid_native_time_series read each time series whole, which cannot work for multi-year hourly 3-D fields (TBs). It now opens each file in dask chunks of ~256 MB of time steps (adf_utils.ts_time_chunk), so the gridding and the write go a block of time steps at a time. The write uses the new write_netcdf_synchronous: dask's synchronous scheduler (threaded to_netcdf writes of large dask datasets hung intermittently) and a temporary file renamed into place only on success, so a job killed mid-write leaves no partial gridded file for a rerun to take as finished. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Jobs split by variable can write into the same time series directory at the same time. Gridding now only touches the files of the variables the run makes or uses (requested ones, derived variables' constituents and the pressure field), so one job no longer grids another's files, possibly mid-write. The plot directory is made with exist_ok, so two jobs starting together do not race on it. The GenTS back end keeps the old behavior (all files of the case and stream). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A native-grid case can now give a 'conservative_weights_file' next to its 'weights_file'. Variables marked 'regrid_conservative: true' in the variable defaults are gridded with it (method conservative); all other variables, and all variables of a case without one, use 'weights_file' and 'regrid_method' as before. So e.g. a bilinear map can be used for the state fields while fluxes keep their area means. The main fluxes are marked in both variable defaults files: precipitation, QFLX, LHFLX, SHFLX, and the top-of-model and surface radiative fluxes (and SWCF, LWCF, RESTOM), and the wind stress (TAUX, TAUY). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
xesmf returns 64-bit floats, so gridded copies of 32-bit model output are twice the size of their data. The new, optional 'gridded_keep_precision' basic setting (default false, i.e. unchanged behavior) casts the gridded fields back to their native float type, via a new keep_dtype keyword of adf_utils.grid_to_latlon. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Zero in middle of diff plots
Collaborator
Author
|
Closes draft PR #427 |
Conflicts: - env/conda_environment.yaml: keep adf_v1.0.6 and uxarray=2026.10.0, take main's alphabetical order and markdown pin. - test_reference_support_vars.py: keep this branch's load_reference_climo_dataset stand-in (main's load_reference_climo_ds does not exist here). - test_adf_file_utils.py: take main's formatting; keep this branch's ts_years_covered/missing_year_ranges/years_tag/year_tagged_names tests. Also point README/AGENTS at adf_v1.0.6. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fields are now compared on the coarser of the two grids, so neither is
interpolated to a resolution it does not have. Resolution is the mean
cell area (adf_utils.mean_cell_area); grids within 10% count as the same
(adf_utils.coarser_grid), and then the test case goes onto the
reference's grid as before.
- regrid_and_vert_interp: when a test case's lat/lon grid is clearly
coarser, the reference (baseline file or obs file) is regridded onto
the case's grid with normalised conservative weights, skipping missing
cells but keeping a coarse cell only if at least half of it has data
(COARSEN_NA_THRES), into {ref}_{case}_{var}_reference_on_case_grid_
YYYY-YYYY.nc; the case keeps its own grid. A field already on the
target grid is no longer regridded (f09 to f09 altered the polar rows).
- UXarray mode still remaps onto the mesh, and native gridding still uses
the case's weights_file, but both warn when that interpolates a coarser
grid onto a finer one.
- AdfData.get_ref_on_case_grid_file; _reference_files prefers it for a
test_case. The lat/lon map, vector map, polar, zonal, meridional and
Taylor scripts load the reference per case when it exists.
- Tests: test_regrid_to_coarser.py, two new cases in
test_native_climo_fallback.py.
Checked: 375 unit tests, black, pylint (9.80-10.00); S03 (same grid:
only the polar rows of the test files change), S13 and a new C01 run
(f09 vs HadISST and OAFlux) exit 0 with the same plots as before, the
obs now on f09 and area means within about 1e-4.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
LDF updates: config and divide by 0
… ldf-native-grid-merge-main
Another config pr
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Native-grid and CLM land diagnostics, merged with main
This PR brings native-grid (ne30pg3) support and the CLM land diagnostics (from
ww-clm-diags) onto currentmain, with the bug fixes found by a 64-run regression suite on Casper. The full results, plots and per-file comparisons are in the LDF Merge Test Log.What this PR does
unstructured_plotting). Native and lat/lon cases, and native cases and observations, can be compared either way: in UXarray mode the lat/lon side is remapped onto the native side's mesh (ESMF conservative weights, uxarray bilinear fallback).clm2inhist_str. Area × landfrac weighting, separate time-series and climatology years, regional climatologies and time series fromregions_lnd.yaml, the LMWG table and website.cam_ts_doneis nowcalc_cam_ts(old key still accepted). Existing time series are reused per variable.meridional_meannow averages over thelatboundsband (default 5°S–5°N), whichmainsets but ignores, averaging over all latitudes. All meridional-mean plots change.lens2_filein the variable defaults, found in the config'slens2_data_loc(asobs_fileandobs_data_loc), instead of a hard-coded directory.…_climo_0011-0013.nc), so a run over other years never reuses a file made for the old ones. Time series files recordts_yrs, climo and regridded filesclimo_yrs. Older names are still read.adf_dataset.py#424).AdfDatahasget_…,load_…_datasetandload_…_dafor time series, climatologies, regridded files, raw history files (by stream) and TEM files, for the test cases and the reference. The reference loaders take only the variable. Scripts that opened files themselves (QBO, regional maps, TEM, tape recorder) now go through them and get the unit conversions.PZM) for their own time steps, so THZM becomes temperature with the right pressure; TEM fields get their unit conversions (WTEMin cm/s,UTENDEPFDin mm/s²).new_unitis plain text in all four variable-defaults files (W/m2, notW m$^{-2}$); the LaTeX moved tompl: colorbar: label, so colorbars look the same.amwg_tableis nowstatistics_table, which also makes the land table;lmwg_tableis gone. Its standard error is now σ/√N with the sample standard deviation.mainit reads the baseline's); regional legends add the climo years. From wwieder (PR additional updates #512), withTOTECOSYSCandTOTSOMCin the land defaults.adf_v1.0.6: uxarray 2026.10.0 (native maps need at least 2025.6.0), plus pylint 4.1.3, astroid 4.3.4 and pytest 9.1.1 as developer tools.Testing
main. Only the meridional band and the new one-side-only time-series plots differ.mainin obs mode (S11–S13 against S00b); only the meridional band and the one-sided global-mean plots differ. Native ts and climo entries match N08 to 1.8e-6. Atmosphere only: no land observations to test against.ww-clm-diagsc6db27e)TEM-2026-PZM)PZM;WTEMandUTENDEPFDnow converted; THZM as temperature from the TEM file's own pressure.REGMAP,CLIMODS)regional_map_multicaseon the loaders (PSL now in hPa); regridding plus 80 lat/lon and zonal plots onload_climo_dataset.8359365caseand plain-text units: every regridded value identical to the suite's runs, the same plots made; only theunitsstrings change (mm d$^{-1}$→mm/d).statistics_tableunder all three namesamwg_tableandlmwg_tableprint a note and run it; standard error checked by hand.92b01f2)conservative_weights_fileandgridded_keep_precision: same 1,014 plots and warnings each time. With the settings off every time series and gridded file holds the same values, and native time series are 0.69× the size; with them on, fluxes use the conservative map (TAUY global mean 0.4% → 0.005% off the native mean) and gridded files are half the size.lib/test/unit_tests)14fb022(380 onda9b821, 375 on92b01f2, 363 on72f4f7e, after main's #508 removed some tests). GitHub CI runs them all in theadf_v1.0.6environment and passes, as does its canary on the latest Python and packages.lib/file)14fb022and passed; locallyadf_dataset.py9.97,adf_diag.py9.81,adf_info.py9.90,adf_utils.py9.97, the new test files 9.74–10.00.lib/Note
The latest clean rerun (all 38 runs, on
1a9fbb7) started from wiped output folders, in an env built fromenv/conda_environment.yaml. S00 ran onmainin its own env; S00 and S01 agree in every value, so the env change moves no result. The full suite has not been rerun since92b1105, so it predates wwieder's PRs #504, #506 and #509, the data-access work, the merge ofmain2fcf147,72f4f7eand theadf_v1.0.6environment.Bugs fixed
All 30 fixes (14 also affect
main); click to opengrid_to_latlon, threesave_to_nc,use_time_bounds_midpointto_raster.zonal_mean.plot_press_levels.lndgridclimo._region_nativeAdfData._native_climo_fallbackget_regrid_fileraised for variables with no reference.main'screate_clean_gridfails on bounds with a time dimension.nanmean.lsmlat/lsmlon).cell_weightsdropped the uxarray grid withisel(time=0).area/landfracareakept the model's time labels and aligned to NaN against a one-step obs file.area,landfracwritten time-independentreturnskipped the remaining pairs; crashes on the mesh without pressure levels orplot_press_levels.global_latlon_map; 30 plots byte-identicalKeyError: 'lat'(lat/lon test vs native reference).prep_ux_for_plotno longer fills NaN with 0diag_var_list): modelWTEMandUTENDEPFDplotted in raw units, while observedUTENDEPFDwas scaled.KeyError: 'label'after the plotting merge frommain.PZMwritten into the TEM fileregional_map_multicaseskipped the unit conversion (PSL in Pa) and the time-bounds correction.cam_climo_loccrashed at start-up.get_reference_climo_file92b01f2)Note
Two more were in the test setup (a config that read its own output, stale files between runs). Details for each are in the test log.
Release notes
cam_ts_doneis nowcalc_cam_ts, with the opposite meaning, matchingcalc_cam_climo:calc_cam_ts: truemakes time series from the history files, ascam_ts_done: falsedid. Replacecam_ts_done: Xwithcalc_cam_ts: not X. The old key still works and prints a deprecation note; giving both with contradicting values stops the run.calc_cam_ts: true, files already incam_ts_locare kept and only the years they do not cover are made, each stretch in a file of its own.calc_cam_ts: falseandcalc_cam_climo: falsea case needs no time series; scripts that need them leave that case out.config_cam_baseline_example.yaml. Indiag_basic_info:unstructured_plotting,regions_file,lens2_data_loc. Per case:climo_start_year/climo_end_year(withts_start_year/ts_end_yearas other names forstart_year/end_year), and for native-grid casesmesh_file,weights_file,regrid_methodand, for CLM,latlon_file. At the top level:region_listandregional_var_list.lens2_data_loc. A config that runsglobal_mean_timeseriesand wants the LENS2 range must setlens2_data_loc(on Derecho/Casper:/glade/campaign/cgd/cas/islas/CESM_DATA/LENS2/global_means/annualmeans). Newlens2_fileentries cover ACTREL, AODVIS, CDNUMC, CLDHGH, CLDLOW, CLDMED, CLDTOT, FSDS, FSDSC, FSNT, FSNTC, SWCF, TGCLDIWP, TGCLDLWP and TS. The aerosol burdens (BURDEN*dn) have LENS2 files but no variable-defaults entry, so they no longer get the range.climo_start_year/climo_end_yearnarrower than the time series, the table still covers all the time-series years; the climatology years no longer steer its file choice or its one-year check.{stem}_YYYY-YYYY.nc). Existing files with the older names are still read, but are never overwritten or reused by a run over other years: that run makes new year-tagged files. Time series files now recordts_yrs.mainusers: meridional means (5°S–5°N band), and single-case time-series, QBO, tape-recorder, zonal and meridional plots for runs where one case has no data, which used to be skipped.load_timeseries_datasetnow takes(case, field)(the old file-list form isload_timeseries_files);load_climo_dsandload_reference_climo_dsare nowload_climo_datasetandload_reference_climo_dataset;load_climo_fileis gone; the reference loaders take only the variable (load_reference_regrid_da(var)), and an old two-argument call stops with aTypeError.amwg_tableandlmwg_tableare nowstatistics_table. Configs listing an old name still work and print a note. The output files keep their names (amwg_table_<case>.csv,lmwg_table_<case>.csv). The standard deviation, standard error and 95% CI columns change (σ/√N, sample standard deviation).new_unitis plain text. For LaTeX on a plot setmpl: colorbar: label. Time-series axis labels now show the plain units.overwrite_tem: true, once) to carry the model pressure; older files still plot, converting THZM with the hybrid coordinate and a warning.WTEMandUTENDEPFDplots change (now converted), as do regional maps (unit conversions).adf_v1.0.6: rebuild withconda env create -f env/conda_environment.yaml. Native-grid maps need uxarray 2025.6.0 or later (the env has 2026.10.0).{ref}_{case}_{var}_reference_on_case_grid_YYYY-YYYY.nc, instead of the case being interpolated up to theirs. For an f09 case this applies to every 1° and 0.25° observation file, so their lat/lon, polar, zonal, meridional and Taylor plots, and the numbers in the plot headers, change slightly. Cases on the same grid are no longer regridded.TOTECOSYSC_1mis nowTOTECOSYSC, andTOTSOMCis new (both summed to PgC in the tables). A config listingTOTECOSYSC_1mneeds the new name.Before this PR is ready
Commit this round's work. The fixes above, the new unit tests and the env change are in five commits,8e60e34toc1e6bd2, pushed to the branch. The two root-level CLM configs below were left out.Commit fix 15. The single-case plot fix inlib/plotting_functions.py: committed and pushed asc711eb7.Commit the year-tagged file names and fixes 16 and 17, with their unit tests: pushed asf45d03d,b2a660dande8a7c65.Rerun the full suite (now 38 runs with L00) on the year-tagged names: done one8a7c65, all 38 pass.Skip regridding when the grids are identical. f09 to f09 conservative regridding changes the rows next to the poles (288 cells each, up to 1.95 m/s in U). Done in92b01f2, with the finer-onto-coarser rule: a field already on the target grid is passed through (S03).Fix two unit strings inldf_variable_defaults.yaml: ELAIm${-2}$ m${-2}$and QBOTkg kg{-1}$come out garbled in tables: fixed in1a9fbb7.Decide on the root-level CLM configs. Kept, and updated to the new config keys (start_year/end_year,calc_cam_ts) ineb0ccfb.Document the new config keys inconfig_cam_baseline_example.yaml. Added, commented out:unstructured_plotting,regions_file, the per-caseclimo_start_year/climo_end_year,mesh_file,weights_file,regrid_methodandlatlon_file, andregion_list/regional_var_list;lens2_data_locis set.adf_v1.0.6, from PR Minor corrections #504) changesmain's environment for every user.CI does not run the science tests. CI installed only PyYAML and pytest, so 141 tests skipped, including every native-grid and land test. Settled by main's PR Add full python unit testing suite to Github Actions workflow #508, merged in as2c08c5b: CI now runs all 363 tests in the full ADF environment and fails on a skip.scripts/is still not linted.New plots and website entries onmain. Runs that used to skip time-series, zonal and meridional plots now make single-case versions.Size. 69 files and about 22,750 added lines. Consider separate PRs for the fixes that also affectmain(1, 5, 6), native grid, land, and the plot-whatever-there-is rule.Open the small PR tomainfor fix 14 (create_clean_gridwith time-dimensioned bounds).mainagain and rerun Suite 1.main2fcf147 (PR Add full python unit testing suite to Github Actions workflow #508) is merged in as2c08c5b, and CI runs and passes every unit test. Suite 1 has not been rerun yet.adf_v1.0.6. The 64 runs predate PRs Minor corrections #504, Improve lat-lon plots #506 and Zero in middle of diff plots #509, the data-access work, the merge ofmain2fcf147 and72f4f7e, and ran on adf_v1.0.5; the data-access work was checked with seven targeted runs.Commit thestatistics_tablerename and the standard-error fix (fix 28), with the twelve configs and the README that now name it: committed in0946a4d.Hard-coded LENS2 path. Replaced by a per-variablelens2_filein the variable defaults, found in the config'slens2_data_loc. Land variables get none (the old search matched atmosphere files for them).Standard error in the tables. Divided by N instead of √N, so the standard error and 95% columns were too small. Fixed here (fix 28,0946a4d): σ/√N with the sample standard deviation, instatistics_table.get_value_convertersonly knows variables indiag_var_list, so the time-series and history loaders do not convert anything else. The TEM loaders work around it (fix 24); a general fix belongs in a follow-up.Multiple test cases. The land regional, global-mean and table scripts use only the first test case.GenTS back end does not reuse existing time series per variable, as the ADF back end now does.lat_bnds/lon_bndsand store area in km² where the f09 files use steradians.Release notes forcam_ts_done→calc_cam_ts, the new env, and the new config keys: written (section above), ready for the GitHub release.LAND-DIAGS_README.mdwith the release-note changes.Choice of remap onto the mesh. In UXarray mode the lat/lon side is now remapped onto the native mesh (fdb06e7,4868272; Suite 5), but the method is fixed in the code: ESMF conservative weights (made once per grid pair, cached inregrid_weights/), uxarray bilinear only when ESMF cannot run, and inverse-distance weighting if bilinear fails (GPCP 2.5°). Gridded mode lets each case choose itsweights_fileandregrid_method; mesh mode has no setting. Conservative keeps area means (fluxes, precipitation, global and regional means) and is the right default; bilinear is smoother (winds, temperature) and matches the bilinear gridded runs. Add a setting such asmesh_regrid_method: conserve | bilinear, one per run or per case, or keep conservative only? PR #510 (bitterbark) suggests a third answer: its per-variableregrid_conservative: trueflag in the variable defaults could choose the method here too, conservative for the flagged fluxes and bilinear for the rest, with no new config key. Make native-grid time series and gridding work at scale, using conservative interpolation when appropriate #510 itself only covers gridded mode (native onto lat/lon); see the next item.Review and merge PR Make native-grid time series and gridding work at scale, using conservative interpolation when appropriate #510 (bitterbark,native-grid-v2into this branch, opened 5 Nov). It makes native time series and gridding work at scale, and every new setting is off by default. Time series are chunked one time step per chunk (--cnk_map=rd1), which shrinks ne30pg3 files that were about 1.6× their data.dateanddatesecare now copied into time series files.grid_native_time_seriesgrids in time chunks of about 256 MB and writes through a temporary file, so a killed job leaves no partial file. Runs split by variable can share a time series directory, each gridding only its own variables. An optional per-caseconservative_weights_fileis used for variables markedregrid_conservative: truein both defaults files (precipitation, surface and top-of-model fluxes, cloud forcing, RESTOM, wind stress). An optionalgridded_keep_precisionkeeps 32-bit fields 32-bit after gridding. Merged on 10 Nov as14fb022(three conflicts, in the defaults files andgrid_native_time_series); GitHub shows Make native-grid time series and gridding work at scale, using conservative interpolation when appropriate #510 as merged. 394 unit tests pass. Three N01 runs: with the new settings off, every time series and gridded file holds the same values as before, and native time series are 0.69× the size; withconservative_weights_fileandgridded_keep_precision, the fluxes use the conservative map (TAUY global mean 0.4% → 0.005% off the native mean) and gridded files are half the size. Theregrid_conservativeflag does not pick the method in the remap onto the mesh or in92b01f2's coarsening, which stay conservative for every variable. Details in the test log (PR Make native-grid time series and gridding work at scale, using conservative interpolation when appropriate #510 merged in).areaandlandfrac, so both sides are weighted over the same cells) are unaffected. Weight the source by its land fraction and mask the result with the mesh's, as the gridded path does withlatlon_file? Same coastal issue as the land question on coastlines.Misleading mesh warning. Gone: with the reference remapped onto the mesh, “declaremesh_file” is no longer printed (N09a, N19).Ocean on native land maps. Native (UXarray) lat/lon maps of land fields showed the ocean as 0 in the test and baseline panels and as “no change” in the difference, becauseprep_ux_for_plotfilled missing cells with 0. Fixed (bug 23,92b1105): missing cells stay blank in all four panels. Redrawn for N04, N09a, L07, L14 and L15: every land map changes (land means unchanged), and so do the atmosphere maps at 850 hPa, where below-ground cells had been drawn as 0 and pulled the means and RMSE far off.Mean: inf, on both the old and the new code.utils.array_diff(percent=True)withoutrel_tolno longer masks a near-zero reference, asmaindid. Passrel_tolfor the atmosphere maps too, or mask non-finite values before the statistics?Science questions · native grid
Native vs observations. In UXarray mode obs are now remapped onto the mesh (N09a, N19, N22). Is that wanted, or should native-vs-obs always be gridded (N08, N12, N13)? Answer (10 Nov): the finer grid should always go onto the coarser one. Done for lat/lon grids in92b01f2; UXarray mode keeps the mesh and warns when the obs are coarser than it.Meridional band. The branch averages meridional means over 5°S–5°N;mainignores its ownlatboundsand averages globally. Which is wanted?scale_factor: -1is never applied becausenew_unitmakes the file look converted, so maps show CAM's sign. Which sign is wanted?lat_bnds/lon_bndsor area in steradians in gridded files?Science questions · land
ncremap -P clmweights by the sub-grid mask. Table means differ by at most 0.88% (SOILWATER_10CM). Which should the ADF match?Regions across grids. A cell counts as inside a region when its centre is in the box, so two grids take in different land at the edges. TOTECOSYSC over CONUS is 1.72% higher on the native grid than on f09 for the same case, while global means are identical. Across grids (L14) that offset is about 17 times the real case difference. Grid both cases first, weight cells by overlap, use mask regions, or note it on the plots?Standard error. The land table used std/N instead of std/√N. Fixed here for both tables (statistics_table, σ/√N with the sample standard deviation).new_unitis km⁻² d⁻¹ (kept as1/km2/din the plain-text change). Area burned should presumably be km² d⁻¹. Which is meant?ENERGY_BALANCE land fraction. The derivedENERGY_BALANCEscales by a hard-codedcpl_landfrac = 0.29251054, which only fits one grid. Should it come from the case's own land fraction instead?C13/C14_TOTSOMC_pmandC13/C14_TOTLITC_pmused to get the plain sum of their constituents; the merge computes them with the per-mil isotope formula from the NCL land diagnostics. Is that right, and do the formulas match Jain et al. 1996?Software questions
Regrid step for UXarray runs. Both setups now work: with the regrid step (native files passed through) and without it (native climo read directly). Keep both, or require one? Answer (6 Nov): keep both. Without a regrid step, the run now says once per variable what it cannot do: a lat/lon side (a lat/lon case or observations) is not put onto the mesh and is skipped, with a warning that namesregridding_scripts, and a 3-D field is plotted on its native levels with a warning that its pressure-level plots are skipped. Before, these failed with misleading “no available mesh file” and “plot_press_levelsvalue not present” messages. With the regrid step nothing changes: N09a made the same 920 plots. Done inda9b821.gridded/folders. The ADF readsgridded/whenever it exists, so pre-made native time series must come from a run that never gridded them. Make it an explicit setting?gridded/beside pre-made native climo, inside a folder the user supplied. Should the ADF only write under its own output folders?is_lndbranches, or split into separate land scripts on shared helpers? CAM time-series plots also get no website category, unlike land.Full test log: https://claude.ai/artifact/31HPszLwAiSNJqV5orBeYy