Skip to content

CESM-compliant time variables and time-average output; output options for an external driver - #92

Merged
Katetc merged 17 commits into
mainfrom
lipscomb/cesm_timevars
Oct 9, 2026
Merged

Katetc merged 17 commits into
mainfrom
lipscomb/cesm_timevars

Conversation

@whlipscomb

@whlipscomb whlipscomb commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

This PR updates CISM's netCDF output, mainly in libglimmer (glimmer_ncio.F90, glimmer_ncparams.F90,
glimmer_ncdf.F90, ncdf_template.F90.in) and utils/build/generate_ncvars.py:

  • Time variables and time-average ('_tavg') output files follow CF and CESM conventions.
  • Several long-standing bugs in time-average output are fixed.
  • Time averages now continue exactly across standard restarts.
  • New output options let an external driver, such as the CESM wrapper, decide when and where CISM
    writes output.

It is the CISM side of ESCOMP/CISM-wrapper issue #75 (and CISM-wrapper #76). The companion
CISM-wrapper PR, ESCOMP/CISM-wrapper#150, uses these changes to write CESM history files
(h0i and h0a) through CISM's [CF output] machinery.

Changes that may require config file changes (standalone)

These checks are new and abort at initialization with a clear message:

  • No mixing in one file: a [CF output] section may not mix instantaneous fields and
    time-average (_tavg) fields. Put the _tavg fields in a separate file.
  • No write_init for averages: time-average files must have write_init = .false., since there
    is nothing to average at initialization.
  • No duplicate _tavg fields: a _tavg field may be listed in only one output file. Each
    average is accumulated in a single model array, so listing it twice gave wrong averages.
  • Missing restart file: with restart = 1, the model aborts if the file named in
    [CF restart] does not exist. Previously it fell back silently to the [CF input] file.
  • _tavg_sum variables: variables ending in _tavg_sum may not be listed in [CF output].
    They are added to restart files automatically (see below).

Time variables and time-average files

  • internal_time and time are double precision, with long names internal time and time
    (no standard_name).
  • Time units: the units are common_years since 0000-01-01 00:00:00. This is the UDUNITS
    spelling (previously common years, which cftime/xarray could not decode), with base year 0
    (previously 0001), so that t = 1950.0 is Jan 1, 1950. Files now open with xarray.open_dataset,
    with times decoded as noleap dates.
  • Time bounds: time-average files have internal_time_bounds and time_bounds, with
    dimensions (time, nbnd), and a bounds attribute on the time variables. Instantaneous and
    restart files have no time bounds.
  • Midpoint times: in time-average files, internal_time and time are the midpoints of the
    averaging interval (CF and CESM practice). The bounds give the start and end.
  • External time: an external driver can pass its own time and units for time (e.g., days since
    0001-01-01 in CESM); internal_time always uses CISM's units.

Bug fixes in time-average output

  1. Accumulation once per timestep: averages were also accumulated at initialization and in
    glide_finalise. This biased the first average of a run, and double-counted the last step of a
    partial interval written with forcewrite_final.
  2. Final average overwritten: the last time-average slice of a run that ended on an output time
    was overwritten by the final-timestep value in glide_finalise.
  3. Time bounds after a restart: the first interval's bounds were wrong after a restart, or with
    tstart /= 0 (e.g., [0,6] instead of [5,6]).
  4. Restart crash: standard restarts with a time-average output file aborted ("NetCDF: Variable
    not found") when the file was reopened.
  5. Skipped output at stop: output at the stop time was sometimes skipped, because of roundoff
    in the model time.
  6. Implicit SAVE: forcewrite had an implicit SAVE attribute in NAME_io_writeall and
    glide_finalise.
  7. Variable list lost: the variable list (vars_copy) was lost when more than one I/O module
    (e.g., glide_io and glad_io) wrote the same file.
  8. bounds attribute: instantaneous files had a bounds attribute pointing to time-bounds
    variables that did not exist.
  9. gnu build: ncdf_template.F90.in had a continued string that the gnu preprocessor could
    not handle.

Exact restart of time averages

Restart files now include the averaging state of each time-average output stream, so averages
continue exactly across a standard restart, including in the middle of an averaging interval:

  • Running sums: for each <field>_tavg in an output file, a <field>_tavg_sum variable is added
    to the restart file automatically. It's the running sum of the field times the timestep, with
    units of the field times years. These variables are generated by generate_ncvars.py.
  • Averaging state: the total time, the start of the interval (internal and external), and the
    timestep of the last accumulation. These are written in variables with dimensions
    (tavgstream, time), and streams are matched by name (global attribute tavg_streams).
  • Restored only on a standard restart (restart = 1):
    • If a current stream has no saved state, the model aborts, suggesting a hybrid restart.
    • Restart files from older versions, which have no averaging state, get a warning and start a
      new interval.
  • Startup and hybrid restart (restart = 2): a new averaging interval starts, with zero sums.
  • Write order: restart files are written after all other output at the same time, so they hold
    the state after any average written at that time. If the final call from glide_finalise writes
    an average after the restart file was written, the restart time slice is rewritten.
  • tavg_restored: a new flag that records whether the state was restored, for use by an
    external driver.

New output options for an external driver ([CF output])

  • one_file_per_write: each write creates a new file with one time slice, while a single output
    object keeps the averaging state from file to file.
    • In standalone runs, the time is inserted in the file name (e.g., out.tavg.0001.nc).
    • Not allowed in [CF restart].
  • external_control: CISM accumulates averages but never writes the file itself. The driver
    decides when to write, using a documented calling sequence (see NAME_io_writeall).
  • Refactoring: glimmer_nc_checkwrite is split into glimmer_nc_advance_timeslice,
    glimmer_nc_write_due and glimmer_nc_write_timeslice, with new routines
    glimmer_nc_checkwrite_init, glimmer_nc_newfile, glimmer_nc_closefile and
    NAME_io_write_slice.

Other changes

  • The maximum length of a config file value is raised from 400 to 4096 characters, to allow long
    output variable lists.
  • GLAD no longer reads the unused esm_history_vars from [esm_output]. Older config files with
    this key still work.

Expected answer changes (standalone)

  • Instantaneous output fields and restart-file fields: bit-for-bit, except that files with
    stop may gain the slice at the stop time (fix 5).
  • Time-average fields: change in the first interval of a startup or hybrid run, in the last
    interval of a run ending on an output time, and in partial intervals written with
    forcewrite_final (fixes 1 and 2).
  • Time values: in time-average files, the time values are midpoints, and the bounds are
    corrected after restarts.
  • Metadata: time units, precision, long names, the nbnd dimension, and bounds attributes.
  • Restart files: gain _tavg_sum and averaging-state variables, if there are time-average
    output files.

Testing

Standalone (derecho, intel), after each step:

  • 8-km Antarctic test (5-year startup plus 5-year standard restart, with instantaneous,
    time-average, scalar and restart files): compared each step with nccmp. Answer changes are only
    those listed above.
  • Restart in the middle of averaging intervals (2-D and scalar time-average files whose
    intervals span the restart): averages are bit-for-bit with a continuous run.
  • Hybrid restarts (restart = 2, tstart = 100) from restart files with and without nonzero
    partial sums are bit-for-bit with each other, with new intervals starting at t = 100.
  • 4-km Greenland test (marine_margin = 1, calving_init = 1): restarts from t = 0 and
    t = 1 are bit-for-bit with a startup run.
  • Builds: intel and gnu. Generated *_io.F90 files were checked for unexpanded NCO/NCI
    macros with the gfortran preprocessor.

CESM: tested through the CISM-wrapper branch. The full aux_glc suite passes, and all exact-restart
comparisons pass. See ESCOMP/CISM-wrapper#150 for details.

Glint: the generated Glint I/O modules compile. Glint was not run (its deprecation is planned).

Known issues, not changed by this PR

After a standard restart, bmlt_float_target is zero, because it is not in restart files. With
inversion, coulomb_c and powerlaw_c differ at vertices with f_ground = 0; they would affect
answers if those cells reground. To be investigated separately.

🤖 Generated with Claude Code

whlipscomb and others added 16 commits September 25, 2026 16:26
In typical CISM runs, most output fields are snapshots of the ice state at a given time.
However, some output fields (including most fluxes) are time-averaged.
All time-averaged fields have the suffix '_tavg'.

Until now, instantanteous and tavg fields could be mixed in the same output file.
To be consistent with CESM practice, it is better to separate them.
With this commit, the model aborts if the user mixes the two in any output variable list.

The model also aborts if write_init = T for a time-average file, since it doesn't
make sense to write time averages at t = 0.

I considered having the model abort if the user lists more than one time-average file
in the config file, but decided this was too strong a requirement.
This means there is a risk that the same tavg field will appear in two different
output files, which will lead to erroneous averaging.
It would take some additional logic to examine multiple tavg variable lists
and find repeat variables.

One other change: The model aborts if the user sets restart = 1 (standard restart)
and there is no file present with the designated restart filename.
Previously, the default was to restart with the [CF input] file instead.
This could lead to non-transparent fatal errors.

These changes have been tested for standalone CISM runs but not in CESM.
Most of the changes are in subroutine glimmer_nc_readparams, and they apply only
to output files in sections called [CF output] or [CF restart].
Although glimmer_nc_readparams is called in CESM runs, the CISM config files
do not have sections called [CF output] or [CF restart].
This commit changes the output time variables in several ways to meet CESM standards:

* The time variables ('internal time' and 'time') are now double-precision.

* These variables now have the long_name attributes 'internal time' and 'time'.
  I removed the standard_name attributes.

* Time-average files (i.e., files containing one of more variables with the '_tavg' suffix)
  now have the additional output variables internal_time_bounds and time_bounds,
  also double precision. These the initial and final time of each averaging interval.

To support clean handling of time-average files, I introduced the requirement that
instantaneous and tavg variables cannot be mixed within a single output file.
At initialization, subroutine glimmer_nc_readparams parses the variable list
(contained in NCO%vars) to make sure that if any variables contain '_tavg',
then all variables contain '_tavg', else the code aborts. Also, the user must set
write_init = .false. for tavg files.

I added some logic using the processed_time property to determine time_bounds
for each tavg file over each averaging period.
Note: It is allowed (though likely not necessary) to have more than one tavg file.
However, a given variable can appear in only one such file, else it is handled incorrectly.
I haven't yet found a simple way to enforce this requirement.

I changed the default internal time units from 'common years since 0001-01-01'
to 'common years since 0000-01-01'.
The old baseline, year 1, implies that t = 0 corresponds to Jan. 1 of year 1,
hence t = 1950.0 is Jan. 1 of year 1951.
With a baseline year of 0, t = 1950.0 corresponds to Jan. 1, 1950, which is more intuitive
and is consistent with most forcing files.

I discovered that as glide_io_create.F90 parses NCO%vars, it removes the variable
substring for every variable it creates. Thus, if all goes well, 'vars' should
contain only empty spaces at the end. I added a warning alerting the reader if
this is not the case. In this way I found and fixed a couple of mistakes in glide_vars.def.

In several places I changed 'processsed' to 'processed'.

I deprecated the use of 'hot' to denote restart files. This was an old Glimmer convention.
…output

This commit continues the work on output time variables and time-average ('_tavg') files.
It refactors subroutine glimmer_nc_checkwrite, in preparation for letting an external
driver (e.g., the CESM wrapper) decide when output is written. It also fixes several bugs,
some new with the recent time-bounds commits and some long-standing.
Most changes are in libglimmer/glimmer_ncio.F90 and libglimmer/ncdf_template.F90.in.

Refactoring (answer-preserving):

* Subroutine glimmer_nc_checkwrite has been split into three steps, each in its own routine:
  (1) glimmer_nc_advance_timeslice: leave define mode; advance the time counter after a write
  (2) glimmer_nc_write_due: a logical function that decides whether a write is due
      (forcewrite, write_init at tstep_count = 0, or the output frequency; not past end_write;
      not already written at this time)
  (3) glimmer_nc_write_timeslice: write internal_time, time, tstep_count and (for tavg files)
      the time bounds, and reset the start of the averaging interval
  glimmer_nc_checkwrite keeps its argument list and calls these three routines,
  so existing callers are unchanged. This change alone is BFB for standalone runs.
  An external driver that controls the timing of output can call step (3) directly.

* glimmer_nc_checkwrite has a new optional output argument, wrote_timeslice,
  which is true if a time slice was written during the call (see bug fix 4 below).

* New subroutine glimmer_nc_checkwrite_init(outfile, time, external_time, tstep_count)
  sets the start of the first averaging interval (processed_time, processed_external_time)
  and the initial value of the new accum_tstep_count (see bug fix 3).
  It is called from openall_out for every output file, with time = tstart.
  An external driver that passes an external time to glimmer_nc_checkwrite should also
  call glimmer_nc_checkwrite_init with the external start time.

Bug fixes:

1. The 'bounds' attribute on internal_time and time is now set only for tavg files.
   Previously it was set for all files, so instantaneous files (including restart files)
   pointed to time-bounds variables that did not exist, which violates CF conventions.

2. Standard restarts with a tavg output file crashed on the first tavg write
   ("NetCDF: Variable not found"). When an existing output file is reopened for appending,
   glimmer_nc_openappend did not look up the varids of internal_time_bounds and time_bounds.
   It now does so for tavg files.

3. Averages were accumulated whenever NAME_io_writeall was called, including calls
   at initialization (tstep_count = 0) and in glide_finalise, which do not correspond
   to a new timestep. As a result:
   * The first averaging interval of a run (startup or hybrid restart) included the initial
     state as an extra sample. For fields that are zero at initialization (e.g., acab_applied,
     calving_rate), this biased the first average low by a factor n/(n+1),
     where n is the number of timesteps in the interval.
   * If glide_finalise forced a write of a partial interval (forcewrite_final = T),
     the final timestep was counted twice.
   Now avg_accumulate does nothing unless tstep_count has increased since the last accumulation.
   The new variable accum_tstep_count (in type glimmer_nc_output) holds the tstep_count
   of the most recent accumulation. This rule also protects the upcoming CESM design,
   in which the wrapper calls writeall every step only to accumulate averages.

4. Long-standing bug: The last tavg slice of each run was overwritten with the
   final-timestep value when the run ended on an output time.
   NAME_io_writeall wrote the model fields whenever NCO%just_processed = T.
   But just_processed remains true for later calls at the same time, e.g. in glide_finalise.
   So the finalise call rewrote the fields of every file written at the final time.
   For tavg files, the average had already been written and reset; the rewrite used only the
   one extra accumulated step (bug 3), replacing the average with the final-timestep value.
   Now NAME_io_writeall writes the fields only if wrote_timeslice = T.
   This also avoids needless rewriting at finalise of all files written at the final time,
   including (potentially large) restart files.

5. Time bounds were wrong for the first averaging interval after a restart, or in any run
   with tstart /= 0. processed_time was initialized to 0, so the first interval after a
   restart at t = 5 was written as [0,6] instead of [5,6].
   This is fixed by calling glimmer_nc_checkwrite_init from openall_out.

6. Output was sometimes skipped at the time specified by 'stop' (end_write)
   because of roundoff in the accumulated model time.
   For example, with dt = 1/6 yr, the model time at year 8 is 8.00000000000001 > 8.
   glimmer_nc_write_due now allows a tolerance of 1.e-3*tinc, the same convention
   used for forcing times in NAME_read_forcing.

7. Implicit SAVE: 'logical :: forcewrite = .false.' in NAME_io_writeall (ncdf_template.F90.in)
   and in glide_finalise and glide_finalise_all (glide_stop.F90) has an implicit SAVE attribute,
   so once forcewrite was set to true, later calls would keep forcing writes.
   forcewrite is now initialized in a separate executable statement.
   This did not change answers in any tests, since these routines are typically called
   only once with forcewrite = T, at the end of the run.

I also added a log message giving the averaging interval and total_time for each tavg write,
and added comments in glimmer_ncparams.F90 noting that the no-mixing rule for
instantaneous and tavg variables does not apply to [CF restart] files.

Testing (standalone):

I tested with an 8-km Antarctic spin-up configuration (DIVA, 256 tasks, dt = 1/6 yr):
a 5-year startup run followed by a 5-year standard restart. Output files were:
* an instantaneous 2D file with frequency = 1 and stop = 3 (stop = 8 on restart)
* a 2D tavg file (acab_applied_tavg, bmlt_applied_tavg, calving_rate_tavg) with frequency = 1
* a scalar tavg file (total_*_flux_tavg) with frequency = 2, so the forced final write
  at t = 5 is a partial interval
* an instantaneous scalar file with frequency = 1
* a restart file with frequency = 5
with forcewrite_final = T. I compared each step with nccmp:
* The checkwrite refactoring (with fixes 1 and 2 applied to both the old and new code)
  was BFB in all files.
* Fixes 3 through 6 changed only the tavg slices and the stop slice identified above.
  Instantaneous and restart files were BFB, apart from the stop fix, which added
  the missing year-8 slice.
* After the changes, total_time equals the interval length for every tavg slice,
  and the time bounds are continuous across the restart ([4,5], [5,6], ...).

Expected answer changes:

Standalone CISM:
* Instantaneous output fields and restart files: BFB, except files with 'stop' set,
  which may gain a slice at the stop time (fix 6).
* Tavg output fields: answer-changing in the first interval of a startup or hybrid run,
  in the last interval of a run ending on an output time, and in partial intervals
  written with forcewrite_final (fixes 3 and 4).
* Tavg time bounds: the first interval after a restart, or of any run with tstart /= 0
  (fix 5).
* Standard restarts with a tavg output file used to abort; they now run (fix 2).
* Metadata: instantaneous files no longer have 'bounds' attributes on internal_time and time
  (fix 1).

CESM (CISM-wrapper tests):
The wrapper creates its own history and restart output objects, calls glimmer_nc_createfile
and glimmer_nc_checkwrite (with forcewrite = .true.) directly, and calls glide_io_write itself.
In CESM configurations there are no [CF output] sections, so the only object in CISM's own
output list is the wrapper's oc_tavg_helper, which is never written.
For the changes in this commit, I expect:
* Instantaneous history fields (the default esm_history_vars): BFB.
* Restart files: BFB field values.
* Metadata: 'bounds' attributes are removed from internal_time and time in instantaneous history
  files and in restart files. This may show up as metadata or header differences in baseline
  comparisons, but not as field differences.
* Tavg history fields (only if the user adds _tavg fields to esm_history_vars):
  expected BFB, but please check. The accumulate-once-per-step rule should not matter in CESM,
  because oc_tavg_helper is created after the initialization call to glide_io_writeall,
  and after a history write nothing reads the extra accumulation in glide_finalise.
  tstep_count is incremented every GLAD step, including when evolve_ice = 0.
* Time bounds in wrapper tavg history files: unchanged, and still incorrect,
  because the wrapper builds a new output object for each file
  (so processed_time = 0 for every file). The wrapper will be fixed in a later commit.
* Exact-restart tests (ERS, ERI): should be unaffected by this commit.

Note for wrapper testing: Commit 8d56fb4 renamed an argument of glimmer_nc_createfile
from baseline_year to external_baseline_year. The wrapper must be updated to use the new name
(e.g., in source_glc/glc_io.F90, glc_io_write_history_tavg_helper and glc_io_write_restart
still pass baseline_year) before it will compile against this branch.

Upcoming work (not in this commit): 'external_control' and 'one file per write' flags
for output objects, and a template routine to write a complete single-slice file,
so the wrapper can use CISM's [CF output] machinery directly for h0i/h0a history files.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…rol flag

This commit adds the option to write each output time slice to a separate file,
while keeping a single persistent output object for each [CF output] section.
It also adds a flag, external_control, which will let an external driver
(e.g., the CESM wrapper) decide when output is written. The external_control flag
is read from the config file but does not yet change model behavior; that will
come in a follow-up commit.

These changes prepare for having CISM-wrapper write CESM history files (h0i and h0a)
through CISM's [CF output] machinery. In CESM, each history file contains one time slice,
whereas standalone CISM output files usually contain many. With one_file_per_write,
the averaging state (accumulated sums, total_time, processed_time for time bounds,
accum_tstep_count) lives in the output object and carries over from one file to the next,
so averages and time bounds are correct without the wrapper's oc_tavg_helper workaround.

New config options (in any [CF output] section; default = .false.):

* one_file_per_write: No file is created at initialization. Each time output is due,
  a new file is created, one time slice is written, and the file is closed.
  In standalone runs, the file name is built by inserting the time (yr) before '.nc':
  e.g., name = out.tavg.nc gives out.tavg.0001.nc, out.tavg.0002.nc, ...
  Integer years are written with at least 4 digits (0006, 1861, -21000);
  other times with 3 decimal places.
  Output timing is set by frequency, write_init, stop and forcewrite_final, as before.

* external_control: Reserved for an external driver. Currently it is read and logged only.

one_file_per_write is not allowed in a [CF restart] section (fatal error), because a
standard restart reads the file named in the config file.

Code changes:

* glimmer_ncdf.F90: New fields external_control, one_file_per_write and base_filename
  in type glimmer_nc_output, and file_open in type glimmer_nc_stat.
  delete_output closes a file only if it is open.

* glimmer_ncparams.F90: handle_output reads the two new flags; readparams aborts if
  one_file_per_write is set for [CF restart].

* glimmer_ncio.F90:
  - createfile and openappend set file_open = T.
  - New subroutines glimmer_nc_newfile (set up an output object for a new file:
    filename, variable list restored from vars_copy, timecounter = 1) and
    glimmer_nc_closefile (close the file but keep the object).
  - New function glimmer_nc_slice_filename (standalone file name for a given time).
  - New function glimmer_nc_output_has_var (see below).
  - openall_out creates no file at initialization for one_file_per_write objects,
    but still calls glimmer_nc_checkwrite_init.
  - glimmer_nc_advance_timeslice leaves define mode only if the file is open,
    and for one_file_per_write objects does not increment timecounter or sync the (closed) file.

* ncdf_template.F90.in:
  - New subroutine NAME_io_write_slice: create a new file, define the variables, fill the
    dimension variables, write the time slice and fields, reset averages, close the file.
    The optional arguments external_time, external_baseline_year and external_time_units
    are passed through for use by an external driver.
  - NAME_io_writeall: for one_file_per_write objects, accumulate averages as usual,
    and if a write is due, call NAME_io_write_slice. The default path is unchanged.
  - NAME_io_createall: for one_file_per_write objects, skip NAME_io_create at initialization,
    but set do_averages from the variable list so averages accumulate before the first write.

* glide_nc_custom.F90: glide_nc_fillall skips one_file_per_write objects
  (their dimension variables are filled when each file is created).

* utils/build/generate_ncvars.py: In the generated *_avg_accumulate and *_avg_reset routines,
  the test for whether a tavg variable belongs to an output object now calls
  glimmer_nc_output_has_var instead of parallel_inq_varid.
  Previously the test asked the open netCDF file whether it contained the variable.
  With one_file_per_write, no file is open between writes, so nothing was accumulated
  and all averages were written as zero. (It also risked querying a stale file ID.)
  glimmer_nc_output_has_var checks the open file, as before, if there is one;
  otherwise it checks the variable list (vars_copy).

Limitations:

* Standalone one_file_per_write assumes that all variables in the file come from one
  I/O module (e.g., glide_io, as in cism_driver). If a GLAD or Glint file set
  one_file_per_write without external_control, each module would create its own copy
  of the file. For CESM, the wrapper will create and write the files itself,
  using external_control.
* As before, a given tavg variable should appear in only one output file.

Testing (standalone):

I used an 8-km Antarctic configuration (DIVA, 256 tasks, dt = 1/6 yr): a 5-year startup run
and a 5-year standard restart, writing an instantaneous 2D file (frequency = 1, stop = 3/8),
a 2D tavg file (frequency = 1), a scalar tavg file (frequency = 2, with a forced partial
interval at the end of the first run), an instantaneous scalar file, and a restart file,
with forcewrite_final = T.

* With the existing config files (no new options), output was BFB with the previous commit
  (0a3fe86): all netCDF files identical with nccmp, and log files identical apart from
  timestamps. This was checked after the infrastructure changes, after the
  one_file_per_write changes, and after the generate_ncvars.py change.
* With one_file_per_write = T for the instantaneous 2D file and both tavg files,
  the run produced the expected 23 single-slice files. Each file matched the corresponding
  slice of the multi-slice file from the BFB run exactly, for all variables, including
  the time variables, tstep_count, the time bounds and the tavg fields.
  This confirms that averages and time bounds carry over correctly from file to file
  and across the restart. The instantaneous scalar file and the restart file were BFB
  (apart from the 'configuration' global attribute, which includes the config file text).

Expected answer changes:

* Standalone CISM: none, unless one_file_per_write is set.
* CESM (CISM-wrapper tests): none expected. The new flags default to .false.,
  the wrapper creates its own output objects, and every output object the wrapper uses
  has an open file, so the tavg accumulation test (glimmer_nc_output_has_var) behaves
  as before.
* Glint: compiles (glint_io and glint_mbal_io are generated from the same template),
  but was not run.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The previous commit (96f6083) added the config option external_control,
which was read and logged but did not yet change model behavior.
With this commit, the option takes effect.

For an output object with external_control = .true., subroutine NAME_io_writeall
(ncdf_template.F90.in) accumulates averages as usual but does nothing else.
It does not decide whether output is due and never writes to the file,
including at initialization (write_init) and at finalization (forcewrite_final).
An external driver (e.g., the CESM wrapper) decides when to write and writes the output.

This separates two responsibilities:
* CISM owns WHAT is written: the variable list, metadata, averaging, and time bounds,
  with the checks already applied in glimmer_nc_readparams (e.g., no mixing of
  instantaneous and tavg fields; write_init = F for tavg files).
* The driver owns WHEN (e.g., based on the CESM clock) and, with one_file_per_write,
  WHERE (the name of each file, e.g., following CESM conventions).

The calling sequence for a driver is documented in a comment in NAME_io_writeall:
(1) With one_file_per_write (the intended CESM case, one time slice per file):
    glimmer_nc_newfile, glimmer_nc_createfile, NAME_io_create (for each I/O module,
    e.g., glide_io and glad_io), glide_nc_filldvars, glimmer_nc_write_timeslice,
    NAME_io_write (for each module), NAME_avg_reset, glimmer_nc_closefile.
    If all the variables belong to one module, NAME_io_write_slice does all of this.
    The driver can add its own global attributes after the file is created.
(2) Without one_file_per_write (the file is created at initialization and holds
    multiple time slices): glimmer_nc_checkwrite with the optional wrote_timeslice argument;
    then, if wrote_timeslice = .true., NAME_io_write (for each module) and NAME_avg_reset.
In either case, a driver that passes an external time should call glimmer_nc_checkwrite_init
at initialization with its external start time, so the first time bounds are in external units.

Next step: modify CISM-wrapper to use this machinery for CESM history files (h0i and h0a),
with buildnml writing [CF output] sections that set external_control and one_file_per_write.

Testing (standalone, same 8-km Antarctic configuration as the previous two commits:
5-year startup run followed by a 5-year standard restart):

* With the existing config files, output was BFB with the previous commit:
  all netCDF files identical with nccmp, and log files identical apart from timestamps.
* Smoke test: external_control = T for the 2D tavg file (multi-slice mode), and
  external_control = T plus one_file_per_write = T for the instantaneous 2D file.
  As expected, the tavg file was created at initialization with all variables defined
  but no time slices, was reopened on restart, and still had no time slices at the end;
  no instantaneous 2D files were created; and the other output files and the restart file
  were BFB (apart from the 'configuration' global attribute, which includes the config text).
  The actual writing by an external driver will be tested with CISM-wrapper.

Expected answer changes:

* Standalone CISM: none, unless external_control is set.
* CESM (CISM-wrapper tests): none. The wrapper does not yet set external_control.
* Glint: compiles, but was not run.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…utput file

Each time-average ('_tavg') variable is accumulated in a single model array.
If the same variable is listed in two or more output files, it is accumulated once
per file per timestep, and the averages in all of those files are wrong.
Until now this was a known limitation that the code did not check.

This commit adds a private subroutine, check_duplicate_tavg, in glimmer_ncparams.F90.
It is called from glimmer_nc_readparams after all the [CF output] and [CF restart]
sections have been read. For each variable name ending in '_tavg' in an output file's
variable list, it searches the variable lists of the other output files
(whole-name matches only, so that e.g. calving_rate_tavg does not match
total_calving_rate_tavg). If a match is found, the model aborts with a message
naming the variable and the two files.

Notes:
* The check is based on names ending in '_tavg', consistent with the existing rule
  that instantaneous and tavg variables cannot be mixed in a [CF output] file.
* Restart files are included in the check, but variables added by expanding the 'restart'
  keyword are not checked (the expansion happens later, in NAME_io_create).
  This is not an issue in practice, since restart files hold the instantaneous state
  and normally contain no tavg variables.
* Listing a non-tavg variable in more than one file is still allowed.
* Once CISM-wrapper writes CESM history files through [CF output] sections, this check
  will also apply to CESM runs.

Testing (standalone, 8-km Antarctic configuration, 5-year startup and 5-year restart runs):
* With the existing config files, output was BFB with the previous commit:
  all netCDF files identical with nccmp, and log files identical apart from timestamps.
* With acab_applied_tavg added to the variable list of a second tavg file,
  the model aborted at initialization with the expected message:
    Error: time-average variable acab_applied_tavg is listed in more than one output file:
      out_Antarctica_8km.tavg.nc
      out_Antarctica_8km.scalars.tavg.nc
    FATAL ERROR : Each tavg variable may appear in only one output file
* The name-matching logic was also tested standalone (duplicates anywhere in the list,
  substring names, non-tavg duplicates, empty lists).

Expected answer changes: none. Runs that list a tavg variable in more than one output file,
which previously gave incorrect averages, now abort at initialization.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…d with '00:00:00'

Two small metadata changes to follow CESM conventions for the time variables
(see ESCOMP/CISM-wrapper issue #75):

* The dimension of the time bounds variables (internal_time_bounds and time_bounds)
  is now called 'nbnd' instead of 'tbnd', as in other CESM components.
  The corresponding netCDF ID in type glimmer_nc_stat is renamed from tbnd_dim to nbnd_dim.
  (The 'tbnd' dimension was added only recently on this branch, so no existing
  production files use it.)

* The units attributes of the time variables now end with '00:00:00' instead of '0:0:0',
  e.g. 'common years since 0000-01-01 00:00:00'. For CESM history files, once CISM-wrapper
  passes time in days since year 1, the units of 'time' will be
  'days since 0001-01-01 00:00:00', as requested in issue #75.
  Both forms are valid under CF/UDUNITS; the new form matches other CESM components.

Testing (standalone, 8-km Antarctic configuration, 5-year startup and 5-year restart runs):
All netCDF files were BFB in their data compared to the previous commit (nccmp -d).
The only header differences were the units strings of internal_time, time and the time bounds,
and (in the tavg files) the name of the bounds dimension. Log files were identical apart
from timestamps.

Expected answer changes: none. Metadata changes only (units strings, and the bounds
dimension name in files containing tavg fields).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
CISM writes the units of internal_time (and, by default, of time and the time bounds)
as '<unit> since 0000-01-01 00:00:00', where the unit is a year of exactly 365 days.
Until now the unit was written as 'common years', with a space. UDUNITS spells this
unit 'common_year' (plural 'common_years'), as noted in an existing comment in
glimmer_nc_createfile. With the space, cftime cannot parse the units, so xarray's
open_dataset fails on every CISM output file:

  ValueError: Failed to decode variable 'internal_time': unable to decode time units
  'common years since 0000-01-01 00:00:00' with "calendar 'noleap'"

With the UDUNITS spelling, 'common_years since 0000-01-01 00:00:00', current versions
of cftime (tested 1.6.4) and xarray (tested 2025.7) decode internal_time and time as
noleap dates, and open_dataset works with no special options.

This resolves the question raised in ESCOMP/CISM-wrapper issue #75 about how to handle
internal_time in CESM history files. The discussion there settled on keeping
internal_time in both history and restart files, for both standalone and coupled runs,
and making sure xarray can open the files. The earlier proposal was to remove 'since'
from the internal_time units; testing showed that the failure was caused only by the space,
so internal_time can remain a proper CF time coordinate.
(In CESM, time will have units 'days since 0001-01-01 00:00:00', supplied by CISM-wrapper,
while internal_time keeps CISM's units. The two can differ, e.g. in accelerated runs.)

Note: Older versions of cftime that do not support 'common_years' will still fail to decode
these units; the workaround is xarray.open_dataset(..., decode_times=False).

Testing (standalone, 8-km Antarctic configuration, 5-year startup and 5-year restart runs):
* All netCDF files were BFB in their data compared to the previous commit (nccmp -d).
  The only header differences were the units of internal_time, time and (in tavg files)
  the two time bounds variables. Log files were identical apart from timestamps.
* xarray.open_dataset failed on all five output files from the previous commit
  and succeeded on all five from this commit, including the restart file and both
  tavg files, with time, internal_time and time_bounds decoded as noleap dates.

Expected answer changes: none. Metadata change only (time units strings).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Commit 8d56fb4 added a warning in NAME_io_create whose character string was continued
onto the next line:

   write(message,*) 'Warning: glide_io_create may have not have created all the requested&
        & output variables in file ', trim(NCO%filename)

With the gnu compiler, the preprocessor runs in traditional mode and does not recognize
Fortran continuation lines. On the second line it sees an unmatched quote and treats the
rest of the line as a string, so the NCO macro is not expanded. All the generated *_io.F90
modules then fail to compile:

   Error: Symbol 'nco' at (1) has no IMPLICIT type

The Intel preprocessor handles this case, so standalone (Intel) builds were not affected.
The failure appeared in CISM-wrapper tests built with gnu
(e.g., ERI_Ly9.f09_g17_ais8gris4.T1850Gag.derecho_gnu.cism-noevolve_ais).

The string is now on a single line, with the continuation after it. I also fixed the typo
"may have not have" and changed 'glide_io_create' to 'NAME_io_create', so that each generated
module reports its own name. A comment explains why the string should not be continued.

This was the only line in CISM with a continued string on a line that uses the NCO or NCI macros.

Testing: Generated glide_io.F90 from the old and new templates and ran gfortran 14.3 -cpp -E.
With the old template, one NCO% reference was left unexpanded (the line above);
with the new template, none. Message text change only; no answer changes.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…iles)

CISM-wrapper exact-restart tests (e.g., ERS_Ly7.f09_g17_gris4.T1850Gg.derecho_intel)
aborted when writing the CESM restart file:

   FATAL ERROR : (glimmer_ncio.F90:827) NetCDF: Variable not found

The CESM restart file is created with the variable list ' restart '. glimmer_nc_createfile
looks for '_tavg' in the list before the 'restart' keyword is expanded, so it correctly defines
no time bounds variables for this file. Then NAME_io_create expands 'restart' into the full
restart variable list, which for GLAD includes the coupling fluxes rofi_tavg, rofl_tavg and
hflx_tavg, and sets do_averages = .true. because these names end in '_tavg'.
glimmer_nc_write_timeslice wrote the time bounds whenever do_averages was true,
so it tried to write variables that did not exist.

Standalone CISM was not affected, because CISM's own restart list contains no '_tavg' names.

The fix separates "this object accumulates averages" from "this file has time bounds":
* New flag NCO%time_bounds (type glimmer_nc_stat), set to .true. by glimmer_nc_createfile when it
  defines the time bounds variables, and by glimmer_nc_openappend when it finds them in an
  existing file.
* glimmer_nc_write_timeslice writes the time bounds only if NCO%time_bounds is true.
For time-average output files the two flags agree, so these files are unchanged.

Testing:
* Compiled with Intel; checked with the gfortran preprocessor (no unexpanded NCO/NCI macros).
* To be tested with the CISM-wrapper test subset and the aux_glc suite (restart tests),
  and with the standalone 8-km Antarctic test (expected BFB).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…te a file

Subroutine check_for_tempstag (glimmer_ncio.F90) converts 'temp', 'flwa' and 'dissip' in an
output variable list to 'tempstag', 'flwastag' and 'dissipstag' (or the reverse for Glide).
It is called from NAME_io_create, and it ended with

   nc%vars_copy = nc%vars

NAME_io_create removes each variable name from nc%vars as it creates the variable.
In CISM-wrapper, a CESM history file is created by calling glide_io_create and then glad_io_create
for the same output object. When check_for_tempstag was called from glad_io_create, nc%vars had
already been mostly consumed by glide_io_create, so vars_copy lost most of the variable list.

This did not matter as long as each output object wrote a single file. But with the new
one_file_per_write option, each file starts from vars_copy (glimmer_nc_newfile). In CISM-wrapper tests
with the new h0i history stream, only the first h0i file written by each output object contained
the history fields; later files contained only the dimension and time variables. Restart tests
(ERS, ERP, ERI) then failed because the base and restart runs had different field lists.

Standalone runs were not affected, since standalone CISM calls only glide_io_create for an output
object.

Now check_for_tempstag applies the same substitutions to nc%vars and to nc%vars_copy separately,
and no longer copies vars into vars_copy. vars_copy is padded with spaces first, so that the first
and last variable names in the list can also be matched. The log messages are unchanged.

Testing: Compiled with Intel (with bounds checking); checked with the gfortran preprocessor.
The replacement logic was tested standalone, including the case of a second I/O module calling
check_for_tempstag after the first has consumed nc%vars. To be tested with the CISM-wrapper
subset (h0i files) and with the standalone 8-km Antarctic test (expected BFB).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
In files with time bounds (i.e., files containing time-average '_tavg' fields), internal_time and
time are now the midpoints of the averaging interval, instead of the end of the interval.
The time bounds (internal_time_bounds and time_bounds) still give the start and end of the interval.

This follows CF conventions and CESM practice for averaged history fields, and was agreed on
ESCOMP/CISM-wrapper issue #75 (resolving CISM-wrapper issue #76). It applies to both standalone
and CESM runs. Instantaneous files are unchanged: internal_time and time are the end of the timestep,
and there are no time bounds.

The change is in glimmer_nc_write_timeslice (glimmer_ncio.F90). The log message
"Writing to file ... at time" still gives the end of the interval (the current model time).

Testing (standalone, 8-km Antarctic configuration, 5-year startup and 5-year restart runs):
* Instantaneous output files and the restart file are BFB with the previous commit
  (nccmp, data and metadata). Log files are identical apart from timestamps.
* In the two tavg files, only internal_time and time differ, and they are exactly the midpoints
  of the time bounds; e.g., for annual averages, 0.5, 1.5, ..., 9.5 (previously 1, 2, ..., 10),
  and for a scalar tavg file with 2-year averages and a partial interval at the end of the first run,
  1, 3, 4.5, 5.5, 7, 9. All tavg fields and time bounds are unchanged.

Expected answer changes: the values of internal_time and time in time-average output files
(metadata only). No changes to output fields.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…acters

CISM's config file reader (glimmer_config.F90) limited each value to 400 characters (valuelen),
and read each line with a fixed format of 450 characters. Longer values caused a fatal error
("The value of config option ... is too long to be read fully").

This limit became a problem in CISM-wrapper tests: with a few time-average fields added to the
default CESM history variables, the esm_history_vars line in cism.<icesheet>.config is
427 characters long, and the model aborted at initialization. History variable lists are
expected to grow (e.g., for CMIP7).

Changes:
* valuelen is now 4096, matching len_history_vars in glad_type.F90 (the maximum length of the
  CESM history variable list). linelen = valuelen + namelen + 1 increases accordingly.
* Each line is read with format '(a)', i.e., up to the length of the line buffer, instead of the
  hardwired '(a450)'. Values that are too long are still caught in InsertValue.

Notes:
* Output variable lists in [CF output] sections can hold up to 1024 characters (glimmer_nc_vars_len);
  CISM-wrapper's buildnml checks this limit.
* ConfigAsString writes the whole config to a string of length 10000 (configstring), which is used
  only for the 'configuration' global attribute of output files. For a very long config file,
  this attribute would be truncated; this is unchanged.

Testing: Compiled with Intel (with bounds checking). To be tested with the CISM-wrapper tests that
use time-average history fields, and with the standalone 8-km Antarctic test (expected BFB).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…tart mid-interval

Previously, the running sums of time-average ('_tavg') fields were not saved in restart files.
After a restart in the middle of an averaging interval, a new interval started at the restart time,
so the first average after the restart did not match the average in a continuous run. This commit
saves the averaging state in restart files and restores it on a standard restart (restart = 1),
so the averages continue exactly (bit-for-bit with a continuous run). This is the CISM part of
ESCOMP/CISM-wrapper issue #75, item 6; the wrapper part (CESM continue runs) will follow.

What is saved:
* Running sums: For each '_tavg' field in a time-average output file, a variable '<name>_tavg_sum'
  is added automatically to the restart variable list. It is written from the same array as
  '<name>_tavg' (e.g., model%climate%acab_applied_tavg), but without dividing by the total time,
  and it is read back into that array. There is no separate '_tavg_sum' array. The units are those
  of the field times years (e.g., '(meter/year)*year' for acab_applied_tavg_sum); the scale_factor is
  the same as for the field, and the restart read divides it out, so the sums are restored exactly.
  The '_tavg_sum' variables are generated by generate_ncvars.py; users should not list them in output
  files, and glimmer_nc_readparams aborts if a [CF output] section lists one. See the note on
  acab_tavg in glide_types.F90.
* Averaging state of each time-average stream: total_time, the start of the averaging interval
  (internal and external time), and the timestep of the most recent accumulation, in new restart
  variables tavg_total_time, tavg_start_time, tavg_start_external_time and tavg_accum_tstep_count,
  with dimensions (tavgstream, time). The global attribute 'tavg_streams' lists the streams by name
  (base_filename for one_file_per_write streams, else the file name), in order.

When it is restored:
* Standard restart (restart = 1; CESM continue runs map to this): the averaging state of each
  stream is restored from the restart file, matched by name. If the restart file has averaging state
  but not for a current stream, the model aborts, suggesting a hybrid restart (restart = 2) to change
  the time-average streams. If the restart file has no averaging state (e.g., an older restart file),
  a warning is written and a new averaging interval starts.
* Startup (restart = 0) and hybrid restart (restart = 2): a new averaging interval starts at the
  initial time, and the running sums are set to zero, including any read from the input file.

Other changes:
* NAME_io_writeall writes restart files in a second pass, after all other output at the same time,
  so the saved averaging state is the state after the time-average files have accumulated this
  timestep and (if they wrote a time slice) been reset. Also, if a restart file was written earlier
  at the same time (in the last timestep), and a time-average file then writes a slice in the final
  call from glide_finalise (forcewrite_final = .true.), the restart time slice is rewritten with the
  state after the reset. Otherwise, a standard restart would continue an interval that had already
  been written.
* The test for whether an output file holds time averages (do_averages) now matches whole names
  ending in '_tavg', so that restart files, which hold '_tavg_sum' variables, do not accumulate
  averages themselves (which would double-count, since the arrays are shared).
* New helper routines in glimmer_ncio.F90: glimmer_nc_is_tavg_stream, glimmer_nc_is_restart,
  glimmer_nc_stream_name, glimmer_nc_tavg_streams, glimmer_nc_write_tavg_state,
  glimmer_nc_restore_tavg_state. New generated routines in each I/O module:
  NAME_add_tavg_sums_to_restart and NAME_is_avg_var (new generator token !GENVAR_AVGLIST!).

Glint and Glad: their I/O modules have no '_tavg' fields, so nothing is added for them, but they
are generated from the same template and compile. (Glint may be deprecated; not tested here.)

Testing (standalone CISM, 8-km Antarctic iotest setup, 256 tasks on derecho, intel):
* 5-yr run plus 5-yr standard restart (iotest, forcewrite_final = .true.): all output files are
  bit-for-bit with the previous commit. Restart files gain the new variables; all other restart
  variables are bit-for-bit. The saved averaging state at t = 5 and t = 10 has zero sums and
  total_time, with the intervals starting at the restart time, since the final call wrote and reset
  the averages.
* Restart in the middle of averaging intervals (forcewrite_final = .false.; 2-D tavg file with
  frequency 3, i.e. interval [3,6], and scalar tavg file with frequency 2, interval [4,6], both
  spanning the restart at t = 5): both time-average files are bit-for-bit with a continuous 10-yr
  run. The other output files are also bit-for-bit, except for differences in inversion-related
  fields that are not new (the previous commit gives exactly the same differences between continuous
  and restarted runs):
  - bmlt_float_target (and total_bmlt_float_target) is zero after the restart; the field is not in
    restart files.
  - coulomb_c, and powerlaw_c, which is a function of coulomb_c here (which_ho_powerlaw_c = 4),
    differ at some vertices, all with f_ground = 0, where basal friction is not applied. coulomb_c
    also differs in the restart file at t = 10.
  All other fields are bit-for-bit. In this short test, the differences in coulomb_c do not affect
  other answers, but they would if those vertices regrounded later. These restart issues are not
  related to this commit and will be revisited later.
  A continuous 10-yr run is bit-for-bit with the previous commit (all output files, and all restart
  variables other than the new ones).
* Hybrid restart (restart = 2, tstart = 100) from the two restart files above, one with nonzero
  partial sums and one with zero sums: all output, including restart files, is bit-for-bit between
  the two runs, and the first averaging intervals start at t = 100.
New logical tavg_restored in glimmer_nc_output, set in NAME_io_createall for each time-average
output stream: true if the averaging state was restored from the restart file (standard restart,
restart = 1), else false.

This is for the CESM wrapper, which needs to know whether CISM restored the state:
* CESM branch and hybrid runs (with an evolving ice sheet) also use restart = 1, reading the
  refcase restart file, but the wrapper starts a new averaging interval in those runs, so it must
  override the restored state.
* In a CESM continue run from a restart file written before the previous commit (with no averaging
  state), the wrapper falls back to setting the start of the averaging interval itself.

No answer changes: the flag is not used in standalone CISM. Builds with intel and gnu.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
GLAD read the CESM history variable list (esm_history_vars) from the [esm_output] section of the
config file and stored it in glad_instance, but it is no longer used: the CESM wrapper now writes
history files through CISM's [CF output] machinery, and the variable lists are in the h0i and h0a
[CF output] sections of the config file. This commit removes esm_history_vars and
len_history_vars from glad_type.F90. GLAD still reads history_option and history_frequency from
[esm_output], for use by the wrapper.

Config files that still have esm_history_vars in [esm_output] (e.g., from an older version of the
CESM wrapper) still work: CISM checks for unused config sections, but not for unused keys in a
section that is used.

The maximum length of a config value stays at 4096 characters (glimmer_config.F90), since
[CF output] variable lists can be long; the comment there is updated.

No answer changes. Builds with intel and gnu.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@whlipscomb
whlipscomb requested review from Katetc and gunterl October 6, 2026 04:10
@whlipscomb whlipscomb changed the title ## CESM-compliant time variables and time-average output; output options for an external driver CESM-compliant time variables and time-average output; output options for an external driver Oct 6, 2026

@Katetc Katetc left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks really nice to me. My only concern is about glide_types

Comment thread libglide/glide_types.F90 Resolved
Comment thread libglide/glide_types.F90 Resolved
@Katetc
Katetc merged commit 6f04e11 into main Oct 9, 2026
@whlipscomb

Copy link
Copy Markdown
Contributor Author

@Katetc, thanks for bringing this in!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants