Skip to content

Failure at restart when DOUT_S_SAVE_INTERIM_RESTART_FILES=TRUE #3351

Closed
10 of 10 issues completed
Closed

Description

@slevis-lmwg

Brief summary of bug

I have confirmed @adrifoster's TRENDY2025 restart failure with my ctsm5.3.062 test (original 20-yr simulation described in NCAR/LMWG_dev#109). I restarted my case with
STOP_N = 2 days
REST_N = 1 day
DOUT_S_SAVE_INTERIM_RESTART_FILES = FALSE
RESUBMIT = 1

and it completed all 4 days correctly. Then I restarted again with
STOP_N = 2 days
REST_N = 1 day
DOUT_S_SAVE_INTERIM_RESTART_FILES = TRUE
RESUBMIT = 1

This time it completed the first 2 days but failed when it resubmitted.

General bug information

CTSM version you are using: ctsm5.3.062 (first version splitting hX files into hXa/hXi)

Does this bug cause significantly incorrect results in the model's science? No

Configurations affected: DOUT_S_SAVE_INTERIM_RESTART_FILES = TRUE

Details of bug

Important output or errors that show the problem

My case ran here
/glade/derecho/scratch/slevis/ctsm53062_f09_1850/run
and archived here
/glade/derecho/scratch/slevis/archive/ctsm53062_f09_1850

So far I have noticed the following behavior:

  1. In .../archive/.../rest I type ls 0021-01-0[5-7]* and get (recall the model restarted successfully from 01-05, and then failed to restart from 01-07):
0021-01-05-00000:
ctsm53062_f09_1850.clm2.h0a.0020-12.nc            ctsm53062_f09_1850.mosart.h0i.0020-12.nc            rpointer.atm.0021-01-02-00000  rpointer.lnd.0006-01-01-00000  rpointer.lnd.0021-01-06-00000  rpointer.rof.0021-01-04-00000
ctsm53062_f09_1850.clm2.h0i.0020-12.nc            ctsm53062_f09_1850.mosart.r.0021-01-05-00000.nc     rpointer.atm.0021-01-03-00000  rpointer.lnd.0011-01-01-00000  rpointer.lnd.0021-01-07-00000  rpointer.rof.0021-01-05-00000
ctsm53062_f09_1850.clm2.r.0021-01-05-00000.nc     ctsm53062_f09_1850.mosart.rh0a.0021-01-05-00000.nc  rpointer.atm.0021-01-04-00000  rpointer.lnd.0016-01-01-00000  rpointer.rof.0006-01-01-00000  rpointer.rof.0021-01-06-00000
ctsm53062_f09_1850.clm2.rh0a.0021-01-05-00000.nc  ctsm53062_f09_1850.mosart.rh0i.0021-01-05-00000.nc  rpointer.atm.0021-01-05-00000  rpointer.lnd.0021-01-01-00000  rpointer.rof.0011-01-01-00000  rpointer.rof.0021-01-07-00000
ctsm53062_f09_1850.clm2.rh0i.0021-01-05-00000.nc  rpointer.atm.0006-01-01-00000                       rpointer.atm.0021-01-06-00000  rpointer.lnd.0021-01-02-00000  rpointer.rof.0016-01-01-00000
ctsm53062_f09_1850.cpl.r.0021-01-05-00000.nc      rpointer.atm.0011-01-01-00000                       rpointer.atm.0021-01-07-00000  rpointer.lnd.0021-01-03-00000  rpointer.rof.0021-01-01-00000
ctsm53062_f09_1850.datm.r.0021-01-05-00000.nc     rpointer.atm.0016-01-01-00000                       rpointer.cpl.0021-01-05-00000  rpointer.lnd.0021-01-04-00000  rpointer.rof.0021-01-02-00000
ctsm53062_f09_1850.mosart.h0a.0020-12.nc          rpointer.atm.0021-01-01-00000                       rpointer.lnd.0001-01-01-01800  rpointer.lnd.0021-01-05-00000  rpointer.rof.0021-01-03-00000

0021-01-06-00000:
 ctsm53062_f09_1850.clm2.h0a.0020-12.nc             ctsm53062_f09_1850.clm2.rh0i.0021-01-06-00000.nc   ctsm53062_f09_1850.mosart.h0i.0020-12.nc             rpointer.atm                   'rpointer.lnd$NINST_STRING'
 ctsm53062_f09_1850.clm2.h0i.0020-12.nc             ctsm53062_f09_1850.cpl.r.0021-01-06-00000.nc       ctsm53062_f09_1850.mosart.r.0021-01-06-00000.nc     'rpointer.atm$NINST_STRING'      rpointer.rof
 ctsm53062_f09_1850.clm2.r.0021-01-06-00000.nc      ctsm53062_f09_1850.datm.r.0021-01-06-00000.nc      ctsm53062_f09_1850.mosart.rh0a.0021-01-06-00000.nc   rpointer.cpl.0021-01-06-00000  'rpointer.rof$NINST_STRING'
 ctsm53062_f09_1850.clm2.rh0a.0021-01-06-00000.nc   ctsm53062_f09_1850.mosart.h0a.0020-12.nc           ctsm53062_f09_1850.mosart.rh0i.0021-01-06-00000.nc   rpointer.lnd

0021-01-07-00000:
ctsm53062_f09_1850.clm2.h0a.0020-12.nc         ctsm53062_f09_1850.clm2.rh0a.0021-01-07-00000.nc  ctsm53062_f09_1850.datm.r.0021-01-07-00000.nc  ctsm53062_f09_1850.mosart.r.0021-01-07-00000.nc     rpointer.cpl.0021-01-07-00000
ctsm53062_f09_1850.clm2.h0i.0020-12.nc         ctsm53062_f09_1850.clm2.rh0i.0021-01-07-00000.nc  ctsm53062_f09_1850.mosart.h0a.0020-12.nc       ctsm53062_f09_1850.mosart.rh0a.0021-01-07-00000.nc
ctsm53062_f09_1850.clm2.r.0021-01-07-00000.nc  ctsm53062_f09_1850.cpl.r.0021-01-07-00000.nc      ctsm53062_f09_1850.mosart.h0i.0020-12.nc       ctsm53062_f09_1850.mosart.rh0i.0021-01-07-00000.nc

01-07 seems to be missing numerous rpointer files.
01-06 has rpointer files with strange suffixes.

  1. In .../run I see an empty rpointer.lnd and the following error in cesm.log:
    end-of-file during read, unit 99, file /glade/derecho/scratch/slevis/ctsm53062_f09_1850/run/./rpointer.lnd

@adrifoster and others please add information that you consider helpful.

Activity

  1. added theissue type on Jul 24, 2025
  2. added
    nextthis should get some attention in the next week or two. Normally each Thursday SE meeting.
    on Jul 24, 2025
  3. slevis-lmwg commented on Jul 24, 2025

    @slevis-lmwg
    ContributorAuthor

    I tried git grep -i DOUT_S_SAVE_INTERIM_RESTART_FILES in various places:

    • clm, ccs_config, share, cdeps, mosart, rtm, mizuroute, cism have nothing
    • cmeps returns
    cime_config/config_component_cesm.xml:  <entry id="DOUT_S_SAVE_INTERIM_RESTART_FILES">
    
    • cime returns
    CIME/case/case_st_archive.py:        datename_is_last or case.get_value("DOUT_S_SAVE_INTERIM_RESTART_FILES")
    CIME/case/case_st_archive.py:        case.get_value("DOUT_S_SAVE_INTERIM_RESTART_FILES"),
    CIME/case/case_st_archive.py:                if case.get_value("DOUT_S_SAVE_INTERIM_RESTART_FILES"):
    CIME/case/case_st_archive.py:    dout_s_save_interim = self.get_value("DOUT_S_SAVE_INTERIM_RESTART_FILES")
    CIME/data/config/config_tests.xml:    <DOUT_S_SAVE_INTERIM_RESTART_FILES>TRUE</DOUT_S_SAVE_INTERIM_RESTART_FILES>
    CIME/data/config/config_tests.xml:    <DOUT_S_SAVE_INTERIM_RESTART_FILES>TRUE</DOUT_S_SAVE_INTERIM_RESTART_FILES>
    
  4. moved this from Todo to In Progress in LMWG: Sprint Planning Boardon Jul 24, 2025
  5. slevis-lmwg commented on Jul 24, 2025

    @slevis-lmwg
    ContributorAuthor
    • Before diving deeper, confirming whether ctsm5.3.061 exhibits the same behavior (same test as above).
  6. slevis-lmwg commented on Jul 25, 2025

    @slevis-lmwg
    ContributorAuthor

    The good news (for me) is that ctsm5.3.061 exhibits the same behavior.
    The bad news (not that bad, considering we hadn't noticed this until now) is that this has been in the model for a little while.

    I could try bisecting until I find the first tag that exhibits the behavior.

  7. slevis-lmwg commented on Jul 25, 2025

    @slevis-lmwg
    ContributorAuthor

    @jedwards4b this may be a CIME bug, are you the right person to notify? If so, shall I hand this over to you?

  8. jedwards4b commented on Jul 25, 2025

    @jedwards4b
    Contributor

    This is because changes in ctsm and mosart have either not been merged or were reverted after merging. I am still waiting for an explanation of why the merge was reverted.

    I have submitted PR's ESCOMP/MOSART#115
    and (again) #3354

    I have also submitted
    ESCOMP/CMEPS#576
    and ESMCI/cime#4827
    to test this interim restart with the ERR test. For example this issue was tested using ERR.f09_t232.I1850Clm60BgcCrop.derecho_intel.drv-interim_restart

  9. 14 remaining items

  10. ekluzek commented on Jul 28, 2025

    @ekluzek
    Contributor

    I linked the unrevert issue to this one. And found the issue that goes into the problems I ran into.

    Notably this test should be run to see if we still see ST_ARCHIVE issues:

    SMS_Ld10_D_Mmpi-serial.CLM_USRDAT.I1PtClm60SpRs.izumi_nag.clm-default--clm-NEON-TOOL

  11. samsrabin commented on Jul 28, 2025

    @samsrabin
    Member

    @ekluzek I get this error with that test:

    ERROR: Command /home/samrabin/ctsm/bld/build-namelist failed rc=255
    out=
    err=ERROR : CLM build-namelist::CLMBuildNamelist::setup_logic_fire_emis() : Fire emission option fire_emis_factors_file can NOT be on when BGC SP (i.e. Satellite Phenology) is also on.
      DON'T use the '--fire_emis' option when '--bgc sp' is activated
    

    Trying without SP mode.

  12. ekluzek commented on Jul 28, 2025

    @ekluzek
    Contributor

    @ekluzek I get this error with that test:

    Trying without SP mode.

    Yes, it should work with a Bgc test.

    Or you add the "nofireemis" testmod to the test as in...

    SMS_Ld10_D_Mmpi-serial.CLM_USRDAT.I1PtClm60SpRs.izumi_nag.clm-default--clm-NEON-TOOL--clm-nofireemis

  13. samsrabin commented on Jul 28, 2025

    @samsrabin
    Member

    Yep @ekluzek this test passes: SMS_Ld10_D_Mmpi-serial.CLM_USRDAT.I1PtClm60SpRs.izumi_nag.clm-default--clm-NEON-TOOL--clm-nofireemis

  14. samsrabin commented on Jul 31, 2025

    @samsrabin
    Member

    @adrifoster confirms that this is working as of Tuesday afternoon on the interim-restart-fix-202507 branch.

  15. changed the title [-]Failure at restart when DOUT_S_SAVE_INTERIM_RESTART_FILES=TRUE[/-] [+]ctsm5.3.067: Failure at restart when DOUT_S_SAVE_INTERIM_RESTART_FILES=TRUE[/+] on Aug 7, 2025
  16. changed the title [-]ctsm5.3.067: Failure at restart when DOUT_S_SAVE_INTERIM_RESTART_FILES=TRUE[/-] [+]Failure at restart when DOUT_S_SAVE_INTERIM_RESTART_FILES=TRUE[/+] on Aug 8, 2025
  17. samsrabin commented on Aug 8, 2025

    @samsrabin
    Member

    This will be closing with PR #3400, but note that child issue #3383 will remain open to be fixed later.

  18. moved this from In progress - master to Done (non release/external) in CTSM: Upcoming tagson Aug 8, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugsomething is working incorrectlypriority: highHigh priority to fix/merge soon, e.g., because it is a problem in important configurations

Type

Projects

Relationships

None yet

Development

No branches or pull requests

Issue actions