Repository navigation
Description
Brief summary of bug
I have confirmed @adrifoster's TRENDY2025 restart failure with my ctsm5.3.062 test (original 20-yr simulation described in NCAR/LMWG_dev#109). I restarted my case with
STOP_N = 2 days
REST_N = 1 day
DOUT_S_SAVE_INTERIM_RESTART_FILES = FALSE
RESUBMIT = 1
and it completed all 4 days correctly. Then I restarted again with
STOP_N = 2 days
REST_N = 1 day
DOUT_S_SAVE_INTERIM_RESTART_FILES = TRUE
RESUBMIT = 1
This time it completed the first 2 days but failed when it resubmitted.
General bug information
CTSM version you are using: ctsm5.3.062 (first version splitting hX files into hXa/hXi)
Does this bug cause significantly incorrect results in the model's science? No
Configurations affected: DOUT_S_SAVE_INTERIM_RESTART_FILES = TRUE
Details of bug
Important output or errors that show the problem
My case ran here
/glade/derecho/scratch/slevis/ctsm53062_f09_1850/run
and archived here
/glade/derecho/scratch/slevis/archive/ctsm53062_f09_1850
So far I have noticed the following behavior:
- In .../archive/.../rest I type
ls 0021-01-0[5-7]*and get (recall the model restarted successfully from 01-05, and then failed to restart from 01-07):
0021-01-05-00000:
ctsm53062_f09_1850.clm2.h0a.0020-12.nc ctsm53062_f09_1850.mosart.h0i.0020-12.nc rpointer.atm.0021-01-02-00000 rpointer.lnd.0006-01-01-00000 rpointer.lnd.0021-01-06-00000 rpointer.rof.0021-01-04-00000
ctsm53062_f09_1850.clm2.h0i.0020-12.nc ctsm53062_f09_1850.mosart.r.0021-01-05-00000.nc rpointer.atm.0021-01-03-00000 rpointer.lnd.0011-01-01-00000 rpointer.lnd.0021-01-07-00000 rpointer.rof.0021-01-05-00000
ctsm53062_f09_1850.clm2.r.0021-01-05-00000.nc ctsm53062_f09_1850.mosart.rh0a.0021-01-05-00000.nc rpointer.atm.0021-01-04-00000 rpointer.lnd.0016-01-01-00000 rpointer.rof.0006-01-01-00000 rpointer.rof.0021-01-06-00000
ctsm53062_f09_1850.clm2.rh0a.0021-01-05-00000.nc ctsm53062_f09_1850.mosart.rh0i.0021-01-05-00000.nc rpointer.atm.0021-01-05-00000 rpointer.lnd.0021-01-01-00000 rpointer.rof.0011-01-01-00000 rpointer.rof.0021-01-07-00000
ctsm53062_f09_1850.clm2.rh0i.0021-01-05-00000.nc rpointer.atm.0006-01-01-00000 rpointer.atm.0021-01-06-00000 rpointer.lnd.0021-01-02-00000 rpointer.rof.0016-01-01-00000
ctsm53062_f09_1850.cpl.r.0021-01-05-00000.nc rpointer.atm.0011-01-01-00000 rpointer.atm.0021-01-07-00000 rpointer.lnd.0021-01-03-00000 rpointer.rof.0021-01-01-00000
ctsm53062_f09_1850.datm.r.0021-01-05-00000.nc rpointer.atm.0016-01-01-00000 rpointer.cpl.0021-01-05-00000 rpointer.lnd.0021-01-04-00000 rpointer.rof.0021-01-02-00000
ctsm53062_f09_1850.mosart.h0a.0020-12.nc rpointer.atm.0021-01-01-00000 rpointer.lnd.0001-01-01-01800 rpointer.lnd.0021-01-05-00000 rpointer.rof.0021-01-03-00000
0021-01-06-00000:
ctsm53062_f09_1850.clm2.h0a.0020-12.nc ctsm53062_f09_1850.clm2.rh0i.0021-01-06-00000.nc ctsm53062_f09_1850.mosart.h0i.0020-12.nc rpointer.atm 'rpointer.lnd$NINST_STRING'
ctsm53062_f09_1850.clm2.h0i.0020-12.nc ctsm53062_f09_1850.cpl.r.0021-01-06-00000.nc ctsm53062_f09_1850.mosart.r.0021-01-06-00000.nc 'rpointer.atm$NINST_STRING' rpointer.rof
ctsm53062_f09_1850.clm2.r.0021-01-06-00000.nc ctsm53062_f09_1850.datm.r.0021-01-06-00000.nc ctsm53062_f09_1850.mosart.rh0a.0021-01-06-00000.nc rpointer.cpl.0021-01-06-00000 'rpointer.rof$NINST_STRING'
ctsm53062_f09_1850.clm2.rh0a.0021-01-06-00000.nc ctsm53062_f09_1850.mosart.h0a.0020-12.nc ctsm53062_f09_1850.mosart.rh0i.0021-01-06-00000.nc rpointer.lnd
0021-01-07-00000:
ctsm53062_f09_1850.clm2.h0a.0020-12.nc ctsm53062_f09_1850.clm2.rh0a.0021-01-07-00000.nc ctsm53062_f09_1850.datm.r.0021-01-07-00000.nc ctsm53062_f09_1850.mosart.r.0021-01-07-00000.nc rpointer.cpl.0021-01-07-00000
ctsm53062_f09_1850.clm2.h0i.0020-12.nc ctsm53062_f09_1850.clm2.rh0i.0021-01-07-00000.nc ctsm53062_f09_1850.mosart.h0a.0020-12.nc ctsm53062_f09_1850.mosart.rh0a.0021-01-07-00000.nc
ctsm53062_f09_1850.clm2.r.0021-01-07-00000.nc ctsm53062_f09_1850.cpl.r.0021-01-07-00000.nc ctsm53062_f09_1850.mosart.h0i.0020-12.nc ctsm53062_f09_1850.mosart.rh0i.0021-01-07-00000.nc
01-07 seems to be missing numerous rpointer files.
01-06 has rpointer files with strange suffixes.
- In .../run I see an empty rpointer.lnd and the following error in cesm.log:
end-of-file during read, unit 99, file /glade/derecho/scratch/slevis/ctsm53062_f09_1850/run/./rpointer.lnd
@adrifoster and others please add information that you consider helpful.
Metadata
Metadata
Assignees
Labels
Type
Projects
- StatusShow more project fieldsDone
- StatusShow more project fieldsDone (non release/external)
Activity
slevis-lmwg commentedon Jul 24, 2025
I tried
git grep -i DOUT_S_SAVE_INTERIM_RESTART_FILESin various places:slevis-lmwg commentedon Jul 24, 2025
slevis-lmwg commentedon Jul 25, 2025
The good news (for me) is that ctsm5.3.061 exhibits the same behavior.
The bad news (not that bad, considering we hadn't noticed this until now) is that this has been in the model for a little while.
I could try bisecting until I find the first tag that exhibits the behavior.
slevis-lmwg commentedon Jul 25, 2025
@jedwards4b this may be a CIME bug, are you the right person to notify? If so, shall I hand this over to you?
jedwards4b commentedon Jul 25, 2025
This is because changes in ctsm and mosart have either not been merged or were reverted after merging. I am still waiting for an explanation of why the merge was reverted.
I have submitted PR's ESCOMP/MOSART#115
and (again) #3354
I have also submitted
ESCOMP/CMEPS#576
and ESMCI/cime#4827
to test this interim restart with the ERR test. For example this issue was tested using ERR.f09_t232.I1850Clm60BgcCrop.derecho_intel.drv-interim_restart
14 remaining items
ekluzek commentedon Jul 28, 2025
I linked the unrevert issue to this one. And found the issue that goes into the problems I ran into.
Notably this test should be run to see if we still see ST_ARCHIVE issues:
SMS_Ld10_D_Mmpi-serial.CLM_USRDAT.I1PtClm60SpRs.izumi_nag.clm-default--clm-NEON-TOOL
samsrabin commentedon Jul 28, 2025
@ekluzek I get this error with that test:
Trying without SP mode.
ekluzek commentedon Jul 28, 2025
Yes, it should work with a Bgc test.
Or you add the "nofireemis" testmod to the test as in...
SMS_Ld10_D_Mmpi-serial.CLM_USRDAT.I1PtClm60SpRs.izumi_nag.clm-default--clm-NEON-TOOL--clm-nofireemis
samsrabin commentedon Jul 28, 2025
Yep @ekluzek this test passes:
SMS_Ld10_D_Mmpi-serial.CLM_USRDAT.I1PtClm60SpRs.izumi_nag.clm-default--clm-NEON-TOOL--clm-nofireemissamsrabin commentedon Jul 31, 2025
@adrifoster confirms that this is working as of Tuesday afternoon on the interim-restart-fix-202507 branch.
[-]Failure at restart when DOUT_S_SAVE_INTERIM_RESTART_FILES=TRUE[/-][+]ctsm5.3.067: Failure at restart when DOUT_S_SAVE_INTERIM_RESTART_FILES=TRUE[/+][-]ctsm5.3.067: Failure at restart when DOUT_S_SAVE_INTERIM_RESTART_FILES=TRUE[/-][+]Failure at restart when DOUT_S_SAVE_INTERIM_RESTART_FILES=TRUE[/+]samsrabin commentedon Aug 8, 2025
This will be closing with PR #3400, but note that child issue #3383 will remain open to be fixed later.