Skip to content

st_archive issues in ctsm5.3.041 with our testing #3098

Description

@ekluzek

Brief summary of bug

Baseline testing shows the st_archive tests as failing.

General bug information

CTSM version you are using: ctsm5.3.041

Does this bug cause significantly incorrect results in the model's science? No

Configurations affected: st_archive in test lists

Details of bug

Tests when run show the the ST_ARCHIVE fails. For example...

FAIL SMS_Ld10_D_Mmpi-serial.CLM_USRDAT.I1PtClm60SpRs.izumi_nag.clm-default--clm-NEON-TOOL SHORT_TERM_ARCHIVER exception

Important details of your setup / configuration so we can reproduce the bug

Normal aux_clm test suite.

Important output or errors that show the problem

 ---------------------------------------------------
2025-04-29 14:41:04: Exception during SHORT_TERM_ARCHIVER:
ERROR: Moved file rpointer.lnd_9999.1976-01-03-00000 with dispostion ignore to directory env_archive_test/archive/rest/1976-01-01-00000
Traceback (most recent call last):
  File "/fs/cgd/data0/erik/ctsm_worktree/externals_update/cime/CIME/SystemTests/system_tests_common.py", line 788, in _phase_modifying_call
    function()
  File "/fs/cgd/data0/erik/ctsm_worktree/externals_update/cime/CIME/SystemTests/system_tests_common.py", line 775, in _st_archive_case_test
    result = self._case.test_env_archive()
  File "/fs/cgd/data0/erik/ctsm_worktree/externals_update/cime/CIME/case/case_st_archive.py", line 1260, in test_env_archive
    _check_disposition(testdir)
  File "/fs/cgd/data0/erik/ctsm_worktree/externals_update/cime/CIME/case/case_st_archive.py", line 1309, in _check_disposition
    _file, root
  File "/fs/cgd/data0/erik/ctsm_worktree/externals_update/cime/CIME/utils.py", line 176, in expect
    raise exc_type(msg)
CIME.utils.CIMEError: ERROR: Moved file rpointer.lnd_9999.1976-01-03-00000 with dispostion ignore to directory env_archive_test/archive/rest/1976-01-01-00000

 ---------------------------------------------------
 ./xmlquery -p DOUT

Results in group run_data_archive
	DOUT_S: FALSE
	DOUT_S_SAVE_INTERIM_RESTART_FILES: FALSE

Activity

  1. added
    bugsomething is working incorrectly
    priority: highHigh priority to fix/merge soon, e.g., because it is a problem in important configurations
    testingadditions or changes to tests
    on Apr 29, 2025
  2. added this to the cesm3_0_beta07 milestone on Apr 29, 2025
  3. self-assigned this
    on Apr 29, 2025
  4. added theissue type on Apr 29, 2025
  5. ekluzek commented on Apr 29, 2025

    @ekluzek
    ContributorAuthor

    This must be due to #3067, but I swear we didn't see this before. Although it the ctsm5.3.041 baselines on both Derecho and Izumi do show it as a problem.

    It seems to be doing something with the tests in config_archive.xml, which I didn't think it exercised.

  6. ekluzek commented on Apr 30, 2025

    @ekluzek
    ContributorAuthor

    OK, first I figured out an easy way to test this. In a test case you just do this

    ./case.st_archive --test-case

    (and also try --test-all to test the config_archive.xml file)

    Next it looks like that cime has to be updated to cime6.1.091 and cmeps to 1.0.47.

    There also seems to be a problem with this line in the config_archive.xml file (in the file testing section).

     <tfile disposition="ignore">rpointer.lnd_9999.1976-01-03-00000</tfile>
    

    This filename should NOT be copied, but seems to being copied anyway. CAM has some files like this that seem to work correctly. So I'm not clear on what's going on here. But, should be able to get something working with at least a workaround.

  7. ekluzek commented on May 1, 2025

    @ekluzek
    ContributorAuthor

    OK, so this is dependent on cmeps and cime externals update, which we aren't quite ready for on master. And I'm not sure the st_archive is working properly in that I thought it shouldn't move files that weren't for the date from the cpl files of 1976-01-01-00000, so there might be a problem there that we should resolved. And most importantly we need to get the tests working in config_archive.xml so that the st_archive part won't fail.

    So we'll revert this both on master and b4b-dev. And then on b4b-dev we'll rebring it in with submodule updates needed as well as changes so that the CTSM config_archive.xml will PASS.

  8. added
    nextthis should get some attention in the next week or two. Normally each Thursday SE meeting.
    and removed
    nextthis should get some attention in the next week or two. Normally each Thursday SE meeting.
    on May 1, 2025
  9. ekluzek commented on May 7, 2025

    @ekluzek
    ContributorAuthor

    I was afraid that this bug meant that st_archive would also fail for production cases that use create_newcase rather than create_test.

    So to check in ctsm5.3.041 I ran this test:

    ./create_newcase --compset I1850Clm60BgcCropCrujra --res f09_f09_mg17 --case st_archive_test_case --run-unsupported
    ./xmlchange STOP_N=1
    ./xmlchange STOP_OPTION=nmonths
    ./case.setup
    vi user_nl_clm.     # Add use_init_interp = TRUE
    ./case.build
    ./case.submit

    The st_archive batch log ends with "st_archive completed" and all of the files seem to be archived correctly.

    ls -l /glade/derecho/scratch/erik/archive/st_archive_test_case/rest/0001-02-01-00000/
    total 2179028
    -rw-r--r-- 1 erik cgdtss        257 May  7 13:59 rpointer.atm.0001-02-01-00000
    -rw-r--r-- 1 erik cgdtss         47 May  7 13:59 rpointer.cpl.0001-02-01-00000
    -rw-r--r-- 1 erik cgdtss        513 May  7 13:57 rpointer.lnd.0001-01-01-01800
    -rw-r--r-- 1 erik cgdtss        257 May  7 13:59 rpointer.lnd.0001-02-01-00000
    -rw-r--r-- 1 erik cgdtss        257 May  7 13:59 rpointer.rof.0001-02-01-00000
    -rw-r--r-- 1 erik cgdtss  327323912 May  7 13:59 st_archive_test_case.clm2.h0.0001-01.nc
    -rw-r--r-- 1 erik cgdtss 1736211611 May  7 13:59 st_archive_test_case.clm2.r.0001-02-01-00000.nc
    -rw-r--r-- 1 erik cgdtss     656140 May  7 13:59 st_archive_test_case.clm2.rh0.0001-02-01-00000.nc
    -rw-r--r-- 1 erik cgdtss  103301164 May  7 13:59 st_archive_test_case.cpl.r.0001-02-01-00000.nc
    -rw-r--r-- 1 erik cgdtss    2318584 May  7 13:59 st_archive_test_case.datm.r.0001-02-01-00000.nc
    -rw-r--r-- 1 erik cgdtss   32157692 May  7 13:59 st_archive_test_case.mosart.h0.0001-01.nc
    -rw-r--r-- 1 erik cgdtss   29037088 May  7 13:59 st_archive_test_case.mosart.r.0001-02-01-00000.nc
    -rw-r--r-- 1 erik cgdtss     257292 May  7 13:59 st_archive_test_case.mosart.rh0.0001-02-01-00000.nc

    I then continued the run setting CONTINUE_RUN=TRUE to verify it could continue on. And it was fine.

  10. ekluzek commented on Jul 28, 2025

    @ekluzek
    ContributorAuthor

    @jedwards4b and @samsrabin FYI this is the issue that documents the problems I ran into when originally bringing in the rpointer changes. It looks like part of it had to do with cmeps and cime updates that we weren't ready to bring in at the time. But, we've now brought those updates in and are bringing in the very latest cime update as well. So I'm hopeful we'll have things working with this go around.

  11. ekluzek commented on Jul 28, 2025

    @ekluzek
    ContributorAuthor

    Since, the "fix" that came in with #3106 was just doing a revert this issue should NOT have been closed at that point and should have stayed open to document that we still needed to unrevert it.

  12. samsrabin commented on Jul 29, 2025

    @samsrabin
    Member

    Closing as completed with #3354 even though it hasn't yet merged to master. Note that #3351 is still open.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugsomething is working incorrectlypriority: highHigh priority to fix/merge soon, e.g., because it is a problem in important configurationstestingadditions or changes to tests

Type

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions