Skip to content

Race condition with a threaded test with matrix on? #3360

Description

@ekluzek

Brief summary of bug

In ctsm5.3.065 I had the test

ERP_P64x2_D_Ld5.f10_f10_mg37.I1850Clm50Bgc.derecho_intel.clm-ciso--clm-matrixcnOn_ignore_warnings

fail at runtime. But, after resubmitting it worked fine.

This is just to bring awareness that this is a potential problem and a simple resubmit solves it.

General bug information

CTSM version you are using: ctsm5.3.064-72-g4037e5a8d

Does this bug cause significantly incorrect results in the model's science? no

Configurations affected: threading with soil matrix on

Details of bug

It dies in a SHR_ASSERT statement that's ensuring the matrix size of two sparse matrices multiplied together are the same size.

Important output or errors that show the problem

cesm.log:

dec0933.hsn.de.hpc.ucar.edu 63:  mosart decomp info proc =        63 begr =    255151 endr =    259200 numr =      4050
dec0933.hsn.de.hpc.ucar.edu 21:  ERROR in SparseMatrixMultiplyMod.F90 at line 973
dec0933.hsn.de.hpc.ucar.edu 21: Image              PC                Routine            Line        Source
dec0933.hsn.de.hpc.ucar.edu 21: cesm.exe           0000000004842AF1  shr_abort_mod_mp_         110  shr_abort_mod.F90
dec0933.hsn.de.hpc.ucar.edu 21: cesm.exe           0000000004842A83  shr_abort_mod_mp_          65  shr_abort_mod.F90
dec0933.hsn.de.hpc.ucar.edu 21: cesm.exe           0000000004843A06  shr_assert_mod_mp          95  shr_assert_mod.F90.in
dec0933.hsn.de.hpc.ucar.edu 21: cesm.exe           0000000004843E0A  shr_assert_mod_mp         112  shr_assert_mod.F90.in
dec0933.hsn.de.hpc.ucar.edu 21: cesm.exe           0000000003544EA7  sparsematrixmulti         973  SparseMatrixMultiplyMod.F90
dec0933.hsn.de.hpc.ucar.edu 21: cesm.exe           0000000001629F29  cnsoilmatrixmod_m         614  CNSoilMatrixMod.F90
dec0933.hsn.de.hpc.ucar.edu 21: cesm.exe           0000000003F2B6D6  cndrivermod_mp_cn        1101  CNDriverMod.F90
dec0933.hsn.de.hpc.ucar.edu 21: cesm.exe           0000000001A77D09  cnvegetationfacad        1125  CNVegetationFacade.F90
dec0933.hsn.de.hpc.ucar.edu 21: cesm.exe           0000000000AA2FA1  clm_driver_mp_clm        1147  clm_driver.F90

Activity

  1. added this to the Eventually milestone on Jul 25, 2025
  2. added
    closed: wontfixWe won't fix this issue, because it would be too difficult and/or isn't important enough to fix
    on Jul 25, 2025
  3. added theissue type on Jul 25, 2025
  4. added
    priority: lowBackground task that doesn't need to be done right away.
    bugsomething is working incorrectly
    on Jul 25, 2025
  5. slevis-lmwg commented on Mar 31, 2026

    @slevis-lmwg
    Contributor
    ERP_P64x2_D_Ld5.f10_f10_mg37.I1850Clm50Bgc.derecho_intel.clm-ciso--clm-matrixcnOn_ignore_warnings
    PET_P64x2_D.f10_f10_mg37.I1850Clm50BgcCrop.derecho_intel.clm-default--clm-matrixcnOn_ignore_warnings
    

    gave the same error in #3894
    ERP passed on the first resubmit and PET passed on the second resubmit.

  6. samsrabin commented on May 21, 2026

    @samsrabin
    Member

    Had the same problem with ERP_P64x2_D_Ld5.f10_f10_mg37.I1850Clm50Bgc.derecho_intel.clm-ciso--clm-matrixcnOn_ignore_warnings in testing #4040.

  7. samsrabin commented on May 28, 2026

    @samsrabin
    Member

    This bit me today with the following tests:

    ERP_D_P64x2_Ld3.f10_f10_mg37.I1850Clm50BgcCrop.derecho_intel.clm-default--clm-matrixcnOn_ignore_warnings
    PET_P64x2_D.f10_f10_mg37.I1850Clm50BgcCrop.derecho_intel.clm-default--clm-matrixcnOn_ignore_warnings
    
  8. samsrabin commented on Nov 13, 2026

    @samsrabin
    Member

    This bit me today with the following test:

    ERP_D_P64x2_Ld3.f10_f10_mt201.I2000Clm50BgcCru.derecho_intel.clm-noFUN_flexCN--clm-matrixcnOn_ignore_warnings
    

    @ekluzek, I see you added the "closed: wontfix" label but you didn't close the issue. Why is that?

  9. ekluzek commented on Nov 13, 2026

    @ekluzek
    ContributorAuthor

    Just an oversight closing now.

  10. samsrabin commented on Nov 13, 2026

    @samsrabin
    Member

    There may be a simple solution to this, which is extremely annoying to us as devs! I sicced Claude Fable on this and it came up with a plausible theory. I'm writing up my summary as I understand it.

    There are four instances of blocks like this in CNSoilMatrix() (one each for C, C13, C14, and N):

    if ( AKXcacc%IsValuesSetSM() )then
    call AKXcacc%SPMP_B_ACC(num_soilc,filter_soilc,AKallsoilc)
    else
    ! This should only happen on the first time-step
    call AKXcacc%SetValueCopySM(num_soilc,filter_soilc,AKallsoilc)
    end if

    Those four blocks are where we end up seeing the "line 973" error. Claude suggested that the race condition happens when:

    1. Thread A calls IsValuesSetSM(), gets .false. and thus calls SetValueCopySM(), which ends by calling SetValueSM(), which starts to fill (in this order) the metadata NE then RI+CI.
    2. Thread B gets to the IsValuesSetSM() call after Thread A has filled NE but before it's finished RI and CI. It enters IsValuesSetSM(), sees that NE isn't empty_int, and returns .true.. Thread B then takes the other path of the above if statement, calling SPMP_B_ACC, which fails at line 973: an assertion involving RI, which Thread A hadn't finished filling.

    Claude suggests that changing the above code (including all four locations) to

    if (is_first_step()) then
       call AKXcacc%SetValueCopySM(num_soilc,filter_soilc,AKallsoilc)
    else
       call AKXcacc%SPMP_B_ACC(num_soilc,filter_soilc,AKallsoilc)
    end if

    would solve the problem by forcing every thread to fill the metadata itself on the first timestep of a (startup/hybrid) run. I think that makes sense, but I haven't tested it.

  11. added
    nextthis should get some attention in the next week or two. Normally each Thursday SE meeting.
    and removed
    closed: wontfixWe won't fix this issue, because it would be too difficult and/or isn't important enough to fix
    on Nov 13, 2026
  12. linked a pull request that will close this issueResolve matrix threaded race condition #4319on Nov 13, 2026
  13. removed
    nextthis should get some attention in the next week or two. Normally each Thursday SE meeting.
    on Nov 13, 2026
  14. added 2 commits that reference this issue on Nov 13, 2026
    e061b45
    22d5176
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugsomething is working incorrectlypriority: lowBackground task that doesn't need to be done right away.

    Type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions