Repository navigation
Labels
bugsomething is working incorrectlysomething is working incorrectlypriority: lowBackground task that doesn't need to be done right away.Background task that doesn't need to be done right away.
Milestone
Description
Brief summary of bug
In ctsm5.3.065 I had the test
ERP_P64x2_D_Ld5.f10_f10_mg37.I1850Clm50Bgc.derecho_intel.clm-ciso--clm-matrixcnOn_ignore_warnings
fail at runtime. But, after resubmitting it worked fine.
This is just to bring awareness that this is a potential problem and a simple resubmit solves it.
General bug information
CTSM version you are using: ctsm5.3.064-72-g4037e5a8d
Does this bug cause significantly incorrect results in the model's science? no
Configurations affected: threading with soil matrix on
Details of bug
It dies in a SHR_ASSERT statement that's ensuring the matrix size of two sparse matrices multiplied together are the same size.
Important output or errors that show the problem
cesm.log:
dec0933.hsn.de.hpc.ucar.edu 63: mosart decomp info proc = 63 begr = 255151 endr = 259200 numr = 4050
dec0933.hsn.de.hpc.ucar.edu 21: ERROR in SparseMatrixMultiplyMod.F90 at line 973
dec0933.hsn.de.hpc.ucar.edu 21: Image PC Routine Line Source
dec0933.hsn.de.hpc.ucar.edu 21: cesm.exe 0000000004842AF1 shr_abort_mod_mp_ 110 shr_abort_mod.F90
dec0933.hsn.de.hpc.ucar.edu 21: cesm.exe 0000000004842A83 shr_abort_mod_mp_ 65 shr_abort_mod.F90
dec0933.hsn.de.hpc.ucar.edu 21: cesm.exe 0000000004843A06 shr_assert_mod_mp 95 shr_assert_mod.F90.in
dec0933.hsn.de.hpc.ucar.edu 21: cesm.exe 0000000004843E0A shr_assert_mod_mp 112 shr_assert_mod.F90.in
dec0933.hsn.de.hpc.ucar.edu 21: cesm.exe 0000000003544EA7 sparsematrixmulti 973 SparseMatrixMultiplyMod.F90
dec0933.hsn.de.hpc.ucar.edu 21: cesm.exe 0000000001629F29 cnsoilmatrixmod_m 614 CNSoilMatrixMod.F90
dec0933.hsn.de.hpc.ucar.edu 21: cesm.exe 0000000003F2B6D6 cndrivermod_mp_cn 1101 CNDriverMod.F90
dec0933.hsn.de.hpc.ucar.edu 21: cesm.exe 0000000001A77D09 cnvegetationfacad 1125 CNVegetationFacade.F90
dec0933.hsn.de.hpc.ucar.edu 21: cesm.exe 0000000000AA2FA1 clm_driver_mp_clm 1147 clm_driver.F90
Metadata
Metadata
Assignees
Labels
bugsomething is working incorrectlysomething is working incorrectlypriority: lowBackground task that doesn't need to be done right away.Background task that doesn't need to be done right away.
Activity
slevis-lmwg commentedon Mar 31, 2026
gave the same error in #3894
ERP passed on the first resubmit and PET passed on the second resubmit.
samsrabin commentedon May 21, 2026
Had the same problem with
ERP_P64x2_D_Ld5.f10_f10_mg37.I1850Clm50Bgc.derecho_intel.clm-ciso--clm-matrixcnOn_ignore_warningsin testing #4040.samsrabin commentedon May 28, 2026
This bit me today with the following tests:
samsrabin commentedon Nov 13, 2026
This bit me today with the following test:
@ekluzek, I see you added the "closed: wontfix" label but you didn't close the issue. Why is that?
ekluzek commentedon Nov 13, 2026
Just an oversight closing now.
samsrabin commentedon Nov 13, 2026
There may be a simple solution to this, which is extremely annoying to us as devs! I sicced Claude Fable on this and it came up with a plausible theory. I'm writing up my summary as I understand it.
There are four instances of blocks like this in
CNSoilMatrix()(one each for C, C13, C14, and N):CTSM/src/soilbiogeochem/CNSoilMatrixMod.F90
Lines 613 to 618 in 2ba8531
Those four blocks are where we end up seeing the "line 973" error. Claude suggested that the race condition happens when:
IsValuesSetSM(), gets.false.and thus callsSetValueCopySM(), which ends by callingSetValueSM(), which starts to fill (in this order) the metadataNEthenRI+CI.IsValuesSetSM()call after Thread A has filledNEbut before it's finishedRIandCI. It entersIsValuesSetSM(), sees thatNEisn'tempty_int, and returns.true.. Thread B then takes the other path of the aboveifstatement, callingSPMP_B_ACC, which fails at line 973: an assertion involvingRI, which Thread A hadn't finished filling.Claude suggests that changing the above code (including all four locations) to
would solve the problem by forcing every thread to fill the metadata itself on the first timestep of a (startup/hybrid) run. I think that makes sense, but I haven't tested it.
Resolve ESCOMP#3360 (matrix threaded race condition)
Resolve ESCOMP#3360 (matrix threaded race condition)