Skip to content

Resolve matrix threaded race condition - #4319

Open
samsrabin wants to merge 1 commit into
ESCOMP:b4b-devfrom
samsrabin:fix-matrix-threaded-race
Open

samsrabin wants to merge 1 commit into
ESCOMP:b4b-devfrom
samsrabin:fix-matrix-threaded-race

Conversation

@samsrabin

@samsrabin samsrabin commented Nov 13, 2026 •

Copy link
Copy Markdown
Member

Description of changes

There were four instances of blocks like this in CNSoilMatrix() (one each for C, C13, C14, and N):

if ( AKXcacc%IsValuesSetSM() )then
call AKXcacc%SPMP_B_ACC(num_soilc,filter_soilc,AKallsoilc)
else
! This should only happen on the first time-step
call AKXcacc%SetValueCopySM(num_soilc,filter_soilc,AKallsoilc)
end if

Those four blocks were where we saw the "line 973" error from #3360. This was the bad result of a race condition that happens when:

  1. Thread A calls IsValuesSetSM(), gets .false. and thus calls SetValueCopySM(), which ends by calling SetValueSM(), which starts to fill (in this order) the metadata NE then RI+CI.
  2. Thread B gets to the IsValuesSetSM() call after Thread A has filled NE but before it's finished RI and CI. It enters IsValuesSetSM(), sees that NE isn't empty_int, and returns .true.. Thread B then takes the other path of the above if statement, calling SPMP_B_ACC, which fails at line 973: an assertion involving RI, which Thread A hadn't finished filling.

By forcing each thread to set or reset the sparse_matrix_type metadata, we can avoid threads seeing partially-filled metadata from other in-progress threads.

Specific notes

Contributors other than yourself, if any:

  • Claude: Fable 5.1 and Opus 5.5 for identifying the problem and reviewing my fix.

CTSM issues resolved or otherwise addressed, if any:

Any user interface changes (namelist or namelist defaults changes)? No

Testing planned or performed, if any:

  • Tested using ERP_D_P64x2_Ld3.f10_f10_mt201.I2000Clm50BgcCru.derecho_intel.clm-noFUN_flexCN--clm-matrixcnOn_ignore_warnings. 5/10 runs failed at ctsm5.4.069; 0/10 runs failed on this commit.
  • aux_clm

Requirements before merge:

  • The code in this PR branch builds with no errors.
  • The code in this PR branch runs with no errors. Briefly describe tested configuration(s): ERP_D_P64x2_Ld3.f10_f10_mt201.I2000Clm50BgcCru.derecho_intel.clm-noFUN_flexCN--clm-matrixcnOn_ignore_warnings
  • This either (a) does not change answers, (b) it only changes answers at roundoff level, or (c) I have performed a scientific evaluation of the answer changes. Which?:
  • I have reviewed relevant parts of the CLM documentation Tech Note or User's Guide to determine if anything needs to be changed or added. If it does, describe:
  • This PR either (a) does not create a need to update the documentation or (b) includes required documentation updates (see guidelines for contributing documentation). Which?: No need

@samsrabin samsrabin self-assigned this Nov 13, 2026
@samsrabin samsrabin added code health improving internal code structure to make easier to maintain (sustainability) next this should get some attention in the next week or two. Normally each Thursday SE meeting. b4b bit-for-bit labels Nov 13, 2026
@samsrabin samsrabin linked an issue Nov 13, 2026 that may be closed by this pull request
@samsrabin
samsrabin force-pushed the fix-matrix-threaded-race branch from 4b98e82 to e061b45 Compare November 13, 2026 22:45
By forcing each thread to set or reset the sparse_matrix_type metadata, we can avoid threads seeing partially-filled metadata from other in-progress threads.

Tested using ERP_D_P64x2_Ld3.f10_f10_mt201.I2000Clm50BgcCru.derecho_intel.clm-noFUN_flexCN--clm-matrixcnOn_ignore_warnings:
- 5/10 runs failed at ctsm5.4.069
- 0/10 runs failed on this commit
@samsrabin
samsrabin force-pushed the fix-matrix-threaded-race branch from e061b45 to 22d5176 Compare November 13, 2026 22:46

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

b4b bit-for-bit code health improving internal code structure to make easier to maintain (sustainability) next this should get some attention in the next week or two. Normally each Thursday SE meeting.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Race condition with a threaded test with matrix on?

1 participant