Repository navigation
enable NVHPC support - #719
Conversation
|
@billsacks - if you have a minute to look at this that would be great i think it's the first use of resize on assignment in CMEPS but its been a feature since Fortran 2003 |
billsacks
left a comment
There was a problem hiding this comment.
The core of this fix seems fine. I have a couple of cleanup requests.
|
@mvertens and @fischer-ncar - I have reviewed this PR and am satisfied with the changes. Given how close we are to release, though, I wanted to also loop you both in to help judge what testing should be done before we merge this. |
|
@billsacks The only NVHPC tests that we do are ERS_C3.ne30_g17.A.derecho_nvhpc.drv-asyncio1pernode |
|
@fischer-ncar - even though this is about fixing nvhpc, the change impacts all compilers. I have reviewed the changes and it looks like it maintains previous behavior, but given how close we are to release, I'll be more comfortable after this has gone through some testing. At the very least, we should keep an eye on aux_cime_baselines after merging this. Does that feel sufficient to you or would you like some additional proactive testing on this? Another possibility is to wait to bring this in until after we've created the cmeps release branch (which I'll probably do in the next week or so) and live with the issue that cesm3's cmeps won't build with particular nvhpc versions. Do you have any preferences? |
|
@billsacks after looking at this closer I would like to have MCC.ne3pg3_t232.BHISTC_LT_DART.derecho_intel.allactive-DART_BHIST_lowres--mom-tx10deg tested before the merge. Then I'll look at the nightly aux_cime_baselines after the merge. I don't want to wait until after a cmeps release branch to merge a fix for a compiler. |
|
I ran MCC.ne3pg3_t232.BHISTC_LT_DART.derecho_intel.allactive-DART_BHIST_lowres--mom-tx10deg and it passed. I'll look at the nightly aux_cime_baselines after this PR is merged. |
This introduces a workaround for a compiler bug in nvhpc (tested with 26.5 and 26.9 - latter being the latest). NVHPC doesn't seem to properly resize with the `deallocate allocate` pattern - NVHPC keeps `petlist` the same size, and accesses cause a segfault due to accessing out-bounds memory. The fix here is to use an auto reallocation idiom instead. This was tested to work with gfortran and ifx as well. Performance shouldn't be a concern as the changed code is only called in initialization.
48bb4d7 to
88ee4a3
Compare
Thanks for reviewing, I've updated the commit to address the cleanup requests. |
billsacks
left a comment
There was a problem hiding this comment.
Thanks for the changes. This looks good now!
A small note for future contributions: I find it much easier to iterate on a PR if you avoid force pushing, but instead add a new cleanup commit on top of the old commit. This post has a good explanation. (There are workarounds that make it slightly easier than that post implies, but it's still much more difficult to review changes following a force push.) You're not at all alone in doing this, and no worries this time.
|
@billsacks |
No worries. I realize different repositories have different preferences there, so I don't hold it against you or anyone... but also like to state my preferences... I should probably put that in a CONTRIBUTING guide rather than just saying it after the fact.... |
Description of changes
This fixes an issue found in the NVHPC nvfortran compiler when trying to execute CMEPS in the ACCESS-OM3 suite.
The specific issue occurs when
pelistis resized at initialization to match the number of procs/threads. NVHPC doesn't seem to do the resize properly when using thedeallocate/allocateidiom. However, the implied do assignment reallocation works.A more specific description is here on the nvidia forums. NVIDIA has opened a bug report, but historically, time frames for fixes vary.
Edit with further details about the nvfortran issue:
The implied do array assignment (re)allocates petlist to the right size. nvfortran 26.5 at -O2 and above miscompiles an explicit deallocate/allocate of petlist here followed by an element-wise fill: the fill writes the new list into the old, freed array, corrupting the heap.
Specific notes
No changes to answers or user interface
Testing performed
ACCESS-OM3 100km ryf was tested with ifx and gfortran for 30 days. bitwise identical answers for ifx.
The repo's own tests don't catch this error, so they pass both before and after this PR.