Skip to content

enable NVHPC support - #719

Merged
billsacks merged 1 commit into
ESCOMP:mainfrom
ACCESS-NRI:feat/nvhpc-support
Nov 9, 2026
Merged

billsacks merged 1 commit into
ESCOMP:mainfrom
ACCESS-NRI:feat/nvhpc-support

Conversation

@edoyango

@edoyango edoyango commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Description of changes

This fixes an issue found in the NVHPC nvfortran compiler when trying to execute CMEPS in the ACCESS-OM3 suite.

The specific issue occurs when pelist is resized at initialization to match the number of procs/threads. NVHPC doesn't seem to do the resize properly when using the deallocate/allocate idiom. However, the implied do assignment reallocation works.

A more specific description is here on the nvidia forums. NVIDIA has opened a bug report, but historically, time frames for fixes vary.

Edit with further details about the nvfortran issue:

The implied do array assignment (re)allocates petlist to the right size. nvfortran 26.5 at -O2 and above miscompiles an explicit deallocate/allocate of petlist here followed by an element-wise fill: the fill writes the new list into the old, freed array, corrupting the heap.

Specific notes

No changes to answers or user interface

Testing performed

ACCESS-OM3 100km ryf was tested with ifx and gfortran for 30 days. bitwise identical answers for ifx.

The repo's own tests don't catch this error, so they pass both before and after this PR.

@anton-seaice

Copy link
Copy Markdown
Contributor

@billsacks - if you have a minute to look at this that would be great

i think it's the first use of resize on assignment in CMEPS but its been a feature since Fortran 2003

@billsacks billsacks left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The core of this fix seems fine. I have a couple of cleanup requests.

Comment thread cesm/driver/esm.F90 Resolved
Comment thread cesm/driver/esm.F90 Resolved
@billsacks

Copy link
Copy Markdown
Member

@mvertens and @fischer-ncar - I have reviewed this PR and am satisfied with the changes. Given how close we are to release, though, I wanted to also loop you both in to help judge what testing should be done before we merge this.

@fischer-ncar

Copy link
Copy Markdown
Collaborator

@billsacks The only NVHPC tests that we do are

ERS_C3.ne30_g17.A.derecho_nvhpc.drv-asyncio1pernode
ERS_Ln9.ne30pg3_ne30pg3_mg17.F2000dev.derecho_nvhpc.cam-outfrq9s_gpu_pcols760

@billsacks

Copy link
Copy Markdown
Member

@fischer-ncar - even though this is about fixing nvhpc, the change impacts all compilers. I have reviewed the changes and it looks like it maintains previous behavior, but given how close we are to release, I'll be more comfortable after this has gone through some testing. At the very least, we should keep an eye on aux_cime_baselines after merging this. Does that feel sufficient to you or would you like some additional proactive testing on this? Another possibility is to wait to bring this in until after we've created the cmeps release branch (which I'll probably do in the next week or so) and live with the issue that cesm3's cmeps won't build with particular nvhpc versions. Do you have any preferences?

@fischer-ncar

Copy link
Copy Markdown
Collaborator

@billsacks after looking at this closer I would like to have MCC.ne3pg3_t232.BHISTC_LT_DART.derecho_intel.allactive-DART_BHIST_lowres--mom-tx10deg tested before the merge. Then I'll look at the nightly aux_cime_baselines after the merge. I don't want to wait until after a cmeps release branch to merge a fix for a compiler.

@fischer-ncar

Copy link
Copy Markdown
Collaborator

I ran MCC.ne3pg3_t232.BHISTC_LT_DART.derecho_intel.allactive-DART_BHIST_lowres--mom-tx10deg and it passed. I'll look at the nightly aux_cime_baselines after this PR is merged.

This introduces a workaround for a compiler bug in nvhpc (tested with 26.5 and 26.9 - latter being the latest).
NVHPC doesn't seem to properly resize  with the `deallocate allocate` pattern - NVHPC keeps `petlist`
the same size, and accesses cause a segfault due to accessing out-bounds memory.

The fix here is to use an auto reallocation idiom instead. This was tested to work with gfortran and ifx as
well. Performance shouldn't be a concern as the changed code is only called in initialization.
@edoyango

edoyango commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor Author

The core of this fix seems fine. I have a couple of cleanup requests.

Thanks for reviewing, I've updated the commit to address the cleanup requests.

@billsacks billsacks left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the changes. This looks good now!

A small note for future contributions: I find it much easier to iterate on a PR if you avoid force pushing, but instead add a new cleanup commit on top of the old commit. This post has a good explanation. (There are workarounds that make it slightly easier than that post implies, but it's still much more difficult to review changes following a force push.) You're not at all alone in doing this, and no worries this time.

@edoyango

edoyango commented Nov 9, 2026

Copy link
Copy Markdown
Contributor Author

@billsacks
apologies! The force pushing is a bad habit I've picked up from working on MOM6 😅.

@billsacks

Copy link
Copy Markdown
Member

apologies! The force pushing is a bad habit I've picked up from working on MOM6

No worries. I realize different repositories have different preferences there, so I don't hold it against you or anyone... but also like to state my preferences... I should probably put that in a CONTRIBUTING guide rather than just saying it after the fact....

@billsacks
billsacks merged commit f83f64b into ESCOMP:main Nov 9, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants