Skip to content

Threaded test that failed hanging on Derecho with intel compiler #614

Open

Description

@ekluzek

Running ctsm5.4.021 with mizuRoute v3.0.1 one test in the mizu testlist failed.

SMS_P64x18.f19_f19_rMERIT_mg17.I2000Clm60SpMizGs.derecho_intel.mizuroute-default ( RUN)

This might just be something about the PE layout for this case It looks like it's actually dying in DATM. riof.log is empty, lnd.log is midway in the initialization phase.

atm.log ends with:

(atm_comp_nuopc):(InitializeAdvertise) datm datamode = CLMNCEP
 ATM: PIO numiotasks=           9
 ATM: PIO stride=           7
 ATM: PIO rearranger=           1
 ATM: PIO root=           1
(dshr_mesh_init) (dshr_mod:dshr_mesh_init) obtained ATM mesh and mask from /glade/campaign/cesm/cesmdata/inputdata/share/meshes/fv1.9x2.5_141008_ESMFmesh.nc
(shr_stream_init_from_xml)  getting calendar for stream 1

cesm.log talks about a fatal error for SIGTERM and process killed.

cesm.log has this in it:

dec1654.hsn.de.hpc.ucar.edu 63: libpioc.so         0000145997411633  PIOc_openfile         Unknown  Unknown
dec1654.hsn.de.hpc.ucar.edu 63: libpiof.so         000014599781851C  piolib_mod_mp_pio     Unknown  Unknown
dec1654.hsn.de.hpc.ucar.edu 63: cesm.exe           000000000119F2FF  dshr_stream_mod_m        1666  dshr_stream_mod.F90
dec1654.hsn.de.hpc.ucar.edu 63: cesm.exe           00000000011A6D60  dshr_stream_mod_m         453  dshr_stream_mod.F90
dec1654.hsn.de.hpc.ucar.edu 63: cesm.exe           0000000001193BB8  dshr_strdata_mod_         235  dshr_strdata_mod.F90
dec1654.hsn.de.hpc.ucar.edu 63: cesm.exe           0000000000573283  atm_comp_nuopc_mp         462  atm_comp_nuopc.F90

Activity

  1. added theissue type on Mar 8, 2026
  2. ekluzek commented on Mar 10, 2026

    @ekluzek
    CollaboratorAuthor

    This is likely a bad PE layout. The standard test that works is:

    SMS.f19_f19_rMERIT_mg17.I2000Clm60SpMizGs.derecho_intel.mizuroute-default

    which has

    ./pelayout 
    Comp  NTASKS  NTHRDS  ROOTPE PSTRIDE
    CPL :   1536/     1;    128      1
    ATM :    128/     1;      0      1
    LND :   1536/     1;    128      1
    ICE :   1536/     1;    128      1
    OCN :   1536/     1;    128      1
    ROF :   1536/     1;    128      1
    GLC :   1536/     1;    128      1
    WAV :   1536/     1;    128      1
    ESP :      1/     1;      0      1
    ESMF_AWARE_THREADING is False
    ROOTPE is with respect to 128.0 tasks per node

    This case is using 64 MPI tasks and 18 threads, so a total of 1152. But, probably 64 tasks is too few to get everything in memory. So how about something like 160 MPI tasks, and 16 threads?

  3. ekluzek commented on Mar 24, 2026

    @ekluzek
    CollaboratorAuthor

    We need to be able to have a setup where, mizuRoute is running with lots of threads, but other components with just MPI tasks. This is something that would go into CTSM into config_pes.xml.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions