Skip to content

PE layout does not work for SMS_P14080x1_Ln9.mpasa3p75_mpasa3p75_mt13.I2000Clm51SpRs.derecho_intel.clm-clm51cam6LndTuningMode #2370

Closed

Description

@slevis-lmwg

Brief summary of bug

In the ctsm_sci test-suite for ctsm5.2, I added new grids that we wish to test for.
I managed to get ctsm_sci to pass, except for ultra high resolution grid mpasa3p75, which is still considered highly experimental.
@briandobbins has run this grid, but as part of a larger run on 40K cores with an active atmosphere.

General bug information

CTSM version you are using:
alpha-ctsm5.2.mksrf.19_ctsm5.1.dev163-55-g1f4611fb7

Activity

  1. added
    investigationNeeds to be verified and more investigation into what's going on.
    testingadditions or changes to tests
    nextthis should get some attention in the next week or two. Normally each Thursday SE meeting.
    on Feb 15, 2024
  2. slevis-lmwg commented on Feb 15, 2024

    @slevis-lmwg
    ContributorAuthor

    Initial discussion regarding this test starts here.

  3. removed
    nextthis should get some attention in the next week or two. Normally each Thursday SE meeting.
    on Feb 22, 2024
  4. added
    closed: wontfixWe won't fix this issue, because it would be too difficult and/or isn't important enough to fix
    and removed
    investigationNeeds to be verified and more investigation into what's going on.
    on Apr 8, 2024
  5. ekluzek commented on Apr 8, 2024

    @ekluzek
    Contributor

    Here's some notes about this from @briandobbins as well as some other issues.

    This can be run with an enormous number of processors, but that isn't the standard thing we are doing in CESM/CTSM. So I'm marking this as a "WONTFIX" since it's experimental, and the number of required processors means we won't be doing this with any standard testing. So I'm removing the test for this as well. We will just make sure the datasets needed are provided in the model and let other test it.

    Hi all,

    And an update on my side -- I've been able to run the 3.75km I2000Clm50Sp case on 10K cores, but going to the 40K - 160K range I do hit problems. As indicated in my earlier email, we've hit a number of these that I believe are related to errors interpolating very small 1-degree input files onto very large grids, where the number of source points is fewer than the number of MPI ranks. An example from the atmosphere is easier - with NDEP, using a 1-degree FV grid (192x288), we have 55,296 grid points - and, when going above 55,296 cores, we crash, because the interpolation code never anticipated having <1 points on a given rank. The land is a bit different, but I suspect the urban time-varying data file ( CLM50_tbuildmax_Oleson_2016_0.9x1.25_simyr1849-2106_c160923.nc ) is also having these problems.

    .
    .
    .

    • Brian
  6. ekluzek commented on Apr 12, 2024

    @ekluzek
    Contributor

    Note, here that @fischer-ncar got this to work with 14k processors for an I compset as noted here:

    #2116 (comment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    closed: wontfixWe won't fix this issue, because it would be too difficult and/or isn't important enough to fixtestingadditions or changes to tests

    Type

    No type

    Projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions