Skip to content

Global moving nests: Fix coarse-to-fine updates - #1917

Open
J-Lentz wants to merge 20 commits into
NOAA-GFDL:mainfrom
J-Lentz:fix_c2f_updates
Open

J-Lentz wants to merge 20 commits into
NOAA-GFDL:mainfrom
J-Lentz:fix_c2f_updates

Conversation

@J-Lentz

@J-Lentz J-Lentz commented Sep 8, 2026 •

Copy link
Copy Markdown
Contributor

Description
In compute_overlap_coarse_to_fine, which determines overlaps for coarse-to-fine updates, the existing tile->tile coordinate transforms do not work when a nest's west or south halo overhangs across the coarse grid's tile boundary. This has been resolved by rewriting the coordinate transforms. The two existing coordinate transform functions, convert_index_to_nest and convert_index_to_coarse, have been replaced with a unified convert_index function.

While the old coordinate transforms assumed that data being transferred from a coarse PE to a fine PE would come either from the east (in the case of a horizontal overhang) or from the north (in the case of a vertical overhang), the new transforms allow data to be transferred from any adjacent tile on the coarse grid. However, large overhangs which extend beyond adjacent tiles are not yet supported with the new transforms. Therefore, the old transforms are kept as an alternative code path when multi-tile overhangs are needed.

How Has This Been Tested?
Existing unit tests pass. A new test has been added which tests C2F updates, halo updates, and F2C updates in a three-tile corner nest.

Checklist:

  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • Any dependent changes have been merged and published in downstream modules
  • New check tests, if applicable, are included
  • make distcheck passes

Jesse Lentz added 4 commits September 8, 2026 11:54
Rewrite the tile->tile coordinate transforms in mpp_define_nest_domains.
Replace `convert_index_to_nest` and `convert_index_to_coarse` with a
single generalized `convert_index` function. The old coordinate
transforms assumed that data being transferred from a coarse PE to a
fine PE would come either from the east (in the case of a horizontal
overhang) or from the north (in the case of a vertical overhang). The
old transforms did not allow data to be transferred from the south or
from the west. The new transforms allow data to be transferred to the
nest from any adjacent coarse-grid tile; however, large overhangs which
span multiple tiles are no longer supported.
The test_mpp_nesting unit test currently has a test case where a nest
spans two full tiles in the horizontal direction. This case is no longer
supported, as the new coordinate transforms can only pull data from
adjacent tiles. Therefore, the size of the nest has been reduced so that
its overhang doesn't extend beyond the adjacent tile.
`compute_overlap_coarse_to_fine` seems to erroneously use `shalo` and
`nhalo` respectively to adjust the size of the north and south halo
buffers in the horizontal dimension, rather than `ehalo` and `whalo` as
one would expect. This bug has no effect in the normal scenario where
all halo sizes are the same, but it would produce incorrect halo regions
if non-equal halo sizes were used.
Modify coordinates using ishift and jshift before, rather than after,
coordiante transforms are performed.
Jesse Lentz added 11 commits September 8, 2026 15:52
Add doxygen comments to `convert_index`.
Add a new test which performs a C2F update-halo update-F2C update cycle
on a corner nest. Add `answers_mod` for the purpose of generating and
verifying checksums of numerical answers.
Rather than the non-standard `ishc` function, use the standard `ishftc`
function to perform a bit-shift in the checksum calculation. This fixes
the build with GFortran.
Restore the old `compute_overlap` routines as an alternative code path
when multi-tile overhangs are needed. Remove these eventually, once the
new `compute_overlap` routines are generalized to handle this case.

@rem1776 rem1776 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks great! just a couple small things, mostly build related.

Comment thread test_fms/common/answers.F90 Outdated
use fms_mod, only: check_nml_error
use fms_string_utils_mod, only: string
#ifdef use_libMPI
use mpi_f08, only: mpi_gather, MPI_INTEGER8

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can this use mpp_gather instead? Then it should work if built without MPI too.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed to mpp_gather in a7c5a80. This depends on #1939 because the send buffer (ref) is 1D while the receive buffer (answers) is 2D.

Comment thread test_fms/mpp/test_mpp_nesting.sh Outdated

cp ../test_corner_nest.nml input.nml
test_expect_success "Three-tile corner nest (C2F, halo update, F2C)" '
mpirun -n 36 ../test_corner_nest

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like this ones failing the GNU + Autotools CI testing. I'm guessing it's from the core count since 36 is past the upper limit of what seems to work while oversubscribed on the runners, the next biggest test uses 32. If it could be scaled down a little it might fix the errors we are seeing.

This is outside the scope of this PR but maybe it would be good to separate some of our bigger tests from the standard suite. FRE-NCtools does something similar with make check and make check-expensive.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reduced to 18 PEs in 30244b7.

Comment thread test_fms/mpp/test_mpp_nesting.sh Resolved
Comment thread test_fms/mpp/Makefile.am Resolved
Jesse Lentz added 5 commits October 29, 2026 11:34
The counting of fine PEs in `mpp_broadcast_domain_nest_fine` has been
simplified and corrected for the case where multiple nests are defined
on multiple tiles of a common domain. Previously, each nest had to be
defined using its own single-tile domain.
Reduce the layout for coarse-grid tiles from 2x2 to 1x1. This reduces
the PE count from 36 to 18. Hopefully this resolves the CI failure.
The non-`pelist` versions of mpp_gather have been modified to take
advantage of mpi_f08's assumed-rank arguments. `MPP_GATHER_1D_` and
`MPP_GATHER_1DV_` have been renamed to `MPP_GATHER_WRAP_` and
`MPP_GATHERV_WRAP_` respectively, since they can now accept send and
receive buffers of any rank. The `pelist` versions of mpp_gather have
not been modified. A pointer-based workaround has been introduced into
the non-MPI versions of MPP_GATHER_ and MPP_GATHERV_ so that the data
buffers can be copied without use of `select rank`.
In answers_mod, use mpp_gather instead of mpi_gather.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants