Repository navigation
Conversation
Rewrite the tile->tile coordinate transforms in mpp_define_nest_domains. Replace `convert_index_to_nest` and `convert_index_to_coarse` with a single generalized `convert_index` function. The old coordinate transforms assumed that data being transferred from a coarse PE to a fine PE would come either from the east (in the case of a horizontal overhang) or from the north (in the case of a vertical overhang). The old transforms did not allow data to be transferred from the south or from the west. The new transforms allow data to be transferred to the nest from any adjacent coarse-grid tile; however, large overhangs which span multiple tiles are no longer supported.
The test_mpp_nesting unit test currently has a test case where a nest spans two full tiles in the horizontal direction. This case is no longer supported, as the new coordinate transforms can only pull data from adjacent tiles. Therefore, the size of the nest has been reduced so that its overhang doesn't extend beyond the adjacent tile.
`compute_overlap_coarse_to_fine` seems to erroneously use `shalo` and `nhalo` respectively to adjust the size of the north and south halo buffers in the horizontal dimension, rather than `ehalo` and `whalo` as one would expect. This bug has no effect in the normal scenario where all halo sizes are the same, but it would produce incorrect halo regions if non-equal halo sizes were used.
Modify coordinates using ishift and jshift before, rather than after, coordiante transforms are performed.
Add doxygen comments to `convert_index`.
Add a new test which performs a C2F update-halo update-F2C update cycle on a corner nest. Add `answers_mod` for the purpose of generating and verifying checksums of numerical answers.
Rather than the non-standard `ishc` function, use the standard `ishftc` function to perform a bit-shift in the checksum calculation. This fixes the build with GFortran.
Restore the old `compute_overlap` routines as an alternative code path when multi-tile overhangs are needed. Remove these eventually, once the new `compute_overlap` routines are generalized to handle this case.
rem1776
left a comment
There was a problem hiding this comment.
looks great! just a couple small things, mostly build related.
| use fms_mod, only: check_nml_error | ||
| use fms_string_utils_mod, only: string | ||
| #ifdef use_libMPI | ||
| use mpi_f08, only: mpi_gather, MPI_INTEGER8 |
There was a problem hiding this comment.
can this use mpp_gather instead? Then it should work if built without MPI too.
|
|
||
| cp ../test_corner_nest.nml input.nml | ||
| test_expect_success "Three-tile corner nest (C2F, halo update, F2C)" ' | ||
| mpirun -n 36 ../test_corner_nest |
There was a problem hiding this comment.
Looks like this ones failing the GNU + Autotools CI testing. I'm guessing it's from the core count since 36 is past the upper limit of what seems to work while oversubscribed on the runners, the next biggest test uses 32. If it could be scaled down a little it might fix the errors we are seeing.
This is outside the scope of this PR but maybe it would be good to separate some of our bigger tests from the standard suite. FRE-NCtools does something similar with make check and make check-expensive.
The counting of fine PEs in `mpp_broadcast_domain_nest_fine` has been simplified and corrected for the case where multiple nests are defined on multiple tiles of a common domain. Previously, each nest had to be defined using its own single-tile domain.
Reduce the layout for coarse-grid tiles from 2x2 to 1x1. This reduces the PE count from 36 to 18. Hopefully this resolves the CI failure.
The non-`pelist` versions of mpp_gather have been modified to take advantage of mpi_f08's assumed-rank arguments. `MPP_GATHER_1D_` and `MPP_GATHER_1DV_` have been renamed to `MPP_GATHER_WRAP_` and `MPP_GATHERV_WRAP_` respectively, since they can now accept send and receive buffers of any rank. The `pelist` versions of mpp_gather have not been modified. A pointer-based workaround has been introduced into the non-MPI versions of MPP_GATHER_ and MPP_GATHERV_ so that the data buffers can be copied without use of `select rank`.
In answers_mod, use mpp_gather instead of mpi_gather.
Description
In
compute_overlap_coarse_to_fine, which determines overlaps for coarse-to-fine updates, the existing tile->tile coordinate transforms do not work when a nest's west or south halo overhangs across the coarse grid's tile boundary. This has been resolved by rewriting the coordinate transforms. The two existing coordinate transform functions,convert_index_to_nestandconvert_index_to_coarse, have been replaced with a unifiedconvert_indexfunction.While the old coordinate transforms assumed that data being transferred from a coarse PE to a fine PE would come either from the east (in the case of a horizontal overhang) or from the north (in the case of a vertical overhang), the new transforms allow data to be transferred from any adjacent tile on the coarse grid. However, large overhangs which extend beyond adjacent tiles are not yet supported with the new transforms. Therefore, the old transforms are kept as an alternative code path when multi-tile overhangs are needed.
How Has This Been Tested?
Existing unit tests pass. A new test has been added which tests C2F updates, halo updates, and F2C updates in a three-tile corner nest.
Checklist:
make distcheckpasses