Skip to content

Regional CTSM Simulations and Capability of Creating Mesh Files - #1892

Merged
adrifoster merged 216 commits into
ESCOMP:b4b-devfrom
negin513:subset_mesh_dask
Apr 25, 2024
Merged

adrifoster merged 216 commits into
ESCOMP:b4b-devfrom
negin513:subset_mesh_dask

Conversation

@negin513

@negin513 negin513 commented Nov 7, 2022 •

Copy link
Copy Markdown
Contributor

Description of changes

Quick Summary:

This PR includes:
1 - Script that enables our users to easily set up and run regional CTSM simulations.
2- Python class and scripts for creating a mesh file from any netCDF file and visualize it.

  • Detailed instructions for running CTSM regional cases are available in this link.

  • Instructions on how to use ./mesh_maker.py for different domains including one example WRF domain are available here.

Additional Details:

In this PR, we first introduced mesh_type python class and its new capabilities.
This new mesh object enables our users to create a mesh file from any netCDF files with either 1D or 2D latitude and longitude coordinates.

  • mesh_type is implemented in ./subset_data to create mesh file for surface dataset for easily running regional cases. Next, we added additional capabilities to automatically set up usermods necessary to run a regional case. This enable our users to create and run a regional case with 2 commands.
  • For other purposes, mesh_type is implemented in a script called mesh_maker.py to create a mesh file from any netCDF file.

Please note that the method that is used for creating a mesh file is different from #1735, where we subset a global mesh file for a region.

Specific notes

Contributors other than yourself, if any: @swensosc

CTSM Issues Fixed :
Fixes #1513: Create ESMF mesh file for regional grids
Fixes #1773: High Resolution Regional Simulations
Fixes #1397: Generating ESMFMesh files for WRF-CTSM cases --> discussions #1832

Are answers expected to change? No

Any User Interface Changes?
1- Changes in subset_data user interface with addition of --create-mesh
2- Adding mesh_maker.py with the following user interface:

usage: mesh_maker.py [-h] --input INPUT [--output OUTPUT] [--outdir OUT_DIR]
                     --lat LAT_NAME --lon LON_NAME [--mask MASK_NAME]
                     [--area AREA_NAME] [--overwrite] [-v]

|------------------------------------------------------------------|
|---------------------  Instructions  -----------------------------|
|------------------------------------------------------------------|
This script creates ESMF unstructured GRID (mesh file) from a netcdf
file with valid lats and lons. Provided lats and lons can be 1D or 2D.

For example for running WRF-CTSM cases, the user can create a mesh
file for their domain :
    ./mesh_maker.py --input wrfinput_d01 --output my_region
        --lat XLAT --lon XLONG --verbose

optional arguments:
  -h, --help        show this help message and exit
  --input INPUT     Netcdf input file for creating ESMF mesh.
  --output OUTPUT   Name of the ESMF mesh created.
  --outdir OUT_DIR  Output directory (only if name of output mesh is not
                    defined)
  --lat LAT_NAME    Name of latitude varibale on netcdf input file. If none
                    given, looks to find variables that include 'lat'.
  --lon LON_NAME    Name of latitude varibale on netcdf input file. If none
                    given, looks to find variables that include 'lon'.
  --mask MASK_NAME  Name of mask varibale on netcdf input file. If none given,
                    create a fake mask with values of 1.
  --area AREA_NAME  Name of area variable on netcdf input file. If none given,
                    ESMF calculates element areas automatically.
  --overwrite       If meshfile exists, overwrite the meshfile.
  -v, --verbose     Increase output verbosity

Testing performed, if any: regional simulations for different regions.

@negin513
negin513 requested a review from ekluzek November 7, 2022 21:56
@adrifoster

Copy link
Copy Markdown
Contributor

@negin513 I am working on bringing this PR in, after removing some of the conda and git workflow-related things @ekluzek was working on. This branch is here.

How would you like me to work from here? Shall I just push to your branch/PR? If so, I think I need collaborator access on your CTSM fork. Otherwise I can just make another PR to CTSM, which would include all of the code mods you made (and still be linked to you).

I'm happy to do whatever you'd like!

@negin513

Copy link
Copy Markdown
Contributor Author

Hello @adrifoster , Thanks! I added you to the collaborators on the fork.

@negin513

negin513 commented Mar 14, 2024 •

Copy link
Copy Markdown
Contributor Author

should we remove the --create-domain feature of this work since mct is no longer being supported for CESM3?

No @wwieder and @samsrabin : for WRF-CTSM runs domain files are needed. So I would suggest keeping --create-domain to create domain files for WRF-CTSM

@negin513

Copy link
Copy Markdown
Contributor Author

@wwieder as I recall we still need the domain files. I think they are to create the mesh files, but don't quite remember.

Yes @ekluzek , domain files are needed for WRF-CTSM (if the procedure is still the same). ;-)

@negin513

Copy link
Copy Markdown
Contributor Author

Hello. Everyone,
Comparing the code with my last commit (3588342) , I noticed the dask arrays are converted to numpy arrays, which means it does not benefit from the lazy loading offered by dask.

I suggest reverting these back to use dask array instead, which should make subsetting for high resolution mesh faster.

For example:

git diff 543e424 python/ctsm/site_and_regional/mesh_type.py

-            self.center_lat2d = da.from_array(self.center_lats)
-            self.center_lon2d = da.from_array(self.center_lons)
+            # self.center_lat2d = da.from_array(np.array(self.center_lats))
+            # self.center_lon2d = da.from_array(np.array(self.center_lons))
+            self.center_lat2d = np.array(self.center_lats)
+            self.center_lon2d = np.array(self.center_lons)
....

I can start from [3588342](https://github.com/ESCOMP/CTSM/pull/1892/commits/358834295a2377bcd8edf03200f10a91ebff63fd) and check the changes made here. 

@adrifoster
adrifoster changed the base branch from master to b4b-dev April 23, 2024 17:36
@adrifoster

Copy link
Copy Markdown
Contributor

All system and python tests are passing. @ekluzek I think this just needs another review before it is ready to be merged.

@ekluzek ekluzek left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome, this is great to see. I resolved the conversations I had after checking.

I do have one change to do from the last conversation. Go ahead fix that and resolve and merge this in.

Comment thread python/ctsm/run_ctsm_py_tests.py Resolved
@samsrabin

samsrabin commented Apr 25, 2024 •

Copy link
Copy Markdown
Member

I think it would be best to get these instructions (once updated, if necessary) moved into the CTSM User's Guide as part of this PR. I can help if needed.

  • Detailed instructions for running CTSM regional cases are available in this link.
  • Instructions on how to use ./mesh_maker.py for different domains including one example WRF domain are available here.

Just wanted to re-raise this. It'd probably be best to get these brought in to the documentation with this PR, rather than letting it wait until users ask us questions about it. I can do this and submit a PR to negin513:subset_mesh_dask if you prefer, @adrifoster, although I'd need you to review to make sure the instructions are all correct.

@negin513

negin513 commented Apr 25, 2024 via email

Copy link
Copy Markdown
Contributor Author

@negin513

negin513 commented Apr 25, 2024 via email

Copy link
Copy Markdown
Contributor Author

@samsrabin

Copy link
Copy Markdown
Member

@negin513 If it's not too much trouble, that'd be great! As long as it can do things like linking to other notes/sections in notes, being included in chapter lists, etc.

@adrifoster
adrifoster merged commit c2fc18f into ESCOMP:b4b-dev Apr 25, 2024
@samsrabin samsrabin added the science Enhancement to or bug impacting science label Aug 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement new capability or improved behavior of existing capability science Enhancement to or bug impacting science

Projects

Status: Ready to eat (Done!)
Status: Done (non release/external)

10 participants