Repository navigation
Open
Labels
enhancementnew capability or improved behavior of existing capabilitynew capability or improved behavior of existing capability
Description
We need a clearer workflow for how users can run high resolution regional cases.
Briefly, Xiulin is wanting to run a high resolution CTSM-FATES case over regions of California with DATM inputs that are archived from a WRF simulation. @lmkueppers and @ckoven is this something you've done previously? Moving forward, encouraged use of nuopc, and thought the following would need to be done:
- Subset surface data (from NLDAS) to the region of interest, given Some issues with creating surface datasets for the nldas2 grid... #1408, I wonder if this will be very straightforward?
- Create a mesh file, Subset mesh files for regional cases and add user-mods for regional cases. #1735 (the mesh file for nuopc is similar in function to the old domain file for mct, but is created differently).
- Create a regional case that points to the surface data, mesh, and datm files. You can point to the correct DATM files using the datm.streams.xml file in a nuopc case. This may be easier to set up with a usermods_dir.
Here are the questions that may trip us up.
- Are the existing surface data and datm files at the same resolution as the NLDAS grid?
- How does CLM handle information on soil properties coming from NLDAS (presumably they aren't the same)?
- @negin513 and @adrifoster, can the ./subsetdata script be manually changed to create a mesh and surface dataset from higher resolution base maps (3 or 9 km NLDAS). If so,
- Do we have an appropriate resolution mesh file to subset for this regional case?
- @ckoven , masking out regions within the region may be a little trickier. I'd assume this is done with the surface dataset and mesh files? @ekluzek may have ideas based on how we're creating the sparse grid for nuopc.
I'm sure there are additional considerations I'm not thinking about here, but wanted to move this discussion onto github to avoid confusion
Reacted by Gregory Lemieux
Metadata
Metadata
Assignees
Labels
enhancementnew capability or improved behavior of existing capabilitynew capability or improved behavior of existing capability
Activity
ckoven commentedon Jun 3, 2022
@wwieder thanks!
No, the resolution is different, and the WRF data is on a non-lat/lon grid. But we can probably make use of the NUOPC runtime interpolation?
Assuming that we also need to translate the native WRF output to DATM format (and subset it at the same time to match the boundaries of the CTSM simulation), and that the format hasn't changed at all with the NUOPC update.
cc @XiulinGao
XiulinGao commentedon Jun 5, 2022
@wwieder Thanks Will! and Charlie is correct. WRF is on curvilinear grid. In the forcing data file, lon/lat are only index of each grid, one needs an external file to refer back to the actual grid center. Also, WRF has a very differnt formating that their time information is hiding in the file name, so each file is only for one hour and there's no actual time dimension in the data (if file name does not count). For running CLM1PT, we might still need to reprocess these huge amount of files so they are in the correct format required by CLM1PT DATM mode.
glemieux commentedon Jun 10, 2022
@XiulinGao during a conversation with @wwieder in a separate email thread, he was wondering if you or Lara have experience with high res regional cases using WFR forcing. Having some example cases using this would be helpful.
XiulinGao commentedon Jun 10, 2022
@glemieux Actually no. This idea about using WRF for high res regional simulation just came up during our coversation few weeks ago. The purpose of doing this is to get more pixels for running our grass only simulations in California so that we can do model benchmark with enough pixels for only grasslands regions.I do have the archived WFR forcing available on cloud if that's something also can be helpful. Polly might be the person to ask as I know she has done some high resolution regional runs but not with WRF forcing I assume.
lmkueppers commentedon Jun 11, 2022
As Xiulin says, Polly has been running regional CLM-FATES simulations in California for an irregular domain. She is not using NUOPC. She is running at a fine resolution (4km) using custom boundary conditions.
I've asked our WRF collaborators (Stefan Rahimi and Alex Hall) if they have any other collaborators who have used their WRF output for land surface model simulations. Will let you know.
wwieder commentedon Jun 13, 2022
@ekluzek mentioned this in an email, but it seemed worth noting here too. Actually following through with this suggestion may require some additional discussion.
The issue already has this, but subset_data should be able to subset our NLDAS2 data fine, but they need to provide WRF data for the forcing. So they'll have to convert WRF history files into datasets that look like our forcing datasets. That's likely to be a non-trivial task.
slevis-lmwg commentedon Jun 14, 2022
I posted a comment in #1408 about successfully generating this surface dataset using the new mksurfdata_esmf tool:
`fsurdat = '/glade/work/slevis/git/mksurfdata_toolchain/tools/mksurfdata_esmf/surfdata_0.125nldas2_hist_78pfts_CMIP6_2000_c220614.nc'
slevis-lmwg commentedon Jun 14, 2022
I think this is the same as the mesh file that I see in the mksurfdata_esmf namelist (mentioned in the previous post when I generated the fsurdat):
/glade/p/cesm/cseg/inputdata/share/meshes/0.125nldas2_ESMFmesh_cd5_241220.ncslevis-lmwg commentedon Jun 14, 2022
True, so generating a nldas2 surface dataset (my second-to-last post) is unnecessary because the subset_data tool can subset an existing nldas2 fsurdat file.
XiulinGao commentedon Jun 15, 2022
According to Mariana's talk this morning at the CESM workshop, it seems that to use WRF forcing we should use CDEPS: https://escomp.github.io/CDEPS/html/index.html. And it also seems like we have to reprocess the WRF forcing into data files that can be used.
mvertens commentedon Jun 15, 2022
@XiulinGao - I want to confirm that you are aiming to using WRF forcing in an "I" compset simulation where the input forcing comes from DATM. Is the WRF output on the same grid as CLM?
XiulinGao commentedon Jun 15, 2022
@mvertens Yes, correct. No, the WRF output is on a curvilinear grid not the regular longitude and latitude used by CLM. So one of the questions we have is that does the NUOPC regridding is able to handle this curvelinear grid used by WRF and do the regridding at runtime or we have to reproject WRF to regular, even spaced grid ourselves offline. I also have a very specific question about what exact format the forcing should be in? I know that with the mct driver to run DATM with CLM1PT (assuming we are still using the same DATM mode for this case?) mode there's specific requirement for how one should prepare their own forcing files in certain way, but I didn't see that information when I went o checkout CDEPS. But very likely that we still have to reprocess our WRF foricng files even reprojection is nor required. Thanks!
1 remaining item
XiulinGao commentedon Jun 15, 2022
Thanks @mvertens ! That makes it more clear now. I think one of the most important things to do now is to reformat all the WRF forcing files into the data files that can be read in. One last question, by any chance that you know what will be the easist way to mask out regions based on vegetation cover? For instance, we'd like to run land only simulations withtin grid cells that have grass cover > 80% in California region. I assume we'll have to make changes to both the mesh and surface files. But how should we do that with NUOPC?
ekluzek commentedon Jun 15, 2022
@XiulinGao we have new capability to collapse vegetation types in gridcells down to remove complexity. I think this might be part of what you are referring to.
If I understand your statement "only run grid cells where grass is > 80% of the gridcell" I think you want to only run grass PFT gridcells. So you'd mask out non-grass gridcells.
One way to do this would be to tell CTSM to run for only the dominant landunit type in the gridcell, as well as only for the dominant vegetation type in the gridcell. This would still run for non-grass grid cells (including urban and lake, and other landunits, as well as other vegetation types), but it would only run with one type so would be faster. The next step would be to mask out running for the non-grass gridcells, which you would then do by changing the mesh file.
So I would suggest start with adding this to your user_nl_clm
n_dom_landunits = 1
n_dom_pfts = 1
and run that way. Then you could figure out what gridcells you want to mask out and create a new mesh file that masks out the non-grass gridcells.
wwieder commentedon Jun 15, 2022
This may be a good idea, @ekluzek, but since they're interested in FATES I'm not sure single PFT grid cells are desired?
ekluzek commentedon Jun 15, 2022
n_dom_landunits should work with FATES (although we don't test it). n_dom_pfts won't help for FATES nor work correctly for FATES, but you could do a non-FATES simulation just to figure out what gridcells to mask out. We are adding something that will prevent users from trying to use n_dom_pfts with FATES. It will likely work that way, just not really do anything. But ,doing a small non-FATES simulation should help you to figure out the grid-cells you want to focus on though.
dlawrenncar commentedon Jun 15, 2022
XiulinGao commentedon Jun 16, 2022
@ekluzek I'm running FATES with only one grass PFT, and interested in using only grassland regions for model benchmarking (so exclude forested area, croplands etc.). So yes, basically we'd want to have a mesh file where non-grass grids are masked out using a threshold value of grass cover. We could of course just run the whole CA regions and then mask out non-grassland grids using land cover data (e.g. NLCD), but since we are doing high resolution runs this would be computationally expensive. I think this might be a good way to go. But what about using land cover information to mask out non-grass grids in the mesh file directly? Is that possible? if it is possible, any change need to be made for surface data as well?
David mentioned sparse grid, which I'm not familiar with. but it seems that with this approach we can turn off selectively run in certain grids with some dort of definition?
Thank you all for the thoughts on this, the discussion has been very informative and helpful!
slevis-lmwg commentedon Jun 22, 2022
Update of progress (working on cheyenne)
TODO This workflow should ultimately appear in the User's Guide.
@XiulinGao provided three WRF files with three grids of interest:
wrfinput_d01, wrfinput_d02, wrfinput_d03A) I renamed the files:
wrfinput_d01.nc, wrfinput_d02.nc, wrfinput_d03.nc...and repeated the following steps for all three files:
ncwa -a Time wrfinput_d01.nc wrfinput_d01_2d_lat_lon.ncncks --rgr infer --rgr scrip=wrfinput_d01_scrip.nc wrfinput_d01_2d_lat_lon.nc wrfinput_d01_foo.nc*foo.nc contains metadata and will not be used.
/glade/u/apps/ch/opt/esmf-netcdf/8.0.0/intel/19.0.5/bin/bing/Linux.intel.64.mpiuni.default/ESMF_Scrip2Unstruct wrfinput_d01_scrip.nc wrfinput_d01_unstruct.nc 0To proceed with steps B, one needs the ctsm5.2.mksurfdata branch where the new mksurfdata_esmf is located:
B) Next made surface datasets (fsurdat files) using the mksurfdata_esmf tool
./gen_mksurfdata_build.sh./gen_mksurfdata_namelist.py --start-year 2000 --end-year 2000 --model-mesh-nx 109 --model-mesh-ny 104 --model-mesh /glade/work/slevis/data_UCB/ca_fates_input_data/wrfinput_to_meshes/wrfinput_d01_unstruct.nc./gen_mksurfdata_jobscript_single.py --number-of-nodes 24 --tasks-per-node 12 --namelist-file surfdata_wrfinput_d01_hist_78pfts_CMIP6_2000_c220622.namelistqsub mksurfdata_jobscript_singleC) Now run the CTSM with the new fsurdat files
./create_newcase --case ~/cases_FATES/test_wrfinput_d01 --compset I2000Clm50Fates --res CLM_USRDAT --run-unsupported./case.setupfsurdat = '/glade/work/slevis/data_UCB/ca_fates_input_data/fsurdat/wrfinput/surfdata_wrfinput_d01_hist_78pfts_CMIP6_2000_c220622.nc'./case.build./case.submitand the simulation test_wrfinput_d01 is still running. I did not change datm settings, so the model is reading global GSWP3 data in this test.env_mach_pes.xmlfor the d02 and d03 cases to run on more nodes and processors because they were running out of memory.P.S. CTSM version:
ctsm5.1.dev099-17-gd7bc3ba48XiulinGao commentedon Jun 22, 2022
@slevisconsulting Thank you Sam for the great work! Just for your information that wrfinput_d01 is in 45km resolution, so it's not a high resolution grids. But the whole workflow seems working pretty well! I'm still working on the reprocessing of the 9km res WRF forcing, which is moving quite slow. Hopefully we can put everything together to do a test run in California soon using WRF forcing. Of course, still have to figure out how to run the 'sparse grid' using some vegetation cover mask. cheers
slevis-lmwg commentedon Jul 15, 2022
Documenting process similar to above but starting from @XiulinGao's
wrf_CAgrid@XiulinGao provided monthly datm files for 1981 for testing.
@XiulinGao provided mesh file
wrf_CA_unstruct.ncgenerated from a datm file using steps A above.I can jump ahead to step A4. I'm documenting here only steps that differ from above:
B2)
./gen_mksurfdata_namelist.py --start-year 1981 --end-year 1981 --res wrf_CA --model-mesh /glade/work/slevis/data_UCB/ca_fates_input_data/datm_from_wrf/wrf_CA_unstruct.nc --model-mesh-nx 147 --model-mesh-ny 151 --nocropB3)
./gen_mksurfdata_jobscript_single.py --number-of-nodes 24 --tasks-per-node 12 --namelist-file surfdata_wrf_CA_hist_16pfts_CMIP6_1981_c220715.namelistC1)
./create_newcase --case ~/cases_FATES/wrf_CA_01 --compset I2000Clm50Fates --res CLM_USRDAT --run-unsupportedC3) Add this line to user_nl_clm:
fsurdat = '/glade/work/slevis/data_UCB/ca_fates_input_data/fsurdat/wrf_CA/surfdata_wrf_CA_hist_16pfts_CMIP6_1981_c220715.nc'C4) Update lines in env_run.xml:
C7) Simulation successful (5 days). CTSM version:
alpha-ctsm5.2.mksrf.06_ctsm5.1.dev090...but update
env_mach_pes.xmlto run faster on more nodes/procs andenv_mach_pes.xmlto run on the regular queue.D1) Update user_nl_datm_streams to include the next 3 lines three times (i.e. repeated for Precip and for TPQW):
D2) Update env_run.xml
Simulation successful (30 days)!
PS. @mvertens recommended changing PIO_TYPENAME to pnetcdf for faster i/o (that is, writing to history and restarts). I haven't tested, yet.
slevis-lmwg commentedon Jul 18, 2022
Next steps from 2022/7/19 meeting with @XiulinGao @lmkueppers @ckoven @pollybuotte
In the case that I described above (wrf_CA_01) which runs on a wrf grid (147x151), I will limit the simulation to grid cells with C3 grass >= 80% according to the fsurdat pft data:
Turns out this step was unnecessary (redundant). It would have been necessary if we were modifying fsurdat variables in the land cells of interest or if the land cells of interest were located where the model normally expects ocean.
/glade/work/slevis/data_UCB/ca_fates_input_data/datm_from_wrf/wrf_CA_c3grass_unstruct.nc
Simulation completed successfully:
/glade/scratch/slevis/wrf_CA_01/run/wrf_CA_01.clm2.h0.2000-01.nc
slevis-lmwg commentedon Jul 28, 2022
I am consolidating the instructions that I have posted over time in this issue
in this google doc.
slevis-lmwg commentedon Oct 19, 2022
I have updated the above mentioned google doc to include a simpler set of instructions for running regional CTSM on a regular 1-degree grid. Comments always welcome.
glemieux commentedon Oct 20, 2022
How about transfering the google doc to a Github Discussion in the Q&A section? I.e similar to @adrifoster single point NUOPC instructions: #1833
slevis-lmwg commentedon Oct 20, 2022
Seems like a good idea. I'm open to that.
slevis-lmwg commentedon Nov 23, 2022
Performance optimization is: