You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Need a process to subset ESMF mesh files from global ones for regional grids #1513
With the update to NUOPC we'll need a way for users to create ESMF mesh files from domain files (or from their grid) for any regional grids they create. This needs to be done for regional cases for CTSM, but also when using LILAC to run WRF with CTSM.
The current subset_data process creates domain files as well as surface datasets for regional areas. We'll need some steps to convert the domain file into an ESMF mesh. Or possibly we should directly create an ESMF mesh file, or maybe the global ESMF mesh file should also be subset (but because ESMF mesh files are 1D vector files and not 2D, this may be difficult).
This isn't a problem with single point simulations as there we provide the latitude and longitude with PTS_LAT and PTS_LON so a mesh file isn't used. But, it is a problem for regional grids of greater than a single point.
This likely isn't the way we want to go long-term, but in case it's helpful, here is the process I inherited from Mariana to convert a scrip grid file to an esmf mesh file:
Here's a script that @swensosc created to create a mesh based on the surface dataset that was created. This bypasses domain files, and bypasses subsetting a SCRIP grid file, so is a good approach to take. It should work for unstructured grids as well since the surface dataset is just a 1D vector in that case. But, we should test it on unstructured grids as well.
#! /usr/bin/env pythonimportsysimportstringimportsubprocessimporttimeimportargparseimportnumpyasnpimportnetCDF4asnetcdf4doTimer=False# input fileinfile='/fs/cgd/csm/inputdata/lnd/clm2/surfdata_map/release-clm5.0.18/surfdata_0.9x1.25_hist_16pfts_Irrig_CMIP6_simyr2000_c190214.nc'# output directorydir_output='./'# creation date command='date "+%y%m%d"'timetag=(subprocess.run(command,stdout=subprocess.PIPE,shell='True').stdout.decode('utf-8')).strip()
# output fileoutfile=dir_output+'ESMFmesh_ctsm_0.9x1.25_'+timetag+'.nc'#-- create mesh file from surface data file# (other file types may have different coordinate variables)f1=netcdf4.Dataset(infile, 'r')
lon_vname='LONGXY'lat_vname='LATIXY'xc=np.asfarray(f1.variables[lon_vname][:,],np.float64)
yc=np.asfarray(f1.variables[lat_vname][:,],np.float64)
xc_units=f1.variables[lon_vname].unitsxc_longname=f1.variables[lon_vname].long_nameyc_units=f1.variables[lat_vname].unitsyc_longname=f1.variables[lat_vname].long_namef1.close()
delx=np.abs(xc[0,1] -xc[0,0])
dely=np.abs(yc[1,0] -yc[0,0])
#-- convert coordinates to 1dlon=np.asarray(xc[0,:])
lat=np.asarray(yc[:,0])
im,jm=lon.size,lat.size#-- subset coordinatestagnum=1iftagnum==1: # NE.USln1=284.ln2=296.lt1=44.lt2=53.xind=np.where((lon>=ln1) & (lon<=ln2))[0]
yind=np.where((lat>=lt1) & (lat<=lt2))[0]
lonc=np.copy(xc) ; latc=np.copy(yc)
lonc=lonc[yind,:]; lonc=lonc[:,xind]
latc=latc[yind,:]; latc=latc[:,xind]
#-- Compute grid informationnum_corners=4nj, ni=lonc.shapegrid_size=ni*njgrid_center_lat=np.reshape(latc,grid_size)
grid_center_lon=np.reshape(lonc,grid_size)
latn=grid_center_lat+0.5*delylats=grid_center_lat-0.5*delylone=grid_center_lon+0.5*delxlonw=grid_center_lon-0.5*delxgrid_corner_lat=np.empty((grid_size,num_corners))
grid_corner_lat[:,0]=latngrid_corner_lat[:,1]=latngrid_corner_lat[:,2]=latsgrid_corner_lat[:,3]=latsgrid_corner_lon=np.empty((grid_size,num_corners))
grid_corner_lon[:,0]=lonegrid_corner_lon[:,1]=lonwgrid_corner_lon[:,2]=lonwgrid_corner_lon[:,3]=lonegrid_imask=np.ones(grid_size, dtype=int)
grid_dims=np.array((ni,nj), dtype=int)
nm=grid_dims.shape[0]
#------------------------------------------coordDim=2elementCount=grid_center_lat.sizemaxNodePElement=grid_corner_lon.shape[1]
numElementConn= [maxNodePElementfor_inrange(elementCount)]
numElementConn=np.asarray(numElementConn,dtype='b')
# create list of all nodesall_nodes_list= []
foriinrange(elementCount):
forjinrange(maxNodePElement):
all_nodes_list.append((grid_corner_lon[i,j],grid_corner_lat[i,j]))
num_all_nodes=len(all_nodes_list)
all_nodes=np.asarray(all_nodes_list)
ifdoTimer:
stime=time.time()
# Identify unique nodesdtr=np.pi/180index_radius=1e-6all_unique_nodes= []
foriinrange(num_all_nodes):
ifnp.mod(i,1000) ==0:
print('i in num_all_nodes ',i,num_all_nodes)
dlon=all_nodes[i+1:,0] -all_nodes[i,0]
dlat=all_nodes[i+1:,1] -all_nodes[i,1]
dlon=dtr*dlondlat=dtr*dlatdist=np.power(np.sin(dlat/2),2) +np.cos(dtr*all_nodes[i+1:,1]) *np.cos(dtr*all_nodes[i,1]) *np.power(np.sin(dlon/2),2)
dist=np.arctan2( np.sqrt(dist), np.sqrt(1-dist))
npoints=np.sum(np.where(dist<index_radius,1,0))
# if no duplicate points found, add to listifnpoints<1:
all_unique_nodes.append(all_nodes[i,:])
nodeCount=len(all_unique_nodes)
print('nodeCount,elementCount, maxNodePElement ', nodeCount,elementCount, maxNodePElement)
#stopifdoTimer:
etime=time.time()
print('Time to create all_unique_nodes: ', etime-stime)
print('')
ifdoTimer:
stime2=time.time()
nodeCoords=np.zeros((nodeCount, coordDim))
foriinrange(nodeCount):
pt=all_unique_nodes[i]
nodeCoords[i,:] = [pt[0],pt[1]]
# Create array describing connectivity (1-based indices)elementConn=np.zeros((elementCount, maxNodePElement),dtype='i')
elementConn[:,] =int(-1)
elementArea=np.zeros((elementCount),dtype='d')
elementMask=np.ones((elementCount),dtype='i')
centerCoords=np.zeros((elementCount, coordDim))
dtr=np.pi/180foriinrange(elementCount):
centerCoords[i,:] = [grid_center_lon[i],grid_center_lat[i]]
dlon=np.max(grid_corner_lon[i,:]) -np.min(grid_corner_lon[i,:])
dlat=np.max(grid_corner_lat[i,:]) -np.min(grid_corner_lat[i,:])
# convert to radians^2pArea=dlon*dlat*np.sin(dtr*grid_center_lat[i])*dtr*dtrelementArea[i] =pArea# record element as indices rather than pointsu_list= []
forjinrange(maxNodePElement):
pt= [grid_corner_lon[i,j],grid_corner_lat[i,j]]
dlon=nodeCoords[:,0] -grid_corner_lon[i,j]
dlat=nodeCoords[:,1] -grid_corner_lat[i,j]
dlon=dtr*dlondlat=dtr*dlatdist=np.power(np.sin(dlat/2),2) +np.cos(dtr*nodeCoords[:,1]) *np.cos(dtr*grid_corner_lat[i,j]) *np.power(np.sin(dlon/2),2)
dist=np.arctan2( np.sqrt(dist), np.sqrt(1-dist))
u_list.append(np.argmin(dist)+1)
# order of points should be counter-clockwiseelementConn[i,:numElementConn[i]] =u_list#elementConn[i,:numElementConn[i]] = np.flip(u_list)ifdoTimer:
etime2=time.time()
print('Time to create elementConn: ', etime2-stime2)
print('')
print('writing file: ', outfile, '\n')
#w = netcdf4.Dataset(outfile, 'w', format='NETCDF4')w=netcdf4.Dataset(outfile, 'w', format='NETCDF3_64BIT')
w.creation_date=timetagw.source_file=infilew.title='ESMF mesh file'w.gridType='unstructured'w.createDimension('nodeCount',int(nodeCount))
w.createDimension('elementCount',int(elementCount))
w.createDimension('maxNodePElement',int(maxNodePElement))
w.createDimension('coordDim',int(coordDim))
w_nodeCoords=w.createVariable('nodeCoords','d',('nodeCount', 'coordDim'))
w_nodeCoords.units='degrees'w_elementConn=w.createVariable('elementConn','i',('elementCount', 'maxNodePElement'),fill_value=-1)
w_elementConn.long_name='Node Indices that define the element connectivity'#w_elementConn._FillValue = -1w_numElementConn=w.createVariable('numElementConn','b',('elementCount'))
w_numElementConn.long_name='Number of nodes per element'w_centerCoords=w.createVariable('centerCoords','d',('elementCount', 'coordDim'))
w_centerCoords.units='degrees'w_elementArea=w.createVariable('elementArea','d',('elementCount'))
w_elementArea.units='radians^2'w_elementArea.long_name='area weights'w_elementMask=w.createVariable('elementMask','i',('elementCount'),fill_value=-9999.)
#w_elementMask._FillValue = -9999.# convert longitude to [0-360] if neededtmp=nodeCoords[:,0]
tmp[tmp<0] +=360nodeCoords[:,0] =tmptmp=centerCoords[:,0]
tmp[tmp<0] +=360centerCoords[:,0] =tmp# write to file --------------------------------------------w_nodeCoords[:,] =nodeCoordsw_elementConn[:,] =elementConnw_numElementConn[:,] =numElementConnw_centerCoords[:,] =centerCoordsw_elementArea[:,] =elementAreaw_elementMask[:,] =elementMaskw.close()
print(outfile,' created')
That's great – thanks @swensosc and @ekluzek ! From looking quickly through this, I have a few thoughts. Sorry, I feel like I'm jumping to some detailed critiques, but overall this seems like a fantastic thing to have! I just can't help myself from noticing some details like this when I read through code:
(1) Regarding unstructured grids: It looks like this assumes that the corners are equidistant from the center. My understanding is that that is not always the case for irregular grids. For example, see the note about corners here https://www.ncl.ucar.edu/Document/Functions/ESMF/curvilinear_to_SCRIP.shtml (which refers to HOMME grids specifically).
(2) What is the performance like? In particular, I see that there is an order N^2 loop over corners (i.e., a doubly nested loop over corners) to identify the unique corners, and I'm wondering how that performs on large grids. If the point of this script is just for small grids, this may be a non-issue, but it would be good to know. If having an O(N^2) loop is unavoidable but leads to too-bad performance in some situations, I'm wondering if this could be reworked a bit to use built-in numpy operations like the rounding-then-unique solution from https://stackoverflow.com/questions/5426908/find-unique-elements-of-floating-point-array-in-numpy-with-comparison-using-a-d.
(3) Should the tolerance for finding identical corners be based somehow on the typical grid cell size?
(4) Most importantly, it would be great if we could ask the ESMF group to provide something like this so that they are the ones to be responsible for maintaining it, ensuring it's robust in a variety of cases, etc. That way we don't have to be the ones to deal with things like (2) and (3).
I should have clarified, too, that my comments were not really meant as criticisms of the script as is: for internal and possibly some external use, this seems like a FANTASTIC thing to have! It was more that I was mentally jumping a step ahead to, "Could this be our standard, supported mechanism for creating mesh files?", so was thinking about what might be needed to get to that point. My mental context was some discussions we've been having in CSEG about the process for mesh file generation, so I was thinking, "could this be a good general process?" And my main thought at this point is that this script could very possibly help in many cases with the general mesh file creation process, even if it needs some adjustments to make it robust in a wider variety of scenarios. Anyway, thank you for sharing this!
@swensosc shared with me another similar script to be used for unstructured grids. So our first use of this might have both scripts in place, and the user will have to figure out which one to use.
@billsacks@ekluzek@swensosc - I think having the script that Sean targeted for regular grids would be very useful. However, I think we need to proceed very cautiously when we talk about unstructured grids and this requires input from a much larger group. Particularly as CAM already has a process for creating meshes for refined grids that could then be leveraged by CTSM. I believe that whatever unstructured mesh CTSM would require would also be required by CAM. When would this not be the case?
@mvertens and I talked about this issue this morning. We both feel that, in general, it makes sense to stick with our current, two-step process:
Create a SCRIP grid file that defines the grid
Convert the SCRIP grid file to an ESMF mesh file
The reason for splitting this is that step (1) is really grid-dependent, particularly in how you define the corners, so for example, there would be a different script used for CAM-SE grids than for the hydrological basin grids that @swensosc has been experimenting with. Step (2), on the other hand, is more generic, but also needs more software engineering to do it in a robust and performant way (e.g., my comments above), so it makes sense to have this be done by a generic tool maintained by the ESMF group. Then you can think of the SCRIP grid file as a relatively simple intermediate representation that is used to communicate between these two steps.
@adrifoster and @jkshuman here's the issue about creating mesh files from domain files. There's a bunch of discussion here that talks about short as well as longer term desires. I think we need a plan that includes a short-term stop gap measure that allows people to do their regional science they need to do now, as well as steps for the longer term goal we want to make it towards.
We are going to have a group meet next Wednesday afternoon to go over this and come up with a plan for a short term solution that will eventually merge into the longer term CESM wide solution.
--
Dr. Mariana Vertenstein
CESM Software Engineering Group Head
National Center for Atmospheric Research
Boulder, Colorado
Office 303-497-1349
Email: ***@***.***
changed the title [-]Need a process to create ESMF mesh files from domain files for regional grids[/-][+]Need a process to subset ESMF mesh files from global ones for regional grids[/+]on Dec 9, 2021
also likely a good idea to loop in @slevisconsulting at the outset for a discussion to see if any of the tools he's been working on are relevant / helpful.
In the big-picture, the mesh_mask_modifier tool that I'm working on (#1677 ) relates to this issue in a similar way that the fsurdat_modifier tool relates to subset_data. I.e. neither mesh_mask_modifier nor fsurdat_modifier perform subsetting.
@adrifoster and @negin513 is this work getting close enough to merge #1735 and start working on documentation for running regional cases? @ckoven would like to try this out for a high resolution CA-FATES case.
In stand-up meeting 2022/7/26 @jkshuman recommended that I share this post which contains a 2-line method for generating mesh files:
Generate SCRIP file from a lat/lon file: ncks --rgr infer --rgr scrip=wrfinput_d01_scrip.nc wrfinput_d01_2d_lat_lon.nc wrfinput_d01_foo.nc
*foo.nc contains metadata and will not be used.
Generate mesh file from the SCRIP grid file: /glade/u/apps/ch/opt/esmf-netcdf/8.0.0/intel/19.0.5/bin/bing/Linux.intel.64.mpiuni.default/ESMF_Scrip2Unstruct wrfinput_d01_scrip.nc wrfinput_d01_unstruct.nc 0
Activity
billsacks commentedon Oct 7, 2021
This likely isn't the way we want to go long-term, but in case it's helpful, here is the process I inherited from Mariana to convert a scrip grid file to an esmf mesh file:
ekluzek commentedon Oct 22, 2021
Here's a script that @swensosc created to create a mesh based on the surface dataset that was created. This bypasses domain files, and bypasses subsetting a SCRIP grid file, so is a good approach to take. It should work for unstructured grids as well since the surface dataset is just a 1D vector in that case. But, we should test it on unstructured grids as well.
billsacks commentedon Oct 22, 2021
That's great – thanks @swensosc and @ekluzek ! From looking quickly through this, I have a few thoughts. Sorry, I feel like I'm jumping to some detailed critiques, but overall this seems like a fantastic thing to have! I just can't help myself from noticing some details like this when I read through code:
(1) Regarding unstructured grids: It looks like this assumes that the corners are equidistant from the center. My understanding is that that is not always the case for irregular grids. For example, see the note about corners here https://www.ncl.ucar.edu/Document/Functions/ESMF/curvilinear_to_SCRIP.shtml (which refers to HOMME grids specifically).
(2) What is the performance like? In particular, I see that there is an order N^2 loop over corners (i.e., a doubly nested loop over corners) to identify the unique corners, and I'm wondering how that performs on large grids. If the point of this script is just for small grids, this may be a non-issue, but it would be good to know. If having an O(N^2) loop is unavoidable but leads to too-bad performance in some situations, I'm wondering if this could be reworked a bit to use built-in numpy operations like the rounding-then-unique solution from https://stackoverflow.com/questions/5426908/find-unique-elements-of-floating-point-array-in-numpy-with-comparison-using-a-d.
(3) Should the tolerance for finding identical corners be based somehow on the typical grid cell size?
(4) Most importantly, it would be great if we could ask the ESMF group to provide something like this so that they are the ones to be responsible for maintaining it, ensuring it's robust in a variety of cases, etc. That way we don't have to be the ones to deal with things like (2) and (3).
swensosc commentedon Oct 22, 2021
billsacks commentedon Oct 22, 2021
I should have clarified, too, that my comments were not really meant as criticisms of the script as is: for internal and possibly some external use, this seems like a FANTASTIC thing to have! It was more that I was mentally jumping a step ahead to, "Could this be our standard, supported mechanism for creating mesh files?", so was thinking about what might be needed to get to that point. My mental context was some discussions we've been having in CSEG about the process for mesh file generation, so I was thinking, "could this be a good general process?" And my main thought at this point is that this script could very possibly help in many cases with the general mesh file creation process, even if it needs some adjustments to make it robust in a wider variety of scenarios. Anyway, thank you for sharing this!
@mvertens @uturuncoglu - FYI - see the discussion above.
ekluzek commentedon Oct 22, 2021
@swensosc shared with me another similar script to be used for unstructured grids. So our first use of this might have both scripts in place, and the user will have to figure out which one to use.
mvertens commentedon Oct 23, 2021
@billsacks @ekluzek @swensosc - I think having the script that Sean targeted for regular grids would be very useful. However, I think we need to proceed very cautiously when we talk about unstructured grids and this requires input from a much larger group. Particularly as CAM already has a process for creating meshes for refined grids that could then be leveraged by CTSM. I believe that whatever unstructured mesh CTSM would require would also be required by CAM. When would this not be the case?
billsacks commentedon Oct 25, 2021
@mvertens and I talked about this issue this morning. We both feel that, in general, it makes sense to stick with our current, two-step process:
The reason for splitting this is that step (1) is really grid-dependent, particularly in how you define the corners, so for example, there would be a different script used for CAM-SE grids than for the hydrological basin grids that @swensosc has been experimenting with. Step (2), on the other hand, is more generic, but also needs more software engineering to do it in a robust and performant way (e.g., my comments above), so it makes sense to have this be done by a generic tool maintained by the ESMF group. Then you can think of the SCRIP grid file as a relatively simple intermediate representation that is used to communicate between these two steps.
ekluzek commentedon Dec 3, 2021
@adrifoster and @jkshuman here's the issue about creating mesh files from domain files. There's a bunch of discussion here that talks about short as well as longer term desires. I think we need a plan that includes a short-term stop gap measure that allows people to do their regional science they need to do now, as well as steps for the longer term goal we want to make it towards.
ekluzek commentedon Dec 3, 2021
We are going to have a group meet next Wednesday afternoon to go over this and come up with a plan for a short term solution that will eventually merge into the longer term CESM wide solution.
mvertens commentedon Dec 4, 2021
[-]Need a process to create ESMF mesh files from domain files for regional grids[/-][+]Need a process to subset ESMF mesh files from global ones for regional grids[/+]9 remaining items
wwieder commentedon Apr 21, 2022
also likely a good idea to loop in @slevisconsulting at the outset for a discussion to see if any of the tools he's been working on are relevant / helpful.
slevis-lmwg commentedon Apr 21, 2022
In the big-picture, the mesh_mask_modifier tool that I'm working on (#1677 ) relates to this issue in a similar way that the fsurdat_modifier tool relates to subset_data. I.e. neither mesh_mask_modifier nor fsurdat_modifier perform subsetting.
I'm happy to discuss more.
wwieder commentedon Jun 2, 2022
@adrifoster and @negin513 is this work getting close enough to merge #1735 and start working on documentation for running regional cases? @ckoven would like to try this out for a high resolution CA-FATES case.
slevis-lmwg commentedon Jul 26, 2022
In stand-up meeting 2022/7/26 @jkshuman recommended that I share this post which contains a 2-line method for generating mesh files:
Generate SCRIP file from a lat/lon file:
ncks --rgr infer --rgr scrip=wrfinput_d01_scrip.nc wrfinput_d01_2d_lat_lon.nc wrfinput_d01_foo.nc*foo.nc contains metadata and will not be used.
Generate mesh file from the SCRIP grid file:
/glade/u/apps/ch/opt/esmf-netcdf/8.0.0/intel/19.0.5/bin/bing/Linux.intel.64.mpiuni.default/ESMF_Scrip2Unstruct wrfinput_d01_scrip.nc wrfinput_d01_unstruct.nc 0ekluzek commentedon Jan 9, 2023
#1892 handles this issue.
ekluzek commentedon Apr 26, 2024
This is completed with #1892 it will autoclose when b4b-dev is next merged to master.
ekluzek commentedon Nov 12, 2024
This was supposed to auto-close but didn't, so closing now.