You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Create all of the NEON surface dataset for CTSM5.2 #2319
Didn't think of this until I was looking at the namelist xml file, but for CTSM5.2 we need to also create all of the NEON surface datasets. And we'll need to make sure the NEON tools all work to create them all.
does the namelist xml need to make the NEON surface data, or is this a post-hoc activity where we update subset_data.py with a default 5.2 surface datatset that's subsequently shared with / hosted by NEON?
There's a config file that points to the default datasets for subset_data to use by default. We will update that file to point to the CTSM5.2 version, and then rerun the NEON surface dataset generation to remake all of the NEON sites for CTSM5.2. We install the NEON surface datasets in CESM inputdata space, so they don't have a tie to the NEON servers. The only tie is the CSV file that gives the surface characteristics of each site.
Maybe? after running subset data we subsequently run modify_singlept_site_neon, which uses a CSV file from NEON servers creates the default surface datasets used in the NEON simulations. Is it these modified surface data you're suggesting we generate, or just the intermediate files? Both are currently handled with neon_surf_wrapper
The final files that are output from modify_singlept_site_neon need to be recreated to with with CTSM5.2 because of changes in the surface datasets for CTSM5.2. So neon_surf_wrapper will be run to create a new set of files.
After we make the first CTSM5.2 tag subset_data will be pointing to new default surface datasets?
Then, as part of the short CTSM5.2 series we can create and upload the new surface data to NEON? Subsequently, an early CTSM5.3 tag could also then be to point to the neonV3 data by default (#2314)?
@wwieder yes subset_data will point to the new datasets, and as part of CTSM5.2 @slevis-lmwg is using subset_data to create the standard single point datasets (like 1x1_numa, 1x1_brazil, 1x1_smallvilleIA, and the urban single point [1x1_mexicocityMEX for example]).
However, the current plan is that we will NOT create NEON surface datasets. Which means that if you try to run a NEON site -- it will fail until we update this in the CTSM5.3 tag you mention. Updating the NEON surface datasets and bringing in neonV3 changes for an early CTSM5.3 tag does make sense to me. But, NEON won't work at all until that happens.
I'm not sure we are saving much by not doing NEON datasets as part of CTSM5.2 though? Since, @slevis-lmwg is really close with CTSM5.2 it might be worth waiting a little bit for him to create NEON surface datasets as part of CTSM5.2. Then we won't have tags where NEON fails. I don't think there is much time to make all the datasets, nor to update them in the NEON user-mods -- unless I'm missing something? Maybe it's an extra day or so?
There's a neon_surf_wrapper runs subset_data and modify_single_point_neon together. Hopefully this is pretty straightforward. We'll also then have to modify /usermods_dirs/NEON/defaults/user_nl_clm to point to these new files.
While waiting for derecho to return to service where I will troubleshoot test-suite failures (in #2318), I have started looking into generating the NEON files.
After a bit of troubleshooting (see corresponding commit from this evening), I ran ./neon_surf_wrapper on casper and it worked. The new files are here for now: /glade/derecho/scratch/slevis/single_point_neon_updated
I will copy to /inputdata and update namelist_defaults...
Activity
wwieder commentedon Jan 16, 2024
does the namelist xml need to make the NEON surface data, or is this a post-hoc activity where we update subset_data.py with a default 5.2 surface datatset that's subsequently shared with / hosted by NEON?
ekluzek commentedon Jan 16, 2024
There's a config file that points to the default datasets for subset_data to use by default. We will update that file to point to the CTSM5.2 version, and then rerun the NEON surface dataset generation to remake all of the NEON sites for CTSM5.2. We install the NEON surface datasets in CESM inputdata space, so they don't have a tie to the NEON servers. The only tie is the CSV file that gives the surface characteristics of each site.
Does that answer your question @wwieder?
wwieder commentedon Jan 16, 2024
Maybe? after running subset data we subsequently run
modify_singlept_site_neon, which uses a CSV file from NEON servers creates the default surface datasets used in the NEON simulations. Is it these modified surface data you're suggesting we generate, or just the intermediate files? Both are currently handled with neon_surf_wrapperekluzek commentedon Jan 16, 2024
The final files that are output from modify_singlept_site_neon need to be recreated to with with CTSM5.2 because of changes in the surface datasets for CTSM5.2. So neon_surf_wrapper will be run to create a new set of files.
wwieder commentedon Feb 1, 2024
After we make the first CTSM5.2 tag subset_data will be pointing to new default surface datasets?
Then, as part of the short CTSM5.2 series we can create and upload the new surface data to NEON? Subsequently, an early CTSM5.3 tag could also then be to point to the neonV3 data by default (#2314)?
ekluzek commentedon Feb 1, 2024
@wwieder yes subset_data will point to the new datasets, and as part of CTSM5.2 @slevis-lmwg is using subset_data to create the standard single point datasets (like 1x1_numa, 1x1_brazil, 1x1_smallvilleIA, and the urban single point [1x1_mexicocityMEX for example]).
However, the current plan is that we will NOT create NEON surface datasets. Which means that if you try to run a NEON site -- it will fail until we update this in the CTSM5.3 tag you mention. Updating the NEON surface datasets and bringing in neonV3 changes for an early CTSM5.3 tag does make sense to me. But, NEON won't work at all until that happens.
I'm not sure we are saving much by not doing NEON datasets as part of CTSM5.2 though? Since, @slevis-lmwg is really close with CTSM5.2 it might be worth waiting a little bit for him to create NEON surface datasets as part of CTSM5.2. Then we won't have tags where NEON fails. I don't think there is much time to make all the datasets, nor to update them in the NEON user-mods -- unless I'm missing something? Maybe it's an extra day or so?
wwieder commentedon Feb 1, 2024
There's a
neon_surf_wrapperrunssubset_dataandmodify_single_point_neontogether. Hopefully this is pretty straightforward. We'll also then have to modify/usermods_dirs/NEON/defaults/user_nl_clmto point to these new files.slevis-lmwg commentedon Feb 7, 2024
While waiting for derecho to return to service where I will troubleshoot test-suite failures (in #2318), I have started looking into generating the NEON files.
After a bit of troubleshooting (see corresponding commit from this evening), I ran
./neon_surf_wrapperon casper and it worked. The new files are here for now:/glade/derecho/scratch/slevis/single_point_neon_updatedI will copy to /inputdata and update namelist_defaults...
slevis-lmwg commentedon Feb 7, 2024
Done. I will need to run the corresponding tests from the test-suites to confirm that things work.
slevis-lmwg commentedon Feb 16, 2024
These datasets will be created again for the actual ctsm5.2 tag (as will all the other fsurdat/landuse files); still, I consider this issue complete.