You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
fsurdat_modifier creates a malformatted file on izumi #1658
When trying to run fsurdat_modifier on izumi with the standard python module loaded, the resulting file cannot be read. ncdump -h gives: ncdump: fsurdat.nc: fsurdat.nc: NetCDF: Unknown file format.
General bug information
CTSM version you are using: Branch a little past master (but I haven't changed the python code in this branch)
Does this bug cause significantly incorrect results in the model's science? No
Configurations affected: Just fsurdat_modifier on izumi
Details of bug
I was hoping we could run the FSURDATMODIFIER test on izumi. So I first tried running FSURDATMODIFYCTSM_D_Mmpi-serial_Ld1.5x5_amazon.I2000Clm50SpRs.izumi_intel. This failed.
Then I loaded a module environment on izumi that is close to what is loaded by default for running CTSM / CESM on izumi:
I then tried running the tool manually: from the case directory of this test (so the .cfg file existed there), I first removed the previously-generated fsurdat.nc and then ran:
This had the same result – even though I got a message saying that the surface dataset was successfully generated.
This is possibly (likely?) a system configuration issue – e.g., maybe the versions of xarray and other packages need to be updated on izumi??? It isn't critical that this is resolved; I'm okay with it being closed as a wontfix if others feel we don't care about supporting this tool on izumi.
@slevisconsulting - thoughts? Or @negin513 would you want to try this with your python environment on izumi?
I would not close as a wontfix for now. Let's first see how the cssi project makes use of the fsurdat_modifier tool. This may give us insight into whether we need it to work on izumi or not.
@adrifoster recommended checking the xarray versions on cheyenne (where fsurdat_modifier works) vs. izumi (where fsurdat_modifier generates a malformatted .nc file). I see differences in libhdf5, libnetcdf, xarray, h5py that seem related. Can others tell whether one or more of these will explain the problem? @adrifoster@negin513@billsacks@ekluzek@jedwards4b
If we're still stuck, @adrifoster recommended posting this issue on the xarray github page.
One more thing that may or may not be related. I tried ./manage_python_env on izumi and got this error (while it seems to work on cheyenne):
Use conda to install the python environment needed to run the CTSM python tools in the conda environment: ctsm_py
Create ctsm_py
CustomValidationError: Parameter channel_priority = 'strict' declared in <<merged>> is invalid.
The value 'strict' cannot be boolified.
Install ctsm_py this can take a long time, be patient....
CustomValidationError: Parameter channel_priority = 'strict' declared in <<merged>> is invalid.
The value 'strict' cannot be boolified.
Trouble installing the ctsm_py python environment
@negin513 uses a more up-to-date environment on izumi and showed me that she gets an error that I'm not seeing when I run fsurdat_modifier with my environment. I will try updating my environment and pursue things from there.
I talked with @slevisconsulting today about this.
I have seen this malformed file issue previously and it happens when writing the netcdf file is not completed successfully.
I suggested @slevisconsulting checking the file size against the successful version from Cheyenne. On Izumi, the netcdf file was smaller which verified that the writing stage of netcdf is not completed successfully.
I tested the code with an updated version of xarray on Izumi (xarray 0.20.2) and it complained about the index size for lsmlat dimension. Here is the error:
INFO: Opening fsurdat_in file to be modified: /fs/cgd/data0/slevis/git/mksurfdata_toolchain/tools/modify_fsurdat/fill_indianocean/surfdata_0.9x1.25_hist_16pfts_Irrig_CMIP6_simyr2000_c190214.nc
Traceback (most recent call last):
File "/home/negins/ctsm_forsam/tools/modify_fsurdat/./fsurdat_modifier", line 68, in <module>
main()
File "/home/negins/ctsm_forsam/tools/modify_fsurdat/../../python/ctsm/modify_fsurdat/fsurdat_modifier.py", line 34, in main
fsurdat_modifier(args.cfg_path)
File "/home/negins/ctsm_forsam/tools/modify_fsurdat/../../python/ctsm/modify_fsurdat/fsurdat_modifier.py", line 115, in fsurdat_modifier
modify_fsurdat.set_idealized() # set 2D variables
File "/home/negins/ctsm_forsam/tools/modify_fsurdat/../../python/ctsm/modify_fsurdat/modify_fsurdat.py", line 285, in set_idealized
self.setvar_lev0('FMAX', max_sat_area)
File "/home/negins/ctsm_forsam/tools/modify_fsurdat/../../python/ctsm/modify_fsurdat/modify_fsurdat.py", line 230, in setvar_lev0
self.file[var] = self.file[var].where(
File "/home/negins/miniconda3/envs/neon/lib/python3.9/site-packages/xarray/core/common.py", line 1291, in where
return ops.where_method(self, cond, other)
File "/home/negins/miniconda3/envs/neon/lib/python3.9/site-packages/xarray/core/ops.py", line 176, in where_method
return apply_ufunc(
File "/home/negins/miniconda3/envs/neon/lib/python3.9/site-packages/xarray/core/computation.py", line 1165, in apply_ufunc
return apply_dataarray_vfunc(
File "/home/negins/miniconda3/envs/neon/lib/python3.9/site-packages/xarray/core/computation.py", line 274, in apply_dataarray_vfunc
args = deep_align(
File "/home/negins/miniconda3/envs/neon/lib/python3.9/site-packages/xarray/core/alignment.py", line 435, in deep_align
aligned = align(
File "/home/negins/miniconda3/envs/neon/lib/python3.9/site-packages/xarray/core/alignment.py", line 323, in align
raise ValueError(f"indexes along dimension {dim!r} are not equal")
ValueError: indexes along dimension 'lsmlat' are not equal
@negin513 our conversation led to a breakthrough, thank you:
I had been loading python with module load lang/python/3.7.0
After our conversation I looked for ways to update xarray, so I unloaded python and activated the ctsm conda environment (just to try): conda activate ctsm
This worked! Now the tool generates an fsurdat that ncdump doesn't complain about.
PS. I don't get the error that we got when you ran the tool on izumi, which is a relief because I also don't get the error on cheyenne. I'm not sure what we did wrong that you got the error, but I don't think that we should worry about it, since my version works. Thanks, Negin!
@slevisconsulting thanks for reporting on your exact conda environment. I played around with it to try to get a version of pu create env that would work. I couldn't get it to work for python 3.7.0 no matter what settings of other packages I tried being informed by your environment. An important part here is that the python version is 3.8. You are really using 3.7.0, but the conda environment thinks it's different from that. So even the working version is somewhat malformed.
But this is really due to the issues of the installed conda version on CGD systems. And I'm working with Joseph to fix those issues.
Activity
slevis-lmwg commentedon Feb 23, 2022
I would not close as a wontfix for now. Let's first see how the cssi project makes use of the fsurdat_modifier tool. This may give us insight into whether we need it to work on izumi or not.
slevis-lmwg commentedon Jun 28, 2022
@adrifoster recommended checking the xarray versions on cheyenne (where fsurdat_modifier works) vs. izumi (where fsurdat_modifier generates a malformatted .nc file). I see differences in
libhdf5, libnetcdf, xarray, h5pythat seem related. Can others tell whether one or more of these will explain the problem? @adrifoster @negin513 @billsacks @ekluzek @jedwards4bCheyenne
Izumi
If we're still stuck, @adrifoster recommended posting this issue on the xarray github page.
One more thing that may or may not be related. I tried
./manage_python_envon izumi and got this error (while it seems to work on cheyenne):slevis-lmwg commentedon Jun 29, 2022
Confirmed (in case I was remembering wrong in yesterday's conversation) that the fsurdat file generated on izumi is unreadable also on cheyenne.
jedwards4b commentedon Jun 29, 2022
Maybe a request to system support to try reinstalling or installing an update on izumi?
slevis-lmwg commentedon Jun 29, 2022
@negin513 uses a more up-to-date environment on izumi and showed me that she gets an error that I'm not seeing when I run fsurdat_modifier with my environment. I will try updating my environment and pursue things from there.
negin513 commentedon Jun 30, 2022
I talked with @slevisconsulting today about this.
I have seen this
malformed fileissue previously and it happens when writing the netcdf file is not completed successfully.I suggested @slevisconsulting checking the file size against the successful version from Cheyenne. On Izumi, the netcdf file was smaller which verified that the writing stage of netcdf is not completed successfully.
negin513 commentedon Jun 30, 2022
I tested the code with an updated version of xarray on Izumi (
xarray 0.20.2) and it complained about the index size forlsmlatdimension. Here is the error:slevis-lmwg commentedon Jun 30, 2022
@negin513 our conversation led to a breakthrough, thank you:
I had been loading python with
module load lang/python/3.7.0After our conversation I looked for ways to update xarray, so I unloaded python and activated the ctsm conda environment (just to try):
conda activate ctsmThis worked! Now the tool generates an fsurdat that ncdump doesn't complain about.
PS. I don't get the error that we got when you ran the tool on izumi, which is a relief because I also don't get the error on cheyenne. I'm not sure what we did wrong that you got the error, but I don't think that we should worry about it, since my version works. Thanks, Negin!
slevis-lmwg commentedon Jun 30, 2022
I'm almost certain that I pointed you to the wrong branch. (Again, it doesn't matter, but the thought was bugging me...)
negin513 commentedon Jun 30, 2022
Perfect! Glad it helped. 👍
slevis-lmwg commentedon Jun 30, 2022
Sanity check:
I have also confirmed that the fsurdat generated on izumi is bfb identical to the fsurdat generated on cheyenne.
slevis-lmwg commentedon Aug 3, 2022
@ekluzek my branch is located on izumi here:
/fs/cgd/data0/slevis/git/mksurfdata_toolchain/tools/modify_input_files
conda activate ctsm_pyuses xarray 0.16and generates the unreadable .nc file, while
has xarray 0.20 and generates a good .nc file.
Rather than looking into the strange behavior requiring the module unload, we should focus on upgrading ctsm_py's xarray to 0.20.
slevis-lmwg commentedon Aug 4, 2022
@ekluzek here's the contents of the ctsm package on izumi:
ekluzek commentedon Aug 8, 2022
@slevisconsulting thanks for reporting on your exact conda environment. I played around with it to try to get a version of pu create env that would work. I couldn't get it to work for python 3.7.0 no matter what settings of other packages I tried being informed by your environment. An important part here is that the python version is 3.8. You are really using 3.7.0, but the conda environment thinks it's different from that. So even the working version is somewhat malformed.
But this is really due to the issues of the installed conda version on CGD systems. And I'm working with Joseph to fix those issues.
ekluzek commentedon Feb 22, 2024
I think this will be resolved in the new conda environment on izumi at this point.