You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I notice this only goes back until 1980. How bad is the global average sea surface temperature this year compared to years past? I think we could answer this with a SQL query.
Use memory optimized VMs with large amounts of RAM (e.g. 64).
Larger chunk sizes (800MiBs)
Coiled Run to launch the job from a remote VM (presumably, to reduce idle VM time given a large task graph).
Other principles:
Have a good understanding of how to minimize the Dask task graph. The size of the graph will determine the performance of the job. The size of the graph will take CPU time due to the scheduler overhead.
Prefer a task graph that minimizes IO overhead. So, try to load bigger batches of data to RAM for the core computation.
era5_ds = xr.open_zarr(
'gs://gcp-public-data-arco-era5/ar/'
'full_37-1h-0p25deg-chunk-1.zarr-v3/',
chunks={'time': 240, 'level': 1},
)
dsl0 = era5_ds.sel(level=1000) # we only care about surface sst
dsl0.nbytes / 2**40 # number of TiBs
# 1119.3059158361393
It looks like the full surface level ERA5 corpus is ~1120 TiBs. i.e., this is ~4.5x larger than the demo that Coiled performed.
🤞🏽 hopefully this means it will only cost ~5 times as much?
Activity
alxmrs commentedon Mar 12, 2024
The query will be something like:
alxmrs commentedon Mar 26, 2024
Lessons for how to scale this up from this article:
https://blog.coiled.io/blog/coiled-xarray.html
Other principles:
alxmrs commentedon Mar 28, 2024
According to this:
It looks like the full surface level ERA5 corpus is ~1120 TiBs. i.e., this is ~4.5x larger than the demo that Coiled performed.
🤞🏽 hopefully this means it will only cost ~5 times as much?
alxmrs commentedon Mar 28, 2024
Wait! We only care about SST! So, this is only 4TiB!