You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
In the PMC on 29 May 2020, there was some discussion about encoding the KGO with some version information. It was agreed that it was a nice idea only if it can be done automatically by the CI test. This issue will explore the possibility of automatically updating that information to the KGO when the CI test fails.
Would it be beneficial to use git lfs for the creation and versioning of the KGO files within the workflows? The files are not that big and can be stored in a separate repo that will be cloned only by a script so that bandwidth and data quotas are not used up that much.
When I looked into this I concluded that we would hit the bandwidth limit easily, but I may not be understanding how it works.
Please can you give more details of your proposed solution?
I just assumed that testing is only needed for contributors and action runs and given the size of all datasets is around 100 Mb, unless you get >10 answer-changing PRs a month - the bandwidth quota won't be met. Especially, if the testing description is moved to the wiki for developers, so the general users won't use it much.
Currently, the test datasets are downloaded every time the CI tests are triggered (push, pull request), so my understanding is that the bandwidth limit will be hit after 10 tests. If my understanding is correct, then we would have hit the bandwidth limit during the last couple of weeks.
Oh. Ok. Then what can be done is: in the workflows and in' driver/download_test_data.sh only a json file containing a list of *.nc files in a google folder (a google apps script can manage that). Your google folder will contain all the KGO files with different versions (I can show you how to set up the google apps scripts). In this case, there will be no need to update the workflows with new google links every time there is an answer-changing PR.
With regards to the versioning of the files, instead of vXXX, the filenames can contain creation date and tag/commit they were created with and in download_test_data.sh that part would be an optional argument if for some reason the older versions of KGO are needed.
Sorry for my slow reply. That sounds like a nice approach. I'm not familiar with google apps scripts, would you be able to provide guidance on how to to do that? For the moment, I'd like to keep the vXXX versioning, and perhaps move to date/tag in the future.
Make an empty .txt file which would contain the list of *.nc files in the folder. It can also be .json file
Get the .txt file Id and folder Id.
Example java_script function:
functionwrite_nc_list(){varlistfile=DriveApp.getFileById('.<txt file id>')varfolder=DriveApp.getFolderById('<folder id>');varlist=[];list.push(['Name','ID','Size']);// comment this out if you do not need the headervarfiles=folder.getFiles();//while(files.hasNext()){file=files.next();varrow=[]varfname=file.getName()if(fname.includes('.nc')){row.push(file.getName(),file.getId(),file.getSize())list.push(row);}}listfile.setContent(JSON.stringify(list))}
This function will fill your .txt with JSON that has filenames, id's size (you can add other entries f.e. date modified) which can later be read in the github workflow or download_test_data.sh to construct download links for curl. There are also options to make triggers for the script (f.e. run it every day) or just execute the script manually once you add new kgo to the folder.
This way you can have the only the .txt file link hardcoded in the workflows. If you have any trouble, we can arrange a zoom.
Activity
mvdebolskiy commentedon Apr 2, 2024
Would it be beneficial to use git lfs for the creation and versioning of the KGO files within the workflows? The files are not that big and can be stored in a separate repo that will be cloned only by a script so that bandwidth and data quotas are not used up that much.
alejandrobodas commentedon Apr 2, 2024
When I looked into this I concluded that we would hit the bandwidth limit easily, but I may not be understanding how it works.
Please can you give more details of your proposed solution?
mvdebolskiy commentedon Apr 4, 2024
I just assumed that testing is only needed for contributors and action runs and given the size of all datasets is around 100 Mb, unless you get >10 answer-changing PRs a month - the bandwidth quota won't be met. Especially, if the testing description is moved to the wiki for developers, so the general users won't use it much.
alejandrobodas commentedon Apr 5, 2024
Currently, the test datasets are downloaded every time the CI tests are triggered (push, pull request), so my understanding is that the bandwidth limit will be hit after 10 tests. If my understanding is correct, then we would have hit the bandwidth limit during the last couple of weeks.
mvdebolskiy commentedon Apr 5, 2024
Oh. Ok. Then what can be done is: in the workflows and in'
driver/download_test_data.shonly a json file containing a list of*.ncfiles in a google folder (a google apps script can manage that). Your google folder will contain all the KGO files with different versions (I can show you how to set up the google apps scripts). In this case, there will be no need to update the workflows with new google links every time there is an answer-changing PR.With regards to the versioning of the files, instead of vXXX, the filenames can contain creation date and tag/commit they were created with and in
download_test_data.shthat part would be an optional argument if for some reason the older versions of KGO are needed.alejandrobodas commentedon Apr 19, 2024
Sorry for my slow reply. That sounds like a nice approach. I'm not familiar with google apps scripts, would you be able to provide guidance on how to to do that? For the moment, I'd like to keep the vXXX versioning, and perhaps move to date/tag in the future.
mvdebolskiy commentedon Apr 22, 2024
You can add a script at https://script.google.com/home/start (documentation).
Before you write a script:
*.ncfiles..txtfile which would contain the list of*.ncfiles in the folder. It can also be.jsonfile.txtfile Id and folder Id..txtwith JSON that has filenames, id's size (you can add other entries f.e. date modified) which can later be read in the github workflow ordownload_test_data.shto construct download links for curl. There are also options to make triggers for the script (f.e. run it every day) or just execute the script manually once you add new kgo to the folder.This way you can have the only the
.txtfile link hardcoded in the workflows. If you have any trouble, we can arrange a zoom.