You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Possibly no safety check when staged file is corrupted with Cntrl-C Interrupt #23
I'm not sure if this is a question or a feature request, but I recently tried to rimport a new runoff dataset. Halfway through the first file, I interrupted it with Cntrl-C, and it looks like the file was corrupted in the staging area (about 8GB too small).
If I had permissions for relinking (which thankfully I didn't have) I think the original file would have be overwritten when I ran rimport again.
Is there anyway to have rimport be able to abort the copy operation cleanly? Or maybe have a check with checksums or at least file sizes to ensure consistency before relinking?
Also, Claude had me do sudo -u cesmdata rm to get rid of the corrupted file, which worked (I hope?)
Great idea! Unfortunately I'm not sure we can implement it here; it all depends on whether we can reliably determine whether the file has been made available publicly by GDEX. For that, I need to know more about the technical details behind their publication process. I'll look into it.
Sorry, I didn't think that through fully—yes, we definitely should add
Something in the relink step that prevents (or at least warns about) deletion of the Campaign-space version if the GDEX-space version doesn't match it exactly, and
Something in the copy step that immediately deletes the GDEX-space version if the copy didn't complete perfectly. (rsync has this option built in.)
@manishvenu Are you still down to implement these changes? I think the "whether we can reliably determine whether the file has been made available publicly by GDEX" question is more or less irrelevant here, as it's pretty improbable that the GDEX uploader would start processing the file before our scripts even finish running.
@manishvenu Are you still down to implement these changes? I think the "whether we can reliably determine whether the file has been made available publicly by GDEX" question is more or less irrelevant here, as it's pretty improbable that the GDEX uploader would start processing the file before our scripts even finish running.
Activity
samsrabin commentedon Jun 30, 2026
Great idea! Unfortunately I'm not sure we can implement it here; it all depends on whether we can reliably determine whether the file has been made available publicly by GDEX. For that, I need to know more about the technical details behind their publication process. I'll look into it.
samsrabin commentedon Jun 30, 2026
Sorry, I didn't think that through fully—yes, we definitely should add
rsynchas this option built in.)samsrabin commentedon Jul 10, 2026
@manishvenu Are you still down to implement these changes? I think the "whether we can reliably determine whether the file has been made available publicly by GDEX" question is more or less irrelevant here, as it's pretty improbable that the GDEX uploader would start processing the file before our scripts even finish running.
manishvenu commentedon Jul 13, 2026
Sure! I'm assuming both of them?
samsrabin commentedon Jul 13, 2026
Yes please! Ideally in separate PRs, or at least one PR with a clear and useful commit history.