Skip to content

Possibly no safety check when staged file is corrupted with Cntrl-C Interrupt #23

Open
0 of 2 issues completed

Description

@manishvenu

Hello,

I'm not sure if this is a question or a feature request, but I recently tried to rimport a new runoff dataset. Halfway through the first file, I interrupted it with Cntrl-C, and it looks like the file was corrupted in the staging area (about 8GB too small).

If I had permissions for relinking (which thankfully I didn't have) I think the original file would have be overwritten when I ran rimport again.

Is there anyway to have rimport be able to abort the copy operation cleanly? Or maybe have a check with checksums or at least file sizes to ensure consistency before relinking?

Also, Claude had me do sudo -u cesmdata rm to get rid of the corrupted file, which worked (I hope?)

Thanks,
Manish V.

Activity

  1. samsrabin commented on Jun 30, 2026

    @samsrabin
    Collaborator

    Great idea! Unfortunately I'm not sure we can implement it here; it all depends on whether we can reliably determine whether the file has been made available publicly by GDEX. For that, I need to know more about the technical details behind their publication process. I'll look into it.

  2. samsrabin commented on Jun 30, 2026

    @samsrabin
    Collaborator

    Sorry, I didn't think that through fully—yes, we definitely should add

    1. Something in the relink step that prevents (or at least warns about) deletion of the Campaign-space version if the GDEX-space version doesn't match it exactly, and
    2. Something in the copy step that immediately deletes the GDEX-space version if the copy didn't complete perfectly. (rsync has this option built in.)
  3. samsrabin commented on Jul 10, 2026

    @samsrabin
    Collaborator

    @manishvenu Are you still down to implement these changes? I think the "whether we can reliably determine whether the file has been made available publicly by GDEX" question is more or less irrelevant here, as it's pretty improbable that the GDEX uploader would start processing the file before our scripts even finish running.

  4. manishvenu commented on Jul 13, 2026

    @manishvenu
    Author

    @manishvenu Are you still down to implement these changes? I think the "whether we can reliably determine whether the file has been made available publicly by GDEX" question is more or less irrelevant here, as it's pretty improbable that the GDEX uploader would start processing the file before our scripts even finish running.

    Sure! I'm assuming both of them?

  5. samsrabin commented on Jul 13, 2026

    @samsrabin
    Collaborator

    Yes please! Ideally in separate PRs, or at least one PR with a clear and useful commit history.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions