Repository navigation
[DevOps]: Skip PACE batch folders in remote-upload staging scans - #507
Merged
tomvothecoder merged 2 commits intoNov 18, 2026
Merged
tomvothecoder merged 2 commits into
tomvothecoder merged 2 commits into
Conversation
Remote-upload staging scans now skip performance_archive_*, pace-exps-* and a nested OLD_PERF folder at the staging root, as NERSC staging scans already do. NERSC staging also skips the nested OLD_PERF. Move the staging pruners into archive_discovery, matching E3SM-Project#436 so it rebases cleanly. Refs E3SM-Project#506
tomvothecoder
marked this pull request as ready for review
November 18, 2026 18:54
6 of 7 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Remote-upload staging scans (
hpc_upload_archive_ingestor, used by Compy and Chrysalis) now skip PACE folders at the staging root that never hold new staging cases:performance_archive_*batches,pace-exps-*work trees, and a nestedOLD_PERFarchive root. NERSC staging already skipped the first two and now skipsOLD_PERFtoo.The staging folder filters move into
archive_discovery.py. The code matches the corresponding hunks of Add Aurora as a remote-upload ingestion site #436 exactly, so Add Aurora as a remote-upload ingestion site #436 rebases without conflicts.Archive scans and callers that pass their own folder filter, such as the LCRC v3 runner, are unchanged.
Behavior change: on upload sites, an
EXECUTION_DIRinside a skipped folder is now rejected, as on NERSC.Not included: limiting upload staging walks to
<user>/<case>/<execution>as NERSC does. Compy uses that layout too, but its case folders hold only execution folders, so the stricter walk would save little and could miss executions at other depths.Closes [DevOps]: Skip PACE batch folders in remote-upload staging scans #506
Checklist
Validation (local):
make backend-test2583 passed, 26 skipped;make pre-commit-runpassed. The new tests cover the upload staging skip, rejection of an uploadEXECUTION_DIRin each skipped folder, caller-filter and archive-mode passthrough, and the NERSCOLD_PERFskip. Each new upload test fails with the filter disabled. CIBackend checkspassed.Deployment Notes (if any)
None. Takes effect on each HPC site's next repository refresh.