Skip to content

[DevOps]: Skip PACE batch folders in remote-upload staging scans - #507

Merged
tomvothecoder merged 2 commits into
E3SM-Project:mainfrom
tomvothecoder:devops/506-upload-staging-filter
Nov 18, 2026
Merged

tomvothecoder merged 2 commits into
E3SM-Project:mainfrom
tomvothecoder:devops/506-upload-staging-filter

Conversation

@tomvothecoder

@tomvothecoder tomvothecoder commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

Description

Remote-upload staging scans (hpc_upload_archive_ingestor, used by Compy and Chrysalis) now skip PACE folders at the staging root that never hold new staging cases: performance_archive_* batches, pace-exps-* work trees, and a nested OLD_PERF archive root. NERSC staging already skipped the first two and now skips OLD_PERF too.

Checklist

  • Code follows project style guidelines
  • Self-reviewed code
  • No new warnings
  • Tests added or updated (if needed)
  • All tests pass (locally and CI/CD)
  • Documentation/comments updated (if needed)
  • Breaking change noted (if applicable)

Validation (local): make backend-test 2583 passed, 26 skipped; make pre-commit-run passed. The new tests cover the upload staging skip, rejection of an upload EXECUTION_DIR in each skipped folder, caller-filter and archive-mode passthrough, and the NERSC OLD_PERF skip. Each new upload test fails with the filter disabled. CI Backend checks passed.

Deployment Notes (if any)

None. Takes effect on each HPC site's next repository refresh.

Remote-upload staging scans now skip performance_archive_*, pace-exps-*
and a nested OLD_PERF folder at the staging root, as NERSC staging scans
already do. NERSC staging also skips the nested OLD_PERF.

Move the staging pruners into archive_discovery, matching E3SM-Project#436 so it
rebases cleanly.

Refs E3SM-Project#506
@tomvothecoder
tomvothecoder marked this pull request as ready for review November 18, 2026 18:54
@tomvothecoder tomvothecoder self-assigned this Nov 18, 2026
@tomvothecoder tomvothecoder added this to the 3. Ongoing: DevOps milestone Nov 18, 2026
@tomvothecoder tomvothecoder added the devops DevOps task (e.g., DOE site ingestion, CI/CD, Docker) label Nov 18, 2026
@tomvothecoder
tomvothecoder merged commit 6cacfcd into E3SM-Project:main Nov 18, 2026
1 check passed
@tomvothecoder
tomvothecoder deleted the devops/506-upload-staging-filter branch November 18, 2026 18:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

devops DevOps task (e.g., DOE site ingestion, CI/CD, Docker)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[DevOps]: Skip PACE batch folders in remote-upload staging scans

1 participant