Repository navigation
Conversation
ops-init-cron copied crontab.example verbatim, and every launcher entry in it named chrysalis. A second site therefore got a file called <site>.crontab that ran Chrysalis's config, wrote Chrysalis-named logs and locks, and reported MACHINE_NAME=chrysalis, with only a comment asking the operator to fix it by hand. Replace the literal with a placeholder and render it through a small initializer that still installs via operations_install, so modes and ACL inheritance are unchanged. Each site config now declares SIMBOARD_DIAGNOSTICS_ENABLED; sites without a reviewed diagnostics archive get no diagnostics jobs rather than ones that fail on every run. Chrysalis keeps its schedule exactly as before.
Aurora is already a seeded machine but had no site config, and the Make entry point accepted only chrysalis and perlmutter and routed everything except Chrysalis down the NERSC path-ingestion branch. Add aurora.config selecting the HPC upload ingestor, and route every site other than Perlmutter through the site launcher. The config sets no filesystem defaults: public sources disagree on Aurora's performance root (E3SM_Dec/performance_archive in planning notes, E3SMinput/baselines in the current E3SM machine config), so a run without explicit roots fails with PERF_ARCHIVE_ROOT is required rather than scanning a guess. Tests stage one complete Aurora execution and run it through the real parser and ingestor offline: machine, execution ID, case, and user parse correctly, staging and archive discovery each accept it, a FAILED status bucket is ignored, and the archive is byte-identical afterward with every network and write callback wired to fail.
Add an operations page listing what is wired up, the facts that must be confirmed on Aurora before scheduling (staging and archive roots, scheduler, owner, credentials, egress, diagnostics), and the validation and rollout steps. Replace the architecture table's inherited Aurora values, which came from a PACE reference and conflict with current E3SM configuration, with a pointer to that page.
PACE's Aurora collector (pace_archive_aurora.sh) stages runs in E3SMinput/baselines/performance_archive and archives processed batches to OLD_PERF inside that same root. Set those as aurora.config defaults. Because OLD_PERF, unarchived performance_archive_* batches, and pace-exps-* upload trees all sit under Aurora's staging root, the remote-upload staging scan would re-collect already processed runs, including FAILED buckets. Move the staging pruner into archive_discovery, exclude a nested OLD_PERF, and apply the top-level exclusions to remote-upload staging scans by default. The Aurora fixture now reproduces the on-disk layout, and the docs record PACE's flow, the history backfill, and what remains to confirm.
This was referenced Nov 18, 2026
Collaborator
I pulled this change in #507 to apply it to all remote sites including PNNL compy. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds Aurora as a remote-upload ingestion site, using the same roots PACE's Aurora collector (
pace_archive_aurora.sh) uses:/lus/flare/projects/E3SMinput/baselines/performance_archive/lus/flare/projects/E3SMinput/baselines/performance_archive/OLD_PERFOn Aurora
OLD_PERFsits inside the staging root, and the staging root also holdsperformance_archive_*batches that never got moved andpace-exps-*upload work trees. The remote-upload staging scan had no pruner, so it would have picked up all of those again, FAILED buckets included. The staging pruner now lives inarchive_discovery, skips a nestedOLD_PERF, and its top-level exclusions are applied to remote-upload staging scans by default. NERSC behavior is unchanged apart from theOLD_PERFexclusion.Other changes:
ops-init-cronrenders the crontab for the selected site instead of copying Chrysalis's entries.run_machine_ingestion.shacceptsauroraand sends every site except Perlmutter through the site launcher.OLD_PERF, unprefixed batch names, a leftover batch, apace-exps-*tree). Without the pruner the staging test finds 5 cases instead of 1.docs/operations/aurora-ingestion.mdcovers PACE's flow on Aurora, validation, loading existing PACE history fromOLD_PERF, and what is still open.Not yet done: an offline dry run on Aurora against the real
OLD_PERF, the scheduler host/account, and a service-account token.Checklist
Deployment Notes (if any)
Nothing is scheduled by this PR. After merge: dry run on Aurora, create the Aurora service account, do a bounded
ingest-applyto dev withscan_mode=archive, then the full archive backfill. Steps are indocs/operations/aurora-ingestion.md.