Skip to content

Add Aurora as a remote-upload ingestion site - #436

Draft
rajeeja wants to merge 4 commits into
E3SM-Project:mainfrom
rajeeja:rajeeja/376-aurora-ingestion
Draft

rajeeja wants to merge 4 commits into
E3SM-Project:mainfrom
rajeeja:rajeeja/376-aurora-ingestion

Conversation

@rajeeja

@rajeeja rajeeja commented Oct 28, 2026

Copy link
Copy Markdown
Collaborator

Description

Adds Aurora as a remote-upload ingestion site, using the same roots PACE's Aurora collector (pace_archive_aurora.sh) uses:

  • staging: /lus/flare/projects/E3SMinput/baselines/performance_archive
  • archive: /lus/flare/projects/E3SMinput/baselines/performance_archive/OLD_PERF

On Aurora OLD_PERF sits inside the staging root, and the staging root also holds performance_archive_* batches that never got moved and pace-exps-* upload work trees. The remote-upload staging scan had no pruner, so it would have picked up all of those again, FAILED buckets included. The staging pruner now lives in archive_discovery, skips a nested OLD_PERF, and its top-level exclusions are applied to remote-upload staging scans by default. NERSC behavior is unchanged apart from the OLD_PERF exclusion.

Other changes:

  • ops-init-cron renders the crontab for the selected site instead of copying Chrysalis's entries.
  • run_machine_ingestion.sh accepts aurora and sends every site except Perlmutter through the site launcher.
  • The Aurora test fixture reproduces the on-disk layout (nested OLD_PERF, unprefixed batch names, a leftover batch, a pace-exps-* tree). Without the pruner the staging test finds 5 cases instead of 1.
  • New docs/operations/aurora-ingestion.md covers PACE's flow on Aurora, validation, loading existing PACE history from OLD_PERF, and what is still open.

Not yet done: an offline dry run on Aurora against the real OLD_PERF, the scheduler host/account, and a service-account token.

Checklist

  • Code follows project style guidelines
  • Self-reviewed code
  • No new warnings
  • Tests added or updated (if needed)
  • All tests pass (locally and CI/CD)
  • Documentation/comments updated (if needed)
  • Breaking change noted (if applicable)

Deployment Notes (if any)

Nothing is scheduled by this PR. After merge: dry run on Aurora, create the Aurora service account, do a bounded ingest-apply to dev with scan_mode=archive, then the full archive backfill. Steps are in docs/operations/aurora-ingestion.md.

ops-init-cron copied crontab.example verbatim, and every launcher entry in
it named chrysalis. A second site therefore got a file called
<site>.crontab that ran Chrysalis's config, wrote Chrysalis-named logs and
locks, and reported MACHINE_NAME=chrysalis, with only a comment asking the
operator to fix it by hand.

Replace the literal with a placeholder and render it through a small
initializer that still installs via operations_install, so modes and ACL
inheritance are unchanged. Each site config now declares
SIMBOARD_DIAGNOSTICS_ENABLED; sites without a reviewed diagnostics archive
get no diagnostics jobs rather than ones that fail on every run. Chrysalis
keeps its schedule exactly as before.
Aurora is already a seeded machine but had no site config, and the Make
entry point accepted only chrysalis and perlmutter and routed everything
except Chrysalis down the NERSC path-ingestion branch.

Add aurora.config selecting the HPC upload ingestor, and route every site
other than Perlmutter through the site launcher. The config sets no
filesystem defaults: public sources disagree on Aurora's performance root
(E3SM_Dec/performance_archive in planning notes, E3SMinput/baselines in
the current E3SM machine config), so a run without explicit roots fails
with PERF_ARCHIVE_ROOT is required rather than scanning a guess.

Tests stage one complete Aurora execution and run it through the real
parser and ingestor offline: machine, execution ID, case, and user parse
correctly, staging and archive discovery each accept it, a FAILED status
bucket is ignored, and the archive is byte-identical afterward with every
network and write callback wired to fail.
Add an operations page listing what is wired up, the facts that must be
confirmed on Aurora before scheduling (staging and archive roots,
scheduler, owner, credentials, egress, diagnostics), and the validation
and rollout steps. Replace the architecture table's inherited Aurora
values, which came from a PACE reference and conflict with current E3SM
configuration, with a pointer to that page.
PACE's Aurora collector (pace_archive_aurora.sh) stages runs in
E3SMinput/baselines/performance_archive and archives processed batches to
OLD_PERF inside that same root. Set those as aurora.config defaults.

Because OLD_PERF, unarchived performance_archive_* batches, and pace-exps-*
upload trees all sit under Aurora's staging root, the remote-upload staging
scan would re-collect already processed runs, including FAILED buckets. Move
the staging pruner into archive_discovery, exclude a nested OLD_PERF, and
apply the top-level exclusions to remote-upload staging scans by default.

The Aurora fixture now reproduces the on-disk layout, and the docs record
PACE's flow, the history backfill, and what remains to confirm.
@tomvothecoder

Copy link
Copy Markdown
Collaborator

On Aurora OLD_PERF sits inside the staging root, and the staging root also holds performance_archive_* batches that never got moved and pace-exps-* upload work trees. The remote-upload staging scan had no pruner, so it would have picked up all of those again, FAILED buckets included. The staging pruner now lives in archive_discovery, skips a nested OLD_PERF, and its top-level exclusions are applied to remote-upload staging scans by default. NERSC behavior is unchanged apart from the OLD_PERF exclusion.

I pulled this change in #507 to apply it to all remote sites including PNNL compy.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[DevOps]: Setup operations at ALCF (Aurora)

2 participants