Skip to content

Resolve the run script that submitted executions with multiple provenance copies - #494

Closed
tomvothecoder wants to merge 5 commits into
E3SM-Project:mainfrom
tomvothecoder:bug/434-multiple-run-scripts
Closed

tomvothecoder wants to merge 5 commits into
E3SM-Project:mainfrom
tomvothecoder:bug/434-multiple-run-scripts

Conversation

@tomvothecoder

Copy link
Copy Markdown
Collaborator

Description

When an execution's CaseDocs holds several run-script provenance copies, SimBoard now records the script that actually submitted the job. If CaseStatus doesn't identify it, the run script stays unset with a warning, as before. It never guesses the newest copy.

Evidence used. The E3SM run script saves a copy of itself on every invocation, and calls ./case.submit only when do_case_submit=true. CIME logs <time>: case.submit success <job_id> in CaseStatus. In the issue's example, three copies (111614, 112114, 113750) were saved and case.submit success 37013148 was logged at 11:38:00, so 113750 is the copy that submitted it. It is also the only copy with do_case_submit=true.

Rule. Pick the latest copy saved at or before the user's submit for this job:

  • A manual submit uses its own time.
  • An automatic resubmit (logged right after case.run success or st_archive success) uses the manual submit that started the chain. Copies saved later that never submitted are excluded.
  • A copy saved between a run's success and the next submit means that submit was manual.

The run script stays unset when CaseStatus has no submit record for the job, copies tie at the chosen time, or the chain's start isn't logged.

Changes

  • parsers/case_status.py: add parse_case_submit_time. Both CaseStatus readers now treat a truncated .gz (EOFError) as unreadable.
  • parsers/parser.py: _resolve_run_script_path uses that time when several copies match. One or zero matches behave as before.
  • backfill/metadata.py: --fields run-script now reads CaseStatus. Without this, the backfill could never resolve executions with several copies. A duplicate CaseStatus match is treated as no evidence, so the record doesn't fail.

Validation

  • Full backend suite: 2566 passed, 26 skipped.
  • make pre-commit-run: all hooks pass.
  • The new tests fail on the old code.
  • On the local copy of the issue's example (qa/cases/OLD_PERF, gitignored), both ingestion parsing and the backfill reader resolve …sh.20250319-113750. Every other QA execution keeps the script it had before.
  • In the QA data, all 463 case.run starting IDs match the preceding submit, and 381 of 484 submits directly follow case.run success. The rest follow user actions.

Risks

Checklist

  • Code follows project style guidelines
  • Self-reviewed code
  • No new warnings
  • Tests added or updated (if needed)
  • All tests pass (locally and CI/CD)
  • Documentation/comments updated (if needed)
  • Breaking change noted (if applicable)

Deployment Notes (if any)

No migration. New executions are resolved by production ingestion after deploy. Executions that are already ingested are skipped as duplicates, so their run script stays unset unless a scoped metadata backfill is run. See the verification comment below.

@tomvothecoder

Copy link
Copy Markdown
Collaborator Author

Verify on Perlmutter with the issue's example case

Example execution: /global/cfs/cdirs/e3sm/OLD_PERF/2025-03/performance_archive_2025_03_21_08_00_29/COMPLETED/njeffery/20250319.v3.LR.CBGC.FrazilEcosys.noSpinu.Icepack1.5.0.pm-cpu/37013148.250320-123519

Expected: …/case_scripts/run_script_provenance/run.20250319.v3.LR.CBGC.FrazilEcosys.noSpinu.Icepack1.5.0.pm-cpu.sh.20250319-113750

Re-running ingestion (scheduled or execution_dir=) won't show the fix for this execution. It's already ingested, and duplicate checks skip it without overwriting metadata. Use one of the checks below from a checkout of this branch (make backend-install first). Both parse on Perlmutter, so the fix doesn't need to be deployed.

1. Read-only parser check (no API, no writes)

cd backend
uv run python -c '
from pathlib import Path
from app.scripts.ingestion.backfill.metadata import _read_execution
path = Path("/global/cfs/cdirs/e3sm/OLD_PERF/2025-03/performance_archive_2025_03_21_08_00_29/COMPLETED/njeffery/20250319.v3.LR.CBGC.FrazilEcosys.noSpinu.Icepack1.5.0.pm-cpu/37013148.250320-123519")
print(_read_execution(path, None, ["run-script"]).run_script_path)
'

2. Scoped run-script backfill for this one case (dry run)

This uses the stored execution-to-directory mappings for this case only. It covers every njeffery execution of this case, not the whole archive. The run script is currently unset, so --overwrite isn't needed.

make metadata-backfill machine=perlmutter env=dev \
  SIMBOARD_ROOT=/path/to/simboard_root \
  args='--fields run-script --sources archive --case-name 20250319.v3.LR.CBGC.FrazilEcosys.noSpinu.Icepack1.5.0.pm-cpu --hpc-username njeffery --dry-run'
  • Expected result: the summary lists planned changes with the …sh.20250319-113750 run script for 37013148.250320-123519.
  • No mapping for the execution? Bootstrap one first with the same filters and --fields source-directories (see docs/operations/metadata-backfill.md).
  • To apply: replace --dry-run with --apply. This writes to that environment's database and needs separate approval.

Not planned: a full metadata backfill for this optional field. After deploy, new executions are resolved by production ingestion.

@tomvothecoder tomvothecoder self-assigned this Nov 17, 2026
@tomvothecoder tomvothecoder added the devops DevOps task (e.g., DOE site ingestion, CI/CD, Docker) label Nov 17, 2026
@tomvothecoder tomvothecoder added this to the 3. Ongoing: DevOps milestone Nov 17, 2026
@tomvothecoder tomvothecoder added the bug Bug fix label Nov 17, 2026
@tomvothecoder
tomvothecoder marked this pull request as ready for review November 17, 2026 18:44
@tomvothecoder

Copy link
Copy Markdown
Collaborator Author

Closing without merging. This only affects one user's executions (njeffery on NERSC, ac.jeffery on LCRC), and the run script is optional. The fix needs submit-chain inference from CaseStatus, and both reviews found cases where that could silently pick the wrong script. That's more logic and risk than this edge case justifies. Current behavior stays: when several provenance scripts match, the run script is left unset and a warning is logged. The branch bug/434-multiple-run-scripts is kept in case this comes up for more users.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Bug fix devops DevOps task (e.g., DOE site ingestion, CI/CD, Docker)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Multiple, ambigiuous run scripts for executions omit artifact ingestion

1 participant