Skip to content

[DevOps]: Report deployment refresh and log archiving job health - #502

Merged
tomvothecoder merged 8 commits into
E3SM-Project:mainfrom
tomvothecoder:devops/501-maintenance-job-health
Nov 17, 2026
Merged

tomvothecoder merged 8 commits into
E3SM-Project:mainfrom
tomvothecoder:devops/501-maintenance-job-health

Conversation

@tomvothecoder

@tomvothecoder tomvothecoder commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

Description

Show the site deployment refresh and weekly launcher-log archiving on the operations dashboard. Before this change, a failed refresh was silent: ingestion kept running the old revision and the dashboard stayed healthy.

  • API: accepts refresh and log_archive job types with default thresholds of 1020/60 and 10200/120 minutes (silence/duration). No migration: scan_mode is a string column.
  • Wrapper: new sites/operations/run_reported_job.sh <site> <job-type> <command...>. It reuses the launcher's job_health start / finish reports, so job_health.py is unchanged.
    • Runs the command once and reports it to each API environment in SIMBOARD_ENV_FILES (colon-separated env.*.sh files).
    • Only the report subshells load the site config and env files, so the command never receives an API token.
    • A missing env file is skipped with a warning; an unreachable API logs job_report_failed (up to 10 s per report). Neither affects the job or the other environment.
    • The script runs in a main function called on its last line, so Bash never rereads it after a refresh replaces it.
  • Cron template: both jobs run through the wrapper and report to development and production.
    • The refresh entry sets DRY_RUN=false inline: cron applies the global DRY_RUN line only to later entries, and the refresh has no dry-run mode.
    • Log archiving follows the global setting.
  • Dashboard: job type filter and table include both types.
  • Docs: job health section, thresholds and upgrade steps in setup-ingestion-operations.md.

Reporting stays best effort: a missing API environment or failed report never stops either job or changes its exit code.

Accepted risks:

  • The refresh now runs through run_reported_job.sh, whose path is fixed in installed crontabs. A break in the wrapper itself would stop site refreshes; report failures cannot.
  • If a refresh breaks the backend runtime, its finish report fails and the run turns stale after 60 minutes.
  • Failed runs show only "Exited with code N"; details remain in simboard-ingestion-provision.log and simboard-log-archiving.log.
  • Site Summary job counts now include these jobs, so a stale refresh flags the site.

Validation: make backend-test (2572 passed, 26 skipped), pnpm test (77 passed), make frontend-lint and make pre-commit-run passed.

Checklist

  • Code follows project style guidelines
  • Self-reviewed code
  • No new warnings
  • Tests added or updated (if needed)
  • All tests pass (locally and CI/CD)
  • Documentation/comments updated (if needed)
  • Breaking change noted (if applicable)

Deployment Notes (if any)

Installed site crontabs don't change on refresh. Post-merge steps for NERSC Spin, LCRC Chrysalis and PNNL Compy are in a PR comment below.

@tomvothecoder tomvothecoder added the devops DevOps task (e.g., DOE site ingestion, CI/CD, Docker) label Nov 17, 2026
@tomvothecoder

tomvothecoder commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator Author

Post-merge updates

Both jobs report to development and production. Until an environment's backend includes this change, its reports get 422; a missing env.*.sh file or an API that is down is skipped. These are logged and never affect the jobs.

1. NERSC Spin

  • Dev: confirm the merge deployed the backend and frontend.
  • Prod: once it is up, release the backend and frontend (component release tags; see docs/operations/deployment-and-release.md).
  • No CronJob changes. The Spin ingestion and diagnostics CronJobs keep job_health run, and Spin has no host crontab, refresh or launcher logs.
  • Verify on each deployed environment: Operations → Job health → Job type lists refresh and log archive.

2. LCRC Chrysalis (SIMBOARD_ROOT=/lcrc/group/e3sm/simboard)

  • Wait for the next 00:00 or 15:00 UTC refresh, or run make ops-refresh SIMBOARD_ROOT=/lcrc/group/e3sm/simboard. Confirm run_reported_job.sh exists under repository/simboard/backend/app/scripts/ingestion/sites/operations/.
  • Confirm operations/env.dev.sh holds the development service-account token. Add env.prod.sh once prod is up; until then prod reports are skipped.
  • Back up the current crontab: crontab -l > <protected-backup>.
  • In operations/chrysalis.crontab, replace the refresh and log-archiving entries with the two from templates/crontab.example. Keep chrysalis as the site argument and in chrysalis-refresh / chrysalis-log-archive. Keep the refresh entry above the DRY_RUN line, as in the template.
  • Reinstall: crontab /lcrc/group/e3sm/simboard/operations/chrysalis.crontab, then check crontab -l.
  • Verify:
    • After the next refresh, operations/simboard-ingestion-provision.log shows event=job_report_sent for start and finish per environment, and each dashboard lists chrysalis-refresh.
    • If global DRY_RUN=false is set, chrysalis-log-archive appears after Sunday 01:00 UTC.

3. PNNL Compy (SIMBOARD_ROOT=/compyfs/simboard)

  • Do the first three Chrysalis items (refresh, env.*.sh files, crontab backup) with this root.
  • In operations/compy.crontab, replace the same two entries. Use compy as the site argument and the job IDs compy-refresh / compy-log-archive. Leaving a chrysalis-* job ID makes reports fail with 409.
  • Reinstall: crontab /compyfs/simboard/operations/compy.crontab, then check crontab -l.
  • Verify as for Chrysalis.
    • The refresh always reports. A missing env.*.sh file logs API environment not readable and is skipped until it exists.
    • Log archiving reports only after global DRY_RUN=false.

Rollback

Restore the backed-up crontab and reinstall it. Registered jobs stay on the dashboard and turn stale.

@tomvothecoder
tomvothecoder marked this pull request as ready for review November 17, 2026 22:26
@tomvothecoder tomvothecoder added this to the 1. FY27Q1: Beta Launch milestone Nov 17, 2026
@tomvothecoder tomvothecoder self-assigned this Nov 17, 2026
@tomvothecoder
tomvothecoder merged commit a0c8c32 into E3SM-Project:main Nov 17, 2026
1 check passed
@tomvothecoder
tomvothecoder deleted the devops/501-maintenance-job-health branch November 17, 2026 22:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

devops DevOps task (e.g., DOE site ingestion, CI/CD, Docker)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[DevOps]: Report deployment refresh and log archiving job health

1 participant