Skip to content

[DevOps]: Tune scheduled job health thresholds from observed run times #469

Open

Description

@tomvothecoder

Task Details

Tune scheduled job health thresholds after real runs are reported. #390 (PR #468) registers each job with starting max_silence_minutes and max_duration_minutes values based only on cron schedules:

Job Schedule max_silence_minutes max_duration_minutes
staging every 15 min 45 60
archive daily at 12:00 UTC 1560 720
diagnostics every 6 h 420 300

Thresholds that are too tight mark healthy jobs stale; thresholds that are too loose hide missed or hung runs.

Jobs: chrysalis-{staging,archive,diagnostics} in the dev and prod APIs, and spin-perlmutter-{staging,archive,diagnostics} in each Spin namespace.

Acceptance criteria

  • Each registered job's thresholds reflect at least one week of observed run times.
  • No job shows stale during normal operation; a missed schedule or hung run shows stale within one schedule interval.
  • The suggested values in docs/operations/nersc-spin-runbook.md (Workload 3, step 7) and docs/operations/setup-ingestion-operations.md match the tuned values.

Steps to Complete

  1. After about one week of reporting, collect run durations per job from run_metrics log events (duration_seconds) or run records from GET /api/v1/ingestions/jobs.
  2. Set max_duration_minutes to about twice the longest observed run, and max_silence_minutes to at least the schedule interval plus a margin for skipped runs.
  3. Re-register each job with the new values (PUT /api/v1/ingestions/jobs/{job_id}, admin only) in every environment.
  4. Update the suggested values in the docs.

Additional Notes

Activity

  1. added
    devopsDevOps task (e.g., DOE site ingestion, CI/CD, Docker)
    on Nov 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    devopsDevOps task (e.g., DOE site ingestion, CI/CD, Docker)

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions