You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[DevOps]: Tune scheduled job health thresholds from observed run times #469
Tune scheduled job health thresholds after real runs are reported. #390 (PR #468) registers each job with starting max_silence_minutes and max_duration_minutes values based only on cron schedules:
Job
Schedule
max_silence_minutes
max_duration_minutes
staging
every 15 min
45
60
archive
daily at 12:00 UTC
1560
720
diagnostics
every 6 h
420
300
Thresholds that are too tight mark healthy jobs stale; thresholds that are too loose hide missed or hung runs.
Jobs: chrysalis-{staging,archive,diagnostics} in the dev and prod APIs, and spin-perlmutter-{staging,archive,diagnostics} in each Spin namespace.
Acceptance criteria
Each registered job's thresholds reflect at least one week of observed run times.
No job shows stale during normal operation; a missed schedule or hung run shows stale within one schedule interval.
The suggested values in docs/operations/nersc-spin-runbook.md (Workload 3, step 7) and docs/operations/setup-ingestion-operations.md match the tuned values.
Steps to Complete
After about one week of reporting, collect run durations per job from run_metrics log events (duration_seconds) or run records from GET /api/v1/ingestions/jobs.
Set max_duration_minutes to about twice the longest observed run, and max_silence_minutes to at least the schedule interval plus a margin for skipped runs.
Re-register each job with the new values (PUT /api/v1/ingestions/jobs/{job_id}, admin only) in every environment.
Activity