Repository navigation
[DevOps]: Report deployment refresh and log archiving job health - #502
Merged
tomvothecoder merged 8 commits intoNov 17, 2026
Merged
tomvothecoder merged 8 commits into
tomvothecoder merged 8 commits into
Conversation
Collaborator
Author
Post-merge updatesBoth jobs report to development and production. Until an environment's backend includes this change, its reports get 1. NERSC Spin
2. LCRC Chrysalis (
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Show the site deployment refresh and weekly launcher-log archiving on the operations dashboard. Before this change, a failed refresh was silent: ingestion kept running the old revision and the dashboard stayed healthy.
refreshandlog_archivejob types with default thresholds of 1020/60 and 10200/120 minutes (silence/duration). No migration:scan_modeis a string column.sites/operations/run_reported_job.sh <site> <job-type> <command...>. It reuses the launcher'sjob_health start/finishreports, sojob_health.pyis unchanged.SIMBOARD_ENV_FILES(colon-separatedenv.*.shfiles).job_report_failed(up to 10 s per report). Neither affects the job or the other environment.mainfunction called on its last line, so Bash never rereads it after a refresh replaces it.DRY_RUN=falseinline: cron applies the globalDRY_RUNline only to later entries, and the refresh has no dry-run mode.setup-ingestion-operations.md.Reporting stays best effort: a missing API environment or failed report never stops either job or changes its exit code.
Accepted risks:
run_reported_job.sh, whose path is fixed in installed crontabs. A break in the wrapper itself would stop site refreshes; report failures cannot.staleafter 60 minutes.simboard-ingestion-provision.logandsimboard-log-archiving.log.Validation:
make backend-test(2572 passed, 26 skipped),pnpm test(77 passed),make frontend-lintandmake pre-commit-runpassed.Checklist
Deployment Notes (if any)
Installed site crontabs don't change on refresh. Post-merge steps for NERSC Spin, LCRC Chrysalis and PNNL Compy are in a PR comment below.