Skip to content

Helix Job Monitor: treat abandoned Azure DevOps jobs as finished and failed - #17823

Draft
premun wants to merge 2 commits into
mainfrom
copilot/detect-abandoned-jobs
Draft

premun wants to merge 2 commits into
mainfrom
copilot/detect-abandoned-jobs

Conversation

@premun

@premun premun commented Nov 20, 2026 •

Copy link
Copy Markdown
Member

When an agent stops responding, its Azure DevOps job can stay inProgress with result: null for hours, even though the timeline already shows an agent-loss error on the job and a completed parent Phase. The Helix Job Monitor waits for every non-monitor job to reach completed, so in that state it keeps running until its own timeout.

Abandoned-job detection (HelixJobMonitorUtilities)

  • Timeline records now deserialize issues (new AzureDevOpsTimelineIssue type).
  • New GetAbandonedNonMonitorJobRecords returns non-monitor jobs that are not completed and match either signal:
    • Agent-lost error: an error issue whose message contains We stopped hearing from agent or Agents can't recover from restarts.
    • Parent phase completed: the parent Phase record is completed. This catches cases where the error text doesn't match, such as the Windows "remote provider deprovision" example in the issue comments.
  • AreNonMonitorJobsComplete counts abandoned jobs as finished, and HasFailedNonMonitorJobs counts them as failed.
  • Jobs in the retrying-submitter ignore set are still excluded from the failure check.

Runner and reporting

  • JobMonitorRunner logs one warning per abandoned job (de-duplicated across polls) through the new StatusReporter.LogAbandonedPipelineJob.
  • The final failure message is the existing Build Analysis-matched line, so no new failure text is introduced.

Docs and tests

  • Design/SemanticBehavior.md §5.4 (termination step) now covers abandoned jobs.
  • HelixJobMonitorUtilitiesTests adds cases for:
    • agent-lost error
    • completed parent phase
    • a running job with an unrelated error, which must stay incomplete
    • the ignored-retry interaction

Review notes

  • The agent-lost check matches Azure DevOps message text. If that wording changes, only the parent-phase check remains.
  • The parent-phase check assumes Azure DevOps never marks a phase completed while one of its jobs is still legitimately running. That held for every reported case, but it is not confirmed with Azure DevOps.

To double check:

Copilot AI balanced review requested due to automatic review settings November 20, 2026 15:08

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot wasn't able to review any files in this pull request. Check if the Files changed in this pull request are included in default exclusions.

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 20, 2026 15:12

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues are recorded for the abandoned-job handling changes.

No open issues 🎉

Review effort: Balanced

@premun premun changed the title [WIP] Detect abandoned jobs and shut down early Helix Job Monitor: treat abandoned Azure DevOps jobs as finished and failed Nov 20, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Helix Job Monitor] Detect abandoned jobs and shut down early

3 participants