Repository navigation
fix(ci): report timed-out main CI runs instead of letting cancelled hide them - #16280
Conversation
…ide them main-ci-failure-issue.yml only admitted `failure` conclusions, but a job that hits its execution cap concludes `cancelled` — so timed-out main runs never filed an issue. That is how QwenLM#16214 (the E2E docker leg rebuilding the sandbox image past its 60-minute cap on cold mainland hosts) hid for weeks (QwenLM#16272). Admit `cancelled` at the trigger, then discriminate in the live step: a cancelled run is reportable only when a job carries the execution-time-timeout annotation; a plain concurrency supersede is skipped, and an unreadable annotation fails safe toward not reporting. The download projections include cancelled jobs so the timed-out leg is named in the report. The infra classifier is unchanged, so a pure-timeout run (no failure jobs) files rather than reruns, and normal failure handling is untouched. Meta-tests: the trigger pin and routing matrix now reflect the admitted cancelled conclusions; three new live-step tests witness timeout→report, concurrency→skip, and unreadable→skip; the download-selector and jobs-fetch pins are updated, as is the ci-platform-lanes trigger pin.
|
|
|
Thanks for the PR — and for the persistence through five rounds on it. Template looks good ✓ (every required heading present; Problem: observed, not theoretical. A job that hits Direction: aligned. This is the repo's own CI observability — a failure tracker that silently ignores one whole class of failure is worse than no tracker, because it manufactures confidence. No auth, sandbox, model-selection, telemetry, release or public-contract surface is touched, so nothing here needs a direction escalation. CHANGELOG: no direct reference (no Size: not applicable — no Approach: this matches what I'd have proposed, and beats it in one place. I would have inlined the annotation discrimination in the One thing to settle before this merges, flagged now so it isn't discovered at merge time: #16277 is a second, competing implementation of the same fix. Different author, also Risk: no elevated risk signals — none of the changed paths match the revert-correlated set. Moving on to code review. 🔍 中文说明感谢这个 PR,也感谢你在五轮评审里的坚持。 模板完整 ✓(必需标题齐全, 问题: 是已观测到的,不是理论性的。撞到 方向: 对齐。这是仓库自身的 CI 可观测性——一个会静默忽略整类失败的追踪器比没有追踪器更糟,因为它制造了虚假的信心。不涉及 auth、sandbox、模型选择、telemetry、发布或公开契约面,因此无需方向层面的升级。CHANGELOG:无直接对应( 规模: 不适用——未触及任何 方案: 与我自己的设想一致,并且有一处更好。我原本会把 annotation 判别逻辑内联在 合并前需要先定下来的一件事,现在就提出来以免合并时才发现:#16277 是同一修复的第二个、相互竞争的实现。 作者不同,同样写 风险: 无升级风险信号——改动路径均未命中与 revert 相关的文件集合。 进入代码审查 🔍 — Qwen Code · qwen3.8-max-2026-09-02 Reviewed at |
Code reviewI read the whole diff and then went back through the workflow at this commit to check the things the diff alone can't show — the step's output contract, the env the new script needs, and which ref the checkout resolves. No blocking defects. What I verified:
The tests are the strongest part of this. They run the real step bash and, in the Three non-blocking notes. Per AGENTS.md's five-round rule I'm recording these rather than asking you to chase them:
Your body's test count also predates the later test additions and should be refreshed whenever you next touch the description. Files changed (10 of 10 shown)
TestingThis is an unattended CI run, so I did not build or execute anything from this PR — the evidence below is the PR's own CI, read through the API for the reviewed commit. All 208 check-runs on
About those 4 cancellations, since a cancelled check on a PR whose whole subject is cancellation deserves an explanation rather than a wave: they are all the Not verified, and why: nothing here has been observed against a real Sandboxed verification would settle this: Real-scenario tmux testing: N/A — unattended CI run, and this change has no user-visible terminal surface (CI workflow internals only). 中文说明代码审查: 我通读了整个 diff,并回到该 commit 上的 workflow 核对 diff 本身看不出来的部分——步骤的输出契约、新脚本所需的环境变量、以及 checkout 解析到哪个 ref。未发现阻塞性缺陷。已核实:提前退出写入了 测试是这份改动最扎实的部分:它执行真实的步骤 bash,并在 real-script 区块中直接 spawn 实际发布的脚本、只对 三条非阻塞意见(按 AGENTS.md 的五轮规则,我只作记录,不要求逐条修复):其一,PR 正文描述的 fail 方向与代码相反——Risk & Scope 说读不到就沉默跳过,Evidence 表也如此,但代码的 正文里的测试计数也早于后续新增的用例,下次改动描述时一并刷新即可。 测试证据: 这是无人值守的 CI 运行,我没有构建或执行本 PR 的任何内容——以下证据是 PR 自身的 CI,通过 API 读取所审 commit 的结果。 未验证的部分及原因:这里没有任何针对真实 沙箱验证可以定这件事: 真实场景 tmux 测试:N/A——无人值守 CI 运行,且本改动没有面向用户的终端界面(仅 CI workflow 内部)。 — Qwen Code · qwen3.8-max-2026-09-02 Reviewed at |
|
Confidence: 3/5 — the code and its tests are good enough to merge; what stops me is a decision that isn't mine to make, not a defect I found. Stepping back: my independent proposal for this problem was to widen the trigger and discriminate on annotations inside the The thing I'd thank you for in six months is the test design. Spawning the shipped script itself against a stubbed So why not approve? Two reasons, and only the first is a real blocker. #16277 is a competing implementation of this same fix, and choosing between them is a maintainer's call. It's by a different author, it also says The description contradicts the code on the fail direction, and the contradiction matters more than a stale sentence usually would. The body tells a reviewer that an unreadable annotation is dropped in favour of silence, and calls that the user-requested tradeoff. The code fails open and reports, with a test asserting it. I happen to think failing open is the correct choice for a tracker whose entire failure mode was silence — but that's my judgement, and the body currently argues the opposite position while claiming a human asked for it. A maintainer should confirm the shipped direction is the wanted one, and then the body and the two design docs should say so. That's a five-minute edit, not a rework. Neither of these is a reason to request changes, so I'm not: per AGENTS.md's five-round rule, hygiene and documentation items don't block a PR this far along, and the two code fragilities I noted above (whitespace-split successor fields, and an empty Two practical notes for whoever picks this up. The standing If the maintainer resolves #16277 in this PR's favour and the body is corrected to describe the fail-open behaviour, my assessment is that this is ready — I found nothing in the code that I'd want changed first. 中文说明Confidence: 3/5 — 代码与测试本身已具备合并条件;让我停下来的不是一个我找到的缺陷,而是一个不该由我做出的决定。 退一步看:我对这个问题的独立设想是放宽触发器、并在 六个月后我会感谢你的地方是测试设计。直接 spawn 实际发布的脚本、只对 那为什么不批准?两个理由,其中只有第一个是真正的阻塞。 #16277 是同一修复的竞争实现,而在两者之间做选择是 maintainer 的决定。 它出自另一位作者,同样写着 描述在 fail 方向上与代码矛盾,而这个矛盾比一般的过期文字更要紧。 正文告诉评审者:annotation 读不到时会被丢弃、选择沉默,并称之为「用户要求的取舍」。代码却是 fail open 上报,并且有测试断言这一点。我个人认为,对一个其唯一失效模式就是沉默的追踪器而言,fail open 是正确的选择——但这是我的判断,而正文目前在主张相反的立场,同时声称这是人提出的要求。maintainer 应当确认落地的方向就是想要的方向,然后让正文和两份设计文档如实描述它。这是五分钟的修改,不是返工。 这两条都不构成 request changes 的理由,所以我没有这么做:按 AGENTS.md 的五轮规则,走到这一步的 PR,卫生与文档类问题不该阻塞;我在上面指出的两处代码脆弱性(后继字段的空白切分、以及 给接手的人两条实用信息。本 PR 上仍然生效的 如果 maintainer 在 #16277 与本 PR 之间选择了本 PR,并且正文被更正为描述 fail-open 行为,我的评估是它可以合并——代码层面我没有找到任何我会要求先改的东西。 — Qwen Code · qwen3.8-max-2026-09-02 Reviewed at |
qwen-code-review-bot
left a comment
There was a problem hiding this comment.
LGTM, looks ready to ship — CI landed green after the review. ✅
qwen-code-review-bot
left a comment
There was a problem hiding this comment.
Reviewed. Suggestions are inline.
Not explored to full depth (tool budget reached): "agent reverse-audit (round 2)": did not read the writer jobs ( analyze outputs consumers, .github/workflows/main-ci-failure-issue.yml ~500-1067) end to end — grepped them for conclusion / ….
Not fully linted (tool limitation, not a blocker): .github/workflows/main-ci-failure-issue.yml — actionlint ran workflow-level checks; embedded run: shell deferred (source mapping not yet supported).
中文说明
已审查。 建议见行内评论。
未探索到全部深度(达到工具调用预算):"agent reverse-audit (round 2)":did not read the writer jobs ( analyze outputs consumers, .github/workflows/main-ci-failure-issue.yml ~500-1067) end to end — grepped them for conclusion / …。
未完整检查(工具限制,非阻断):.github/workflows/main-ci-failure-issue.yml——actionlint 已运行 workflow 级检查;内嵌 run: shell 延迟处理(源映射尚未支持)。
— deepseek-v4.1-flash-1017 via Qwen Code /review (v0.26.7)
… unknown Review on QwenLM#16280: real concurrency cancels carry the runner shutdown / higher- priority / operation-canceled annotations, not just "operation was canceled", and a timed-out job can carry a shutdown annotation alongside the timeout. The previous discriminator only looked for the timeout text and skipped everything else, so it rested on "absence of timeout = concurrency" and dropped unreadable cancellations — re-creating the hiding problem QwenLM#16272 fixes. Classify by positive evidence on both sides instead: a timeout annotation reports (even with a coexisting shutdown annotation); a concurrency annotation with no timeout skips; neither, or unreadable jobs/annotations, fails open and reports like the rest of the live step. Fixtures cover the shutdown and higher-priority concurrency forms, timeout+shutdown coexistence, and the fail-open paths; breaking the concurrency signature reddens the skip tests.
|
⏳ Historical-head review — head moved to 中文说明⏳ 历史 head 评审 —— 本次评审进行中 head 移动到了 |
…r dedup, docs R1-4 (security): only trusted triggers (push/schedule/dispatch on main) run the cancellation classifier; PR-triggered cancellations stay on the supersede path so contributor-authored annotation text cannot drive the PAT-backed reporter. R1-1: renderFailureComment now keeps cancelled jobs, so a timed-out leg is named in the issue/PR comment even when no job concluded failure (the QwenLM#16214 shape). R1-2: factor the classifier into cancelled_class with a total_count receipt and explicit unreadable handling; unknown fails open with a warning instead of asserting a supersede it never established. R1-3: a timeout-backed cancelled successor owns the older run's result (same shared classifier), preventing a duplicate report; log wording corrected. R1-6: the classifier runs after the stale-attempt check so a stale event is attributed to staleness, not to a cancellation cause. R1-7: the test stub resolves the check-run id (job id != check-run id) and applies the caller's --jq with real jq; RUN_CONCLUSION env is pinned; fixtures cover shutdown / higher-priority / coexisting timeout / unreadable / PR arms. R1-5: sync the two design docs (both languages) with the new behaviour.
Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com> Patrol-Run: qwen-pr-closeout/jmw1wuw3xco
main-ci-failure-issue.yml grew to 63933 bytes, 6793 over its recorded
57140 (allowance 4096), so the workflow-size ratchet failed. Move the new
cancelled_class helper into .github/scripts/ci/classify-cancelled-run.sh,
which is what the ratchet asks for ("long steps into .github/scripts/"),
and trim review-narrative prose to the load-bearing why. Behaviour is
unchanged: both call sites invoke the script.
The workflow-shape pin that read the classifier's attempt-jobs fetch out
of the workflow now reads the script.
Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
Patrol-Run: qwen-pr-closeout/jmw1wuw3xco
qwen-code-review-bot
left a comment
There was a problem hiding this comment.
Not explored to full depth (tool budget reached): "agent 5": did not execute vitest on scripts/tests/main-ci-failure-issue-workflow.test.js ; the mutation verdicts above are reading-based, not run..
Not fully linted (tool limitation, not a blocker): .github/workflows/main-ci-failure-issue.yml — actionlint ran workflow-level checks; embedded run: shell deferred (source mapping not yet supported).
中文说明
未探索到全部深度(达到工具调用预算):"agent 5":did not execute vitest on scripts/tests/main-ci-failure-issue-workflow.test.js ; the mutation verdicts above are reading-based, not run.。
未完整检查(工具限制,非阻断):.github/workflows/main-ci-failure-issue.yml——actionlint 已运行 workflow 级检查;内嵌 run: shell 延迟处理(源映射尚未支持)。
— deepseek-v4.1-flash-1017 via Qwen Code /review (v0.26.7)
Local verification: PASS
Claims
Key evidenceTrigger gate (
Renderer: for the #16214 shape (1 cancelled job, 0 failure) the before arm renders a 265-byte comment naming no job; the head renders 405 bytes naming Suites: before Static: Harness provenance / discarded runsProbes, extracted step body, stubbed Residuals
中文版(完整对照)本地验证:PASS
声明
关键读数
缺口
|
…R1-2, R2-1, R2-4) Critical R1-2: the classifier returned unknown (fail-open report) for a cancelled run whose attempt has no cancelled job at all — a run displaced while queued certifies 0==0 and reads no annotation, so it filed a "Main CI failed" issue for a run that tested nothing. Track whether any cancelled job was seen; none means supersede (concurrency), while unknown now only covers cancelled jobs whose annotations are unreadable/unmatched or uncertifiable pages. R2-1: tests for the certification false branch (partial pages -> unknown) and the no-jobs case (-> concurrency). R2-4: a negative successor case with a real id whose cancelled job is a supersede, exercising the == 'timeout' false branch.
qwen-code-review-bot
left a comment
There was a problem hiding this comment.
Convergence: round 3 posted 6 inline comment(s), 4 of them reported for the first time; the previous round posted 5 (5 new). Findings keep coming back to the same files: .github/scripts/ci/classify-cancelled-run.sh (findings in round 2; 2 more now); scripts/tests/main-ci-failure-issue-workflow.test.js (findings in round 2; 2 more now). A cluster that keeps producing siblings usually means the fixes are treating instances of a shared root cause — triaging that cause before the next round, or splitting an independent cluster into its own pull request, tends to end the loop faster than fixing them one at a time. (Observation only — nothing was withheld from this review because of this observation.)
中文说明
收敛情况:第 3 轮发布了 6 条行内评论,其中 4 条是首次提出;上一轮发布了 5 条(其中 5 条首次提出)。发现反复回到同一批文件:.github/scripts/ci/classify-cancelled-run.sh(第 2 轮已出过发现,本轮又有 2 条);scripts/tests/main-ci-failure-issue-workflow.test.js(第 2 轮已出过发现,本轮又有 2 条)。一个不断再生兄弟发现的簇,通常意味着逐条修复只在处理同一根因的实例——先定位并处理该根因,或把独立的簇拆成单独的 PR,通常比逐条修复更快结束循环。(仅为观察——本轮评审未因此扣留任何内容。)
— deepseek-v4.1-flash-1017 via Qwen Code /review (v0.26.7)
Local verification: PASS
Claims
Key evidenceTrigger gate (
Before arm: all ten yield
Suites: before Harness provenance / discarded runsProbes, extracted step body, stubbed Residuals
中文版(完整对照)本地验证:PASS
五条主张
关键读数门禁(
分类器(13 形态): 测试:before 残余与缺口
|
…M#16280) R3-1 (Critical): the classifier read only `cancelled` jobs, so a run whose job already concluded `failure` before the rest was cancelled classified as a deliberate supersede and dropped the regression. Read `failure`/`timed_out` jobs first and fail open on them. R1-2 (Critical): a cancelled run whose jobs all stopped before their first step still fell through to `unknown`, which reports. Treat "no cancelled job ran a step" as nothing-executed, preserving the zero-job behaviour. R3-2: give that outcome its own token and case arm, so the caller's notice no longer claims an annotation this path never read. R3-4: the annotation-error test now uses two cancelled jobs, because with one the fall-through prints the same `unknown` as the guard and cannot see it removed. Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com> Patrol-Run: qwen-pr-closeout/jmw2mktggdp
…l fetch R3-3: the comment no longer claims the gate admits only failures. R2-4: pin the classifier fetch including ?per_page=100 and --paginate so flag-stripping reddens.
|
Qwen Code review did not complete successfully. The review pipeline failed before a review could be posted. A transient error is retried automatically; if you are seeing this, retry with |
Refresh base to ecc64a7. Main now carries ef9c891 (QwenLM#16351), whose ci.yml corepack cache fix targets the pool-side "Cannot find module qwen-ci-corepack/v1/pnpm/11.24.0/bin/pnpm.mjs" failure that killed Linux unit shard 8/8 on run 41925293939 attempt 4. Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com> Patrol-Run: qwen-pr-closeout/jmw3vl6sufe
|
merge-ready — 94/94 scripted assertions passed; verified head 中文摘要结论:merge-ready。 中心主张成立:因 job 执行超时而被取消的 main run 现在会进入 analyze 并上报;并发取消与「没有任何 job 执行过步骤」的取消仍然沉默,failure 路径不变。发布过程中 head 从 A/B(见 01-ab-matrix.png 与下表): 10 个取消类场景在 base 臂全部被触发闸门拒绝(0 个上报);head 臂按设计 7 个上报、3 个沉默。其中真实超时 run 真实语料: 最近 12 个 main 上 发现(均为 Suggestion,不阻塞): ① 未覆盖: 真实 Central claim and A/BClaim. A main CI run that concluded The A/B runs the real workflow at each arm: the
Counts. 10 cancelled-class scenarios: base reports 0/10, head reports 7/10 and keeps the 3 that must stay silent silent. The central flip is Real corpus. The 12 newest Renderer. On the same real jobs payload, the base renderer names no job; head names the timed-out docker leg and its cancelled step ( Head movementThe head moved from
Corrections
FindingsThe mutation matrix holds 19 guards: 17 are pinned by the live suites (17/19 killed, including the positive-control row that breaks the timeout regex and reddens three tests). The two survivors are adjudicated below — one is dead code, one is a real coverage gap. The matrix is per guard, not per commit. All three findings are Suggestion-severity and non-blocking; none is a failed assertion. S1 —
|
|
Updated the verification report above for the head movement: |
|
Status at
CI is green (27 passed, 0 failed), no conflicts. Re-review requested from @chiga0 (the earlier approval was dismissed by the test-only head move). Ready for approval. |
|
@qwen-code /triage All review threads are resolved (24/24) and the required lanes are green on The standing |
评审结论(head
|
|
Sandboxed verification: The verification job did not complete (checkout, runner, or setup error) and produced no report. See the workflow run for details. 中文 — 判定:
|
|
⏸️ Deferring to the maintainers — needs a human call on this one. Escalating this one to a maintainer — I reviewed it fully and found no defect that would justify requesting changes, but there is a decision here that the gate should not make on its own. 1. #16277 is a competing implementation of the same fix, and only one can land. Both PRs are open against 2. The PR description states the opposite fail direction from the code. Risk & Scope says the change "fails safe toward silence" when jobs or annotations cannot be read, and calls that "the user-requested tradeoff"; the Evidence block lists unreadable annotations as a skip. The shipped code fails open and reports in that case, and a test asserts exactly that behaviour. I think failing open is the right call for a tracker whose original failure mode was silence, but the body currently argues the other side while attributing it to a human decision — so whoever signs off should confirm the shipped direction is the intended one, then correct the body and the two design docs this PR touches (both the nightly and managed-agent-gate docs describe only the timeout-vs-concurrency split and omit the fail-open Neither item is a code defect, and per AGENTS.md's five-round rule I am not requesting changes over documentation at this point in a PR's life. Two smaller fragilities are recorded in the review comment above and are unreachable or theoretical today. For context on merge mechanics: the standing If #16277 is resolved in this PR's favour and the description is corrected, my assessment is that it is ready to merge — I found nothing in the code I would want changed first. 中文说明把这个 PR 上交 maintainer——我做了完整评审,没有发现足以支持 request changes 的缺陷,但这里有一个不该由闸门自行做出的决定。 1. #16277 是同一修复的竞争实现,二者只能合入一个。 两个 PR 都开向 2. PR 描述陈述的 fail 方向与代码相反。 Risk & Scope 说在 jobs 或 annotation 读不到时「朝沉默方向 fail-safe」,并称之为「用户要求的取舍」;Evidence 表把 annotation 读不到列为跳过。而实际发布的代码在这种情况下 fail open 并上报,且有一条测试正是断言该行为。我认为对一个原本失效模式就是沉默的追踪器而言,fail open 是正确选择,但正文目前在主张相反立场、并将其归因于人的决定——所以签署的人应当确认落地方向就是意图方向,然后更正正文以及本 PR 改动的两份设计文档(nightly 与 managed-agent-gate 两份都只描述了 timeout 与 concurrency 的二分,遗漏了 fail-open 的 两项都不是代码缺陷,按 AGENTS.md 的五轮规则,在 PR 走到这个阶段我不会因文档问题提出 request changes。另有两处较小的脆弱性已记录在上方评审评论中,目前不可达或仅为理论风险。 关于合并机制的背景:仍然生效的 如果 #16277 的取舍结果有利于本 PR、且描述被更正,我的评估是它可以合并——代码层面我没有找到任何我会要求先改的东西。 |
|
Triage re-run completed without a new review.
The stage comments above were updated with the latest result. View workflow run. 上方各阶段评论已更新为最新结果。查看工作流运行。 |
|
Sandboxed verification: The verification job was cancelled before producing a report. 中文 — 判定:
|
…LM#16280 R4-1) The R4-1 fix widened the successor-ownership rule from `timeout` to `timeout | unknown`, and the classifier now skips on `nothing-executed` as well as `concurrency`, but both design docs this PR touches still described the pre-fix rule: only a timeout-backed cancelled successor owns the result, and a cancelled run reports only when a job carries a timeout annotation. Restate the shipped rule in all four files (both languages): the two verdicts that stay silent, the fail-open `unknown`/`unreadable` reports, a cancelled run that also carries a `failure`/`timed_out` job, and the `unreadable` successor that deliberately does not take ownership because a failed read certifies nothing about what that run concluded. Docs only — no workflow or script bytes change, so the workflow-size ratchet is unaffected.
…M#16280) Merging main into this branch brought in 7cb46d9 (QwenLM#13969), which added a seven-line comment — 501 bytes — to main-ci-failure-issue.yml. The merged file is 61381 bytes, 145 past the 57140 + 4096 ratchet, so the workflow-size gate fails on growth this PR did not author. This PR's own contribution to that file is untouched: the merge's diff on it is exactly main's seven added lines and nothing else. Record the merged size so the ratchet reflects the tree this branch now carries.
|
merge-ready — 143/143 scripted assertions passed; verified head 中文摘要结论:merge-ready。 中心主张在新 head 上重新测过并成立:因 job 执行超时被取消的 main run 现在进入 analyze 并上报;并发取消、以及没有任何 cancelled job 跑过步骤的取消仍然沉默; 上一轮发现: 三项 Suggestion( 本轮新增一条 Suggestion: 判别脚本里 jobs 读取失败返回的 未覆盖: 真实 Previous findings at this head
Central claim and A/BClaim. A main CI run that concluded The A/B runs the real workflow at each arm: the
Counts. The 14 cancelled-class cells: base reports 0/14, head reports 9/14 and keeps the 5 that must stay silent silent. The 10 successor cells: base reports 9, head reports 6 — the three flips are exactly the duplicate reports R4-1 removes. Real dataThe shipped classifier was run against the live API, not a fixture, on the 25 newest cancelled push runs on The replay was calibrated three ways on the same runs: the shipped classifier against the real API, the replayed Head movementThe head advanced from
Corrections
FindingsAll findings are non-blocking Suggestions; none is a failed assertion. The mutation matrix holds 20 guards: 15 killed, 5 survived and adjudicated below. N1 — the jobs-read failure token (
|
| build | successor-jobs-unreadable |
|---|---|
head (c8c21f3f) |
reported (…does not own this result; reporting this run) |
M05 (unreadable → unknown) |
skipped (the newer push run owns its result) |
This is the same failure class as R4-1, one call site over: the author correctly separated unreadable from unknown for annotation reads and pinned that, but not for the jobs read. The code at head is correct; the gap is coverage. A one-line classifier case (jobs read fails → unreadable) plus a successor cell with jobs_error would pin it.
Repro: node harness/run-mutations.mjs --ids M05-jobs-read-unknown-token --label b6 → survived; node harness/run-cell.mjs --tree head --cell successor-jobs-unreadable → reported at head, and the mutant's own run is saved as mutant-out/m05-successor-jobs-unreadable.json (skipped).
R1 — cancelled_seen is a dead disjunct (prior S1)
classify-cancelled-run.sh:107 tests "${cancelled_seen}" != 'true' || "${executed_seen}" != 'true'. executed_seen can only become true inside the loop iteration that already set cancelled_seen = true, so the first test is subsumed. M02 (drop it) survives. This is inferred from the loop structure, not re-enumerated this round: the prior round measured 3 reachable (cancelled_seen, executed_seen) states with 0 mismatches, and this file is byte-identical to that round's (3ec397040948… at both heads). A clause that cannot decide an outcome; removing it is a readability call, not a behavioural one.
R2 — the successor's run_attempt is still unpinned (prior S2)
main-ci-failure-issue.yml:208 passes ${successor_attempt:-1}. M10 (hardcode 1) survives. The new R5-5 test pins the jq projection's \(.id // "") and \(.run_attempt // "") fields, but not the caller's use of the bound attempt: no test supplies an attempt-2 successor whose attempt 1 classifies differently.
Driving the mutant: successor-attempt-2 (attempt 1 = concurrency, attempt 2 = timeout) reads attempt 2 at head and skips; the mutant reads attempt 1, classifies concurrency, and reports — a duplicate report for one sustained failure. Re-running a cancelled run is the normal infra recovery, so attempt 2 is exactly when the gate runs.
Repro: node harness/run-mutations.mjs --ids M10-successor-attempt-hardcode-1 --label b3 shows the suite stays green. Then edit "${successor_attempt:-1}" to "1" in the workflow, run node harness/run-cell.mjs --tree head --cell successor-attempt-2 (the mutant calls /attempts/1 and reports; saved as mutant-out/m10-successor-attempt-2.json), and git -C trees/head checkout HEAD -- . to restore.
R3 — the TSV field shift is latent, not fixed (prior S3)
classify-cancelled-run.sh:71 reads id, check_run_id, steps_run from an @tsv row with IFS=$'\t'. Tab is IFS whitespace, so a run of tabs collapses: with an empty check_run_url, the step count slides into check_run_id and a 24-step job queries check-run 24. The cancelled-no-checkrun-url cell still does exactly that; the read now fails, so the verdict is unreadable and the run is reported (fail-open). It was not observed in the 25-run corpus — every real cancelled job carried check_run_url — so it is latent. A non-whitespace delimiter (or a |-joined row) would make the empty field unambiguous.
R4 — the pull_request_review half of the PR guard is unpinned (review R5-3)
main-ci-failure-issue.yml:124 guards both pull_request and pull_request_review. M13 (drop the second disjunct) survives all 260 tests. The cells managed-agent-pr-cancelled and managed-agent-pr-review-cancelled both pass at head, but no test distinguishes the arms. The Managed Agent gate does run on pull_request_review, and its annotation text is contributor-authored; the guard is the only barrier, and it has no pin on that half.
Mutation matrix
15/20 killed. The positive control M01 (break the timeout signature) is killed by reports a cancelled run backed by a job execution-time timeout, so the runner and the selection can make the suite fail. Vacuity: the tests this round depends on are killed by their own guards — M03/M04 by keeps a confirmed timeout when a sibling annotation read fails, M11/M12 by the two successor-ownership tests, M17/M18 by evaluates the successor jq projection end to end, M06 by fails open when the jobs pages cannot be certified complete, M07 by reports a timeout even when a shutdown annotation coexists on the job, M08 by reports a cancelled run that also carries a failed job.
Survivors are classified, not just listed: M02 and M19 are clauses that cannot decide an outcome (M19's empty-id guard: with the guard removed the classifier is called with an empty id, gh fails, unreadable is returned, and the caller still reports — same outcome); M05, M10 and M13 are coverage gaps with a demonstrated or reachable behavioural difference, each described above. None is a defect at head: every flip above is what the mutant does.
CI on this head
At the time this round posted, c8c21f3f had 28 pass, 1 pending, 7 skipping, 0 failed. The pending check was TUI parity snapshots (ink vs opentui), unrelated to this diff. The review decision is still CHANGES_REQUESTED from round 4; the only Critical there (R4-1) is fixed and pinned, and the author reports all review threads resolved. Apart from this report, no new third-party reviews or comments arrived during the round.
Coordination, not a code finding: #16277 is still open and implements the same issue with different semantics (rerun vs file). Two implementations cannot both land; this needs a maintainer decision independent of this round's evidence.
Not covered
- A live
workflow_runcompletion event. Event delivery and the Actions runtime were not exercised; the gate expression was evaluated with GitHub-expression semantics and everything downstream of the event was run. download,summary,file_issueend-to-end. The renderer and the download projections are pinned by the author's tests and unchanged by this PR; the full chain from a live run to a filed issue was not replayed.- Annotation variants outside the recorded runs. Runner-shutdown text, localized text, and run-level
timed_outconclusions were not observed onmain; the suite covers the shutdown form synthetically. workflow_dispatchandpull_request_reviewarms with real events. Their gate and guard cells are evaluated and replayed, but no live event of either shape was captured.- Per-commit attribution. The mutation matrix is per guard. This round did not enumerate the PR's commits one row each, so no per-commit coverage is claimed.
Methodology
The round ran on macOS (Darwin 25.x, Node 24.18.1, jq 1.7.1) in two detached worktrees under the artifact directory, seeded from the repository's installed dependencies. Base is the PR's baseRefOid 1b9aef22; head is c8c21f3f. Cells are .mjs harnesses: harness/run-cell.mjs extracts the analyze gate and the live step from each arm's YAML with the yaml parser, evaluates the gate, and runs the step bash with harness/gh-stub.mjs replaying recorded payloads; harness/assert-gate.mjs prints the gate truth table; harness/run-mutations.mjs applies one anchored mutant at a time, runs the same suites (--bail=1), restores with git checkout HEAD --, and refuses to continue on a dirty tree; harness/assertions.mjs derives every number in this report from the recorded files and writes assertions.json. Raw logs are in logs/, per-cell results in cells-out/, fixtures (unmodified API responses) in fixtures/, and mutation logs in mutation-logs/.
— via Qwen Code /verify-pr (v0.26.9).
|
Closeout at 1. The three Criticals, checked in code and by mutation at this head
A green suite proves nothing unless it can redden, so I reverted each fix in a scratch tree and re-ran the suite that pins it (restoring and confirming a clean
2. Description and design docs correctedThis closes item 2 of the bot's 2026-11-07 deferral to maintainers: the body argued the opposite fail direction from the code and attributed it to a human decision.
3. Base sync and the size ratchet
4. What I ran at this headDedicated worktree at
CI at this head, settled 2026-11-07T11:27:40Z and read back at 11:29:02Z (run 42365470676): 37 checks — 30 success / 7 skipped / 0 failure. The 7 skips are the platform-conditional lanes (macOS and Windows unit tests, SDK Java, installed-npm acceptance, CLI-no-sandbox integration). Counting basis, so the numbers are reproducible: the check-runs API for this head SHA ( The count rose from the 5. Deferrals and the one open human decision
中文说明在 三个 Critical 的代码级落点:R1-2 —— 变异实证:五个修复我逐个在临时树里改回去再跑钉住它的套件(每次跑完 描述与文档:bot 在 2026-11-07 上交维护者时点的第 2 项已收口——正文不再主张「朝沉默方向 fail-safe」,改为写清真实方向:只有 base 同步与尺寸棘轮: 实测:两套件 vitest 260 passed / 0 failed( CI 终态:2026-11-07T11:27:40Z 全部落定、11:29:02Z 回读,37 个 check —— 30 success / 7 skipped / 0 failure;7 条 skipped 都是平台条件 lane(macOS 与 Windows 单测、SDK Java、installed-npm 验收、CLI-no-sandbox 集成)。 defer 与唯一待裁项:R5-2/R5-3/R5-4 仍按五轮规则 defer 在 #16319;另补记两条(记录:N3 attempt 交接无测试钉、D4 |
|
@qwen-code /triage Triage on
Only the verdict needs to catch up with the code on this head. |
|
Local verification: PASS
This head is four commits past the last verified head
Causal evidence and controls:
Residuals:
Local verdict ≠ merge approval: reviewers and required approvals still decide. 中文摘要(完整版)本地验证结论:PASS(merge-reference,run 5)
这轮的关键判断:当前 head 比上一个已验证 head 五条声明:
因果证据与控制项:两臂都跑真实的 workflow 步骤正文 + 真实的分类脚本; 缺口(不算已验证):没有制造真实的 本地 PASS 不等于合并批准。 |
Stale bot review anchored to an old commit; all Critical findings independently verified fixed on head c8c21f3 (team double audit + wenshao public merge-ready verification). Re-review requested on the current head.
|
Sandboxed verification: ✅ passed — merge-ready (agent verdict) - workflow run Ran the PR in an isolated, token-free container: A/B against the base build, mock-free harness assertions, targeted gates. Advisory evidence for human reviewers — not a review, an approval, or a CI check. Scripted assertions: 177 passed · 0 failed · 177 total Targeted gates (existing tests, not counted individually above): 328 passed · 0 failed across 3 suite(s) Flakiness gate: ✅ 3 changed test file(s) x 5 identical rounds, no divergence 中文 — 判定:✅ 通过 · 可合入(agent 判定)沙箱验证在隔离、无凭证的容器中执行了该 PR 的代码(与 base 构建 A/B 对照、无 mock harness 断言、定向门禁)。仅作为评审证据,不构成评审、批准或 CI 检查。 脚本断言:177 通过 · 0 失败 · 177 总计 定向门禁(已有测试,不逐条计入上面的断言):328 通过 · 0 失败,共 3 个套件 抖动门:✅ 3 changed test file(s) x 5 identical rounds, no divergence Verification reportPR 16280 round 1 verification — Verdict: merge-ready — 177/177 scripted assertions passed, 0 failed. Verified head: 中文摘要结论:merge-ready(可合并) — 177/177 条脚本断言全部通过,0 失败。 A/B 结论:同一个 stubbed-
变异矩阵:7 个守卫各一个变异,7/7 被钉住(harness 翻转 + 契约套件变红双重验证);作者声称的两个实证完全复现(超时签名改写 → 恰好 5 例变红;优先级重排 → 优先级用例变红)。 门禁:契约套件 260/260、签名套件 63/63、size ratchet、prettier、actionlint、shellcheck、bash -n 全绿。 未覆盖:真实 Central claim and A/B proofClaim: a main-CI run cancelled by a job execution-time timeout now files a failure report, while non-breakage cancellations stay silent and Instrument: the
Gate-level matrix: 24 trigger-shape cells × both arms — cancelled admitted exactly on the push/schedule/dispatch/managed-gate-PR arms, Captures: Review findings at the verified head
Resolved Suggestions re-measured in passing: R1-1 (comment names the cancelled leg — C1), R1-4 trust gate (C8 + M6), R1-6 ordering (C10), R1-7 check-run id + real CorrectionsNone — no earlier review or bot description was found inaccurate at this head. FindingsNo blocking finding; no non-blocking defect reproduced. Two measured observations, both consistent with the PR's stated intent:
Not covered
MethodologyCI verify lane ( Flakiness gate logEvidence imagesHarness scripts and raw logs are in the workflow run artifacts (7-day retention). — qwen3.8-max-2026-09-02 via Qwen Code /verify-pr (v0.26.9; skill 1766260e45d7) |
|
Triage re-run completed without a new review.
The stage comments above were updated with the latest result. View workflow run. 上方各阶段评论已更新为最新结果。查看工作流运行。 |
|
merge-ready — 236/236 scripted assertions passed; verified head 中文摘要结论:merge-ready(236/236 断言通过)。 本轮在同一 head 试合并(本轮新增的关键测量): 真实语料: 用发布的判别脚本直连 API 跑今天最新的 25 条 cancelled push/main run(扫描 237 条已完成的 push run):17 timeout / 3 concurrency / 3 unknown / 2 nothing-executed,0 unreadable、0 错误。17 条 timeout 全部来自 上一轮发现(同一 head 重新测量): R1( 本轮新增变异: 新增四条单点变异( 未覆盖: 真实 Previous findings at this headRound 2 (2026-11-07 18:38–19:29, same head
Correction to round 2's own record
Central claim and A/BClaim. A main CI run that concluded The A/B runs the real workflow at each arm: the
Counts. The 14 originally-recorded cancelled-class cells (12 Trial merge into current main
Measured on the merged tree: contract suites 260/260, signature 63/0, Real dataThe shipped classifier was run against the live API on today's 25 newest completed cancelled push/main runs across the watched workflows (237 completed push runs scanned). Verdicts: 17 All 17 timeouts are the All three Calibration: the live-API verdict and this round's replay of the live step agree on all five recorded runs ( FindingsAll findings are non-blocking Suggestions; none is a failed assertion. The mutation matrix holds 24 single-point guards: 18 killed, 6 survived, each survivor adjudicated below. N1 — the jobs-read failure token (
|
| build | successor-jobs-unreadable |
|---|---|
head (c8c21f3f) |
reported (…does not own this result; reporting this run) |
M05 (unreadable → unknown) |
skipped (the newer push run owns its result) |
This is the same failure class as R4-1, one call site over: the author separated unreadable from unknown for annotation reads and pinned that, but not for the jobs read. The code at head is correct; the gap is coverage. A one-line classifier case (jobs read fails → unreadable) plus a successor cell with jobs_error would pin it.
Repro: apply M05's one-token edit in trees/head, then node harness/run-cell.mjs --tree head --cell successor-jobs-unreadable (mutant output saved as mutant-out/m05-successor-jobs-unreadable.json; restore with git -C trees/head checkout HEAD -- .).
M23 — the timed_out conjunct of the failed-jobs guard is unpinned (new this round)
classify-cancelled-run.sh:55 treats a job concluding failure or timed_out under a cancelled run as fail-open evidence; both the code comment and the body name both conclusions. Only the failure half has a witness: M22 (drop failure) is killed by reports a cancelled run that also carries a failed job, while M23 (drop timed_out) survives every suite. I had predicted M23 would be killed; the measurement overruled the prediction.
| build | cancelled-timedout-job-concurrency |
|---|---|
head (c8c21f3f) |
reported (failing open and reporting) |
M23 (failure only) |
skipped (concurrency supersede … not reporting) |
The shape is a cancelled run whose cancelled jobs carry a concurrency annotation while one job concluded timed_out. Head reports it through the fail-open guard; without the conjunct the run is certified as a supersede and its only report is dropped — the #16272 hiding shape, one clause over. Severity is Suggestion because the code at head is right; a fixture whose timed_out job sits beside a concurrency annotation would pin it.
Repro: node harness/run-mutations.mjs --ids M23-failed-jobs-drop-timed-out-conjunct --label m23 → survived; then the same one-token edit plus node harness/run-cell.mjs --tree head --cell cancelled-timedout-job-concurrency (mutant output mutant-out/m23-cancelled-timedout-job-concurrency.json).
R1 — cancelled_seen is a dead disjunct (round-2 S1)
classify-cancelled-run.sh:107 tests "${cancelled_seen}" != 'true' || "${executed_seen}" != 'true'. executed_seen can only become true inside the loop iteration that already set cancelled_seen = true, so the first test is subsumed. M02 (drop it) survives. A clause that cannot decide an outcome; removing it is a readability call, not a behavioural one.
R2 — the successor's run_attempt is still unpinned (round-2 S2)
main-ci-failure-issue.yml:208 passes ${successor_attempt:-1}. M10 (hardcode 1) survives: no test supplies an attempt-2 successor whose attempt 1 classifies differently. Driving it earlier showed the consequence — successor-attempt-2 reads attempt 2 at head and skips; hardcoded to 1 it reads attempt 1, classifies concurrency, and reports a duplicate. Re-running a cancelled run is the normal infra recovery, so attempt 2 is exactly when the gate runs.
R3 — the TSV field shift is latent, not fixed (round-2 S3)
classify-cancelled-run.sh:71 reads id, check_run_id, steps_run from an @tsv row with IFS=$'\t'. Tab is IFS whitespace, so a run of tabs collapses: with an empty check_run_url, the step count slides into check_run_id and a 24-step job queries check-run 24. Re-measured at this head: the cell still issues check-runs/24/annotations, the read fails, the verdict is unreadable, and the run reports (fail-open). Every real cancelled job in today's 25-run corpus carried a check_run_url, so it remains latent.
R5-3 — the pull_request_review half of the PR guard has no test (round-2)
M13 (drop pull_request_review from the trust guard) survives. The only live-step guard test is skips a PR-triggered cancellation without trusting contributor annotations, which drives the pull_request event; the pull_request_review cells pass only because the surviving pull_request clause catches them, so the review clause can be deleted with every suite green.
S2 — the empty-successor-id guard has no witness (round-2 survivor, carried forward)
M19 rewrites if [[ -n "${successor_id}" ]] to admit the empty id and survives. Inferred from the code (not driven this round): with an empty id the classifier is invoked with no run id, its gh call fails, it answers unreadable, which does not own the result — so both builds report and the only difference is one wasted failing API call. That is why it stays a Suggestion: the guard is defensive and nothing observable changes without it.
Review comments and the body's claims
The PR's own Reviewer Test Plan is walked row by row: every line of its Evidence table is a measured cell, including failure leaving the classifier unengaged (its call log carries no attempts/ or check-runs/ request). The workflow_dispatch residual #16319 records is measured as behaviour: the gate admits a dispatch on any ref (cancelled-dispatch-feature-sdk-java, gate admit) and the live step classifies it, because the trust guard covers only pull_request/pull_request_review. Bounding it: a dispatch needs write access, and a fork's branch is not dispatchable in the base repository; what it covers is a maintainer dispatching on a branch that exists in the repo, which is already listed as a deferral and is not a regression here.
Not covered
- No live
workflow_runcancellation event was delivered end-to-end; the cancellation shapes are replayed from recorded payloads plus aghstub. - The download → summary → issue-writer pipeline past the
analyzegate's live step is exercised only by the unit suites; no issue was filed against GitHub (no PAT in this round). - Of the 25 real corpus runs, one (
42382919050) was inspected down to its annotation text; the other 24 were classified but their annotations were not read individually. - The
workflow_dispatchtrust residual (ci: failure-report refinements — name only timed-out cancelled legs; skip cancelled PR runs at the trigger #16319) is measured as behaviour only. - Windows/macOS lane behaviour and the Java lane are not exercised here; CI at this head skips them by design.
- The mutation matrix is per-guard, not per-commit: 24 rows over the guards this PR adds or changes, not one row per commit of the PR.
- S2 (M19) is inferred from code, not driven; its behavioural difference is unobservable in the cells available.
Methodology
Maintainer-local round on macOS, in the host working copy (prepare reported isolation: none detected); every arm executes in a scratch git worktree under tmp/pr16280-verify-20261108-020028/trees/, and no host-level install or auth state was modified. Head c8c21f3f, base 1b9aef22, trial-merge tip 1feab51d; the merge commit is fa0037209df5 with parents exactly 1feab51d + c8c21f3f. Harness scripts and raw logs live in the artifact directory (harness/, logs/, cells-out/, mutation-logs/); the evidence captures are evidence/01-ab-matrix.png … 04-real-corpus.png. Node 24.18.1, vitest 3.2.7, Bash 5.3.
qqqys
left a comment
There was a problem hiding this comment.
APPROVE — reviewed at head c8c21f3fd2d8baffa6f83ee008443dae26fca43e. All four previously filed Criticals are fixed in the code at this head, and I found no new blocking correctness, security, data-loss, regression, or compatibility issue.
Prior blocking findings, each re-checked against this head
R1-2 — unknown reported a cancellation that proved nothing (zero-job runs, then steps: [] runs): fixed. The classifier now has a dedicated nothing-executed verdict (classify-cancelled-run.sh:107-113) reached when cancelled_seen != true (no cancelled job at all — displaced while queued) or executed_seen != true (every cancelled job stopped before its first step), and the live step maps it to superseded=true (main-ci-failure-issue.yml:134-137). executed_seen is set from the real step count projected by the loop's jq (((.steps // []) | length)), so both shapes the two rounds named now stay silent, and the zero-job {"total_count":0,"jobs":[]} payload still certifies (0 == 0) and reaches the same verdict instead of falling to unknown.
R3-1 — a cancelled run carrying a failure/timed_out job was silently dropped: fixed. The failed-job guard (classify-cancelled-run.sh:54-63) runs before the annotation loop and emits unknown when any job concluded failure or timed_out, so the generic The operation was canceled. annotation on a sibling can no longer classify a real regression as a supersede. A jq failure in that guard also emits unknown, not a false negative. The verdict reaches the *) arm and reports.
R4-1 — successor ownership credited only timeout while the live step also reports unknown: fixed. The successor case is now timeout | unknown) successor_owns=true (main-ci-failure-issue.yml:208-210), fed by the same script as the current-run gate, so the two cannot drift. The projection was widened to carry .id and .run_attempt for that second classification, and the cancelled arm is guarded on a non-empty successor_id.
R5-1 — a confirmed timeout downgraded to unreadable by a sibling's failed annotation read: fixed and unchanged since it was verified by execution. The read failure now sets read_failed=true and continues (classify-cancelled-run.sh:85-86), and the verdict block ranks timeout above unreadable (:99-104). I confirmed via compare/1ca9a1d05...c8c21f3fd that classify-cancelled-run.sh is absent from the diff — the script is byte-identical to the head at which @chiga0 executed it and killed 9/9 mutants, including the three around this fix, the failed-job guard and the successor timeout|unknown set.
Also re-checked: R1-4 — the pull_request / pull_request_review arms set superseded=true and exit before the classifier is invoked (main-ci-failure-issue.yml:124-126), so contributor-authored annotation text never decides a PAT-backed filing; R1-6 — the classifier runs only after the stale-attempt check and is gated on superseded != 'true' (:123); R1-1 — renderFailureComment now keeps cancelled jobs (main-failure-signature.mjs:414-417), matching the download-side projection widened to failure|cancelled at both jq sites, so a timed-out leg is actually named in the filed body; R1-5 — both design docs now state the shipped direction in both languages, including the unreadable non-ownership rule.
Independent scan of the current diff
No injection surface. Every untrusted-ish value reaches the script through env: (RUN_CONCLUSION, WORKFLOW_RUN_ID, RUN_ATTEMPT, EVENT_NAME, HEAD_SHA, HEAD_BRANCH) rather than inline ${{ }} in a shell body, and is expanded quoted; check_run_id comes from a jq TSV projection and is quoted in the annotations call; successor_id/successor_attempt come from a jq projection and are quoted. REPO is :?-required.
The script cannot abort the step or fake a verdict. It runs set -uo pipefail without -e, so each gh api/jq failure is handled by an explicit if ! branch rather than killing the substitution inside the set -euo pipefail step; every exit path ends in echo <verdict> with status 0. The pagination receipt (([.[].total_count] | max) vs the merged job count) means a partial page list reads unreadable rather than masquerading as a complete list that happens to lack the timed-out job.
Widening the gate and the download projections adds no untrusted path. A cancelled PR-arm run exits before both the classifier and the download steps (if: steps.live.outputs.superseded != 'true'), so the newly admitted conclusion reaches log analysis only for push/schedule/workflow_dispatch runs of the repo's own workflows.
.size-baseline is exact. The ledger entry for main-ci-failure-issue.yml is 61381 and the blob at this head is 61381 bytes — the growth came from main and this PR records it rather than drifting the gate.
One noted trade-off, explicitly not a blocker
The successor-ownership set is timeout | unknown, while the live step also reports on unreadable. During a jobs/annotations API outage an older run and its successor can therefore each file for one incident. This is reasoned in three places rather than overlooked — the script header (unreadable "says nothing about what the run concluded, so the successor check must not defer to it"), the inline comment at :202-207, and the design-doc sentence added by 34ed530fb — and deferring to an uncertifiable read would drop the only report of a real regression, which is the costlier error. Left as recorded, not requested.
CI
Lint & Static, Test (ubuntu-latest, Node 22.x), all eight Linux unit shards, the three integration shards and the TUI lanes pass at this head. The only non-pass entry is a pending authorize job in the review workflow, which does not gate this change. @wenshao approved at this same head.
Keep main's ['failure','cancelled'] job semantics while landing the PR's coverage line: namedJobs is used both as the named-job list and as the coverage denominator, so a timed-out (cancelled) lane is named and the body stays self-consistent. F1: semantic conflict resolved; main's cancelled-job test stays green. F2: logCount falls back to 0. F3: new fixture pins the cancelled/timed_out exclusion (kills M4).
|
Released in v0.27.1. |
main QwenLM#16280 landed the same gate widening this PR was built on, so the conflicts were structural rather than semantic: take main's hoisted conclusion check and its larger schedule-arm name list, and keep this PR's classifier work in the signature module, which merged cleanly. The size baseline moves to the merged file's actual byte size, and the meta-test's substring assertion now pins the `fromJSON` form the conclusion check took on main.












What this PR does
Makes the main-CI failure tracker report a run that was cancelled by a job execution-time timeout, while still ignoring the cancellations that carry no breakage. The trigger now admits
cancelledconclusions alongsidefailure, and a new shared classifier script decides which cancellations are real: thelivestep calls it after the stale-attempt check, and it reads the cancelled jobs' annotations to report a run whose job carriesexceeded the maximum execution time. The successor-ownership check asks the same script about the newer run, so the two call sites cannot drift — and a cancelled successor that reports itself now owns the older run's result, which is what keeps one sustained timeout from filing twice. The download projections also include cancelled jobs so the timed-out leg is named in the filed issue.Why it's needed
main-ci-failure-issue.ymlonly admittedfailure. A job that hits itstimeout-minutescap concludescancelled, notfailure— so a timed-out main run never filed an issue. That is exactly how #16214 hid: the E2Esandbox:dockerleg rebuilt the sandbox image past its 60-minute cap on cold mainland hosts, run after run reportedcancelled, and the tracker stayed silent for weeks. This is the follow-up tracked as #16272.A
cancelledconclusion covers two very different things, so the trigger alone cannot decide: a deliberate concurrency supersede (should not file) and a job timeout (a real breakage, should file). Positive annotation evidence is the discriminator — a timeout carriesThe job has exceeded the maximum execution time of …; a concurrency cancel carries onlyThe operation was canceled.; and a run cancelled before any job ran a step carries no evidence to read at all.Reviewer Test Plan
How to verify
Read the cancellation block the
livestep gained. For acancelledrun that is not PR-triggered, and only once the stale-attempt check has had its say, it calls the new shared classifier (.github/scripts/ci/classify-cancelled-run.sh), which fetches the attempt's jobs (certified complete againsttotal_count), reads each cancelled job's check-run annotations, and answers with one of five verdicts. Two of them skip:concurrency(a shutdown / higher-priority / operation-canceled annotation and no timeout) andnothing-executed(no cancelled job ran a step).timeoutreports on positive evidence. The remaining two fail open and report:unknown(the run was read and matched neither pattern, or it also carries afailure/timed_outjob) andunreadable(a jobs or annotation read failed, or the pages could not be certified). The successor check yields only to a verdict that reports —timeoutorunknown. The rest of the pipeline is unchanged; the infra classifier still selectsfailurejobs only, so a pure-timeout run (no failure jobs) files an issue rather than rerunning, and normal failure handling is untouched.Evidence (Before & After)
The shipped classifier and the real
livestep bash, exercised against a stubbedgh:scripts/tests/main-ci-failure-issue-workflow.test.js+scripts/tests/ci-platform-lanes.test.jsunder the CI scripts config: 260 passed..github/scripts/ci/main-failure-signature.test.mjsundernode --test: 63 passed. The suites spawn the shipped classifier itself instead of restating its logic, and apply the caller's real--jqprojection with realjq, so verdict precedence and the successor hand-off (run id, thenrun_attempt) are pinned where they live. Mutation witnesses: rewriting the timeout signature reddens 5 cases; reordering the verdict block so a failed annotation read outranks a confirmed timeout reddens the case that pins that precedence. YAML parses;bash -nclean on the step and on the classifier; prettier clean.The branch also carries a merge of current
main: the lint-gate freshness check requires the branch to hold main's current gate definitions before this lane's green means anything. That merge is where the extra contract cases in the count above come from, and it also brought main's own #13969 growth of this workflow file (501 bytes), which the size ratchet's baseline now records in the same push — so the workflow-size gate passes. This PR's own edits to that file are unchanged by the sync.Tested on
Local: the two contract suites (which execute the real step bash and the shipped classifier with a stubbed
gh), the signature suite, and the two mutation checks. Linux marked not-tested because the workflow was not exercised on a liveworkflow_runevent here; the behavior is covered by the bash-level tests.Risk & Scope
cancelledlets more runs enteranalyze, so the classifier must reliably skip the cancellations that carry no breakage. It skips on two certified verdicts only —concurrency, andnothing-executed, which is the queued-displacement shape ci: main-ci-failure-issue.yml ignores cancelled runs — timed-out main CI never alerts (how #16214 hid) #16272 asks to keep silent. Everything it cannot certify fails open into a report. That direction is deliberate for a tracker whose original failure mode was weeks of silence, and it is the rule this workflow already states for its own stale reports: a report that turns out to be wrong costs one close, while a dropped report costs the regression. The accepted cost is the mirror image of silence — an unreadable or unclassifiable cancellation files, and when the read that failed was the successor's, this run keeps its own fail-open report rather than deferring to a verdict nobody established, so that one shape can file twice.workflow_runcancellation event end-to-end; the unrelated reporter-queueing gap in ci: main-push failures file no issues — the reporter waits 20–100 min for a hosted runner and is then dropped as superseded #15470 (failure runs whose reporter job is itself superseded). Three known residuals are recorded as deferrals in ci: failure-report refinements — name only timed-out cancelled legs; skip cancelled PR runs at the trigger #16319 rather than fixed here: a runner-shutdown cancellation reading asconcurrency, the annotation-trust guard not covering theworkflow_dispatcharm, and the wording of the two skip notices.failureruns — their path through the pipeline is unchanged. The new reports are cancellations that previously produced nothing: a confirmed job timeout, plus the fail-open verdicts. One report goes away: a cancelled successor that reports itself now owns the older run's result, so a sustained timeout files once instead of once per run.Linked Issues
Fixes #16272
中文说明
这个 PR 做了什么
让 main CI 失败追踪器上报因 job 执行超时而被取消的 run,同时仍然忽略不带任何破坏的取消。触发器现在在
failure之外也接纳cancelled结论,并由一个新的共享判别脚本决定哪些取消是真的:live步骤在 stale-attempt 检查之后调用它,它读取被取消 job 的 annotation,只有当某个 job 带有exceeded the maximum execution time时才上报。后继归属检查也问同一个脚本,因此两个调用点不会漂移——会自我上报的 cancelled 后继现在拥有旧 run 的结果,这正是一次持续超时不会重复建单的原因。download 投影也纳入 cancelled job,以便在提交的 issue 里点名超时的 leg。为什么需要
main-ci-failure-issue.yml以前只接纳failure。撞到timeout-minutes上限的 job 结论是cancelled而非failure——所以超时的 main run 从不提交 issue。这正是 #16214 藏起来的方式:E2Esandbox:dockerleg 在冷的大陆主机上重建 sandbox 镜像超过 60 分钟上限,一个又一个 run 报cancelled,追踪器却沉默了几周。这是记为 #16272 的后续项。cancelled结论涵盖两种截然不同的情况,所以仅凭触发器无法判断:故意的并发顶替(不该提交)和 job 超时(真实破坏,该提交)。正向的 annotation 证据是判别依据——超时带The job has exceeded the maximum execution time of …;并发取消只带The operation was canceled.;而在任何 job 跑起步骤之前就被取消的 run 根本没有证据可读。评审测试计划
如何验证
读
live步骤新增的取消处理块。对非 PR 触发的cancelledrun,且在 stale-attempt 检查表态之后,它调用新的共享判别脚本.github/scripts/ci/classify-cancelled-run.sh:脚本抓取该 attempt 的 jobs(用total_count确证完整),读取每个 cancelled job 的 check-run annotation,给出五个结论之一。其中两个跳过:concurrency(带 shutdown / higher-priority / operation-canceled annotation 且无超时)与nothing-executed(没有任何 cancelled job 跑过步骤)。timeout按正向证据上报。剩下两个 fail open 上报:unknown(run 读到了但两种模式都不匹配,或它同时带failure/timed_outjob)与unreadable(jobs 或 annotation 读取失败,或分页无法确证)。后继检查只让位给会上报的结论——timeout或unknown。流水线其余部分不变;infra 分类器仍只选failurejob,所以纯超时 run(无 failure job)会提交 issue 而非重跑,正常失败处理不受影响。证据(前后对比)
真实发布的判别脚本与真实
live步骤 bash,对着 stubgh演练:CI 同式的 scripts 配置下,
scripts/tests/main-ci-failure-issue-workflow.test.js+scripts/tests/ci-platform-lanes.test.js:260 通过;node --test下的.github/scripts/ci/main-failure-signature.test.mjs:63 通过。套件直接 spawn 发布的判别脚本而不是在测试里重述其逻辑,并用真jq执行调用方真实的--jq投影,所以结论优先级与后继交接(run id,然后run_attempt)都钉在它们真正所在的位置。变异实证:改写超时签名会让 5 例变红;把结论块里「读取失败」提到「已确证超时」之前,会让钉住该优先级的那一例变红。YAML 可解析;步骤与判别脚本bash -n干净;prettier 干净。分支同时并入了当前
main:lint gate freshness 检查要求分支先带上 main 当前的 gate 定义,这条 lane 的绿灯才有意义。上面计数中多出的契约用例来自这次合并;合并也带进了 main 侧 #13969 给该 workflow 文件加的 501 字节,size 棘轮的 baseline 已在同一次 push 里记录合并后的尺寸,因此 workflow-size 门禁通过。本 PR 自己对该文件的改动未因这次同步而变化。测试平台
本地:两个契约套件(用 stub
gh执行真实步骤 bash 与发布的判别脚本)、签名套件、以及两项变异检查。Linux 标记未测试,因为没有在真实workflow_run事件上跑过该 workflow;行为已由 bash 级测试覆盖。风险与范围
cancelled让更多 run 进入analyze,所以判别脚本必须可靠地跳过不带破坏的取消。它只在两种被确证的结论上跳过——concurrency,以及nothing-executed(后者正是 ci: main-ci-failure-issue.yml ignores cancelled runs — timed-out main CI never alerts (how #16214 hid) #16272 要求保持沉默的排队顶替形状)。凡不能确证的一律 fail open 上报。对一个原本的失效模式就是数周沉默的追踪器来说,这个方向是刻意的,也正是该 workflow 对自己的过期报告已经写下的规则:一份事后证明不该发的报告只花掉一次关闭,漏掉一份则漏掉整个回归。代价是沉默的反面——读不到或无法归类的取消会建单;而当读取失败的正是后继那一次时,本次运行保留自己的 fail-open 上报,而不是让位给一个谁都没有确证过的结论,所以这一种形状可能建两份单。workflow_run取消事件的端到端;ci: main-push failures file no issues — the reporter waits 20–100 min for a hosted runner and is then dropped as superseded #15470 里无关的 reporter 排队缺口(reporter job 自身被顶替的 failure run)。三处已知残余记在 ci: failure-report refinements — name only timed-out cancelled legs; skip cancelled PR runs at the trigger #16319 作为 defer,本轮不修:runner shutdown 的取消被归为concurrency、annotation 信任边界未覆盖workflow_dispatch臂、以及两条跳过提示的措辞。failurerun 无——它们走流水线的路径不变。新增的上报是以前什么都不产生的取消:已确证的 job 超时,加上 fail-open 的结论。少掉的是一份:会自我上报的 cancelled 后继现在拥有旧 run 的结果,所以一次持续超时只建一份单,而不是每个 run 一份。关联 Issue
Fixes #16272