Skip to content

fix(ci): report timed-out main CI runs instead of letting cancelled hide them - #16280

Merged
wenshao merged 17 commits into
QwenLM:mainfrom
yiliang114:fix/ci-tracker-cancelled
Nov 7, 2026
Merged

wenshao merged 17 commits into
QwenLM:mainfrom
yiliang114:fix/ci-tracker-cancelled

Conversation

@yiliang114

@yiliang114 yiliang114 commented Nov 4, 2026 •

Copy link
Copy Markdown
Collaborator

What this PR does

Makes the main-CI failure tracker report a run that was cancelled by a job execution-time timeout, while still ignoring the cancellations that carry no breakage. The trigger now admits cancelled conclusions alongside failure, and a new shared classifier script decides which cancellations are real: the live step calls it after the stale-attempt check, and it reads the cancelled jobs' annotations to report a run whose job carries exceeded the maximum execution time. The successor-ownership check asks the same script about the newer run, so the two call sites cannot drift — and a cancelled successor that reports itself now owns the older run's result, which is what keeps one sustained timeout from filing twice. The download projections also include cancelled jobs so the timed-out leg is named in the filed issue.

Why it's needed

main-ci-failure-issue.yml only admitted failure. A job that hits its timeout-minutes cap concludes cancelled, not failure — so a timed-out main run never filed an issue. That is exactly how #16214 hid: the E2E sandbox:docker leg rebuilt the sandbox image past its 60-minute cap on cold mainland hosts, run after run reported cancelled, and the tracker stayed silent for weeks. This is the follow-up tracked as #16272.

A cancelled conclusion covers two very different things, so the trigger alone cannot decide: a deliberate concurrency supersede (should not file) and a job timeout (a real breakage, should file). Positive annotation evidence is the discriminator — a timeout carries The job has exceeded the maximum execution time of …; a concurrency cancel carries only The operation was canceled.; and a run cancelled before any job ran a step carries no evidence to read at all.

Reviewer Test Plan

How to verify

Read the cancellation block the live step gained. For a cancelled run that is not PR-triggered, and only once the stale-attempt check has had its say, it calls the new shared classifier (.github/scripts/ci/classify-cancelled-run.sh), which fetches the attempt's jobs (certified complete against total_count), reads each cancelled job's check-run annotations, and answers with one of five verdicts. Two of them skip: concurrency (a shutdown / higher-priority / operation-canceled annotation and no timeout) and nothing-executed (no cancelled job ran a step). timeout reports on positive evidence. The remaining two fail open and report: unknown (the run was read and matched neither pattern, or it also carries a failure/timed_out job) and unreadable (a jobs or annotation read failed, or the pages could not be certified). The successor check yields only to a verdict that reports — timeout or unknown. The rest of the pipeline is unchanged; the infra classifier still selects failure jobs only, so a pure-timeout run (no failure jobs) files an issue rather than rerunning, and normal failure handling is untouched.

Evidence (Before & After)

The shipped classifier and the real live step bash, exercised against a stubbed gh:

cancelled + timeout annotation          -> superseded=false (report)
cancelled + "operation was canceled"    -> superseded=true  (skip: concurrency)
cancelled + no job ran a step           -> superseded=true  (skip: nothing-executed)
cancelled + annotations unreadable      -> superseded=false (fail open: report)
cancelled + matched neither pattern     -> superseded=false (fail open: report)
cancelled + a job concluded failure     -> superseded=false (fail open: report)
PR-triggered cancellation               -> superseded=true  (skip: contributor text untrusted)
failure                                 -> classifier not engaged (unchanged)

cancelled successor, timeout or unknown -> older run yields (one report, not two)
cancelled successor, concurrency        -> older run still reports
cancelled successor, unreadable         -> older run still reports (a failed read certifies nothing)

scripts/tests/main-ci-failure-issue-workflow.test.js + scripts/tests/ci-platform-lanes.test.js under the CI scripts config: 260 passed. .github/scripts/ci/main-failure-signature.test.mjs under node --test: 63 passed. The suites spawn the shipped classifier itself instead of restating its logic, and apply the caller's real --jq projection with real jq, so verdict precedence and the successor hand-off (run id, then run_attempt) are pinned where they live. Mutation witnesses: rewriting the timeout signature reddens 5 cases; reordering the verdict block so a failed annotation read outranks a confirmed timeout reddens the case that pins that precedence. YAML parses; bash -n clean on the step and on the classifier; prettier clean.

The branch also carries a merge of current main: the lint-gate freshness check requires the branch to hold main's current gate definitions before this lane's green means anything. That merge is where the extra contract cases in the count above come from, and it also brought main's own #13969 growth of this workflow file (501 bytes), which the size ratchet's baseline now records in the same push — so the workflow-size gate passes. This PR's own edits to that file are unchanged by the sync.

Tested on

OS Status
🍏 macOS ✅
🪟 Windows N/A
🐧 Linux ⚠️

Local: the two contract suites (which execute the real step bash and the shipped classifier with a stubbed gh), the signature suite, and the two mutation checks. Linux marked not-tested because the workflow was not exercised on a live workflow_run event here; the behavior is covered by the bash-level tests.

Risk & Scope

  • Main risk or tradeoff: admitting cancelled lets more runs enter analyze, so the classifier must reliably skip the cancellations that carry no breakage. It skips on two certified verdicts only — concurrency, and nothing-executed, which is the queued-displacement shape ci: main-ci-failure-issue.yml ignores cancelled runs — timed-out main CI never alerts (how #16214 hid) #16272 asks to keep silent. Everything it cannot certify fails open into a report. That direction is deliberate for a tracker whose original failure mode was weeks of silence, and it is the rule this workflow already states for its own stale reports: a report that turns out to be wrong costs one close, while a dropped report costs the regression. The accepted cost is the mirror image of silence — an unreadable or unclassifiable cancellation files, and when the read that failed was the successor's, this run keeps its own fail-open report rather than deferring to a verdict nobody established, so that one shape can file twice.
  • Not validated / out of scope: a live workflow_run cancellation event end-to-end; the unrelated reporter-queueing gap in ci: main-push failures file no issues — the reporter waits 20–100 min for a hosted runner and is then dropped as superseded #15470 (failure runs whose reporter job is itself superseded). Three known residuals are recorded as deferrals in ci: failure-report refinements — name only timed-out cancelled legs; skip cancelled PR runs at the trigger #16319 rather than fixed here: a runner-shutdown cancellation reading as concurrency, the annotation-trust guard not covering the workflow_dispatch arm, and the wording of the two skip notices.
  • Breaking changes / migration notes: none for failure runs — their path through the pipeline is unchanged. The new reports are cancellations that previously produced nothing: a confirmed job timeout, plus the fail-open verdicts. One report goes away: a cancelled successor that reports itself now owns the older run's result, so a sustained timeout files once instead of once per run.

Linked Issues

Fixes #16272

中文说明

这个 PR 做了什么

让 main CI 失败追踪器上报因 job 执行超时而被取消的 run,同时仍然忽略不带任何破坏的取消。触发器现在在 failure 之外也接纳 cancelled 结论,并由一个新的共享判别脚本决定哪些取消是真的:live 步骤在 stale-attempt 检查之后调用它,它读取被取消 job 的 annotation,只有当某个 job 带有 exceeded the maximum execution time 时才上报。后继归属检查也问同一个脚本,因此两个调用点不会漂移——会自我上报的 cancelled 后继现在拥有旧 run 的结果,这正是一次持续超时不会重复建单的原因。download 投影也纳入 cancelled job,以便在提交的 issue 里点名超时的 leg。

为什么需要

main-ci-failure-issue.yml 以前只接纳 failure。撞到 timeout-minutes 上限的 job 结论是 cancelled 而非 failure——所以超时的 main run 从不提交 issue。这正是 #16214 藏起来的方式:E2E sandbox:docker leg 在冷的大陆主机上重建 sandbox 镜像超过 60 分钟上限,一个又一个 run 报 cancelled,追踪器却沉默了几周。这是记为 #16272 的后续项。

cancelled 结论涵盖两种截然不同的情况,所以仅凭触发器无法判断:故意的并发顶替(不该提交)和 job 超时(真实破坏,该提交)。正向的 annotation 证据是判别依据——超时带 The job has exceeded the maximum execution time of …;并发取消只带 The operation was canceled.;而在任何 job 跑起步骤之前就被取消的 run 根本没有证据可读。

评审测试计划

如何验证

读 live 步骤新增的取消处理块。对非 PR 触发的 cancelled run,且在 stale-attempt 检查表态之后,它调用新的共享判别脚本 .github/scripts/ci/classify-cancelled-run.sh:脚本抓取该 attempt 的 jobs(用 total_count 确证完整),读取每个 cancelled job 的 check-run annotation,给出五个结论之一。其中两个跳过:concurrency(带 shutdown / higher-priority / operation-canceled annotation 且无超时)与 nothing-executed(没有任何 cancelled job 跑过步骤)。timeout 按正向证据上报。剩下两个 fail open 上报:unknown(run 读到了但两种模式都不匹配,或它同时带 failure/timed_out job)与 unreadable(jobs 或 annotation 读取失败,或分页无法确证)。后继检查只让位给会上报的结论——timeout 或 unknown。流水线其余部分不变;infra 分类器仍只选 failure job,所以纯超时 run(无 failure job)会提交 issue 而非重跑,正常失败处理不受影响。

证据(前后对比)

真实发布的判别脚本与真实 live 步骤 bash,对着 stub gh 演练:

cancelled + 超时 annotation             -> superseded=false(上报)
cancelled + "operation was canceled"    -> superseded=true (跳过:concurrency)
cancelled + 没有 job 跑过步骤           -> superseded=true (跳过:nothing-executed)
cancelled + annotation 读不到           -> superseded=false(fail open:上报)
cancelled + 两种模式都不匹配            -> superseded=false(fail open:上报)
cancelled + 某个 job 结论为 failure     -> superseded=false(fail open:上报)
PR 触发的取消                           -> superseded=true (跳过:贡献者文本不可信)
failure                                 -> 判别脚本不介入(不变)

cancelled 后继为 timeout 或 unknown     -> 旧 run 让位(一次故障一份报告)
cancelled 后继为 concurrency            -> 旧 run 仍然上报
cancelled 后继为 unreadable             -> 旧 run 仍然上报(读取失败什么都证明不了)

CI 同式的 scripts 配置下,scripts/tests/main-ci-failure-issue-workflow.test.js + scripts/tests/ci-platform-lanes.test.js:260 通过;node --test 下的 .github/scripts/ci/main-failure-signature.test.mjs:63 通过。套件直接 spawn 发布的判别脚本而不是在测试里重述其逻辑,并用真 jq 执行调用方真实的 --jq 投影,所以结论优先级与后继交接(run id,然后 run_attempt)都钉在它们真正所在的位置。变异实证:改写超时签名会让 5 例变红;把结论块里「读取失败」提到「已确证超时」之前,会让钉住该优先级的那一例变红。YAML 可解析;步骤与判别脚本 bash -n 干净;prettier 干净。

分支同时并入了当前 main:lint gate freshness 检查要求分支先带上 main 当前的 gate 定义,这条 lane 的绿灯才有意义。上面计数中多出的契约用例来自这次合并;合并也带进了 main 侧 #13969 给该 workflow 文件加的 501 字节,size 棘轮的 baseline 已在同一次 push 里记录合并后的尺寸,因此 workflow-size 门禁通过。本 PR 自己对该文件的改动未因这次同步而变化。

测试平台

OS 状态
🍏 macOS ✅
🪟 Windows N/A
🐧 Linux ⚠️

本地:两个契约套件(用 stub gh 执行真实步骤 bash 与发布的判别脚本)、签名套件、以及两项变异检查。Linux 标记未测试,因为没有在真实 workflow_run 事件上跑过该 workflow;行为已由 bash 级测试覆盖。

风险与范围

  • 主要风险或取舍:接纳 cancelled 让更多 run 进入 analyze,所以判别脚本必须可靠地跳过不带破坏的取消。它只在两种被确证的结论上跳过——concurrency,以及 nothing-executed(后者正是 ci: main-ci-failure-issue.yml ignores cancelled runs — timed-out main CI never alerts (how #16214 hid) #16272 要求保持沉默的排队顶替形状)。凡不能确证的一律 fail open 上报。对一个原本的失效模式就是数周沉默的追踪器来说,这个方向是刻意的,也正是该 workflow 对自己的过期报告已经写下的规则:一份事后证明不该发的报告只花掉一次关闭,漏掉一份则漏掉整个回归。代价是沉默的反面——读不到或无法归类的取消会建单;而当读取失败的正是后继那一次时,本次运行保留自己的 fail-open 上报,而不是让位给一个谁都没有确证过的结论,所以这一种形状可能建两份单。
  • 未验证 / 范围之外:真实 workflow_run 取消事件的端到端;ci: main-push failures file no issues — the reporter waits 20–100 min for a hosted runner and is then dropped as superseded #15470 里无关的 reporter 排队缺口(reporter job 自身被顶替的 failure run)。三处已知残余记在 ci: failure-report refinements — name only timed-out cancelled legs; skip cancelled PR runs at the trigger #16319 作为 defer,本轮不修:runner shutdown 的取消被归为 concurrency、annotation 信任边界未覆盖 workflow_dispatch 臂、以及两条跳过提示的措辞。
  • 破坏性变更 / 迁移说明:对 failure run 无——它们走流水线的路径不变。新增的上报是以前什么都不产生的取消:已确证的 job 超时,加上 fail-open 的结论。少掉的是一份:会自我上报的 cancelled 后继现在拥有旧 run 的结果,所以一次持续超时只建一份单,而不是每个 run 一份。

关联 Issue

Fixes #16272

…ide them

main-ci-failure-issue.yml only admitted `failure` conclusions, but a job that
hits its execution cap concludes `cancelled` — so timed-out main runs never
filed an issue. That is how QwenLM#16214 (the E2E docker leg rebuilding the sandbox
image past its 60-minute cap on cold mainland hosts) hid for weeks (QwenLM#16272).

Admit `cancelled` at the trigger, then discriminate in the live step: a
cancelled run is reportable only when a job carries the execution-time-timeout
annotation; a plain concurrency supersede is skipped, and an unreadable
annotation fails safe toward not reporting. The download projections include
cancelled jobs so the timed-out leg is named in the report. The infra
classifier is unchanged, so a pure-timeout run (no failure jobs) files rather
than reruns, and normal failure handling is untouched.

Meta-tests: the trigger pin and routing matrix now reflect the admitted
cancelled conclusions; three new live-step tests witness timeout→report,
concurrency→skip, and unreadable→skip; the download-selector and jobs-fetch
pins are updated, as is the ci-platform-lanes trigger pin.
@github-actions github-actions Bot added the review/self-reported The linked issue was opened by the PR author (self-reported) label Nov 4, 2026
@qwen-code-review-bot

qwen-code-review-bot commented Nov 4, 2026 •

Copy link
Copy Markdown
Collaborator

🚫 Qwen Triage was cancelled — view run. The run was cancelled before finishing. Check for a newer run before re-running.

🚫 Qwen Triage 已取消 —— 查看运行。运行未完成即被取消。重跑前请先确认是否有更新的运行。

@wenshao

wenshao commented Nov 4, 2026

Copy link
Copy Markdown
Collaborator

⚠️ Failed to process this request. Please re-mention the bot to retry.

@qwen-code-review-bot

qwen-code-review-bot commented Nov 4, 2026 •

Copy link
Copy Markdown
Collaborator

Thanks for the PR — and for the persistence through five rounds on it.

Template looks good ✓ (every required heading present; Environment is the optional one).

Problem: observed, not theoretical. A job that hits timeout-minutes concludes cancelled, and the trigger admitted only failure, so #16214 stayed invisible for weeks and #16272 tracks the follow-up. I checked the mechanism rather than taking the framing for it, and it holds: on.workflow_run in this workflow carries no conclusions: filter at all — only workflows and types: ['completed'] — so GitHub already delivers every conclusion and the job-level if: is the only gate. That is visible in the workflow's own history: 98 of its last 100 runs concluded skipped, i.e. events arriving and being turned away by the if:. So widening the if: is the right lever, and it is not dead code waiting on a trigger filter nobody changed.

Direction: aligned. This is the repo's own CI observability — a failure tracker that silently ignores one whole class of failure is worse than no tracker, because it manufactures confidence. No auth, sandbox, model-selection, telemetry, release or public-contract surface is touched, so nothing here needs a direction escalation. CHANGELOG: no direct reference (no workflow_run / timeout-minutes hits), which is expected — this is internal CI plumbing with no user-facing product surface, not a signal against it.

Size: not applicable — no packages/** core paths are touched, so the core-module gate does not engage. For the record the 733/36 splits into ≈194 production lines (the new 116-line classifier, 5 in the signature renderer, 73 in the workflow), ≈559 test lines, ≈16 doc lines. Well under the large-PR advisory.

Approach: this matches what I'd have proposed, and beats it in one place. I would have inlined the annotation discrimination in the live step; you extracted classify-cancelled-run.sh because two sites need it (the current-run gate and the successor-ownership check), and sharing one implementation is what keeps those two from drifting apart later. Good call. The five-verdict taxonomy is more machinery than the binary split I had in mind, but each verdict maps to a shape that actually occurs and each one is pinned by a test, so I don't think it's speculative. Widening the download projection and renderFailureComment is required for the stated goal — otherwise the filed issue cannot name the leg that timed out. No drive-by refactors or formatting churn; the two design docs are updated in both languages, which AGENTS.md asks for.

One thing to settle before this merges, flagged now so it isn't discovered at merge time: #16277 is a second, competing implementation of the same fix. Different author, also Fixes #16272, also editing main-ci-failure-issue.yml and main-failure-signature.mjs and the same contract test — but with different semantics (rerun the timed-out run rather than file an issue). Only one of the two can land. I've written more about this in the final decision below; it needs a maintainer's call, not a gate's.

Risk: no elevated risk signals — none of the changed paths match the revert-correlated set.

Moving on to code review. 🔍

中文说明

感谢这个 PR,也感谢你在五轮评审里的坚持。

模板完整 ✓(必需标题齐全,Environment 是可选的)。

问题: 是已观测到的,不是理论性的。撞到 timeout-minutes 的 job 结论是 cancelled,而触发器只接纳 failure,所以 #16214 隐藏了几周,#16272 记录了后续项。我没有直接采信这个叙述,而是核对了机制,结论成立:本 workflow 的 on.workflow_run 完全没有 conclusions: 过滤——只有 workflows 和 types: ['completed']——因此 GitHub 本来就会投递所有结论,job 级 if: 是唯一的闸门。这一点在该 workflow 自己的历史里可见:最近 100 次运行中有 98 次结论为 skipped,即事件已投递、被 if: 挡掉。所以放宽 if: 是正确的着力点,它不是在等一个没人改过的触发器过滤条件、因而失效的死代码。

方向: 对齐。这是仓库自身的 CI 可观测性——一个会静默忽略整类失败的追踪器比没有追踪器更糟,因为它制造了虚假的信心。不涉及 auth、sandbox、模型选择、telemetry、发布或公开契约面,因此无需方向层面的升级。CHANGELOG:无直接对应(workflow_run / timeout-minutes 均无命中),这是预期内的——它属于内部 CI 管道,没有面向用户的产品面,不构成反对信号。

规模: 不适用——未触及任何 packages/** 核心路径,核心模块闸门不启动。作为记录,733/36 的构成约为 194 行生产代码(新增的 116 行分类器、signature 渲染器 5 行、workflow 73 行)、约 559 行测试、约 16 行文档,远低于大 PR 提示阈值。

方案: 与我自己的设想一致,并且有一处更好。我原本会把 annotation 判别逻辑内联在 live 步骤里;你把它抽成 classify-cancelled-run.sh,是因为有两个调用点(当前 run 的闸门与后继归属检查),共用一份实现才能防止两者日后漂移。这个取舍是对的。五种 verdict 的分类比我设想的二分法更重,但每一种都对应真实出现的形态、且都有测试钉住,所以不认为是过度设计。放宽 download 投影与 renderFailureComment 是达成目标所必需的——否则提交的 issue 无法点名超时的 leg。没有顺手重构或格式化噪声;两份设计文档中英文同步更新,符合 AGENTS.md 的要求。

合并前需要先定下来的一件事,现在就提出来以免合并时才发现:#16277 是同一修复的第二个、相互竞争的实现。 作者不同,同样写 Fixes #16272,同样修改 main-ci-failure-issue.yml、main-failure-signature.mjs 和同一份契约测试——但语义不同(重跑超时的 run,而不是提交 issue)。两者只能合入一个。下面的最终结论里我写了更多;这需要 maintainer 决定,不是闸门能定的。

风险: 无升级风险信号——改动路径均未命中与 revert 相关的文件集合。

进入代码审查 🔍

— Qwen Code · qwen3.8-max-2026-09-02

Reviewed at e6eb154d398bd2fb7e065c0e39d91e6a9878ad47 · re-run with @qwen-code /triage

@qwen-code-review-bot

qwen-code-review-bot commented Nov 4, 2026 •

Copy link
Copy Markdown
Collaborator

Code review

I read the whole diff and then went back through the workflow at this commit to check the things the diff alone can't show — the step's output contract, the env the new script needs, and which ref the checkout resolves. No blocking defects. What I verified:

  • The early exit honours the output contract. The new skip path writes both superseded=true and provisional=false before exit 0, which is exactly the pair the normal path writes at the end of the step. Every downstream step gates on steps.live.outputs.superseded != 'true', and the job-level provisional output feeds the infra-rerun job — which additionally requires infra_only and file_issue, both left unset on the early exit, so a skipped cancellation can't accidentally request a rerun.
  • The new script has what it needs at both call sites. It requires REPO and reads RUNNER_TEMP; REPO is in the live step's env, and both invocations (the current-run gate and the successor check) are inside that step. Its pages file is keyed on run id and attempt, so the two calls can't clobber each other.
  • It resolves from the right tree. actions/checkout here has no ref:, so on a workflow_run event it takes the default branch — the script is present once this merges. That's the same convention the existing main-failure-signature.mjs helper already follows.
  • It cannot abort the step. Every path through the script ends in an echo and status 0, including the failed-read paths, so cancel_class="$(…)" never trips the step's set -euo pipefail.
  • A pure timeout files rather than rerun-looping. The infra classifier still selects conclusion == "failure" jobs only, and infra_only=true needs total > 0 && infra == total, so a run whose only problem job concluded cancelled gets an issue. This is the claim the PR body makes and it holds.
  • A cancelled run that also carries a failed job still reports. The classifier checks failure/timed_out jobs before anything else and returns unknown, which fails open. Without that, "a job failed, then the next push cancelled the rest" would read as a deliberate supersede and drop a real regression.
  • The contributor-text boundary is respected. pull_request / pull_request_review cancellations skip the classifier outright, so annotation text a fork author controls can never drive a PAT-backed filing. That's the right place to draw it.
  • The page-certification is copied from the right precedent. Requiring total_count to match the merged job count mirrors the download step, so a truncated listing reads as unreadable instead of masquerading as a complete list that happens to lack the timed-out job.
  • Verdict precedence is the R5-1 fix and it's correct. timeout outranks read_failed, so one job's confirmed timeout isn't downgraded by an unrelated sibling's failed annotation read; unreadable still outranks concurrency, because an unread job could have carried the timeout that would have overturned the supersede.

The tests are the strongest part of this. They run the real step bash and, in the cancelled-run classifier (real script) block, spawn the actual shipped script against a stubbed gh — so verdict precedence is measured in the artifact that ships rather than re-implemented in the test. The R5-5 case is a good catch on your own work: handing the stub raw JSON and letting real jq apply the caller's filter is what makes dropping \(.id // "") from the projection actually fail, where a pre-formatted string would have stayed green.

Three non-blocking notes. Per AGENTS.md's five-round rule I'm recording these rather than asking you to chase them:

  1. The PR body describes the opposite fail direction from the code. Risk & Scope says it "fails safe toward silence — if the jobs or annotations cannot be read, no timeout is proven and the run is skipped", and the Evidence block lists unreadable annotations as a skip. The shipped *) branch does the reverse: it warns and reports, and you have a test asserting exactly that (fails open and reports when the annotations cannot be read). The body also attributes the silence direction to "the user-requested tradeoff", so a maintainer reasoning from the body would sign off on a different tradeoff than the one that lands. Same gap in the two design docs this PR touches — the nightly doc now says a cancelled run reports on a timeout annotation and a concurrency cancel doesn't, without covering the fail-open unknown/unreadable paths, and the managed-agent-gate doc says a timeout-backed cancelled successor owns the result while the code also defers to unknown. Worth correcting before merge, since the body is what a reviewer reads; it's documentation, not behaviour.
  2. The successor field split is whitespace-fragile. The projection emits "\(.status) \(.conclusion // "") \(.id // "") \(.run_attempt // "")", so a null conclusion yields a double space, and read -r collapses whitespace runs — shifting the run id into successor_conclusion. I traced it and it can't produce a wrong outcome today: only the completed branch reads those fields, a completed run always carries a conclusion, and an unsettled successor takes the *) path regardless. It's a trap for the next person who adds a field, not a bug.
  3. A cancelled job with an empty check_run_url is skipped before cancelled_seen is set, so a run where every cancelled job lacked one would classify as nothing-executed and stay silent — failing in the direction this PR exists to eliminate. The API always returns that field, so this is theoretical; I'm noting it only because of which way it fails.

Your body's test count also predates the later test additions and should be refreshed whenever you next touch the description.

Files changed (10 of 10 shown)
File What changed
.github/scripts/ci/classify-cancelled-run.sh New 116-line shared classifier: fetches an attempt's jobs, certifies the pages against total count, then reads cancelled jobs' annotations into one of five verdicts.
.github/workflows/main-ci-failure-issue.yml Trigger admits cancelled; live step gains the discriminator and its early exit; successor check defers to a self-reporting cancelled successor; download projections include cancelled jobs and steps.
.github/scripts/ci/main-failure-signature.mjs Comment renderer keeps cancelled jobs so a timed-out leg is named even when nothing concluded failure.
.github/scripts/ci/main-failure-signature.test.mjs New case for the incident shape: one cancelled job, rest success or skipped, zero failures.
scripts/tests/main-ci-failure-issue-workflow.test.js Bulk of the change: trigger pins updated, matrix relabelled as a trigger-gate matrix, plus live-step and real-script cases for each verdict and the successor paths.
scripts/tests/ci-platform-lanes.test.js Trigger pin updated to the new conclusion expression.
docs/design/2026-10-16-managed-agent-gate.md Successor-ownership prose updated for timeout-backed cancellations.
docs/design/2026-10-16-managed-agent-gate.zh-CN.md Chinese counterpart, kept in sync.
docs/design/2026-10-27-installed-acceptance-nightly.md Reporting rule restated: skipped and successful stay silent, a timeout-backed cancellation reports.
docs/design/2026-10-27-installed-acceptance-nightly.zh-CN.md Chinese counterpart, kept in sync.

Testing

This is an unattended CI run, so I did not build or execute anything from this PR — the evidence below is the PR's own CI, read through the API for the reviewed commit. All 208 check-runs on e6eb154d are settled: 27 success, 177 skipped, 4 cancelled, 0 failure, 0 pending.

Check Conclusion
Test (ubuntu-latest, Node 22.x) success
Linux unit tests (shard 1/8 … 8/8) success (all 8)
Lint & Static (ubuntu-latest, Node 22.x) success
Build unit-test tree (ubuntu-latest, Node 22.x) success
Integration Tests (no-AK, No Sandbox) success
Managed Agent gate success
Check TUI impact / TUI parity snapshots (ink vs opentui) success
OpenTUI no-flicker gate success
web-shell E2E Smoke (shard 1/4 … 4/4, plus the aggregate) success (all 5)
Classify PR / assign / label / route / Select the Managed Agent tier / Remind on force-push success
route (4 bot-orchestration jobs on superseded runs) cancelled
177 further matrix legs not selected for this commit skipped
failures 0

About those 4 cancellations, since a cancelled check on a PR whose whole subject is cancellation deserves an explanation rather than a wave: they are all the route job, from four distinct runs created minutes apart (42187453024, 42187460616, 42187468255, 42187475840), and route also appears among the successes. That is bot orchestration under pull_request_target/issue_comment being displaced by its own concurrency group across successive pushes — not PR CI, and not something this diff can influence. Fittingly it's the exact annotation shape the new classifier is built to ignore. Nothing red is attributable to this change, so there is no failing-job log to quote.

Not verified, and why: nothing here has been observed against a real workflow_run cancellation. The suites execute the genuine step bash and the genuine shipped script, but with a stubbed gh, so the annotation strings they match against are fixtures the author chose. The claim that real GitHub annotations for a job timeout say exceeded the maximum execution time, and that real concurrency cancels say only The operation was canceled., rests on the author's reading — and their Linux row is marked not-tested for precisely this reason. I attempted an independent read-only sweep of historical cancelled main runs to measure how often the fail-open unknown/unreadable paths would actually be hit (which is what bounds the issue-spam risk of failing open), and that probe was denied by this environment's permission rules, so I have no first-hand number. The author reports 0 unknown and 0 unreadable across 97 such runs; that is their claim, not evidence I reproduced.

Sandboxed verification would settle this: @qwen-code /verify — whether the discriminator classifies real GitHub annotations correctly is not observable from the diff or from suites that stub the API client, and the fail-open frequency that determines the spam risk is currently unmeasured by anyone but the author. The author has write access, so this is a direct run rather than a sponsored one; there is no TUI surface here, so /tmux is not the applicable lane.

Real-scenario tmux testing: N/A — unattended CI run, and this change has no user-visible terminal surface (CI workflow internals only).

中文说明

代码审查: 我通读了整个 diff,并回到该 commit 上的 workflow 核对 diff 本身看不出来的部分——步骤的输出契约、新脚本所需的环境变量、以及 checkout 解析到哪个 ref。未发现阻塞性缺陷。已核实:提前退出写入了 superseded 与 provisional 两个输出,与正常路径一致,因此不会误触发 infra 重跑;新脚本在两个调用点都能拿到 REPO,且分页文件按 run id 与 attempt 命名,两次调用互不覆盖;actions/checkout 未指定 ref:,workflow_run 事件下取默认分支,合入后脚本即在树内,与既有 helper 的用法一致;脚本每条路径都以状态 0 结束,不会触发该步骤的 set -euo pipefail;纯超时 run 会建单而非陷入重跑,因为 infra 分类器仍只选 failure job 且要求 total > 0;同时带 failure job 的 cancelled run 仍会上报(分类器先行返回 unknown 并 fail open),避免把「先失败后被顶替」误判为主动取消而丢掉真实回归;PR 触发的取消直接跳过分类器,贡献者可控的 annotation 文本无法驱动 PAT 建单;分页完整性以 total_count 校验,沿用了 download 步骤的既有做法;verdict 优先级 timeout 高于 read_failed(R5-1),同时 unreadable 高于 concurrency,两个方向都正确。

测试是这份改动最扎实的部分:它执行真实的步骤 bash,并在 real-script 区块中直接 spawn 实际发布的脚本、只对 gh 打桩,因此 verdict 优先级是在真正会上线的产物上被测量的,而不是在测试里重新实现一遍。R5-5 让桩返回原始 JSON、由真实 jq 应用调用方的 filter,这才能真的钉住投影字段。

三条非阻塞意见(按 AGENTS.md 的五轮规则,我只作记录,不要求逐条修复):其一,PR 正文描述的 fail 方向与代码相反——Risk & Scope 说读不到就沉默跳过,Evidence 表也如此,但代码的 *) 分支是 fail open 上报,并且有一条测试正是断言这一点;正文还把沉默方向称为「用户要求的取舍」,因此照正文评审的 maintainer 会批准一个与实际落地不同的取舍。本 PR 改动的两份设计文档也有同样的遗漏。建议合并前更正,因为正文是评审者的依据;这属于文档而非行为问题。其二,后继字段用空白切分较脆弱:conclusion 为 null 时投影会产生连续空格,而 read -r 会合并空白,使 run id 落到 successor_conclusion。我追踪后确认目前不会导致错误结果——只有 completed 分支读这些字段,而完成的 run 必有 conclusion,未定论的后继走 *) 分支——但这是留给后来者的陷阱。其三,check_run_url 为空的 cancelled job 会在设置 cancelled_seen 之前被跳过,因此若某 run 的所有 cancelled job 都缺该字段,会判为 nothing-executed 而保持沉默,恰好失败在本 PR 要消除的方向上;该字段 API 总会返回,所以这只是理论风险,我提出来仅因其失败方向。

正文里的测试计数也早于后续新增的用例,下次改动描述时一并刷新即可。

测试证据: 这是无人值守的 CI 运行,我没有构建或执行本 PR 的任何内容——以下证据是 PR 自身的 CI,通过 API 读取所审 commit 的结果。e6eb154d 上全部 208 个 check-run 均已定论:27 success、177 skipped、4 cancelled、0 failure、0 pending。 上方表格中的 4 个 cancelled 全是 route job,来自相隔数分钟的四个不同 run,而 route 同时也出现在 success 中——这是 pull_request_target/issue_comment 下的 bot 编排作业被自身并发组顶替,不是 PR CI,也不受本 diff 影响;讽刺的是,它正是新分类器被设计来忽略的那种 annotation 形态。没有可归因于本改动的红项,因此没有失败日志可引用。

未验证的部分及原因:这里没有任何针对真实 workflow_run 取消事件的观测。测试套件执行的是真实步骤 bash 与真实脚本,但 gh 被打桩,因此匹配的 annotation 字符串是作者选定的 fixture。「真实 GitHub 的超时 annotation 写作 exceeded the maximum execution time、真实并发取消只写作 The operation was canceled.」这一前提依赖作者的判读,其 Linux 一栏也正是因此标记为未测试。我尝试独立做一次只读的历史 cancelled main run 抽样,以测量 fail-open 的 unknown/unreadable 路径实际命中频率(这决定了 fail open 的刷单风险边界),但该探测被本环境的权限规则拒绝,所以我没有第一手数字。作者报告 97 个此类 run 中 0 个 unknown、0 个 unreadable;这是作者的说法,不是我复现的证据。

沙箱验证可以定这件事:@qwen-code /verify——判别器对真实 GitHub annotation 的分类是否正确,无法从 diff、也无法从对 API 客户端打桩的套件中观测,而决定刷单风险的 fail-open 频率目前除作者外无人测量。作者具备写权限,因此这是直接运行而非 sponsored run;此处没有 TUI 界面,所以 /tmux 不是适用的通道。

真实场景 tmux 测试:N/A——无人值守 CI 运行,且本改动没有面向用户的终端界面(仅 CI workflow 内部)。

— Qwen Code · qwen3.8-max-2026-09-02

Reviewed at e6eb154d398bd2fb7e065c0e39d91e6a9878ad47 · re-run with @qwen-code /triage

@qwen-code-review-bot

qwen-code-review-bot commented Nov 4, 2026 •

Copy link
Copy Markdown
Collaborator

Confidence: 3/5 — the code and its tests are good enough to merge; what stops me is a decision that isn't mine to make, not a defect I found.

Stepping back: my independent proposal for this problem was to widen the trigger and discriminate on annotations inside the live step, which is essentially what you built. You went one better by extracting the discriminator into a shared script, because two call sites need the same answer and a copy in each would drift. That's the right instinct and it's why I have no simpler path to offer you — I looked for one and didn't find it.

The thing I'd thank you for in six months is the test design. Spawning the shipped script itself against a stubbed gh, rather than re-implementing its logic in the test, means the verdict precedence is pinned where it actually lives. And R5-5 — making the stub return raw JSON so real jq applies the caller's projection — is the kind of detail that turns a test from decorative into load-bearing. Most of what I'd normally worry about in a 500-line bash-and-test CI change is already answered by that.

So why not approve? Two reasons, and only the first is a real blocker.

#16277 is a competing implementation of this same fix, and choosing between them is a maintainer's call. It's by a different author, it also says Fixes #16272, it edits the same workflow, the same signature module and the same contract test — and it disagrees on the remedy: where this PR files an issue for a timed-out run, that one reruns it. Those are different operational behaviours for the same incident, and they cannot both land. Approving here would settle a contest between two contributors by bot vote, which is exactly the kind of thing this gate should hand up rather than decide. Whoever owns CI needs to pick one, and the other author deserves to hear that from a human.

The description contradicts the code on the fail direction, and the contradiction matters more than a stale sentence usually would. The body tells a reviewer that an unreadable annotation is dropped in favour of silence, and calls that the user-requested tradeoff. The code fails open and reports, with a test asserting it. I happen to think failing open is the correct choice for a tracker whose entire failure mode was silence — but that's my judgement, and the body currently argues the opposite position while claiming a human asked for it. A maintainer should confirm the shipped direction is the wanted one, and then the body and the two design docs should say so. That's a five-minute edit, not a rework.

Neither of these is a reason to request changes, so I'm not: per AGENTS.md's five-round rule, hygiene and documentation items don't block a PR this far along, and the two code fragilities I noted above (whitespace-split successor fields, and an empty check_run_url failing toward silence) are unreachable or theoretical today. Recording them, not gating on them.

Two practical notes for whoever picks this up. The standing CHANGES_REQUESTED on this PR is the bot's own review of c8afa19b, two commits behind the head I reviewed, and all 24 review threads are resolved — so the branch-protection block is stale rather than substantive, and a fresh review on e6eb154d supersedes it. Separately, the workflow file lands at 60,880 bytes against a ratchet ceiling of 61,236 (57,140 recorded plus the 4,096 allowance), so this passes without a .size-baseline bump — but with only 356 bytes to spare, and the recorded baseline already 832 bytes behind what's on main, the next PR that grows this file will trip the ratchet and have to bump it. Worth knowing, since #16277 does bump it.

If the maintainer resolves #16277 in this PR's favour and the body is corrected to describe the fail-open behaviour, my assessment is that this is ready — I found nothing in the code that I'd want changed first.

中文说明

Confidence: 3/5 — 代码与测试本身已具备合并条件;让我停下来的不是一个我找到的缺陷,而是一个不该由我做出的决定。

退一步看:我对这个问题的独立设想是放宽触发器、并在 live 步骤内基于 annotation 做判别,这基本就是你实现的方案。你更进一步,把判别逻辑抽成共享脚本,因为有两个调用点需要同一个答案,各写一份迟早会漂移。这个判断是对的,所以我没有更简的路径可以给你——我找过,没找到。

六个月后我会感谢你的地方是测试设计。直接 spawn 实际发布的脚本、只对 gh 打桩,而不是在测试里重新实现它的逻辑,这意味着 verdict 优先级被钉在它真正所在的地方。而 R5-5——让桩返回原始 JSON、由真实 jq 应用调用方的投影——正是把一个测试从装饰性变成承重性的那种细节。我通常会对一个 500 行的 bash + 测试的 CI 改动担心的大部分问题,这个设计已经回答了。

那为什么不批准?两个理由,其中只有第一个是真正的阻塞。

#16277 是同一修复的竞争实现,而在两者之间做选择是 maintainer 的决定。 它出自另一位作者,同样写着 Fixes #16272,改的是同一个 workflow、同一个 signature 模块和同一份契约测试——并且在补救方式上不一致:本 PR 对超时 run 提交 issue,那一个则重跑它。对同一次事故这是两种不同的运维行为,二者不能同时合入。在这里批准,等于用 bot 的一票裁定两位贡献者之间的竞争,而这正是本闸门应当上交、而非自行决定的事。CI 的负责人需要选定其一,而另一位作者应当从人那里听到这个结论。

描述在 fail 方向上与代码矛盾,而这个矛盾比一般的过期文字更要紧。 正文告诉评审者:annotation 读不到时会被丢弃、选择沉默,并称之为「用户要求的取舍」。代码却是 fail open 上报,并且有测试断言这一点。我个人认为,对一个其唯一失效模式就是沉默的追踪器而言,fail open 是正确的选择——但这是我的判断,而正文目前在主张相反的立场,同时声称这是人提出的要求。maintainer 应当确认落地的方向就是想要的方向,然后让正文和两份设计文档如实描述它。这是五分钟的修改,不是返工。

这两条都不构成 request changes 的理由,所以我没有这么做:按 AGENTS.md 的五轮规则,走到这一步的 PR,卫生与文档类问题不该阻塞;我在上面指出的两处代码脆弱性(后继字段的空白切分、以及 check_run_url 为空时朝沉默方向失败)在今天要么不可达、要么只是理论风险。记录在案,不作为闸门。

给接手的人两条实用信息。本 PR 上仍然生效的 CHANGES_REQUESTED 是 bot 自己对 c8afa19b 的评审,比我审的 head 落后两个 commit,而全部 24 条 review thread 都已 resolved——所以分支保护上的阻塞是过期的,而非实质性的,在 e6eb154d 上的一次新评审即可取代它。另外,workflow 文件在该 commit 上为 60,880 字节,ratchet 上限为 61,236(记录的 57,140 加上 4,096 的允许增量),因此本 PR 无需修改 .size-baseline 即可通过——但只剩 356 字节余量,且记录的 baseline 已比 main 上的实际大小落后 832 字节,所以下一个使该文件增长的 PR 会触发 ratchet 并必须更新它。值得知道,因为 #16277 确实更新了它。

如果 maintainer 在 #16277 与本 PR 之间选择了本 PR,并且正文被更正为描述 fail-open 行为,我的评估是它可以合并——代码层面我没有找到任何我会要求先改的东西。

— Qwen Code · qwen3.8-max-2026-09-02

Reviewed at e6eb154d398bd2fb7e065c0e39d91e6a9878ad47 · re-run with @qwen-code /triage

@qwen-code-review-bot qwen-code-review-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, looks ready to ship — CI landed green after the review. ✅

@qwen-code-review-bot qwen-code-review-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed. Suggestions are inline.

Not explored to full depth (tool budget reached): "agent reverse-audit (round 2)": did not read the writer jobs ( analyze outputs consumers, .github/workflows/main-ci-failure-issue.yml ~500-1067) end to end — grepped them for conclusion / ….

Not fully linted (tool limitation, not a blocker): .github/workflows/main-ci-failure-issue.yml — actionlint ran workflow-level checks; embedded run: shell deferred (source mapping not yet supported).

中文说明

已审查。 建议见行内评论。

未探索到全部深度(达到工具调用预算):"agent reverse-audit (round 2)":did not read the writer jobs ( analyze outputs consumers, .github/workflows/main-ci-failure-issue.yml ~500-1067) end to end — grepped them for conclusion / …。

未完整检查(工具限制,非阻断):.github/workflows/main-ci-failure-issue.yml——actionlint 已运行 workflow 级检查;内嵌 run: shell 延迟处理(源映射尚未支持)。

— deepseek-v4.1-flash-1017 via Qwen Code /review (v0.26.7)

… unknown

Review on QwenLM#16280: real concurrency cancels carry the runner shutdown / higher-
priority / operation-canceled annotations, not just "operation was canceled",
and a timed-out job can carry a shutdown annotation alongside the timeout. The
previous discriminator only looked for the timeout text and skipped everything
else, so it rested on "absence of timeout = concurrency" and dropped unreadable
cancellations — re-creating the hiding problem QwenLM#16272 fixes.

Classify by positive evidence on both sides instead: a timeout annotation
reports (even with a coexisting shutdown annotation); a concurrency annotation
with no timeout skips; neither, or unreadable jobs/annotations, fails open and
reports like the rest of the live step. Fixtures cover the shutdown and
higher-priority concurrency forms, timeout+shutdown coexistence, and the
fail-open paths; breaking the concurrency signature reddens the skip tests.
@qwen-code-review-bot

Copy link
Copy Markdown
Collaborator

⏳ Historical-head review — head moved to e02fcc840a95a9d21822ef1d1928ab66e64d277d while this review was in flight (past the salvage threshold), so the run finished and posted against the head it reviewed: 7ab6a568f31b5ee178069fd833617a8c2306b86b (#10110). The next automatic review covers the delta from that anchor. Full log in the workflow run.

中文说明

⏳ 历史 head 评审 —— 本次评审进行中 head 移动到了 e02fcc840a95a9d21822ef1d1928ab66e64d277d(已过 salvage 阈值),因此评审跑完并针对其实际评审的 head 7ab6a568f31b5ee178069fd833617a8c2306b86b 发布(#10110)。下一次自动评审将从该锚点起评审增量。完整日志见 workflow 运行。

yiliang114 and others added 3 commits November 4, 2026 17:03
…r dedup, docs

R1-4 (security): only trusted triggers (push/schedule/dispatch on main) run the
cancellation classifier; PR-triggered cancellations stay on the supersede path
so contributor-authored annotation text cannot drive the PAT-backed reporter.

R1-1: renderFailureComment now keeps cancelled jobs, so a timed-out leg is named
in the issue/PR comment even when no job concluded failure (the QwenLM#16214 shape).

R1-2: factor the classifier into cancelled_class with a total_count receipt and
explicit unreadable handling; unknown fails open with a warning instead of
asserting a supersede it never established.

R1-3: a timeout-backed cancelled successor owns the older run's result (same
shared classifier), preventing a duplicate report; log wording corrected.

R1-6: the classifier runs after the stale-attempt check so a stale event is
attributed to staleness, not to a cancellation cause.

R1-7: the test stub resolves the check-run id (job id != check-run id) and
applies the caller's --jq with real jq; RUN_CONCLUSION env is pinned; fixtures
cover shutdown / higher-priority / coexisting timeout / unreadable / PR arms.

R1-5: sync the two design docs (both languages) with the new behaviour.
Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
Patrol-Run: qwen-pr-closeout/jmw1wuw3xco
main-ci-failure-issue.yml grew to 63933 bytes, 6793 over its recorded
57140 (allowance 4096), so the workflow-size ratchet failed. Move the new
cancelled_class helper into .github/scripts/ci/classify-cancelled-run.sh,
which is what the ratchet asks for ("long steps into .github/scripts/"),
and trim review-narrative prose to the load-bearing why. Behaviour is
unchanged: both call sites invoke the script.

The workflow-shape pin that read the classifier's attempt-jobs fetch out
of the workflow now reads the script.

Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
Patrol-Run: qwen-pr-closeout/jmw1wuw3xco

@qwen-code-review-bot qwen-code-review-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not explored to full depth (tool budget reached): "agent 5": did not execute vitest on scripts/tests/main-ci-failure-issue-workflow.test.js ; the mutation verdicts above are reading-based, not run..

Not fully linted (tool limitation, not a blocker): .github/workflows/main-ci-failure-issue.yml — actionlint ran workflow-level checks; embedded run: shell deferred (source mapping not yet supported).

中文说明

未探索到全部深度(达到工具调用预算):"agent 5":did not execute vitest on scripts/tests/main-ci-failure-issue-workflow.test.js ; the mutation verdicts above are reading-based, not run.。

未完整检查(工具限制,非阻断):.github/workflows/main-ci-failure-issue.yml——actionlint 已运行 workflow 级检查;内嵌 run: shell 延迟处理(源映射尚未支持)。

— deepseek-v4.1-flash-1017 via Qwen Code /review (v0.26.7)

@yiliang114

yiliang114 commented Nov 4, 2026 •

Copy link
Copy Markdown
Collaborator Author

Local verification: PASS

  • Mode: merge-reference
  • After arm (PR head): 26a5fe92d7a8bcbd09b928cbc08ef39ba3cac969
  • Before arm (merge-base): b383efb8543965707de8a44a5057254cca623ebb (equals the PR's base, not today's main tip e838c4c0)
  • Surface: Build / CI workflow (trigger gate + live step bash + failure-comment renderer)
  • Runtime: Node v24.19.0, vitest 3.2.7 and bare node --test from a symlinked donor node_modules; three detached worktrees, one per arm; gh replaced by a stub that applies the caller's --jq with real jq
  • Provider boundary: GitHub API is stubbed with realistic payloads (total_count, check_run_url → check-runs/<id>, run_attempt); the workflow YAML, the step body, and classify-cancelled-run.sh are the real head files
  • A local PASS is not a merge approval; no product code was modified, pushed, or re-run in CI.

Claims

# Required claim Observation / evidence Result
1 Runtime identity both arms pinned to the SHAs above; clean trees (git status --porcelain = 0 lines) PASS
2 Observed change timed-out run (#16214 shape) goes from unreported to reported PASS
3 Causal evidence real live step bash driven against a stubbed gh; classify-cancelled-run.sh exercised directly on 11 annotation shapes PASS
4 Controls plain failure path unchanged; concurrency cancel still skipped; PR-triggered cancel skipped; successor ownership both ways PASS
5 Regression pin hybrid arm (head tree + merge-base workflow/renderer, no classifier): 15 failed / 151 passed and 1 failed / 62 passed PASS

Key evidence

Trigger gate (jobs.analyze.if, evaluated with a synthetic github context):

                       before (merge-base)   after (head)
E2E Tests push main failure      ADMITTED           ADMITTED
E2E Tests push main cancelled    rejected           ADMITTED   <- the fix
E2E Tests push main success      rejected           rejected
E2E Tests push main timed_out    rejected           rejected

live step, real step body, stubbed gh — 8 scenarios, all as declared:

cancelled + timeout annotation          -> superseded=false (report)
cancelled + "operation was canceled"    -> superseded=true  (skip)
cancelled + shutdown signal             -> superseded=true  (skip)
cancelled + higher-priority             -> superseded=true  (skip)
cancelled + timeout AND shutdown        -> timeout wins, superseded=false
cancelled + unrelated/empty annotations -> superseded=false (fail open)
cancelled on pull_request               -> superseded=true  (skip, untrusted text)
failure                                 -> superseded=false (unchanged)
push + successor completed/cancelled(timeout)    -> superseded=true  (owns result)
push + successor completed/cancelled(concurrency) -> superseded=false (reports)

classify-cancelled-run.sh directly: timeout→timeout; operation canceled / shutdown signal / higher priority→concurrency; coexisting timeout+shutdown→timeout; unrelated or empty annotations→unknown; uneven pagination (total_count 3, 1 job returned)→unknown; unreadable jobs or annotations→unknown. The classifier file does not exist at the merge-base, so the before arm has no discriminator at all.

Renderer: for the #16214 shape (1 cancelled job, 0 failure) the before arm renders a 265-byte comment naming no job; the head renders 405 bytes naming sandbox:docker, its /job/9001 link, and the cancelled step, while still dropping success/skipped jobs.

Suites: before 226/226 (2 files) and 62/62; head 238/238 and 63/63. Pin arm reds map to this PR's own new/updated assertions: gate pin, both cancelled routing rows, the RUN_CONCLUSION env pin, the eight cancellation tests, the successor-ownership test, the changed successor warning wording, and the cancelled-job download projection.

Static: bash -n on the new script rc=0, node --check on the renderer rc=0, prettier --check clean on all five changed code files, git diff --check <mb> HEAD rc=0.

Harness provenance / discarded runs

Probes, extracted step body, stubbed gh, and logs are retained in the verification bundle. Two earlier probe runs were discarded as harness bugs (a missing GitHub-expression contains/fromJSON stub, then a mock that re-applied jq to an already-projected successor line); all numbers above come from the corrected runs.

Residuals

  • No live workflow_run cancellation event was exercised end-to-end; the author marks Linux ⚠️ for the same reason. This run proves the gate, the discriminator, and the renderer, not a hosted filing.
  • By design (fail-safe), a real timeout whose annotations or job list cannot be read is skipped rather than reported.
  • Docs changes (2026-10-16-managed-agent-gate*.md, 2026-10-27-installed-acceptance-nightly*.md) describe the same behavior; they are text-only and were not rendered.
中文版(完整对照)

本地验证:PASS

  • 模式: merge-reference(合并参考)
  • 后臂(PR head): 26a5fe92d7a8bcbd09b928cbc08ef39ba3cac969
  • 前臂(merge-base): b383efb8543965707de8a44a5057254cca623ebb(等于本 PR 的 base,不是今天的 main tip e838c4c0)
  • 面: 构建 / CI workflow(触发门 + live step 的 bash + 失败评论渲染器)
  • 运行时: Node v24.19.0,vitest 3.2.7 与裸 node --test,node_modules 软链到 donor;三个 detached worktree 每个臂一个;gh 用桩替换,桩会按调用方的 --jq 用真 jq 过滤
  • 边界: GitHub API 是桩,payload 形态真实(total_count、check_run_url → check-runs/<id>、run_attempt);workflow YAML、step 正文、classify-cancelled-run.sh 都是 head 的真实文件
  • 本地 PASS 不等于合并批准;未修改产品代码、未推分支、未重跑 CI。

声明

# 需证声明 观测 / 证据 结果
1 运行时身份 两臂都钉在上面的 SHA;工作树干净 PASS
2 行为变化 #16214 形态的超时 run 由「不上报」变成「上报」 PASS
3 因果证据 真实 live step bash + gh 桩;分类器直接跑 11 种 annotation 形态 PASS
4 对照 普通 failure 路径不变;并发取消仍跳过;PR 触发的取消跳过;后继归属两向 PASS
5 回归钉 混合臂(head 树 + merge-base 的 workflow/渲染器、无分类器):15 红 / 151 绿、1 红 / 62 绿 PASS

关键读数

  • 触发门:前臂只接纳 failure,cancelled 被拒;后臂接纳 ["failure","cancelled"],main push 的 cancelled 放行,success/timed_out 仍拒。
  • live step 八条路径全部符合声明:超时→上报;并发(operation canceled / shutdown / higher priority)→跳过;超时与 shutdown 共存→超时优先;无法判读→fail open 上报;PR 触发→跳过(不信任贡献者文本);failure→不受影响;后继为超时取消→拥有结果,后继为并发取消→本 run 仍上报。
  • 分类器直跑:超时→timeout;三种并发文本→concurrency;不相关/空 annotation→unknown;分页不全(total_count 3 仅返回 1 个 job)→unknown;jobs 或 annotation 读取失败→unknown。前臂没有这个脚本。
  • 渲染器:ci: E2E "sandbox:docker" leg hits its 60-minute cap on pool hosts that rebuild the sandbox image cold #16214 形态前臂只渲染 265 字节、不点名任何 job;head 渲染 405 字节,点名 sandbox:docker、/job/9001 与被取消的 step,仍不输出 success/skipped job。
  • 套件:前臂 226/226、62/62;head 238/238、63/63。pin 臂的红逐条对应本 PR 新增/改写的断言(门 pin、两行 cancelled 路由、RUN_CONCLUSION env pin、八条取消测试、后继归属测试、后继告警措辞、cancelled job 的下载投影)。
  • 静态:新脚本 bash -n rc=0,渲染器 node --check rc=0,五个改动代码文件 prettier --check 全干净,git diff --check rc=0。

缺口

  • 未跑真实 workflow_run 取消事件的端到端链路(作者同样标 Linux ⚠️);本报告证明的是门、判别器与渲染器,不是 hosted 建单。
  • 按 fail-safe 设计:annotation 或 job 列表读不到的真实超时会被跳过而非上报。
  • 文档改动仅文本,未渲染。

Superseded by #16280 (comment) for head 53fef7b (2026-11-05T20:2xZ) — the findings above cover the stale head 26a5fe9, which gained a further commit (fix(ci): do not report a cancelled run that ran no job).

…R1-2, R2-1, R2-4)

Critical R1-2: the classifier returned unknown (fail-open report) for a
cancelled run whose attempt has no cancelled job at all — a run displaced while
queued certifies 0==0 and reads no annotation, so it filed a "Main CI failed"
issue for a run that tested nothing. Track whether any cancelled job was seen;
none means supersede (concurrency), while unknown now only covers cancelled jobs
whose annotations are unreadable/unmatched or uncertifiable pages.

R2-1: tests for the certification false branch (partial pages -> unknown) and
the no-jobs case (-> concurrency). R2-4: a negative successor case with a real
id whose cancelled job is a supersede, exercising the == 'timeout' false branch.

@qwen-code-review-bot qwen-code-review-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Convergence: round 3 posted 6 inline comment(s), 4 of them reported for the first time; the previous round posted 5 (5 new). Findings keep coming back to the same files: .github/scripts/ci/classify-cancelled-run.sh (findings in round 2; 2 more now); scripts/tests/main-ci-failure-issue-workflow.test.js (findings in round 2; 2 more now). A cluster that keeps producing siblings usually means the fixes are treating instances of a shared root cause — triaging that cause before the next round, or splitting an independent cluster into its own pull request, tends to end the loop faster than fixing them one at a time. (Observation only — nothing was withheld from this review because of this observation.)

中文说明

收敛情况:第 3 轮发布了 6 条行内评论,其中 4 条是首次提出;上一轮发布了 5 条(其中 5 条首次提出)。发现反复回到同一批文件:.github/scripts/ci/classify-cancelled-run.sh(第 2 轮已出过发现,本轮又有 2 条);scripts/tests/main-ci-failure-issue-workflow.test.js(第 2 轮已出过发现,本轮又有 2 条)。一个不断再生兄弟发现的簇,通常意味着逐条修复只在处理同一根因的实例——先定位并处理该根因,或把独立的簇拆成单独的 PR,通常比逐条修复更快结束循环。(仅为观察——本轮评审未因此扣留任何内容。)

— deepseek-v4.1-flash-1017 via Qwen Code /review (v0.26.7)

@yiliang114

Copy link
Copy Markdown
Collaborator Author

Local verification: PASS

  • Mode: merge-reference
  • After arm (PR head): 53fef7b5b8f55b0f279f9d305de0c128faaa0c28
  • Before arm (merge-base): b383efb8543965707de8a44a5057254cca623ebb (equals the PR's base, not today's main tip)
  • Surface: Build / CI workflow (trigger gate + live step bash + classify-cancelled-run.sh + failure-comment renderer)
  • Runtime: Node v24.19.0, vitest 3.2.7 and bare node --test; four detached worktrees (before / after / two hybrid pins); gh replaced by a stub that applies the caller's --jq with real jq
  • Provider boundary: GitHub API stubbed with realistic payloads (total_count, check_run_url → check-runs/<id>, run_attempt); workflow YAML, step bodies, the classifier script and the renderer are the real head files
  • A local PASS is not a merge approval; no product code was modified, pushed, or re-run in CI.

Claims

# Required claim Observation / evidence Result
1 Runtime identity both arms pinned to the SHAs above; trees clean PASS
2 Observed change main-CI cancelled runs go from rejected-at-gate to admitted, then discriminated by positive evidence PASS
3 Causal evidence real live step body driven against a stubbed gh; classifier exercised directly on 13 payload shapes PASS
4 Controls plain failure path unchanged; PR-triggered and non-main cancellations still rejected; successor ownership both ways PASS
5 Regression pin hybrid arms (head tree + merge-base workflow / renderer): 19 failed / 222 passed and 1 failed / 62 passed PASS

Key evidence

Trigger gate (jobs.analyze.if, synthetic context):

                                        before    after
E2E Tests   push main failure           ADMITTED  ADMITTED
E2E Tests   push main cancelled         rejected  ADMITTED   <- the fix
E2E Tests   push main success           rejected  rejected
E2E Tests   push main timed_out         rejected  rejected
tui-parity  schedule main cancelled     rejected  ADMITTED
tui-parity  schedule main failure       ADMITTED  ADMITTED
E2E Tests   pull_request cancelled      rejected  rejected
E2E Tests   push feature cancelled      rejected  rejected

live step, real head body, stubbed gh — 10 scenarios:

cancelled + timeout annotation           superseded=false  report
cancelled + operation-canceled           superseded=true   skip
cancelled + unclassifiable annotations   superseded=false  fail open, report
cancelled on pull_request                superseded=true   skip (contributor text)
cancelled with no cancelled job          superseded=true   skip (ran nothing)
failure                                  superseded=false  unchanged
cancelled+timeout, successor timeout-cxl superseded=true   successor owns
cancelled+concurrency                    superseded=true   skip
failure, successor timeout-cancelled     superseded=true   owns
failure, successor concurrency-cancelled superseded=false  reports

Before arm: all ten yield superseded=false — no discriminator exists at the merge-base (no classifier file, no RUN_CONCLUSION input), so the gate alone was the only filter.

classify-cancelled-run.sh directly (13 shapes): timeout; operation-canceled / shutdown-signal / higher-priority → concurrency; timeout+shutdown → timeout; unrelated or empty annotations → unknown; two jobs one timeout → timeout; uneven pagination (total_count 3, 1 job) → unknown; unreadable annotations → unknown; no cancelled job → concurrency; cancelled job with no matching annotation → unknown; unreadable job list → unknown.

Suites: before 226/226 (2 files) and 62/62; head 241/241 and 63/63. Pin reds map to this PR's own assertions: 13 new workflow test titles + 2 new it.each gate rows + 4 rewritten existing assertions = 19; the renderer pin red is exactly the new cancelled-job test.

Harness provenance / discarded runs

Probes, extracted step body, stubbed gh and logs are retained in the verification bundle. Three runs were discarded as harness bugs (a synthetic context missing event.workflow.name, twice, and an unbound fixture variable in the live probe); every number above comes from the corrected runs.

Residuals

  • No live hosted workflow_run cancellation event was exercised end-to-end; this proves the gate expression, the discriminator and the renderer, not a hosted filing.
  • contains(fromJSON(...)) was evaluated with a local shim, not GitHub's expression engine.
  • Annotation strings are GitHub's documented wording, not captured from a real cancelled run.
  • By design (fail-safe), a real timeout whose annotations or job list cannot be read is skipped.
  • Docs edits are text-only and were not rendered.
中文版(完整对照)

本地验证:PASS

  • 模式: merge-reference(两臂对照)
  • after 臂(PR head): 53fef7b5b8f55b0f279f9d305de0c128faaa0c28
  • before 臂(merge-base): b383efb8543965707de8a44a5057254cca623ebb(等于本 PR 的 base,不是今天的 main tip)
  • 面: 构建 / CI workflow(触发门禁 + live step 的 bash + classify-cancelled-run.sh + 失败评论渲染器)
  • 运行时: Node v24.19.0、vitest 3.2.7、node --test;四个 detached worktree(before / after / 两个混合 pin);gh 用桩替换(桩会按调用方的 --jq 用真 jq 处理)
  • 服务边界: GitHub API 用真实形态的桩数据(total_count、check_run_url → check-runs/<id>、run_attempt);workflow YAML、step 正文、分类器与渲染器都是 head 的真实文件
  • 本地 PASS 不等于可合并;没有改动产品代码、没有推分支、没有重跑 CI。

五条主张

  1. 运行身份:两臂锁定在上述 SHA,树干净 —— PASS
  2. 观察到的变化:main CI 的 cancelled 运行从「门禁直接拒绝」变成「进入分析」,再由正证据区分 —— PASS
  3. 因果证据:真实 live step 正文 + gh 桩驱动;分类器直接跑 13 种载荷形态 —— PASS
  4. 对照:普通 failure 路径不变;PR 触发与非 main 的取消仍被门禁拒绝;后继归属两个方向都对 —— PASS
  5. 回归钉:混合臂(head 的树 + merge-base 的 workflow / 渲染器)分别 19 红 / 222 绿、1 红 / 62 绿 —— PASS

关键读数

门禁(jobs.analyze.if,合成上下文):failure 两臂都准入;cancelled 从拒绝变为准入;success/timed_out 仍拒绝;tui-parity/schedule/main/cancelled 从拒绝变为准入;PR 触发与非 main 的取消仍拒绝。

live step(10 个场景):超时 → 上报;并发取消 / PR 触发 / 未跑任何 job 的取消 → 跳过;注释不可读或不可判 → fail open 上报;普通 failure 不变;后继归属(超时取消的后继拥有、并发取消的后继不拥有)两个方向都对。before 臂十项全部 superseded=false——merge-base 上根本没有这套判别(没有分类器文件、没有 RUN_CONCLUSION),只有门禁一层。

分类器(13 形态):timeout;operation-canceled / shutdown-signal / higher-priority → concurrency;超时+shutdown 并存 → timeout;无关或空注释 → unknown;两个 job 一个超时 → timeout;分页不全(total_count 3、只到 1)→ unknown;注释不可读 → unknown;没有 cancelled job → concurrency;cancelled job 但注释不匹配 → unknown;job 列表不可读 → unknown。

测试:before 226/226(2 文件)与 62/62;head 241/241 与 63/63。pin 臂 19 红 = 13 条新增 workflow 用例 + 2 条新增门禁参数行 + 4 条被改写的既有断言;渲染器 pin 的 1 红正是新增的「cancelled job 也要点名」用例。

残余与缺口

  • 没有真实的托管 workflow_run 取消事件端到端跑过;本报告证明的是门禁表达式、判别逻辑与渲染器,不是托管端的真实建单。
  • contains(fromJSON(...)) 由本地 shim 求值,不是 GitHub 的表达式引擎。
  • 注释文本用的是 GitHub 文档措辞,不是从真实取消运行里抓的。
  • 按设计(fail-safe),真实超时但注释或 job 列表读不到时会跳过。
  • 文档改动是纯文本,未渲染。

yiliang114 and others added 3 commits November 5, 2026 05:30
…M#16280)

R3-1 (Critical): the classifier read only `cancelled` jobs, so a run whose
job already concluded `failure` before the rest was cancelled classified as a
deliberate supersede and dropped the regression. Read `failure`/`timed_out`
jobs first and fail open on them.

R1-2 (Critical): a cancelled run whose jobs all stopped before their first
step still fell through to `unknown`, which reports. Treat "no cancelled job
ran a step" as nothing-executed, preserving the zero-job behaviour.

R3-2: give that outcome its own token and case arm, so the caller's notice no
longer claims an annotation this path never read.

R3-4: the annotation-error test now uses two cancelled jobs, because with one
the fall-through prints the same `unknown` as the guard and cannot see it
removed.

Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
Patrol-Run: qwen-pr-closeout/jmw2mktggdp
…l fetch

R3-3: the comment no longer claims the gate admits only failures. R2-4: pin the
classifier fetch including ?per_page=100 and --paginate so flag-stripping reddens.
@qwen-code-review-bot

Copy link
Copy Markdown
Collaborator

Qwen Code review did not complete successfully. The review pipeline failed before a review could be posted. A transient error is retried automatically; if you are seeing this, retry with @qwen-code /review. See workflow logs.

Refresh base to ecc64a7. Main now carries ef9c891 (QwenLM#16351), whose ci.yml
corepack cache fix targets the pool-side "Cannot find module
qwen-ci-corepack/v1/pnpm/11.24.0/bin/pnpm.mjs" failure that killed Linux unit
shard 8/8 on run 41925293939 attempt 4.

Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
Patrol-Run: qwen-pr-closeout/jmw3vl6sufe
@wenshao

wenshao commented Nov 5, 2026 •

Copy link
Copy Markdown
Collaborator

merge-ready — 94/94 scripted assertions passed; verified head 268447d59ab5f6cb6cbdd748b6af596e66bd49e4 against base b9e20830f7727c97a5db6e129d542b0d572cadd3 (32 scenario cells, four live suites, 19-row mutation matrix). The head advanced to c8afa19bbffdd85bd7cf832e742f576f10587e45 while the report was being published; that commit is a mechanical merge of main over this PR, its production closure is byte-identical, and the affected suites were re-run green there (see Head movement).

中文摘要

结论:merge-ready。 中心主张成立:因 job 执行超时而被取消的 main run 现在会进入 analyze 并上报;并发取消与「没有任何 job 执行过步骤」的取消仍然沉默,failure 路径不变。发布过程中 head 从 268447d5 前进到 c8afa19b(把 main 合并进来);该合并是纯机械合并,本 PR 生产文件逐字节相同,受影响套件已在新 head 重跑通过。

A/B(见 01-ab-matrix.png 与下表): 10 个取消类场景在 base 臂全部被触发闸门拒绝(0 个上报);head 臂按设计 7 个上报、3 个沉默。其中真实超时 run 41743539611(E2E docker leg,annotation 同时含 timeout 与 operation-canceled)由 base 的静默翻转为 head 的上报,这是中心主张的承重翻转。后继判定另有一处翻转:main 前进后,带超时的 cancelled 后继现在拥有结果(旧 run 不再重复上报),并发取消的后继仍不上报。

真实语料: 最近 12 个 main 上 E2E Tests 的 cancelled push run,12/12 都带 The job has exceeded the maximum execution time of 1h0m0s,且 12/12 同时带 The operation was canceled.——超时优先于并发正是真实形状,不是假想边界。

发现(均为 Suggestion,不阻塞): ① classify-cancelled-run.sh 的 cancelled_seen 分句是死子句(可达状态枚举 3/3 与化简式等价);② 后继 run 的 run_attempt 传播没有被测试钉住(变异构建会把 attempt 1 的并发取消读成「后继不上报」,从而重复上报);③ read 以制表符分割,check_run_url 为空时字段会左移,当前实测落到 unknown(fail-open 上报),且本仓库 check-run 24 为 404,未观察到真实触发。

未覆盖: 真实 workflow_run 事件端到端、download/summary 步骤全链路、file_issue 写入器(本 PR 未改)、12 个真实 run 之外的 annotation 变体;变异矩阵按守卫而非按提交计。CI 本 head 的两条红均为环境:shard 8/8 在 Install dependencies 因 corepack pnpm 模块缺失失败(同日 main 同步骤因 registry ENETUNREACH 失败),Test (ubuntu-latest) 只是依赖 shard 全绿的汇总门;delay-automatic-review 是 api.github.com 连接失败。

Central claim and A/B

Claim. A main CI run that concluded cancelled because one of its jobs hit its timeout-minutes cap now files a failure report, while a plain concurrency cancellation and a cancellation in which no job ran a step stay silent. The failure path is unchanged.

The A/B runs the real workflow at each arm: the analyze gate expression is read from that arm's YAML and evaluated, then the arm's real live step bash runs under the workflow's default shell (bash --noprofile --norc -eo pipefail) with a replaying gh that serves payloads recorded from this repository's API. Every cancelled scenario therefore exercises the exact code the workflow ships.

scenario base head what changed
cancelled-timeout-real (run 41743539611) gate-reject reported FIX: timed-out run now reaches analyze
cancelled-timeout-two-pages gate-reject reported 2-page job list certifies against total_count
cancelled-pages-short gate-reject reported partial pages → unknown → fail-open report
cancelled-failure-job gate-reject reported failed job under a cancelled run → fail-open
cancelled-unreadable-annotations gate-reject reported unreadable annotation → fail-open
cancelled-unclassified-annotation gate-reject reported unmatched annotation → fail-open
cancelled-no-checkrun-url gate-reject reported missing URL → field shift → unknown
cancelled-concurrency-real gate-reject skipped real higher-priority cancel stays silent
cancelled-nothing-real (run 40177685642) gate-reject skipped 0-job displacement stays silent
managed-agent-pr-cancelled gate-reject skipped PR-triggered cancellation: untrusted annotations
failure-real (run 42059106624) reported reported unchanged path
successor-timeout reported skipped FIX: newer timeout-backed run owns the result
successor-attempt-2 reported skipped FIX: reads successor attempt 2, not attempt 1
successor-concurrency reported reported successor cannot report → older run reports
successor-success skipped skipped settled green successor suppresses
successor-in-progress reported (provisional) reported (provisional) unsettled successor: provisional

Counts. 10 cancelled-class scenarios: base reports 0/10, head reports 7/10 and keeps the 3 that must stay silent silent. The central flip is cancelled-timeout-real; the successor flip is successor-timeout (and its attempt-2 sibling). failure-real and the three successor controls are byte-identical across arms.

A/B matrix

Real corpus. The 12 newest E2E Tests push runs on main that concluded cancelled were sampled through the API: 12/12 carry The job has exceeded the maximum execution time of 1h0m0s, and 12/12 also carry The operation was canceled. — the exact coexistence the timeout-wins precedence handles. Two also carry a download-retry annotation, which matches neither pattern and changes nothing. The real concurrency annotation was taken from run 42059630017 (Canceling since a higher priority waiting request …), and the 0-job displacement shape from run 40177685642.

Real corpus and classifier tokens

Renderer. On the same real jobs payload, the base renderer names no job; head names the timed-out docker leg and its cancelled step (Run E2E tests). The infra classifier is unchanged and returns infra_only=false on that payload on both arms, so a timeout-only run files rather than reruns, exactly as the description says.

Renderer A/B

Head movement

The head moved from 268447d59ab5f6cb6cbdd748b6af596e66bd49e4 to c8afa19bbffdd85bd7cf832e742f576f10587e45 while this report was being published. The new commit is Merge remote-tracking branch 'origin/main' into fix/ci-tracker-cancelled (parents 268447d5 + ecc64a735a).

  • Mechanical merge, proven by tree. git merge-tree --write-tree 268447d5 ecc64a73 yields tree c0e2e70e656bbdd668360226e78108c4378a8d04, which equals git rev-parse c8afa19b^{tree}. git show --remerge-diff c8afa19b prints no resolution diff, so no hand-written merge content exists.
  • Production closure unchanged. Every file the measurements read is byte-identical at both heads: classify-cancelled-run.sh 6901ad9969…, main-failure-issue.yml 1880b541bf…, main-failure-signature.mjs ae41e434d5…, and main-ci-failure-issue-workflow.test.js. The only PR-closure file that changed is scripts/tests/ci-platform-lanes.test.js, which gained 152 lines from main; the merged file was re-run.
  • Re-measured at c8afa19b: vitest 244/244 (the one new test is main's), and main-failure-signature.test.mjs 63/63. The scenario cells and mutation matrix are carried forward by that proven-identical input closure, not by a re-run; their results remain attributed to 268447d5.
  • The merge message says main now carries the ci.yml corepack-cache fix (fix(ci): clear the pinned corepack cache before every install #16351), which is the pool-side failure attributed to the environment in CI on this head; that red should not recur on this head.

Corrections

  • The description's fail-safe direction is inverted. The Risk section says: "It fails safe toward silence — if the jobs or annotations cannot be read, no timeout is proven and the run is skipped … A real timeout whose annotations are unreadable is therefore dropped." The code does the opposite: the classifier returns unknown, and the live step warns "failing open and reporting" (main-ci-failure-issue.yml:142), leaving superseded=false. The author's own test is named "fails open and reports when the annotations cannot be read", and the cancelled-unreadable-annotations cell measures reported. This is a correction to the description, not a request to change the code: the implemented direction matches the classifier's header comment ("callers fail open on this so an unclassifiable cancellation cannot hide"), and it is the safer one for the reported outage. But a reviewer accepting the stated tradeoff is currently accepting its opposite.
  • The test count in the description is stale. It cites 224 passed for the two contract suites; at the verified head 268447d5 they report 243 (main-ci-failure-issue-workflow.test.js 171 + ci-platform-lanes.test.js 72), plus 63 in main-failure-signature.test.mjs (delta over base +17 / +0 / +1). At the merged c8afa19b head the same suites report 244, because main added one test to ci-platform-lanes.test.js.

Findings

The mutation matrix holds 19 guards: 17 are pinned by the live suites (17/19 killed, including the positive-control row that breaks the timeout regex and reddens three tests). The two survivors are adjudicated below — one is dead code, one is a real coverage gap. The matrix is per guard, not per commit.

Mutation matrix

All three findings are Suggestion-severity and non-blocking; none is a failed assertion.

S1 — cancelled_seen is a dead disjunct in the nothing-executed branch

classify-cancelled-run.sh:92 tests "${cancelled_seen}" != 'true' || "${executed_seen}" != 'true'. executed_seen can only become true inside the same loop iteration that already set cancelled_seen = true, so the second test subsumes the first. Mutation M9 (drop the cancelled_seen disjunct) survives the whole suite, and an enumeration of the reachable (cancelled_seen, executed_seen) states (3 states, 0 mismatches) shows the branch is identical to executed_seen != true on every one of them. Outcome: a clause that cannot decide any outcome; removing it (or keeping it with a comment saying it is documentary) is a readability call, not a behavioural one.

Repro: node harness/run-mutations.mjs "$PWD" → M9 survived; node harness/dead-disjunct.mjs → "mismatch": 0.

S2 — the successor's run_attempt is not pinned by any test

main-ci-failure-issue.yml:204 passes ${successor_attempt:-1} to the classifier, and the successor lookup supplies .run_attempt. Mutation M15 (hardcode 1) survives the suite. Running the mutant build against a successor on attempt 2 — whose attempt 1 was a concurrency cancel and whose attempt 2 is a timeout — makes it read attempt 1, classify concurrency, and report the older run, while head reads attempt 2, classifies timeout, and suppresses the duplicate.

The code at head is correct; the gap is that no test distinguishes the two. A test with completed cancelled 555 2 and differently-classified per-attempt jobs would pin it. This matters because re-running a cancelled run is the normal infra recovery, so attempt 2 is exactly when the successor gate runs.

Repro: node harness/run-cell.mjs --tree tree-head --cell cells/successor-attempt-2.json … → skipped; with M15 applied → reported.

S3 — tab-delimited read shifts fields when check_run_url is empty

classify-cancelled-run.sh:63 reads id, check_run_id, steps_run from an @tsv row with IFS=$'\t'. Tab is IFS whitespace, so a run of tabs collapses: when check_run_url is absent, the step count slides into check_run_id (a 24-step job queries check-run 24). Measured on a synthesized cell: token unknown, and the live step reports (fail-open) — the safe direction for this repository, because check-run 24 returns 404 and the read error is caught. It was not observed in the 12/12 real corpus (every real cancelled job carried check_run_url), so this is latent rather than live. A non-whitespace delimiter (or |-joined row) would make the empty field unambiguous instead of reading an unrelated id.

Repro: node harness/run-cell.mjs --tree tree-head --cell cells/cancelled-no-checkrun-url.json … → reported with the failing open warning; logs/classifier.tsv → unknown.

CI on this head

The verified head 268447d5 has three red checks; none is caused by this diff.

  • Linux unit tests (shard 8/8) fails in Install dependencies after 49 s with Error: Cannot find module '<runner>/qwen-ci-corepack/v1/pnpm/11.24.0/bin/pnpm.mjs'. No test executes. Shards 1–7 pass on the same head.
  • Test (ubuntu-latest, Node 22.x) fails in its single step Require the build and every Linux test shard to pass — a dependent gate, not a test result.
  • The same job on main (run 42059106624, head 5d45b429) fails in the same Install dependencies step with connect ENETUNREACH 2606:4700::6810:b22:443, i.e. the registry, confirming the install path is the failing component rather than this diff.
  • delay-automatic-review fails with error connecting to api.github.com — the review pipeline, unrelated to this workflow.
  • Locally, the affected suites are green on both arms: head 243/243 vitest + 63/63 node, base 226/226 + 62/62; bash -n on all extracted steps (6 × 2 arms), YAML parse, actionlint and Prettier are clean, with negative controls proving Prettier and actionlint can fail.

Not covered

  • A live workflow_run completion event. The Actions expression engine and event delivery were not exercised; the gate expression was evaluated with GitHub-expression semantics in the harness. Everything downstream of the event is covered.
  • download and summary end-to-end. The download selector strings are pinned by the author's tests and the renderer was run directly on the real jobs payload, but the actual log download and analyze pass were not replayed.
  • file_issue writer. Unchanged by this PR and not exercised.
  • Annotation variants outside the 12 sampled runs. The runner-shutdown form, localized text, and run-level timed_out conclusions were not observed on main; the suite covers the shutdown form synthetically.
  • Per-commit attribution. The mutation matrix is per guard. At the verified head the PR has 7 non-merge commits (5 touching production files) and 2 merge commits; the guards those commits introduced were mutated at the aggregate head, not one row per commit.
  • The cancelled-no-checkrun-url shape is synthesized, not recorded: no real cancelled job in the sample lacked check_run_url.

Methodology

Environment: Linux x86-64, Node 24.22.1, jq 1.7.1; base and head checked out as detached worktrees under this artifact directory with the repository's installed dependencies. Cells are .mjs harnesses: harness/run-cell.mjs extracts the analyze gate and the live step verbatim from each arm's .github/workflows/main-ci-failure-issue.yml with the yaml parser, evaluates the gate, and executes the step bash under bash --noprofile --norc -eo pipefail with harness/gh-stub.mjs replaying recorded API payloads; harness/run-mutations.mjs applies one mutant at a time with byte-exact restore; harness/assertions.mjs re-checks every report claim and writes assertions.json. Raw logs and per-cell result.json files are in logs/ and cells-out/; fixtures are the unmodified API responses in fixtures/.

— DeepSeek-V4.1-Flash-1017@fda99a87 via Qwen Code /verify-pr (v0.26.8; skill c5273cc80df5)

@wenshao

wenshao commented Nov 5, 2026

Copy link
Copy Markdown
Collaborator

Updated the verification report above for the head movement: c8afa19bbffdd85bd7cf832e742f576f10587e45 is a mechanical merge of main over this PR (git merge-tree yields tree c0e2e70e656bbdd668360226e78108c4378a8d04, equal to the merge tree), the production closure is byte-identical, and the affected suites re-ran green there (vitest 244/244, node 63/63). Verdict unchanged: merge-ready (94/94 assertions).

@yiliang114

Copy link
Copy Markdown
Collaborator Author

Status at e6eb154d398bd: all review threads across rounds 1–5 are resolved. The round-5 items were handled as follows:

  • R5-1 (timeout downgraded by a sibling's failed read): fixed in ad18b323a — the loop records read_failed and continues; the verdict emits timeout before consulting it.
  • R5-5 (successor jq projection unpinned): fixed in e6eb154d — the new test feeds raw workflow_runs JSON through the caller's real --jq filter; dropping \(.id // "") reddens it (mutation-verified).
  • R5-2 / R5-3 / R5-4 (shutdown→infra routing, workflow_dispatch trust arm, notice wording): Suggestion-level, deferred to ci: failure-report refinements — name only timed-out cancelled legs; skip cancelled PR runs at the trigger #16319 per the 5-round scope rule, each recorded with its mechanism.

CI is green (27 passed, 0 failed), no conflicts. Re-review requested from @chiga0 (the earlier approval was dismissed by the test-only head move). Ready for approval.

@yiliang114

Copy link
Copy Markdown
Collaborator Author

@qwen-code /triage

All review threads are resolved (24/24) and the required lanes are green on e6eb154d398b (Qwen Code CI run 42187357561, tui-parity, Managed Agent gate; 0 failed / 0 pending).

The standing CHANGES_REQUESTED is bound to c8afa19bbf, three commits behind this head — R4-1 from that round was fixed in 555e7780 and R5-1/R5-5 were closed in ad18b323a and e6eb154d39. Round-5 Suggestions R5-2/R5-3/R5-4 stay deferred on #16319 (recorded there), so nothing is silently dropped.

@wenshao

wenshao commented Nov 6, 2026

Copy link
Copy Markdown
Collaborator

评审结论(head e6eb154d):代码无阻塞项

范围:10 个改动文件——新增 classify-cancelled-run.sh、workflow 的 trigger / live / download / infra 与评论渲染、4 个设计文档、2 个契约测试套件。

己方验证

  • 测试:scripts/tests/main-ci-failure-issue-workflow.test.js + ci-platform-lanes.test.js 251 passed;main-failure-signature.test.mjs 63 passed;bash -n、workflow YAML 解析、prettier 均干净。
  • 变异:改掉超时签名 exceeded the maximum execution time → 5 例变红;把 verdict 里的 read_failed 分支提到 timeout_found 之前 → 5 例变红。R5-1 的修复确实被测试钉住,不是只靠读代码的判断。
  • 真实数据(只读):对 main 上 116 个历史 cancelled run(覆盖 Qwen Code CI / E2E / SDK Java / Desktop / tui-parity / Managed Agent gate)直接跑真实判别脚本——97 个不含 failure job 的 run 得 45 timeout / 42 nothing-executed / 10 concurrency,0 unknown、0 unreadable;19 个带 failure/timed_out job 的 run 按设计得 unknown 并 fail open 上报。
  • 真实 run 直跑 live 步:42273598606(annotation 为 The job has exceeded the maximum execution time of 1h0m0s)→ superseded=false;42217945806(无 job 执行)→ true;42224009046(取消,annotation 含 operation was canceled)→ true;同一 timeout run 走 push 分支、successor 未定 → provisional=true 但仍上报(file_issue 不读 provisional),不存在「既不重跑也不建单」的静默丢弃。
  • CI:27 SUCCESS / 122 SKIPPED / 4 CANCELLED(均为 Qwen Autofix 的 route,已被后续 run 顶替),无红、无 pending。

非阻塞建议

  1. PR 正文的 fail 方向与代码相反。 Risk & Scope 说 annotations 读不到时被 skip,Evidence 表写 unreadable → superseded=true;实际 live 的 *) 分支是 fail open 上报,并有 fails open and reports when the annotations cannot be read 用例。同一遗漏也在 docs/design/2026-10-27-installed-acceptance-nightly.md 与 .zh-CN.md:句子只说「带执行超时 annotation 才上报」,未覆盖 unknown/unreadable 的 fail-open。建议按当前行为改正文与这两行文档(第 4 轮已记录该项)。
  2. 正文证据计数已过期。 「224 passed」在 head 上是 251(R5-5 加了用例后未同步)。
  3. PR-arm 守卫的 pull_request_review 半边无测试。 删掉 || "${EVENT_NAME}" == 'pull_request_review' 后 178 例仍全绿(实测)。代码本身正确,建议补一条 EVENT_NAME=pull_request_review 的用例。

合并前需要明确的冲突

#16277 与 #16280 是同一功能的两个实现:两者都写 Fixes #16272,都改 main-ci-failure-issue.yml 与 main-failure-signature.mjs,但语义不同——#16277 对超时 run 整轮重跑,#16280 则是提交 issue。两者不能同时合入,需先明确保留哪一个,否则同一次超时会出现两套处置。

合并状态

代码与证据已可合并。当前 reviewDecision 仍为 CHANGES_REQUESTED,来源是 round 4 在 c8afa19b 上的评审——其唯一 C 级项 R4-1(successor 只认 timeout)已在 head 上按 timeout | unknown 修好,之后 bot 的评审均为 COMMENTED,状态没有翻过来。所有 review thread 已 resolved。建议请该评审人在当前 head 上重评以解除阻塞。

@qwen-code-review-bot

Copy link
Copy Markdown
Collaborator

Sandboxed verification: ⚠️ incomplete — infrastructure failure - workflow run

The verification job did not complete (checkout, runner, or setup error) and produced no report. See the workflow run for details.

中文 — 判定:⚠️ 未完成 · 基础设施故障

验证作业未完成(检出、runner 或初始化错误),未生成报告。详见工作流运行日志。

— Qwen Code · sandboxed verification

@qwen-code-review-bot

Copy link
Copy Markdown
Collaborator

⏸️ Deferring to the maintainers — needs a human call on this one.

Escalating this one to a maintainer — I reviewed it fully and found no defect that would justify requesting changes, but there is a decision here that the gate should not make on its own.

1. #16277 is a competing implementation of the same fix, and only one can land.

Both PRs are open against main, both say Fixes #16272, and both edit .github/workflows/main-ci-failure-issue.yml, .github/scripts/ci/main-failure-signature.mjs and scripts/tests/main-ci-failure-issue-workflow.test.js. They disagree on the remedy: this one files an issue for a run cancelled by a job timeout, #16277 reruns the run instead. Those are different operational behaviours for the same incident, so merging both would leave a single timeout with two dispositions. The authors are different people (yiliang114 here, 2SinceJuly there), which is why I don't think a bot should pick the winner — approving this PR would effectively close out the other contributor's work by automation. This needs a human to choose, and to tell the other author.

2. The PR description states the opposite fail direction from the code.

Risk & Scope says the change "fails safe toward silence" when jobs or annotations cannot be read, and calls that "the user-requested tradeoff"; the Evidence block lists unreadable annotations as a skip. The shipped code fails open and reports in that case, and a test asserts exactly that behaviour. I think failing open is the right call for a tracker whose original failure mode was silence, but the body currently argues the other side while attributing it to a human decision — so whoever signs off should confirm the shipped direction is the intended one, then correct the body and the two design docs this PR touches (both the nightly and managed-agent-gate docs describe only the timeout-vs-concurrency split and omit the fail-open unknown/unreadable paths).

Neither item is a code defect, and per AGENTS.md's five-round rule I am not requesting changes over documentation at this point in a PR's life. Two smaller fragilities are recorded in the review comment above and are unreachable or theoretical today.

For context on merge mechanics: the standing CHANGES_REQUESTED is the bot's own review of c8afa19b, two commits behind the head I reviewed (e6eb154d), and all 24 review threads are resolved — so that block is stale rather than substantive and a fresh review on current head supersedes it. CI on the reviewed head is fully settled with 0 failures (27 success, 177 skipped, 4 cancelled route bot jobs from concurrency supersedes). The workflow file also passes the size ratchet with only 356 bytes of headroom, so the next PR to grow it will need a .size-baseline bump.

If #16277 is resolved in this PR's favour and the description is corrected, my assessment is that it is ready to merge — I found nothing in the code I would want changed first.

中文说明

把这个 PR 上交 maintainer——我做了完整评审,没有发现足以支持 request changes 的缺陷,但这里有一个不该由闸门自行做出的决定。

1. #16277 是同一修复的竞争实现,二者只能合入一个。

两个 PR 都开向 main,都写着 Fixes #16272,都修改 .github/workflows/main-ci-failure-issue.yml、.github/scripts/ci/main-failure-signature.mjs 和 scripts/tests/main-ci-failure-issue-workflow.test.js。它们在补救方式上不一致:本 PR 对因 job 超时而被取消的 run 提交 issue,#16277 则改为重跑该 run。对同一次事故这是两种不同的运维行为,因此同时合入会让一次超时得到两套处置。两位作者不是同一人(这里是 yiliang114,那里是 2SinceJuly),这也是我认为不该由 bot 裁定胜负的原因——批准本 PR 实际上等于用自动化终结另一位贡献者的工作。这需要人来选择,并告知另一位作者。

2. PR 描述陈述的 fail 方向与代码相反。

Risk & Scope 说在 jobs 或 annotation 读不到时「朝沉默方向 fail-safe」,并称之为「用户要求的取舍」;Evidence 表把 annotation 读不到列为跳过。而实际发布的代码在这种情况下 fail open 并上报,且有一条测试正是断言该行为。我认为对一个原本失效模式就是沉默的追踪器而言,fail open 是正确选择,但正文目前在主张相反立场、并将其归因于人的决定——所以签署的人应当确认落地方向就是意图方向,然后更正正文以及本 PR 改动的两份设计文档(nightly 与 managed-agent-gate 两份都只描述了 timeout 与 concurrency 的二分,遗漏了 fail-open 的 unknown/unreadable 路径)。

两项都不是代码缺陷,按 AGENTS.md 的五轮规则,在 PR 走到这个阶段我不会因文档问题提出 request changes。另有两处较小的脆弱性已记录在上方评审评论中,目前不可达或仅为理论风险。

关于合并机制的背景:仍然生效的 CHANGES_REQUESTED 是 bot 自己对 c8afa19b 的评审,比我审的 head(e6eb154d)落后两个 commit,且全部 24 条 review thread 都已 resolved——因此该阻塞是过期的而非实质性的,在当前 head 上的一次新评审即可取代它。所审 head 上的 CI 已全部定论、0 失败(27 success、177 skipped、4 个因并发顶替而 cancelled 的 route bot 作业)。workflow 文件也通过了 size ratchet,但只剩 356 字节余量,因此下一个使其增长的 PR 需要更新 .size-baseline。

如果 #16277 的取舍结果有利于本 PR、且描述被更正,我的评估是它可以合并——代码层面我没有找到任何我会要求先改的东西。

@qwen-code-review-bot

Copy link
Copy Markdown
Collaborator

Triage re-run completed without a new review.

⚠️ The bot has neither a verdict nor a deferral on e6eb154d398bd2fb7e065c0e39d91e6a9878ad47 — no APPROVED, CHANGES_REQUESTED, or COMMENTED review of its own. A DISMISSED one does not count: dismiss_stale_reviews voids the bot's approval on every push, which is exactly when a fresh one is needed. If this re-run was meant to review or approve, it did not, and an approval left by another account is a separate vote that does not count as the bot's own.

⚠️ 机器人在 e6eb154d398bd2fb7e065c0e39d91e6a9878ad47 上既没有裁决也没有 defer —— 没有属于它自己的 APPROVED、CHANGES_REQUESTED 或 COMMENTED 评审。DISMISSED 不算:dismiss_stale_reviews 会在每次推送时作废机器人的批准,而那恰恰是需要一次新批准的时刻。如果这次重跑本应评审或批准,那么它没有做到;而其他账号留下的批准是另一张票,不能算作机器人自己的。

The stage comments above were updated with the latest result. View workflow run.

上方各阶段评论已更新为最新结果。查看工作流运行。

@qwen-code-review-bot

qwen-code-review-bot commented Nov 7, 2026 •

Copy link
Copy Markdown
Collaborator

Sandboxed verification: ⚠️ incomplete — cancelled - workflow run

The verification job was cancelled before producing a report.

中文 — 判定:⚠️ 未完成 · 已取消

验证作业在生成报告前被取消。

— Qwen Code · sandboxed verification

yiliang114 and others added 3 commits November 7, 2026 18:09
…LM#16280 R4-1)

The R4-1 fix widened the successor-ownership rule from `timeout` to
`timeout | unknown`, and the classifier now skips on `nothing-executed` as
well as `concurrency`, but both design docs this PR touches still described
the pre-fix rule: only a timeout-backed cancelled successor owns the result,
and a cancelled run reports only when a job carries a timeout annotation.

Restate the shipped rule in all four files (both languages): the two verdicts
that stay silent, the fail-open `unknown`/`unreadable` reports, a cancelled
run that also carries a `failure`/`timed_out` job, and the `unreadable`
successor that deliberately does not take ownership because a failed read
certifies nothing about what that run concluded.

Docs only — no workflow or script bytes change, so the workflow-size ratchet
is unaffected.
…M#16280)

Merging main into this branch brought in 7cb46d9 (QwenLM#13969), which added a
seven-line comment — 501 bytes — to main-ci-failure-issue.yml. The merged file
is 61381 bytes, 145 past the 57140 + 4096 ratchet, so the workflow-size gate
fails on growth this PR did not author.

This PR's own contribution to that file is untouched: the merge's diff on it is
exactly main's seven added lines and nothing else. Record the merged size so
the ratchet reflects the tree this branch now carries.
@wenshao

wenshao commented Nov 7, 2026 •

Copy link
Copy Markdown
Collaborator

merge-ready — 143/143 scripted assertions passed; verified head c8c21f3fd2d8baffa6f83ee008443dae26fca43e against base 1b9aef22646905339667982df7c202aaed935d32 (26 scenario cells on both arms, a 20-row mutation matrix, and four live suites).

中文摘要

结论:merge-ready。 中心主张在新 head 上重新测过并成立:因 job 执行超时被取消的 main run 现在进入 analyze 并上报;并发取消、以及没有任何 cancelled job 跑过步骤的取消仍然沉默;failure 路径不变。后继归属的 R4-1/R5-1 修复也各自被变异钉住(见 Central claim and A/B 与 Mutation matrix 两张表)。

上一轮发现: 三项 Suggestion(cancelled_seen 死子句、后继 run_attempt 未钉、TSV 字段左移)在新 head 上仍然成立。PR 正文里「fail 方向写反」的更正已被作者修正——正文现在明确写 fail-open,证据表也写 unreadable → superseded=false;正文的测试计数仍是旧值(见 Corrections)。评审 round-5 的 R5-3(pull_request_review 守卫无测试)同样仍然成立。

本轮新增一条 Suggestion: 判别脚本里 jobs 读取失败返回的 unreadable token 没有任何测试钉住。把它改成 unknown 会让「后继 jobs 读不到」的格从「上报」翻成「跳过」——正是 R4-1 刚修好的失败模式,在相邻调用点重新打开。代码本身正确,缺的是钉子。

未覆盖: 真实 workflow_run 事件投递、download/summary/file_issue 全链路、本 PR 之外的 annotation 变体;变异矩阵按守卫而非按提交计。证据图见 Central claim and A/B、Mutation matrix、Real data 三节。

Previous findings at this head

# Finding Severity Status at c8c21f3f
R1 (was S1) cancelled_seen is a dead disjunct in the nothing-executed branch Suggestion stands — M02 survived; file unchanged
R2 (was S2) the successor's run_attempt is not pinned by any test Suggestion stands — M10 survived; the mutant flips successor-attempt-2 from skipped to reported
R3 (was S3) tab-delimited read shifts fields when check_run_url is empty Suggestion stands — cancelled-no-checkrun-url still queries check-run 24; the verdict is now unreadable instead of unknown (still fail-open, still reports)
C1 the description's fail-safe direction was inverted correction fixed — the body now says "Everything it cannot certify fails open into a report", and its evidence table says unreadable -> superseded=false
C2 the description's test count is stale correction stands — the body says 251; the two contract suites report 260 at this head
R5-1 a confirmed timeout was downgraded by a sibling's failed annotation read Critical (round 5) fixed and pinned — M03 and M04 are killed by keeps a confirmed timeout when a sibling annotation read fails; cancelled-timeout-sibling-read-fails reports
R4-1 the successor gate deferred to unreadable Critical (round 4) fixed and pinned — M11 and M12 are killed; successor-unreadable reports, successor-unknown skips
R5-3 the pull_request_review half of the PR guard has no test Suggestion stands — M13 survived

Central claim and A/B

Claim. A main CI run that concluded cancelled because one of its jobs hit its timeout-minutes cap now files a failure report, while a plain concurrency cancellation and a cancellation in which no job ran a step stay silent. The failure path is unchanged, and a cancelled successor that reports itself owns the older run's result.

The A/B runs the real workflow at each arm: the analyze gate expression is read from that arm's YAML and evaluated with GitHub-expression semantics, then the arm's real live step bash runs under the workflow's default shell (bash --noprofile --norc -eo pipefail) with a gh stand-in that replays payloads recorded from this repository's API and applies the caller's own --jq projection with real jq. Every cell below therefore exercises the code the workflow ships, not a restatement of it.

scenario base head what it proves
cancelled-timeout-real (42273598606) gate-reject reported FIX — a real timed-out E2E run now reports
cancelled-timeout-real-coexisting (41743539611) gate-reject reported timeout wins when the same job also carries The operation was canceled.
cancelled-timeout-sibling-read-fails gate-reject reported R5-1 — a sibling's failed read does not bury a confirmed timeout
cancelled-concurrency-real (42224009046) gate-reject skipped a deliberate concurrency supersede stays silent
cancelled-nothing-executed-real (42217945806) gate-reject skipped a cancelled job with 0 steps stays silent
cancelled-zero-jobs (40177685642) gate-reject skipped a run displaced while queued stays silent
cancelled-manual-cancel-executed gate-reject reported a manual cancel that ran steps matches neither pattern → fail open
cancelled-sibling-reads-fail-no-timeout gate-reject reported unreadable → fail open
cancelled-unreadable-jobs gate-reject reported a jobs-read failure → fail open
cancelled-pages-short gate-reject reported uncertified pages → fail open
cancelled-failure-job gate-reject reported a failure job under a cancelled run → fail open
cancelled-no-checkrun-url gate-reject reported R3's latent field shift still reports
managed-agent-pr-cancelled gate-reject skipped PR-triggered cancellation: untrusted annotations
managed-agent-pr-review-cancelled gate-reject skipped same guard on the pull_request_review arm
failure-real (42059106624) reported reported unchanged path
stale-attempt skipped skipped a later attempt owns the run
successor-timeout reported skipped FIX — a timeout-backed successor owns the result
successor-unknown reported skipped R4-1 — a fail-open successor owns the result
successor-attempt-2 reported skipped the classifier reads attempt 2, not attempt 1
successor-concurrency reported reported a concurrency successor cannot report → older run reports
successor-unreadable reported reported R4-1 boundary — an unreadable successor certifies nothing
successor-jobs-unreadable reported reported the same rule when the successor's job list cannot be read
successor-success skipped skipped a settled green successor suppresses
successor-skipped reported reported a skipped successor cannot report
successor-in-progress reported (provisional) reported (provisional) unsettled successor: provisional report
successor-none reported reported no successor at the new tip

Counts. The 14 cancelled-class cells: base reports 0/14, head reports 9/14 and keeps the 5 that must stay silent silent. The 10 successor cells: base reports 9, head reports 6 — the three flips are exactly the duplicate reports R4-1 removes. failure-real and the three successor controls are identical across arms. The base reports 0 because its trigger admits failure only; this is a gate flip plus a semantics flip, not a formatting change.

A/B matrix

Real data

The shipped classifier was run against the live API, not a fixture, on the 25 newest cancelled push runs on main across the watched workflows (1,000 cancelled runs scanned; 25 push/main). The verdicts: 12 timeout, 6 concurrency, 4 nothing-executed, 3 unknown, 0 unreadable, 0 errors. All three unknown runs carry a failure job under the cancelled run (two Managed Agent gate, one Qwen Code CI), which is the designed fail-open path — the accepted cost, now measured on real data rather than argued.

The replay was calibrated three ways on the same runs: the shipped classifier against the real API, the replayed live step, and the author's independently published live-step outputs at e6eb154d (42273598606 → superseded=false, 42217945806 → true, 42224009046 → true). All three agree on all five runs.

Real corpus and calibration

Head movement

The head advanced from 42ebf37768 to c8c21f3fd2 while this round ran. The new commit is ci: record main's 501-byte growth of main-ci-failure-issue.yml (#16280) and changes exactly one line of .github/workflows/.size-baseline.

  • Production closure byte-identical. git diff --stat 42ebf37768 c8c21f3f over the classifier, the workflow, the signature module and both contract suites is empty, and the sha256 of each file is equal at both heads (classify-cancelled-run.sh 3ec397040948…, main-ci-failure-issue.yml 67664b90a50a…, main-failure-signature.mjs bf01e0b098b1…, main-ci-failure-issue-workflow.test.js 90bdce4832f2…, ci-platform-lanes.test.js 1a715fb063d9…).
  • Positive control for that comparison. The same command over .size-baseline reports the one-line 57140 → 61381 change, so the comparator can report a difference.
  • Re-measured at c8c21f3f: the 26-cell matrix (identical outcomes), the two contract suites 260/260, the signature suite 63/63, and the workflow-size suite 216 passed | 23 skipped (239). The mutation matrix and the corpus are carried forward by the proven-identical input closure; their results are attributed to 42ebf37768 and the identical bytes at c8c21f3f.

Corrections

  • The previous correction about the description is resolved, not standing. The body now says "It skips on two certified verdicts only … Everything it cannot certify fails open into a report", and its evidence table says unreadable -> superseded=false (fail open: report). That matches the code and the measured cells. This is a note that the earlier correction no longer applies, not a request to change anything.
  • The stale test count is still stale, and has moved. The body cites 251 passed for the two contract suites; at this head they report 260 (main-ci-failure-issue-workflow.test.js + ci-platform-lanes.test.js). The signature suite is correctly cited at 63.

Findings

All findings are non-blocking Suggestions; none is a failed assertion. The mutation matrix holds 20 guards: 15 killed, 5 survived and adjudicated below.

N1 — the jobs-read failure token (unreadable) is unpinned

classify-cancelled-run.sh:38 returns unreadable when the jobs list cannot be read. No test pins that token: M05 (change it to unknown) survives the whole suite. Driving the mutant shows why it matters: the successor gate defers only to timeout | unknown, so a successor whose jobs list cannot be read would own the older run's result and drop its only report.

build successor-jobs-unreadable
head (c8c21f3f) reported (…does not own this result; reporting this run)
M05 (unreadable → unknown) skipped (the newer push run owns its result)

This is the same failure class as R4-1, one call site over: the author correctly separated unreadable from unknown for annotation reads and pinned that, but not for the jobs read. The code at head is correct; the gap is coverage. A one-line classifier case (jobs read fails → unreadable) plus a successor cell with jobs_error would pin it.

Repro: node harness/run-mutations.mjs --ids M05-jobs-read-unknown-token --label b6 → survived; node harness/run-cell.mjs --tree head --cell successor-jobs-unreadable → reported at head, and the mutant's own run is saved as mutant-out/m05-successor-jobs-unreadable.json (skipped).

R1 — cancelled_seen is a dead disjunct (prior S1)

classify-cancelled-run.sh:107 tests "${cancelled_seen}" != 'true' || "${executed_seen}" != 'true'. executed_seen can only become true inside the loop iteration that already set cancelled_seen = true, so the first test is subsumed. M02 (drop it) survives. This is inferred from the loop structure, not re-enumerated this round: the prior round measured 3 reachable (cancelled_seen, executed_seen) states with 0 mismatches, and this file is byte-identical to that round's (3ec397040948… at both heads). A clause that cannot decide an outcome; removing it is a readability call, not a behavioural one.

R2 — the successor's run_attempt is still unpinned (prior S2)

main-ci-failure-issue.yml:208 passes ${successor_attempt:-1}. M10 (hardcode 1) survives. The new R5-5 test pins the jq projection's \(.id // "") and \(.run_attempt // "") fields, but not the caller's use of the bound attempt: no test supplies an attempt-2 successor whose attempt 1 classifies differently.

Driving the mutant: successor-attempt-2 (attempt 1 = concurrency, attempt 2 = timeout) reads attempt 2 at head and skips; the mutant reads attempt 1, classifies concurrency, and reports — a duplicate report for one sustained failure. Re-running a cancelled run is the normal infra recovery, so attempt 2 is exactly when the gate runs.

Repro: node harness/run-mutations.mjs --ids M10-successor-attempt-hardcode-1 --label b3 shows the suite stays green. Then edit "${successor_attempt:-1}" to "1" in the workflow, run node harness/run-cell.mjs --tree head --cell successor-attempt-2 (the mutant calls /attempts/1 and reports; saved as mutant-out/m10-successor-attempt-2.json), and git -C trees/head checkout HEAD -- . to restore.

R3 — the TSV field shift is latent, not fixed (prior S3)

classify-cancelled-run.sh:71 reads id, check_run_id, steps_run from an @tsv row with IFS=$'\t'. Tab is IFS whitespace, so a run of tabs collapses: with an empty check_run_url, the step count slides into check_run_id and a 24-step job queries check-run 24. The cancelled-no-checkrun-url cell still does exactly that; the read now fails, so the verdict is unreadable and the run is reported (fail-open). It was not observed in the 25-run corpus — every real cancelled job carried check_run_url — so it is latent. A non-whitespace delimiter (or a |-joined row) would make the empty field unambiguous.

R4 — the pull_request_review half of the PR guard is unpinned (review R5-3)

main-ci-failure-issue.yml:124 guards both pull_request and pull_request_review. M13 (drop the second disjunct) survives all 260 tests. The cells managed-agent-pr-cancelled and managed-agent-pr-review-cancelled both pass at head, but no test distinguishes the arms. The Managed Agent gate does run on pull_request_review, and its annotation text is contributor-authored; the guard is the only barrier, and it has no pin on that half.

Mutation matrix

Mutation matrix

15/20 killed. The positive control M01 (break the timeout signature) is killed by reports a cancelled run backed by a job execution-time timeout, so the runner and the selection can make the suite fail. Vacuity: the tests this round depends on are killed by their own guards — M03/M04 by keeps a confirmed timeout when a sibling annotation read fails, M11/M12 by the two successor-ownership tests, M17/M18 by evaluates the successor jq projection end to end, M06 by fails open when the jobs pages cannot be certified complete, M07 by reports a timeout even when a shutdown annotation coexists on the job, M08 by reports a cancelled run that also carries a failed job.

Survivors are classified, not just listed: M02 and M19 are clauses that cannot decide an outcome (M19's empty-id guard: with the guard removed the classifier is called with an empty id, gh fails, unreadable is returned, and the caller still reports — same outcome); M05, M10 and M13 are coverage gaps with a demonstrated or reachable behavioural difference, each described above. None is a defect at head: every flip above is what the mutant does.

CI on this head

At the time this round posted, c8c21f3f had 28 pass, 1 pending, 7 skipping, 0 failed. The pending check was TUI parity snapshots (ink vs opentui), unrelated to this diff. The review decision is still CHANGES_REQUESTED from round 4; the only Critical there (R4-1) is fixed and pinned, and the author reports all review threads resolved. Apart from this report, no new third-party reviews or comments arrived during the round.

Coordination, not a code finding: #16277 is still open and implements the same issue with different semantics (rerun vs file). Two implementations cannot both land; this needs a maintainer decision independent of this round's evidence.

Not covered

  • A live workflow_run completion event. Event delivery and the Actions runtime were not exercised; the gate expression was evaluated with GitHub-expression semantics and everything downstream of the event was run.
  • download, summary, file_issue end-to-end. The renderer and the download projections are pinned by the author's tests and unchanged by this PR; the full chain from a live run to a filed issue was not replayed.
  • Annotation variants outside the recorded runs. Runner-shutdown text, localized text, and run-level timed_out conclusions were not observed on main; the suite covers the shutdown form synthetically.
  • workflow_dispatch and pull_request_review arms with real events. Their gate and guard cells are evaluated and replayed, but no live event of either shape was captured.
  • Per-commit attribution. The mutation matrix is per guard. This round did not enumerate the PR's commits one row each, so no per-commit coverage is claimed.

Methodology

The round ran on macOS (Darwin 25.x, Node 24.18.1, jq 1.7.1) in two detached worktrees under the artifact directory, seeded from the repository's installed dependencies. Base is the PR's baseRefOid 1b9aef22; head is c8c21f3f. Cells are .mjs harnesses: harness/run-cell.mjs extracts the analyze gate and the live step from each arm's YAML with the yaml parser, evaluates the gate, and runs the step bash with harness/gh-stub.mjs replaying recorded payloads; harness/assert-gate.mjs prints the gate truth table; harness/run-mutations.mjs applies one anchored mutant at a time, runs the same suites (--bail=1), restores with git checkout HEAD --, and refuses to continue on a dirty tree; harness/assertions.mjs derives every number in this report from the recorded files and writes assertions.json. Raw logs are in logs/, per-cell results in cells-out/, fixtures (unmodified API responses) in fixtures/, and mutation logs in mutation-logs/.

— via Qwen Code /verify-pr (v0.26.9).

@yiliang114

Copy link
Copy Markdown
Collaborator Author

Closeout at c8c21f3fd2d8. The three Criticals from rounds 1–4 are re-verified in the code at this head rather than from the resolved flags, the description and the four design-doc copies now say what the code does, and the branch carries current main with the size ratchet settled. Earlier status comments on this thread were written at older heads (ad18b323, e6eb154d); this one is the record for this head and does not overwrite any of them.

1. The three Criticals, checked in code and by mutation at this head

Finding Fixed in Where it is pinned at c8c21f3fd2d8
R1-2 — unknown reported a cancellation that proved nothing 193a310ea6 classify-cancelled-run.sh:107-113 emits a distinct nothing-executed token when no cancelled job ran a step, and the workflow gives it its own arm whose notice no longer claims a concurrency supersede (main-ci-failure-issue.yml:134-137); tests at main-ci-failure-issue-workflow.test.js:626 and :646
R3-1 — the classifier read only cancelled jobs, so a run that also carried a failed job was certified as a supersede 193a310ea6 classify-cancelled-run.sh:50-64: a job concluding failure/timed_out under a cancelled run yields unknown, which the live step reports instead of skipping; test at :671
R4-1 — successor ownership yielded only on timeout while the successor's own gate also reports a fail-open unknown 555e7780 main-ci-failure-issue.yml:200-212: the cancelled-successor arm asks the shared classifier and yields on timeout | unknown (:209) — the same two verdicts the successor reports itself at (:138-139 timeout, :141-143 fail-open). concurrency / nothing-executed still own the result (:130-137), and unreadable deliberately does not, with the reasoning at :201-206; test at :869

A green suite proves nothing unless it can redden, so I reverted each fix in a scratch tree and re-ran the suite that pins it (restoring and confirming a clean git status after each):

Mutation Result Case that reddened
:209 yields on timeout only (revert R4-1) 1 failed / 178 passed, exit 1 lets a fail-open cancelled successor own the result (no duplicate)
verdict block reads read_failed before timeout_found (revert R5-1) 1 failed / 178 passed, exit 1 keeps a confirmed timeout when a sibling annotation read fails
drop the failure/timed_out guard (revert R3-1) 1 failed / 178 passed, exit 1 reports a cancelled run that also carries a failed job
nothing-executed folds back into concurrency (revert R1-2) 2 failed / 177 passed, exit 1 skips a cancelled run that ran no job (displaced while queued), skips a cancelled job that stopped before its first step
successor --jq drops \(.id // "") (revert R5-5, e6eb154d) 1 failed / 178 passed, exit 1 evaluates the successor jq projection end to end, including the id and attempt fields

2. Description and design docs corrected

This closes item 2 of the bot's 2026-11-07 deferral to maintainers: the body argued the opposite fail direction from the code and attributed it to a human decision.

  • The body no longer says the change "fails safe toward silence". It states the shipped direction: only concurrency and nothing-executed — two certified conclusions — skip; timeout reports on positive evidence; unknown and unreadable fail open and report. The cost is named instead of hidden: an unreadable successor does not take ownership, so that one shape can file twice.
  • The discriminator is the shared .github/scripts/ci/classify-cancelled-run.sh, called by the live step after the stale-attempt check, with the successor-ownership arm asking the same script — so the two call sites cannot drift.
  • The stale 224 passed count is replaced with the numbers in section 4, measured at this head.
  • Successor-ownership yielding is in the Evidence table (timeout/unknown yield, concurrency still reports, unreadable still reports) and in Breaking changes: one sustained timeout files one issue.
  • 34ed530f aligned all four design-doc copies with the same verdicts — docs/design/2026-10-16-managed-agent-gate.md (:191-197) and its .zh-CN.md (:158-163), plus docs/design/2026-10-27-installed-acceptance-nightly.md (:30) and its .zh-CN.md (:30). Both documents previously described only the timeout-vs-concurrency split and omitted the fail-open unknown/unreadable paths and the nothing-executed skip.

3. Base sync and the size ratchet

  • main is merged at 42ebf37768 (main tip 1b9aef2264), and c8c21f3fd2 records .size-baseline 57140 → 61381 for main-ci-failure-issue.yml. The ratchet only downgrades to a warning when a PR leaves the file byte-identical to base, and this PR is the change to that file, so main-side growth lands in the hard-fail branch: at the merge-only head the gate was red by 145 B (61381 actual vs 57140 + 4096 = 61236). I reproduced both sides locally — exit 1 with the pre-bump baseline and exit 0 after it, using CI's own WORKFLOW_SIZE_BASE_SHA.
  • The sync changed nothing this PR authored: git diff 34ed530fbb c8c21f3fd2 -- .github/workflows/main-ci-failure-issue.yml is 7 insertions, and those 7 lines are byte-for-byte main's own 7cb46d9f (feat(hooks): add FileChanged notifications for local sessions #13969) comment block — the added content of that diff and of git diff 2666e3552d 1b9aef2264 -- <same file> are identical.
  • Check lint gate freshness was red at 34ed530f because main's gate moved after the branch last incorporated it (ci.yml in 8660f754, fix(ci): refresh host assets on older PR checkouts #16476, 2026-11-06T17:42Z; the branch's last main merge was 1ca9a1d0). At this head all nine gate files' latest main-side commits are contained in the branch (ci.yml → 8660f754, eslint.legacy-core-barrel-imports.mjs → 17ad9cdb, the rest older), and the lane's step 11 is green in CI.

4. What I ran at this head

Dedicated worktree at refs/pull/16280/head, corepack pnpm install --frozen-lockfile exit 0 in 2m44.6s, no symlink into any other checkout.

Command Result
npx vitest run --config ./scripts/tests/vitest.config.ts scripts/tests/main-ci-failure-issue-workflow.test.js scripts/tests/ci-platform-lanes.test.js 260 passed / 0 failed, exit 0 (main-ci-failure-issue-workflow.test.js 179 + ci-platform-lanes.test.js 81)
node --test .github/scripts/ci/main-failure-signature.test.mjs 63 passed / 0 failed, exit 0
WORKFLOW_SIZE_BASE_SHA=1b9aef22… bash .github/scripts/check-workflow-size.sh exit 0 — "every workflow file is under the 470000-byte gate and within 4096 bytes of its recorded baseline"
the five mutations above 5/5 reddened the case that pins them; none survived
behaviour probe: the shipped classifier against 12 stubbed fixtures (timeout / concurrency / no-step / no-cancelled-job / unmatched / cancelled+failed job / annotation 5xx / uncertifiable pages / jobs 5xx / timeout-with-unread-sibling / two-page merge) 12/12 matched the verdicts the description claims

CI at this head, settled 2026-11-07T11:27:40Z and read back at 11:29:02Z (run 42365470676): 37 checks — 30 success / 7 skipped / 0 failure. The 7 skips are the platform-conditional lanes (macOS and Windows unit tests, SDK Java, installed-npm acceptance, CLI-no-sandbox integration). Lint & Static is green (job 128382697715), including step 9 Check workflow file size and step 11 Check lint gate freshness, plus ESLint / actionlint / shellcheck / yamllint / Prettier and the .github/scripts helper tests.

Counting basis, so the numbers are reproducible: the check-runs API for this head SHA (GET /repos/QwenLM/qwen-code/commits/c8c21f3f…/check-runs). Matrix lanes register progressively, so a snapshot taken mid-run shows a smaller total, and gh pr checks may show a different row count again because it merges matrix legs.

The count rose from the 251 published before the sync because the merge brought main's test changes with it: four commits touching scripts/tests/ci-platform-lanes.test.js (+350 lines), and #15274 (e99286813e) adding one case to the workflow suite — files when a demoted lock wait sits beside a suite failure. This PR's own tests did not change.

5. Deferrals and the one open human decision

中文说明

在 c8c21f3fd2d8 收口。1–4 轮的三个 Critical 我是在这个 head 的代码里重新核的,不看 resolved 标记;描述与四份设计文档已与代码同向;分支已带上当前 main,尺寸棘轮结清。本条是这个 head 的记录,不覆盖线程里任何人的既有评论(此前几条状态评论写在 ad18b323、e6eb154d 上)。

三个 Critical 的代码级落点:R1-2 —— classify-cancelled-run.sh:107-113 有独立的 nothing-executed token,workflow 有独立 case 臂(:134-137),notice 不再声称并发顶替;R3-1 —— :50-64 的守卫:cancelled run 里只要存在 failure/timed_out 的 job 就先返回 unknown,不再被当成顶替而静默;R4-1 —— main-ci-failure-issue.yml:200-212 的后继臂问同一个共享脚本,并在 timeout | unknown(:209)上让位,与后继自己会上报的两个臂(:138-139、:141-143)严格对齐;concurrency/nothing-executed 仍不让位,unreadable 有意不让位(理由在 :201-206)。

变异实证:五个修复我逐个在临时树里改回去再跑钉住它的套件(每次跑完 git checkout -- 还原并确认 git status 干净),5/5 变红,红的具体用例见上表——包括 R4-1 让位规则、R5-1 的结论优先级、R3-1 的守卫、R1-2 的独立 token、R5-5 的后继 --jq 投影。没有一个变异存活。

描述与文档:bot 在 2026-11-07 上交维护者时点的第 2 项已收口——正文不再主张「朝沉默方向 fail-safe」,改为写清真实方向:只有 concurrency 与 nothing-executed 两个被确证的结论跳过,timeout 按正向证据上报,unknown/unreadable fail open 上报;代价也写明(unreadable 后继不接手,这一种形状可能建两份单)。判别器实名 .github/scripts/ci/classify-cancelled-run.sh,调用位置在 stale-attempt 检查之后,后继归属问同一脚本,两处不会漂移。过期的 224 passed 换成本 head 实测值。34ed530f 把四份设计文档(managed-agent-gate 与 installed-acceptance-nightly 各中英一份)对齐到同一套判定,此前两份都只写了 timeout 与 concurrency 的二分。

base 同步与尺寸棘轮:main 已在 42ebf37768 并入(main tip 1b9aef2264),c8c21f3fd2 把 .size-baseline 的 main-ci-failure-issue.yml 从 57140 顶到 61381。棘轮只在「PR 未改该文件(与 base 逐字节相同)」时降级为 warning,而本 PR 改的正是它,所以 main 侧增长走 hard-fail:只 merge 不顶 baseline 时红 145 B(实测 61381 vs 57140+4096=61236)。两面我都在本地按 CI 的 WORKFLOW_SIZE_BASE_SHA 实跑复现过(顶之前 exit 1、顶之后 exit 0)。同步没有改动本 PR 自己写的任何一行:git diff 34ed530fbb c8c21f3fd2 -- <该文件> 是 7 行新增,且与 git diff 2666e3552d 1b9aef2264 -- <同一文件> 的新增内容逐行相同,全部来自 main 侧 7cb46d9f(#13969) 的注释块。Check lint gate freshness 在 34ed530f 上红是 main 侧 gate 移动(ci.yml 的 8660f754/#16476)造成的;本 head 上九个 gate 文件的 main 侧最新提交都已被分支包含,CI 的 step 11 也是绿的。

实测:两套件 vitest 260 passed / 0 failed(main-ci-failure-issue-workflow.test.js 179 + ci-platform-lanes.test.js 81,JSON reporter 实取),node --test 63 passed / 0 failed;尺寸门禁 exit 0;五条变异全红;判别脚本行为探针 12/12 与描述一致;专用 worktree + corepack pnpm install --frozen-lockfile exit 0(2m44.6s),未软链任何其他 checkout。计数从此前的 251 上升,是合并带进 main 的测试改动:ci-platform-lanes.test.js 四个提交(+350 行),以及 #15274(e99286813e)给 workflow 套件新增的一例 files when a demoted lock wait sits beside a suite failure;本 PR 自己的测试没有变。

CI 终态:2026-11-07T11:27:40Z 全部落定、11:29:02Z 回读,37 个 check —— 30 success / 7 skipped / 0 failure;7 条 skipped 都是平台条件 lane(macOS 与 Windows 单测、SDK Java、installed-npm 验收、CLI-no-sandbox 集成)。Lint & Static 绿(含 step 9 尺寸门禁与 step 11 lint gate freshness,以及 ESLint/actionlint/shellcheck/yamllint/Prettier 与 .github/scripts helper tests)。取数口径:本 head SHA 的 check-runs API;矩阵 lane 是逐步注册的,跑中快照的 total 会更小,gh pr checks 因归并矩阵腿也可能显示不同行数。

defer 与唯一待裁项:R5-2/R5-3/R5-4 仍按五轮规则 defer 在 #16319;另补记两条(记录:N3 attempt 交接无测试钉、D4 unreadable 后继双报是有意取舍),使被接受的残余不被静默丢掉。#16277 是同问题域内一个仍开放的 PR,同样引用 Fixes #16272;两者关系由作者与维护者裁定,本条记录不作任何处置判断。

@yiliang114

Copy link
Copy Markdown
Collaborator Author

@qwen-code /triage

Triage on c8c21f3fd2d8, which has been the head since 11:39Z:

  • Threads: 24 total, hasNextPage=false, 0 unresolved.
  • Checks on this head: 29 pass / 0 fail / 7 skipped (skips are platform-only lanes). Qwen Code CI, tui-parity and Managed Agent gate all success.
  • The open CHANGES_REQUESTED is stale: latestOpinionatedReviews still holds the bot's own review from 2026-11-06T00:41Z bound to c8afa19bbf, five heads behind. Every finding it names is fixed and test-pinned on c8c21f3fd2d8 (record: fix(ci): report timed-out main CI runs instead of letting cancelled hide them #16280 (comment)).
  • Human review is already in: wenshao reviewed this exact head and called it merge-ready at 11:27Z.
  • Nothing is in flight to race with: the 🧐 Qwen Pull Request Review run queued on this head (42365469072) never received a job (status=pending, total_count=0, updated_at unchanged since 10:48Z).

Only the verdict needs to catch up with the code on this head.

@yiliang114

Copy link
Copy Markdown
Collaborator Author

Local verification: PASS

  • Mode: merge-reference
  • Tested revision: head c8c21f3fd2d8baffa6f83ee008443dae26fca43e; before arm = merge base 1b9aef22646905339667982df7c202aaed935d32 (also main's tip; the branch merged main at 42ebf377680b)
  • Runtime: text-contract harness built from exact blobs at each SHA (gh api contents?ref=<sha>), vitest 3.2.7 + node --test, bash steps spawned from the workflow YAML itself with a gh stub
  • CI gate: 37 check runs on this head, 0 failing, 0 in flight; the only non-success items are the by-design skips (macOS/Windows legs, SDK Java (none), Installed-npm lanes)

This head is four commits past the last verified head e6eb154d398b: one docs commit (34ed530fbbcd), a main merge (42ebf377680b), and the size-baseline bump (c8c21f3fd2d8). The three load-bearing files are byte-identical to that head — classify-cancelled-run.sh ebe255cb8a, main-failure-signature.mjs ae41e434d5, main-failure-signature.test.mjs 1442023636 — so the behaviour evidence below was re-run against the merged tree rather than assumed.

# Claim Observation Result
1 The trigger gate admits a cancelled workflow_run (#16272 shape) head: contains(fromJSON('["failure","cancelled"]'), ...) present and pinned; the it.each matrix returns true for cancelled (tui-parity, E2E Tests, ...). Pin arm (head tests × base YAML) returns false PASS
2 A cancelled run is reported only when a job actually timed out live-step scenarios in the shipped YAML: execution-time timeout reported; concurrency / PR-trigger / no-job cancellations stay superseded=true; failed annotation read does not defer the report PASS
3 Signature rendering names a timed-out cancelled job names a timed-out (cancelled) job even when no job concluded failure passes at head; fails on the base module (pin arm) PASS
4 Full contract suites stay green on the merged tree workflow suite 179/179, platform-lane suite 81/81, signature suite 63/63 at head PASS
5 .size-baseline ratchet is satisfied after main's 501-byte growth all 65 entries within the 4096-byte growth allowance (max delta +3972, min −768); the changed entry is exact: main-ci-failure-issue.yml baseline 61381 = actual 61381 PASS

Causal evidence and controls:

  • Both arms run the real main-ci-failure-issue.yml steps through bash with a stubbed gh, plus the real .github/scripts/ci/classify-cancelled-run.sh — not a re-implementation of the logic.
  • Unaffected conclusions keep their behaviour in both arms: failure / success / timed_out gate rows and the curated MAX_TOKENS notice are identical; ci-platform-lanes is 81/81 in both.
  • Regression pin: head's test files with base's workflow YAML and base's signature module → 23 failed / 156 passed (workflow) and 3 failed / 62 passed (signature), against 179/179 and 63/63 at head. Arms differ only in those two files; sha256 of both is recorded in the run log.

Residuals:

  • No real workflow_run cancellation event was produced; the cancellation shapes are replayed locally from the shipped step bodies plus a gh stub. Same boundary as the prior verified head.
  • The docs commit 34ed530fbbcd (R4-1) is editorial and was read, not rendered.
  • claim 5 was checked numerically against the head tree (65/65 entries); the full workflow-size.test.js needs all 69 workflow files, so the complete gate run is left to CI's Lint & Static lane, which is green on this head. My partial harness could only supply 9 of them.

Local verdict ≠ merge approval: reviewers and required approvals still decide.

中文摘要(完整版)

本地验证结论:PASS(merge-reference,run 5)

  • 被测版本:head c8c21f3fd2d8baffa6f83ee008443dae26fca43e;对照臂 = 真实 merge base 1b9aef22646905339667982df7c202aaed935d32(同时是 main 尖端,分支在 42ebf377680b 合过 main)
  • 运行环境:按各自 SHA 用 gh api contents?ref 精确取 blob 搭文本契约 harness,vitest 3.2.7 + node --test,workflow 步骤由 YAML 正文里真实 spawn bash 并挂 gh 桩
  • CI 门禁:该 head 37 条 check,0 红、0 在飞;非 success 的只有按设计跳过的腿(macOS/Windows、SDK Java (none)、Installed npm)

这轮的关键判断:当前 head 比上一个已验证 head e6eb154d398b 多四个提交(docs、合 main、size-baseline)。三个承重文件与那次 PASS 时逐字节相同(classify-cancelled-run.sh ebe255cb8a、main-failure-signature.mjs ae41e434d5、其测试 1442023636),所以行为证据是对合后的树重跑出来的,不是照搬旧结论。

五条声明:

  1. 触发门禁接纳 cancelled(#16272 形态):head 的表达式与矩阵为 true;pin 臂(head 测试 × base YAML)为 false。
  2. 只有「确实超时的 job」才上报:live step 的十来个取消形态里,执行超时上报、并发/PR 触发/无 job 取消保持 superseded、注解读取失败不推迟上报。
  3. 签名渲染会点名「超时取消」的 job:head 通过、base 模块下失败。
  4. 合后全套契约测试仍绿:workflow 179/179、platform-lane 81/81、signature 63/63。
  5. .size-baseline 棘轮达标:65 条全部在 4096 字节允差内(最大 +3972、最小 −768),被改的那条精确相等(61381 = 61381)。

因果证据与控制项:两臂都跑真实的 workflow 步骤正文 + 真实的分类脚本;failure/success/timed_out 与既有 MAX_TOKENS 提示在两臂一致。回归钉:head 测试 + base 的 workflow 与签名模块 → 23 红/156 绿(workflow)、3 红/62 绿(signature),而 head 是 179/179 与 63/63;两臂只差那两个文件,sha256 已记录。

缺口(不算已验证):没有制造真实的 workflow_run 取消事件,取消形态是本地复刻;docs 提交只做阅读不渲染;第 5 条是数值核对而非跑完整 workflow-size.test.js(该套件需要全部 69 个 workflow,本机 harness 只供了 9 个),完整门禁交给该 head 上已绿的 Lint & Static。

本地 PASS 不等于合并批准。

@yiliang114
yiliang114 dismissed stale reviews from qwen-code-review-bot, qwen-code-review-bot, and qwen-code-review-bot November 7, 2026 12:48

Stale bot review anchored to an old commit; all Critical findings independently verified fixed on head c8c21f3 (team double audit + wenshao public merge-ready verification). Re-review requested on the current head.

@qwen-code-review-bot

qwen-code-review-bot commented Nov 7, 2026 •

Copy link
Copy Markdown
Collaborator

Sandboxed verification: ✅ passed — merge-ready (agent verdict) - workflow run

Ran the PR in an isolated, token-free container: A/B against the base build, mock-free harness assertions, targeted gates. Advisory evidence for human reviewers — not a review, an approval, or a CI check.

Scripted assertions: 177 passed · 0 failed · 177 total

Targeted gates (existing tests, not counted individually above): 328 passed · 0 failed across 3 suite(s)

Flakiness gate: ✅ 3 changed test file(s) x 5 identical rounds, no divergence

中文 — 判定:✅ 通过 · 可合入(agent 判定)

沙箱验证在隔离、无凭证的容器中执行了该 PR 的代码(与 base 构建 A/B 对照、无 mock harness 断言、定向门禁)。仅作为评审证据,不构成评审、批准或 CI 检查。

脚本断言:177 通过 · 0 失败 · 177 总计

定向门禁(已有测试,不逐条计入上面的断言):328 通过 · 0 失败,共 3 个套件

抖动门:✅ 3 changed test file(s) x 5 identical rounds, no divergence

Verification report

PR 16280 round 1 verification — Verdict: merge-ready — 177/177 scripted assertions passed, 0 failed. Verified head: c8c21f3fd2d8baffa6f83ee008443dae26fca43e (merge 00cbac4, base tip c2e77d1). Review state: 24 review threads, all resolved; 0 standing change requests; 0 of 4 blocking-tagged threads open, 4/4 re-measured at the verified head, 0 left unmeasured.

中文摘要

结论:merge-ready(可合并) — 177/177 条脚本断言全部通过,0 失败。

A/B 结论:同一个 stubbed-gh 场景矩阵跑在 base 与 head 两臂上(workflow 步骤 bash 逐字提取、真实 jq、GitHub bash 契约):

  • 核心翻转成立:cancelled + 超时 annotation 的 main CI run,base 完全沉默(gate 只接纳 failure,ci: E2E "sandbox:docker" leg hits its 60-minute cap on pool hosts that rebuild the sandbox image cold #16214 藏了几周的机制),head 上报——issue 建单条件为真、infra_only=false(纯超时走建单而非重跑)、报告点名超时的 leg(sandbox:docker … : Run E2E tests)。
  • 不该报的保持沉默:并发顶替、排队期被顶替(无 job / 无步骤)、PR 触发的取消(信任门禁,wire 日志证实根本没读贡献者的 annotation)。
  • 读不到/匹配不了的取消一律 fail open 上报;带 failure job 的取消 run 也上报(R3-1);sibling 读失败不降级已确证的超时(R5-1)。
  • failure run 两臂行为逐字节一致(A/A 对照),分类器不介入;stale 事件归因于 staleness 且不调用分类器(R1-6)。
  • 后继归属:timeout/unknown 后继接管旧 run(一次故障一份单,R4-1),concurrency/unreadable/skipped 不接管。
  • gate 触发矩阵 24 格(含 schedule/dispatch/Managed-gate 各臂)与作者 it.each 一致。

变异矩阵:7 个守卫各一个变异,7/7 被钉住(harness 翻转 + 契约套件变红双重验证);作者声称的两个实证完全复现(超时签名改写 → 恰好 5 例变红;优先级重排 → 优先级用例变红)。

门禁:契约套件 260/260、签名套件 63/63、size ratchet、prettier、actionlint、shellcheck、bash -n 全绿。

未覆盖:真实 workflow_run 事件端到端(无 GitHub token);replay 未对照真实产物校准(无 token、无上一轮报告——校准物应为一真实的 Main-CI 失败 issue 正文);yamllint(容器内 pip3 无权限装不上);per-commit 归因(depth-2 检出,本地只有 1 个 commit 可达 vs 元数据 17 个)。

Central claim and A/B proof

Claim: a main-CI run cancelled by a job execution-time timeout now files a failure report, while non-breakage cancellations stay silent and failure runs behave exactly as before.

Instrument: the live/destination/download/infra/summary step scripts and the job-gate expression extracted verbatim (via the yaml parser) from both arms' main-ci-failure-issue.yml, driven under the GitHub bash contract (bash --noprofile --norc -eo pipefail) with a fixture-backed gh stub that applies every caller --jq with the real jq and wire-logs each call. The gate expression is evaluated per arm; a gate-blocked cell means the analyze job never starts — the base arm's only silence mechanism for cancelled runs.

Cell (identical fixtures on both arms) Base c2e77d1 Head (merge @​ c8c21f3)
C1 cancelled + timeout annotation / push / main silent — gate admits failure only (the #16214 hiding) reports — timeout notice, file_issue=true, infra_only=false, record-job condition true, comment names E2E Test (Linux) - sandbox:docker - shard 1/1: Run E2E tests
C2 cancelled + The operation was canceled. silent (gate) silent — concurrency skip
C3 cancelled, displaced while queued (0 jobs) silent (gate) silent — nothing-executed
C4 cancelled, job stopped before its first step silent (gate) silent — nothing-executed (notice names no annotation)
C5 cancelled, annotation reads fail silent (gate) reports — fail open
C5b cancelled + timeout, jobs spread over 2 pages silent (gate) reports — timeout (pages certified via total_count, timeout wins over coexisting shutdown)
C6 cancelled, annotation matches neither pattern silent (gate) reports — fail open
C7 cancelled run carrying a failure job silent (gate) reports — fail open (R3-1)
C8 cancelled / Managed Agent gate / pull_request silent (gate) silent — trust gate; wire log shows zero jobs/annotations reads (contributor text never consulted)
C9 failure / push / main (A/A control) reports reports — comment byte-identical across arms, classifier not engaged (1 jobs fetch: download only)
C10 stale attempt / cancelled + timeout silent (gate) silent — attributed to staleness; classifier not consulted (R1-6 ordering)
C11a successor cancelled + timeout n/a (gate) older run yields — one report, not two
C11b successor cancelled + concurrency n/a older run reports
C11c successor cancelled + unreadable n/a older run reports (failed read certifies nothing)
C11d successor cancelled + unknown n/a older run yields (fail-open successor reports itself)
C11e/f/g successor skipped / in_progress / success n/a reports / provisional report / yields (pre-existing shapes preserved)
C15/C15b timeout + sibling job's failed read silent (gate) timeout preserved at both the current-run and successor level (R5-1)
C12 failure run + cancelled-timeout successor reports defers to the successor — intended R4-1 change (successor files its own report; total reports: one)
C13/C14 cancelled + timeout / schedule / dispatch arms silent (gate) reports

Gate-level matrix: 24 trigger-shape cells × both arms — cancelled admitted exactly on the push/schedule/dispatch/managed-gate-PR arms, success/skipped/timed_out still blocked on both arms. The PR does not touch package.json/pnpm-lock.yaml, so the A/B has no dependency confound.

Captures: 01-ab-matrix-base-vs-head.png (the 168-assertion matrix as it ran), 02-mutation-matrix.png (the mutation matrix as it ran).

Review findings at the verified head

review-threads.json holds 24 threads, all resolved; changeRequests is empty. Blocking-tagged threads (first comment tagged Critical, bot association COLLABORATOR), all resolved and all re-measured at the head:

# Thread Status at head
R1-2 (a) unknown reports a cancellation that proved nothing (queued displacement files "Main CI failed") not reproduced — C3/C4 skip via nothing-executed; M4 mutation of that guard flips C3/C4 and reddens 2 suite tests
R1-2 (b) unknown still reports a cancellation that proved nothing (second round) not reproduced — same measurement as above
R3-1 classifier reads only cancelled jobs, so a cancelled run with a failure job is misread as a supersede not reproduced — C7 reports (fail open); M3 mutation flips C7 and reddens 1 suite test
R4-1 successor ownership (timeout only) drifts from the successor's own gate (timeout | unknown) not reproduced — C11a/C11d defer, C11b/C11c report; M5 mutation flips C11d and reddens 1 suite test

Resolved Suggestions re-measured in passing: R1-1 (comment names the cancelled leg — C1), R1-4 trust gate (C8 + M6), R1-6 ordering (C10), R1-7 check-run id + real --jq (wire log asserts check-runs/9001 fetched, never the job id; the stub applies the caller's filter), R2-1 page certification (C5b + suite), R2-3/R5-3 (PR/dispatch trust boundary — C8 measures the PR arm; the dispatch gap is a recorded deferral in #16319), R2-4/R3-3/R5-5 (fetch/projection pins — M7 harness kill relies on the gate pin; C11a's wire assertion proves the successor id/attempt projection feeds the classifier), R5-1 (C15/C15b + M2), R5-2/R5-4 (known deferrals in #16319; C3/C4 assert the notice wording the R5-4 fix produced). No thread claims a defect this round could not either measure or name as a recorded deferral.

Corrections

None — no earlier review or bot description was found inaccurate at this head.

Findings

No blocking finding; no non-blocking defect reproduced. Two measured observations, both consistent with the PR's stated intent:

  1. Intended delta beyond the central claim (measured, C12): a failure run whose newer push successor is cancelled-with-timeout now defers to that successor (base reported). The successor's own gate files its report, so the total is one report instead of one — this is the R4-1 dedup by construction, and the PR description states it ("a cancelled successor that reports itself now owns the older run's result").
  2. Successor projection field-shift with an empty conclusion (measured, does not hold as a defect): read -r successor_status successor_conclusion successor_id successor_attempt collapses the double space an empty conclusion produces, shifting id into conclusion. Verified this only occurs for non-completed successors, where those fields are never read (C11f: in_progress → provisional=true regardless). completed successors always carry a non-empty conclusion.

Not covered

  • Live workflow_run end-to-end: no GitHub token in this environment; the trigger, the real Actions API and the filed issue were exercised only through the replay harness. The PR's own "Tested on" table already marks the live Linux path untested.
  • Replay calibration: uncalibrated against a real emitted artifact — no token to fetch a filed Main-CI-failure issue, and this is round 1 (no previous-report.md). What would calibrate it: the body of a real issue this workflow filed, or a previous verify report. Mitigations: the stub's wire shapes and --jq projections are taken from the repo's own committed contract suite (evolved with the production workflow), the harness applies the caller's real --jq with real jq, and the failure path (C9) reproduces byte-identical output on both arms.
  • yamllint: not installed in this container (pip3: Permission denied during node scripts/lint.js --setup). YAML validity is established separately by the yaml-parser extraction and actionlint.
  • Per-commit attribution: the checkout is depth-2 — git rev-list HEAD^1..HEAD^2 reaches 1 commit vs 17 in the PR metadata. The aggregate HEAD^1..HEAD diff was verified; the mutation matrix is per-guard, not per-commit (coverageClaim: "per-guard" in mutations.json).
  • Out of scope per the PR: the ci: main-push failures file no issues — the reporter waits 20–100 min for a hosted runner and is then dropped as superseded #15470 reporter-queueing gap; the three deferrals recorded in ci: failure-report refinements — name only timed-out cancelled legs; skip cancelled PR runs at the trigger #16319 (runner-shutdown cancellation reading as concurrency, the annotation-trust guard not covering workflow_dispatch, skip-notice wording).

Methodology

CI verify lane (node:22-bookworm): merge-ref checkout, npm ci + npm run build done before the round; no network. Harness (harness/ in this artifact dir): extract.mjs pulls the gate and step scripts verbatim from each arm's YAML; gh-stub.sh serves fixture JSON through the real endpoints the steps call, applies every --jq with real jq, fails per-fixture via *.fail markers, and wire-logs all calls; pipeline.mjs renders step env, evaluates step/job conditions, runs steps under bash --noprofile --norc -eo pipefail with fresh per-cell RUNNER_TEMP; ab-matrix.mjs (168 assertions, ab-results.json) and mutation-matrix.mjs (7 guards, mutations.json, anchors asserted to match exactly once, mutant arms are copies, repo-tree mutations restored and git status --porcelain verified clean after each row, arm checksums re-verified byte-identical at the end). Gate logs in logs/; both suites re-run clean from the unmutated tree after the matrix. Raw counts: A/B 168 pass / 0 fail; mutation rows 7/7 detected (M1 = 5 suite reds, exactly the author's claimed witness; M2 = 1, the precedence case); shellcheck liveness probe (planted SC2086 caught) 1 pass; arm-integrity checksum check 1 pass — 177/177 total. The PR leaves the dependency tree untouched, so both arms share one node_modules with no confound.

Flakiness gate log

rounds=5 files=3 skipped=0
file .github/scripts/ci/main-failure-signature.test.mjs: (cd .) node --test ./.github/scripts/ci/main-failure-signature.test.mjs
file scripts/tests/ci-platform-lanes.test.js: (cd .) npx --no-install vitest run --config ./scripts/tests/vitest.config.ts ./scripts/tests/ci-platform-lanes.test.js
file scripts/tests/main-ci-failure-issue-workflow.test.js: (cd .) npx --no-install vitest run --config ./scripts/tests/vitest.config.ts ./scripts/tests/main-ci-failure-issue-workflow.test.js


per-file results (P=pass F=fail I=infra-exit, one letter per run):
  .github/scripts/ci/main-failure-signature.test.mjs: PPPPP
  scripts/tests/ci-platform-lanes.test.js: PPPPP
  scripts/tests/main-ci-failure-issue-workflow.test.js: PPPPP

verdict: pass
summary: 3 changed test file(s) x 5 identical rounds, no divergence

--- per-invocation detail (full copy in the artifact) ---
round 1 · .github/scripts/ci/main-failure-signature.test.mjs: P (exit 0)
round 1 · scripts/tests/ci-platform-lanes.test.js: P (exit 0)
round 1 · scripts/tests/main-ci-failure-issue-workflow.test.js: P (exit 0)
round 2 · .github/scripts/ci/main-failure-signature.test.mjs: P (exit 0)
round 2 · scripts/tests/ci-platform-lanes.test.js: P (exit 0)
round 2 · scripts/tests/main-ci-failure-issue-workflow.test.js: P (exit 0)
round 3 · .github/scripts/ci/main-failure-signature.test.mjs: P (exit 0)
round 3 · scripts/tests/ci-platform-lanes.test.js: P (exit 0)
round 3 · scripts/tests/main-ci-failure-issue-workflow.test.js: P (exit 0)
round 4 · .github/scripts/ci/main-failure-signature.test.mjs: P (exit 0)
round 4 · scripts/tests/ci-platform-lanes.test.js: P (exit 0)
round 4 · scripts/tests/main-ci-failure-issue-workflow.test.js: P (exit 0)
round 5 · .github/scripts/ci/main-failure-signature.test.mjs: P (exit 0)
round 5 · scripts/tests/ci-platform-lanes.test.js: P (exit 0)
round 5 · scripts/tests/main-ci-failure-issue-workflow.test.js: P (exit 0)

Evidence images

01-ab-matrix-base-vs-head

02-mutation-matrix

Harness scripts and raw logs are in the workflow run artifacts (7-day retention).

— qwen3.8-max-2026-09-02 via Qwen Code /verify-pr (v0.26.9; skill 1766260e45d7)

@qwen-code-review-bot

Copy link
Copy Markdown
Collaborator

Triage re-run completed without a new review.

⚠️ The bot has neither a verdict nor a deferral on c8c21f3fd2d8baffa6f83ee008443dae26fca43e — no APPROVED, CHANGES_REQUESTED, or COMMENTED review of its own. A DISMISSED one does not count: dismiss_stale_reviews voids the bot's approval on every push, which is exactly when a fresh one is needed. If this re-run was meant to review or approve, it did not, and an approval left by another account is a separate vote that does not count as the bot's own.

⚠️ 机器人在 c8c21f3fd2d8baffa6f83ee008443dae26fca43e 上既没有裁决也没有 defer —— 没有属于它自己的 APPROVED、CHANGES_REQUESTED 或 COMMENTED 评审。DISMISSED 不算:dismiss_stale_reviews 会在每次推送时作废机器人的批准,而那恰恰是需要一次新批准的时刻。如果这次重跑本应评审或批准,那么它没有做到;而其他账号留下的批准是另一张票,不能算作机器人自己的。

The stage comments above were updated with the latest result. View workflow run.

上方各阶段评论已更新为最新结果。查看工作流运行。

@wenshao

wenshao commented Nov 7, 2026

Copy link
Copy Markdown
Collaborator

merge-ready — 236/236 scripted assertions passed; verified head c8c21f3fd2d8baffa6f83ee008443dae26fca43e against base 1b9aef22646905339667982df7c202aaed935d32, with a third arm merging that head into today's main tip 1feab51dd7f1d839f6f3fd9df9f5f4c805f5980a. The head is unchanged since round 2 (2026-11-07 11:27Z), so every round-2 finding is re-measured here rather than quoted.

中文摘要

结论:merge-ready(236/236 断言通过)。 本轮在同一 head c8c21f3f 上独立重测中心主张,并把 PR 头合并进今天的 main 后重跑:31 个场景格 × base / head / 试合并三臂的结果见 Central claim and A/B 与 Trial merge into current main。base 臂对所有 cancelled 类场景都在 gate 被拒(19 格),head 与试合并臂逐格一致(20 reported / 10 skipped / 1 gate-reject)。

试合并(本轮新增的关键测量): main 已从 PR 的 base 1b9aef22 前进到 1feab51d(28 个提交、233 个文件),而 PR 与 main 同时改了 .size-baseline 和 ci-platform-lanes.test.js。本地合并无冲突,合并结果就是两边的并集:对 main 只多出 57140 → 61381 一行,对 head 只多出 main 的三条基线。合并树上契约套件 260/260、签名 63/0、workflow-size 242(219 passed | 23 skipped)、shell 棘轮绿,记录值 61381 与实际字节数 61381 精确相等。

真实语料: 用发布的判别脚本直连 API 跑今天最新的 25 条 cancelled push/main run(扫描 237 条已完成的 push run):17 timeout / 3 concurrency / 3 unknown / 2 nothing-executed,0 unreadable、0 错误。17 条 timeout 全部来自 E2E Tests(2026-11-06T14:58Z 至 2026-11-07T14:43Z),正是 #16214 的形态;抽查 42382919050,其 cancelled job 的 annotation 同时含 The job has exceeded the maximum execution time of 1h0m0s 与 The operation was canceled.,与 cancelled-timeout-real-coexisting 钉住的重合规则一致。3 条 unknown 都带一个 failure job——设计内的 fail-open 原因,不是噪声。

上一轮发现(同一 head 重新测量): R1(cancelled_seen 死子句)、R2(后继 run_attempt 未钉)、N1(jobs 读取失败的 unreadable token 未钉)、R3(TSV 字段左移,仍潜伏)、R5-3(pull_request_review 守卫无测试)全部仍然成立,均为 Suggestion;上一轮的 C2(正文测试计数过期)已修复:正文现在写 260 / 63,与本轮实测一致。

本轮新增变异: 新增四条单点变异(read_failed 与 concurrency 的先后、failure / timed_out 两个合取项各自、gate 的 dispatch 臂)。其中 read_failed 顺序、failure 合取项、dispatch 臂都被现有测试杀死;timed_out 合取项存活(M23),是本轮唯一的新发现:把它删掉后,「一个 job 结论为 timed_out、被取消 job 带 concurrency annotation」的运行会从「上报」翻成「跳过」——即丢掉一条带超时 job 的运行报告。代码在 head 上是对的,缺的是钉子。

未覆盖: 真实 workflow_run 取消事件投递;issue 写入链路(本轮无 PAT,未真的建单);25 条真实 run 中只完整抽检了 1 条的 annotation;#16319 记录的 workflow_dispatch 信任残余只按行为测量并给出边界。证据图见 01-ab-matrix.png … 04-real-corpus.png。

Previous findings at this head

Round 2 (2026-11-07 18:38–19:29, same head c8c21f3f) published merge-ready, 143/143. Every one of its findings is re-measured here at the same head rather than quoted.

# Finding Severity Status at c8c21f3f (this round)
R1 cancelled_seen is a dead disjunct in the nothing-executed branch Suggestion stands — M02 survives the suites (re-run)
R2 the successor's run_attempt is not pinned Suggestion stands — M10 survives
N1 the jobs-read unreadable token is unpinned Suggestion stands — M05 survives; driving the mutant flips successor-jobs-unreadable from reported to skipped
R3 tab-delimited read shifts fields when check_run_url is empty Suggestion stands — cancelled-no-checkrun-url still queries check-run 24 and reports via unreadable
R5-3 the pull_request_review half of the PR guard has no test Suggestion stands — M13 survives
C2 the body's test count was stale correction fixed — the body now says 260 and 63; measured 260/260 and 63/63

Correction to round 2's own record

  • C2 is closed, not standing. Round 2 reported the body's 251 passed as stale; the body now reads 260 passed / 63 passed, which is exactly what this round measures at this head. Nothing is left to correct.

Central claim and A/B

Claim. A main CI run that concluded cancelled because one of its jobs hit its timeout-minutes cap now files a failure report, while a plain concurrency cancellation and a cancellation in which no job ran a step stay silent. The failure path is unchanged, fail-open verdicts (unknown, unreadable) report, and a cancelled successor that reports itself owns the older run's result.

The A/B runs the real workflow at each arm: the analyze gate expression is read from that arm's YAML and evaluated with GitHub-expression semantics, then the arm's real live step bash runs under the workflow's default shell (bash --noprofile --norc -eo pipefail) with a gh stand-in that replays payloads recorded from this repository's API and applies the caller's own --jq projection with real jq. A third arm merges the PR head into today's main and runs the same cells.

scenario base head merge what it proves
cancelled-timeout-real-coexisting gate-reject reported reported timeout wins when the job also carries The operation was canceled.
cancelled-timeout-real gate-reject reported reported FIX — a real timed-out E2E run now reports
cancelled-concurrency-real gate-reject skipped skipped a deliberate concurrency supersede stays silent
cancelled-nothing-executed-real gate-reject skipped skipped a cancelled run with no executed step stays silent
cancelled-zero-jobs gate-reject skipped skipped a run displaced while queued stays silent
cancelled-manual-cancel-executed gate-reject reported reported a manual cancel that ran steps matches neither pattern → fail open
cancelled-timeout-sibling-read-fails gate-reject reported reported R5-1 — a sibling read failure does not bury a confirmed timeout
cancelled-sibling-reads-fail-no-timeout gate-reject reported reported unreadable → fail open
cancelled-unreadable-jobs gate-reject reported reported a jobs-read failure → fail open
cancelled-pages-short gate-reject reported reported uncertified pages → fail open
cancelled-failure-job gate-reject reported reported a failure job under a cancelled run → fail open
cancelled-no-checkrun-url gate-reject reported reported R3's latent field shift still reports
managed-agent-pr-cancelled gate-reject skipped skipped PR-triggered cancellation: untrusted annotations
managed-agent-pr-review-cancelled gate-reject skipped skipped same guard on the pull_request_review arm
failure-real reported reported reported unchanged path
stale-attempt skipped skipped skipped a later attempt owns the run
successor-timeout reported skipped skipped FIX — a timeout-backed successor owns the result
successor-unknown reported skipped skipped R4-1 — a fail-open successor owns the result
successor-concurrency reported reported reported a concurrency successor cannot report → older run reports
successor-unreadable reported reported reported R4-1 boundary — an unreadable successor certifies nothing
successor-jobs-unreadable reported reported reported the same rule when the successor jobs list cannot be read
successor-attempt-2 reported skipped skipped the classifier reads attempt 2, not attempt 1
successor-success skipped skipped skipped a settled green successor suppresses
successor-skipped reported reported reported a skipped successor cannot report
successor-in-progress reported reported reported unsettled successor: provisional report
successor-none reported reported reported no successor at the new tip
cancelled-schedule-timeout gate-reject reported reported new arm — a nightly schedule timeout reports
cancelled-schedule-sdk-python gate-reject gate-reject gate-reject a schedule run the gate does not name stays out
cancelled-dispatch-sdk-java gate-reject reported reported new arm — the SDK Java dispatch is classified
cancelled-dispatch-feature-sdk-java gate-reject reported reported the dispatch arm has no branch clause (#16319 residual)
cancelled-timedout-job-concurrency gate-reject reported reported a timed_out job under a concurrency cancellation → fail open (the conjunct M23 leaves unpinned)

Counts. The 14 originally-recorded cancelled-class cells (12 cancelled-* plus the two Managed Agent PR cells): base reports 0/14, head reports 9/14 and keeps the 5 that must stay silent silent (concurrency, nothing-executed, zero-jobs, and the two PR events). The 10 successor cells: base reports 9, head reports 6 — the three flips are exactly the duplicate reports R4-1 removes. failure-real and the three successor controls are identical across arms. The base reports 0 because its gate admits failure only; this is a gate flip plus a semantics flip, not a formatting change. The merge arm reproduces head cell-for-cell (31/31).

A/B matrix

Trial merge into current main

main moved from this PR's base 1b9aef22 to 1feab51d (28 commits, 233 files) while the PR sat, and the PR shares two changed files with that movement: .size-baseline and scripts/tests/ci-platform-lanes.test.js. The local merge of the PR head into that tip is conflict-free, and the merged tree is exactly the union:

  • .size-baseline versus main: only the one-line 57140 → 61381 bump.
  • .size-baseline versus head: only main's three post-base additions (ephemeral-runner-smoke.yml, qwen-triage.yml, qwen-code-pr-review.yml).
  • scripts/tests/ci-platform-lanes.test.js: carries both main's pr_test_runner change and this PR's cancelled-trigger pin at line 1974.

Measured on the merged tree: contract suites 260/260, signature 63/0, workflow-size.test.js 219 passed | 23 skipped (242), the shell ratchet green, the merged baseline entry exactly 61381 against an actual 61381, and all 31 A/B cells identical to head. bash -n on the classifier and prettier --check on the nine text files are clean.

Trial merge

Real data

The shipped classifier was run against the live API on today's 25 newest completed cancelled push/main runs across the watched workflows (237 completed push runs scanned). Verdicts: 17 timeout, 3 concurrency, 3 unknown, 2 nothing-executed, 0 unreadable, 0 errors.

All 17 timeouts are the E2E Tests lane between 2026-11-06T14:58Z and 2026-11-07T14:43Z — the #16214 shape, live. 42382919050 is one of them: its cancelled job (E2E Test (Linux) - sandbox:docker - shard 1/1, 24 steps) carries both The job has exceeded the maximum execution time of 1h0m0s and The operation was canceled., which is the coexistence rule cancelled-timeout-real-coexisting pins.

All three unknown runs carry a failure job under the cancelled run (two Managed Agent gate, one Qwen Code CI shard), which is the designed fail-open cause — on today's data the unknown arm reports real breakage, not noise.

Calibration: the live-API verdict and this round's replay of the live step agree on all five recorded runs (42273598606 → timeout, 42217945806 → nothing-executed, 42224009046 → concurrency, 41743539611 → timeout, 40177685642 → nothing-executed).

Real corpus

Findings

All findings are non-blocking Suggestions; none is a failed assertion. The mutation matrix holds 24 single-point guards: 18 killed, 6 survived, each survivor adjudicated below.

N1 — the jobs-read failure token (unreadable) is unpinned

classify-cancelled-run.sh:38 returns unreadable when the jobs list cannot be read. No test pins that token: M05 (change it to unknown) survives the whole suite. Driving the mutant shows why it matters: the successor gate defers only to timeout | unknown, so a successor whose jobs list cannot be read would own the older run's result and drop its only report.

build successor-jobs-unreadable
head (c8c21f3f) reported (…does not own this result; reporting this run)
M05 (unreadable → unknown) skipped (the newer push run owns its result)

This is the same failure class as R4-1, one call site over: the author separated unreadable from unknown for annotation reads and pinned that, but not for the jobs read. The code at head is correct; the gap is coverage. A one-line classifier case (jobs read fails → unreadable) plus a successor cell with jobs_error would pin it.

Repro: apply M05's one-token edit in trees/head, then node harness/run-cell.mjs --tree head --cell successor-jobs-unreadable (mutant output saved as mutant-out/m05-successor-jobs-unreadable.json; restore with git -C trees/head checkout HEAD -- .).

M23 — the timed_out conjunct of the failed-jobs guard is unpinned (new this round)

classify-cancelled-run.sh:55 treats a job concluding failure or timed_out under a cancelled run as fail-open evidence; both the code comment and the body name both conclusions. Only the failure half has a witness: M22 (drop failure) is killed by reports a cancelled run that also carries a failed job, while M23 (drop timed_out) survives every suite. I had predicted M23 would be killed; the measurement overruled the prediction.

build cancelled-timedout-job-concurrency
head (c8c21f3f) reported (failing open and reporting)
M23 (failure only) skipped (concurrency supersede … not reporting)

The shape is a cancelled run whose cancelled jobs carry a concurrency annotation while one job concluded timed_out. Head reports it through the fail-open guard; without the conjunct the run is certified as a supersede and its only report is dropped — the #16272 hiding shape, one clause over. Severity is Suggestion because the code at head is right; a fixture whose timed_out job sits beside a concurrency annotation would pin it.

Repro: node harness/run-mutations.mjs --ids M23-failed-jobs-drop-timed-out-conjunct --label m23 → survived; then the same one-token edit plus node harness/run-cell.mjs --tree head --cell cancelled-timedout-job-concurrency (mutant output mutant-out/m23-cancelled-timedout-job-concurrency.json).

R1 — cancelled_seen is a dead disjunct (round-2 S1)

classify-cancelled-run.sh:107 tests "${cancelled_seen}" != 'true' || "${executed_seen}" != 'true'. executed_seen can only become true inside the loop iteration that already set cancelled_seen = true, so the first test is subsumed. M02 (drop it) survives. A clause that cannot decide an outcome; removing it is a readability call, not a behavioural one.

R2 — the successor's run_attempt is still unpinned (round-2 S2)

main-ci-failure-issue.yml:208 passes ${successor_attempt:-1}. M10 (hardcode 1) survives: no test supplies an attempt-2 successor whose attempt 1 classifies differently. Driving it earlier showed the consequence — successor-attempt-2 reads attempt 2 at head and skips; hardcoded to 1 it reads attempt 1, classifies concurrency, and reports a duplicate. Re-running a cancelled run is the normal infra recovery, so attempt 2 is exactly when the gate runs.

R3 — the TSV field shift is latent, not fixed (round-2 S3)

classify-cancelled-run.sh:71 reads id, check_run_id, steps_run from an @tsv row with IFS=$'\t'. Tab is IFS whitespace, so a run of tabs collapses: with an empty check_run_url, the step count slides into check_run_id and a 24-step job queries check-run 24. Re-measured at this head: the cell still issues check-runs/24/annotations, the read fails, the verdict is unreadable, and the run reports (fail-open). Every real cancelled job in today's 25-run corpus carried a check_run_url, so it remains latent.

R5-3 — the pull_request_review half of the PR guard has no test (round-2)

M13 (drop pull_request_review from the trust guard) survives. The only live-step guard test is skips a PR-triggered cancellation without trusting contributor annotations, which drives the pull_request event; the pull_request_review cells pass only because the surviving pull_request clause catches them, so the review clause can be deleted with every suite green.

S2 — the empty-successor-id guard has no witness (round-2 survivor, carried forward)

M19 rewrites if [[ -n "${successor_id}" ]] to admit the empty id and survives. Inferred from the code (not driven this round): with an empty id the classifier is invoked with no run id, its gh call fails, it answers unreadable, which does not own the result — so both builds report and the only difference is one wasted failing API call. That is why it stays a Suggestion: the guard is defensive and nothing observable changes without it.

Review comments and the body's claims

The PR's own Reviewer Test Plan is walked row by row: every line of its Evidence table is a measured cell, including failure leaving the classifier unengaged (its call log carries no attempts/ or check-runs/ request). The workflow_dispatch residual #16319 records is measured as behaviour: the gate admits a dispatch on any ref (cancelled-dispatch-feature-sdk-java, gate admit) and the live step classifies it, because the trust guard covers only pull_request/pull_request_review. Bounding it: a dispatch needs write access, and a fork's branch is not dispatchable in the base repository; what it covers is a maintainer dispatching on a branch that exists in the repo, which is already listed as a deferral and is not a regression here.

Not covered

  • No live workflow_run cancellation event was delivered end-to-end; the cancellation shapes are replayed from recorded payloads plus a gh stub.
  • The download → summary → issue-writer pipeline past the analyze gate's live step is exercised only by the unit suites; no issue was filed against GitHub (no PAT in this round).
  • Of the 25 real corpus runs, one (42382919050) was inspected down to its annotation text; the other 24 were classified but their annotations were not read individually.
  • The workflow_dispatch trust residual (ci: failure-report refinements — name only timed-out cancelled legs; skip cancelled PR runs at the trigger #16319) is measured as behaviour only.
  • Windows/macOS lane behaviour and the Java lane are not exercised here; CI at this head skips them by design.
  • The mutation matrix is per-guard, not per-commit: 24 rows over the guards this PR adds or changes, not one row per commit of the PR.
  • S2 (M19) is inferred from code, not driven; its behavioural difference is unobservable in the cells available.

Methodology

Maintainer-local round on macOS, in the host working copy (prepare reported isolation: none detected); every arm executes in a scratch git worktree under tmp/pr16280-verify-20261108-020028/trees/, and no host-level install or auth state was modified. Head c8c21f3f, base 1b9aef22, trial-merge tip 1feab51d; the merge commit is fa0037209df5 with parents exactly 1feab51d + c8c21f3f. Harness scripts and raw logs live in the artifact directory (harness/, logs/, cells-out/, mutation-logs/); the evidence captures are evidence/01-ab-matrix.png … 04-real-corpus.png. Node 24.18.1, vitest 3.2.7, Bash 5.3.

@wenshao
wenshao enabled auto-merge (squash) November 7, 2026 23:50

@qqqys qqqys left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

APPROVE — reviewed at head c8c21f3fd2d8baffa6f83ee008443dae26fca43e. All four previously filed Criticals are fixed in the code at this head, and I found no new blocking correctness, security, data-loss, regression, or compatibility issue.

Prior blocking findings, each re-checked against this head

R1-2 — unknown reported a cancellation that proved nothing (zero-job runs, then steps: [] runs): fixed. The classifier now has a dedicated nothing-executed verdict (classify-cancelled-run.sh:107-113) reached when cancelled_seen != true (no cancelled job at all — displaced while queued) or executed_seen != true (every cancelled job stopped before its first step), and the live step maps it to superseded=true (main-ci-failure-issue.yml:134-137). executed_seen is set from the real step count projected by the loop's jq (((.steps // []) | length)), so both shapes the two rounds named now stay silent, and the zero-job {"total_count":0,"jobs":[]} payload still certifies (0 == 0) and reaches the same verdict instead of falling to unknown.

R3-1 — a cancelled run carrying a failure/timed_out job was silently dropped: fixed. The failed-job guard (classify-cancelled-run.sh:54-63) runs before the annotation loop and emits unknown when any job concluded failure or timed_out, so the generic The operation was canceled. annotation on a sibling can no longer classify a real regression as a supersede. A jq failure in that guard also emits unknown, not a false negative. The verdict reaches the *) arm and reports.

R4-1 — successor ownership credited only timeout while the live step also reports unknown: fixed. The successor case is now timeout | unknown) successor_owns=true (main-ci-failure-issue.yml:208-210), fed by the same script as the current-run gate, so the two cannot drift. The projection was widened to carry .id and .run_attempt for that second classification, and the cancelled arm is guarded on a non-empty successor_id.

R5-1 — a confirmed timeout downgraded to unreadable by a sibling's failed annotation read: fixed and unchanged since it was verified by execution. The read failure now sets read_failed=true and continues (classify-cancelled-run.sh:85-86), and the verdict block ranks timeout above unreadable (:99-104). I confirmed via compare/1ca9a1d05...c8c21f3fd that classify-cancelled-run.sh is absent from the diff — the script is byte-identical to the head at which @chiga0 executed it and killed 9/9 mutants, including the three around this fix, the failed-job guard and the successor timeout|unknown set.

Also re-checked: R1-4 — the pull_request / pull_request_review arms set superseded=true and exit before the classifier is invoked (main-ci-failure-issue.yml:124-126), so contributor-authored annotation text never decides a PAT-backed filing; R1-6 — the classifier runs only after the stale-attempt check and is gated on superseded != 'true' (:123); R1-1 — renderFailureComment now keeps cancelled jobs (main-failure-signature.mjs:414-417), matching the download-side projection widened to failure|cancelled at both jq sites, so a timed-out leg is actually named in the filed body; R1-5 — both design docs now state the shipped direction in both languages, including the unreadable non-ownership rule.

Independent scan of the current diff

No injection surface. Every untrusted-ish value reaches the script through env: (RUN_CONCLUSION, WORKFLOW_RUN_ID, RUN_ATTEMPT, EVENT_NAME, HEAD_SHA, HEAD_BRANCH) rather than inline ${{ }} in a shell body, and is expanded quoted; check_run_id comes from a jq TSV projection and is quoted in the annotations call; successor_id/successor_attempt come from a jq projection and are quoted. REPO is :?-required.

The script cannot abort the step or fake a verdict. It runs set -uo pipefail without -e, so each gh api/jq failure is handled by an explicit if ! branch rather than killing the substitution inside the set -euo pipefail step; every exit path ends in echo <verdict> with status 0. The pagination receipt (([.[].total_count] | max) vs the merged job count) means a partial page list reads unreadable rather than masquerading as a complete list that happens to lack the timed-out job.

Widening the gate and the download projections adds no untrusted path. A cancelled PR-arm run exits before both the classifier and the download steps (if: steps.live.outputs.superseded != 'true'), so the newly admitted conclusion reaches log analysis only for push/schedule/workflow_dispatch runs of the repo's own workflows.

.size-baseline is exact. The ledger entry for main-ci-failure-issue.yml is 61381 and the blob at this head is 61381 bytes — the growth came from main and this PR records it rather than drifting the gate.

One noted trade-off, explicitly not a blocker

The successor-ownership set is timeout | unknown, while the live step also reports on unreadable. During a jobs/annotations API outage an older run and its successor can therefore each file for one incident. This is reasoned in three places rather than overlooked — the script header (unreadable "says nothing about what the run concluded, so the successor check must not defer to it"), the inline comment at :202-207, and the design-doc sentence added by 34ed530fb — and deferring to an uncertifiable read would drop the only report of a real regression, which is the costlier error. Left as recorded, not requested.

CI

Lint & Static, Test (ubuntu-latest, Node 22.x), all eight Linux unit shards, the three integration shards and the TUI lanes pass at this head. The only non-pass entry is a pending authorize job in the review workflow, which does not gate this change. @wenshao approved at this same head.

@wenshao
wenshao merged commit 274c8a5 into QwenLM:main Nov 7, 2026
84 of 85 checks passed
indexGin added a commit to indexGin/qwen-code that referenced this pull request Nov 9, 2026
Keep main's ['failure','cancelled'] job semantics while landing the PR's coverage line: namedJobs is used both as the named-job list and as the coverage denominator, so a timed-out (cancelled) lane is named and the body stays self-consistent.

F1: semantic conflict resolved; main's cancelled-job test stays green. F2: logCount falls back to 0. F3: new fixture pins the cancelled/timed_out exclusion (kills M4).
@qwen-code-review-bot

Copy link
Copy Markdown
Collaborator

Released in v0.27.1.

2SinceJuly added a commit to 2SinceJuly/qwen-code that referenced this pull request Nov 16, 2026
main QwenLM#16280 landed the same gate widening this PR was built on, so the
conflicts were structural rather than semantic: take main's hoisted
conclusion check and its larger schedule-arm name list, and keep this
PR's classifier work in the signature module, which merged cleanly.

The size baseline moves to the merged file's actual byte size, and the
meta-test's substring assertion now pins the `fromJSON` form the
conclusion check took on main.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

review/self-reported The linked issue was opened by the PR author (self-reported)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ci: main-ci-failure-issue.yml ignores cancelled runs — timed-out main CI never alerts (how #16214 hid)

5 participants