Skip to content

fix(ci): scope the sdk-java host lock to the cache write-back - #15128

Merged
yiliang114 merged 7 commits into
mainfrom
autofix/issue-15117
Oct 22, 2026
Merged

yiliang114 merged 7 commits into
mainfrom
autofix/issue-15117

Conversation

@qwen-code-dev-bot

@qwen-code-dev-bot qwen-code-dev-bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator

What this PR does

Scope the per-host sdk-java lock in the MariaDB integration lane to the shared Maven cache write-back. The lane seeds its job-local Maven repository before taking the lock, runs both Maven commands unlocked, and holds one acquisition only for the write-back. The write-back's mkdir/timeout/find/cp children close fd 9 so a cancelled job's survivor cannot retain the lock. Raise the lane's ceiling to 310 minutes to cover the ~3-hour verify plus one lock wait and setup.

The fast tier and the release fallback keep the same lock helper, 5400-second budget, and lock not acquired within error signature.

Why it's needed

Run 39627609266's fast-tier job arrived at 17:38 and stayed head of the host-lock queue until it failed at 19:15, without starting Maven. The same lock was held by the MariaDB integration lane's install and full MySQL verify from 15:54 to 18:57 (run 39628985638). The fast tier's 90-minute budget cannot outlast a multi-hour hold, so every fast-tier job arriving during that window fails before Maven.

Both MariaDB Maven runs already resolve from a job-local repository, so the long hold protects nothing beyond the shared cache write-back — which is short. This is the same starvation class as #15108's run; this change narrows the hold for the #15117 failure.

Reviewer Test Plan

How to verify

  1. Read the integration lane's step: seed → SDK install → MySQL verify → single lock acquisition → mkdir/timeout/find/cp write-back, with 9>&- on the write-back children.
  2. Confirm the fast-tier step still sources the helper and calls sdk_java_lock_acquire 5400, and the fallback still contains flock --wait 5400 9.
  3. Run npx vitest run --config ./scripts/tests/vitest.config.ts sdk-java-workflow workflow-size sdk-java-lock; verify the ordering, descriptor-closure, and workflow-size assertions.
  4. Optional mutation check: move the lock above Maven, or remove 9>&- from a write-back child; the workflow pins should go red.

Evidence (Before & After)

N/A — CI workflow change with no user-visible UI.

Tested on

OS Status
🍏 macOS ⚠️
🪟 Windows ⚠️
🐧 Linux ✅

Environment (optional)

Sandboxed Linux container. npm run build, npm run typecheck, and npm run lint passed; the workflow meta-tests run under the repository's scripts vitest config. Java/Maven are unavailable locally, so the Java suites run in CI.

Risk & Scope

  • Main risk or tradeoff: the cache seed now runs unlocked. It is read-only and best-effort, and the write-back remains under the lock; a concurrent write-back can race the seed, so a rare cache miss or re-resolve is possible rather than corrupted host state. The Maven runs themselves touch the job-local repository, not host-shared state.
  • Not validated / out of scope: the Java suites and a real ECS host run. The fast-tier run does not execute the changed MySQL/MariaDB integration lane; a full-tier run remains necessary for that runtime evidence.
  • Breaking changes / migration notes: none. The lock file, helper, 5400-second budget, and error signature are unchanged. The integration lane's ceiling grows from 200 to 310 minutes to fit the ~3-hour verify plus one lock wait and setup.
  • Design docs: English · 简体中文

Linked Issues

Fixes #15117

中文说明

这个 PR 做了什么

把 MariaDB 集成通道的 sdk-java 宿主机锁缩小到只包住共享 Maven 缓存写回。该通道现在先种子复制 job 本地 Maven 仓库,再取锁;两次 Maven 命令都不持锁运行;只对缓存写回取一次锁。写回的 mkdir/timeout/find/cp 子进程会关闭 fd 9,避免被取消 job 的幸存进程继续持有锁。该通道上限从 200 分钟提高到 310 分钟,容纳约 3 小时的 verify 加一次等锁和 setup。

fast 通道和发布回退路径保持不变:同一个锁辅助脚本、同一个 5400 秒预算、同一个 lock not acquired within 错误签名。

为什么需要

run 39627609266 的 fast 通道 job 于 17:38 到达,一直排在宿主机锁队列的队首,直到 19:15 失败,Maven 一次都没启动。同一把锁从 15:54 到 18:57 被 MariaDB 集成通道的 install 和完整 MySQL verify 持有(run 39628985638)。fast 通道的 90 分钟预算熬不过数小时的持锁,因此在这个窗口内到达的 fast 通道 job 都会在 Maven 之前失败。

MariaDB 通道的两次 Maven 运行本来就都从 job 本地仓库解析,长时间持锁除了共享缓存写回之外保护不了什么 —— 而写回很短。这与 #15108 那次运行属于同一类饥饿;本改动针对 #15117 的失败收窄持锁范围。

评审测试计划

如何验证

  1. 阅读集成通道的步骤:种子复制 → SDK install → MySQL verify → 单次取锁 → mkdir/timeout/find/cp 写回,写回子进程带 9>&-。
  2. 确认 fast 通道仍 source 该辅助脚本并调用 sdk_java_lock_acquire 5400,回退分支仍包含 flock --wait 5400 9。
  3. 运行 npx vitest run --config ./scripts/tests/vitest.config.ts sdk-java-workflow workflow-size sdk-java-lock;核对执行顺序、描述符关闭与工作流体积断言。
  4. 可选变异检查:把锁移到 Maven 之前,或去掉写回子进程的 9>&-;工作流 pin 应转红。

证据(修改前 / 修改后)

N/A —— CI 工作流改动,无用户可见 UI。

测试平台

OS 状态
🍏 macOS ⚠️
🪟 Windows ⚠️
🐧 Linux ✅

环境(可选)

沙箱 Linux 容器。npm run build、npm run typecheck、npm run lint 均通过;工作流元测试使用仓库的 scripts vitest 配置。本地没有 Java/Maven,因此 Java 套件由 CI 运行。

风险与范围

  • 主要风险或取舍:缓存种子复制现在不持锁。它是只读且尽力而为的,写回仍在锁内;并发写回可能与种子复制竞争,因此可能出现罕见的缓存未命中或重新解析,而不会破坏宿主机状态。Maven 运行本身只访问 job 本地仓库,不碰宿主机共享状态。
  • 未验证 / 超出范围:Java 套件和真实 ECS 宿主机运行。fast-tier 不执行被改动的 MySQL/MariaDB 集成通道;该运行时证据仍需 full-tier 运行。
  • 破坏性变更 / 迁移说明:无。锁文件、辅助脚本、5400 秒预算和错误签名都不变。集成通道上限从 200 分钟增至 310 分钟,以容纳约 3 小时的 verify 加一次等锁和 setup。
  • 设计文档:English · 简体中文

关联 issue

Fixes #15117

Run 39627609266's fast-tier job arrived at 17:38, stayed head of the
sdk-java host-lock queue, and failed at 19:15 without starting Maven.
The MariaDB integration lane held the same lock across its install and
full MySQL clean verify; under pool contention that hold ran from 15:54
to 18:57 (run 39628985638), past the fast tier's 90-minute budget.

Both MariaDB Maven runs already resolve from a job-local repository, so
they touch no host-shared state. Seed that repository before the lock,
run both Maven commands unlocked, and take the host lock only for the
cache write-back, the sole remaining host-shared mutation. Drop fd 9 on
the write-back children so a cancelled job's survivor cannot retain the
lock. Raise the lane ceiling to 300 minutes for the ~3-hour verify plus
one lock wait and setup, and pin the new order and fd closes.

Fixes #15117
@qwen-code-dev-bot

Copy link
Copy Markdown
Collaborator Author

E2E Test Report — issue #15117

Failure

Run 39627609266's SDK Java (fast) / ubuntu-latest / Java 21 job failed in Run Java SDK tests (self-hosted) after 1h 36m. The step never reached Maven: it waited the full 90-minute host-lock budget as queue ticket 93 and timed out with itself at the head of the queue. The job arrived at 17:38 and failed at 19:15; the MariaDB integration lane held the same per-host lock from 15:54 to 18:57 (run 39628985638) across its install and full MySQL verify.

The ticketed queue was behaving as designed — the head was blocked by a holder, not by queue order. The defect is the hold scope: a fast-tier waiter with a 90-minute budget cannot outlast a multi-hour integration hold.

Fix

Both MariaDB Maven runs already resolve from a job-local repository, so only the cache write-back mutates host-shared state. The lane now:

  • seeds the job-local repository before the lock (read-only, best-effort);
  • runs the SDK install and MySQL verify unlocked;
  • takes one host-lock acquisition only for the cache write-back;
  • closes fd 9 on the write-back's mkdir/tar children so a survivor cannot retain the lock;
  • raises the lane ceiling from 200 to 300 minutes to cover the ~3-hour verify plus one lock wait and setup.

The fast-tier and release fallback paths are unchanged: same helper, same 5400 s budget, same lock not acquired within signature.

Verification

  • npm run build — passed (rerun with COREPACK_HOME=/tmp/corepack-home after the sandbox's read-only corepack cache blocked the first attempt).
  • npm run typecheck — passed.
  • npm run lint — passed.
  • npx vitest run --config ./scripts/tests/vitest.config.ts sdk-java-workflow workflow-size sdk-java-lock — 3 files, 338 passed, 1 skipped.
  • npx vitest run --config ./scripts/tests/vitest.config.ts workflow-size managed-agent-gate-workflow hosted-process-ci main-ci-failure-issue-workflow managed-agent-server-e2e qwen-triage-workflow spotbugs-gate event-transport-broker-ci event-transport-multi-node-ci — 9 files, 686 passed, 1 skipped.
  • Mutation probes (temporary edits, reverted):
    • lock acquisition moved above Maven → holds the per-host sdk-java lock once around the cache write-back failed (1 failed / 95 passed); restored → green.
    • 9>&- removed from the write-back mkdir → keeps the host lock out of every locked child failed; restored → green.
    • 9>&- removed from the first write-back tar → the same test failed; restored → green.
  • Java suites — not run: no JDK/Maven in this sandbox. The change is workflow-only; CI's Java lanes are the final verification.
中文说明

E2E 测试报告 — issue #15117

失败

Run 39627609266 的 SDK Java (fast) / ubuntu-latest / Java 21 job 在 Run Java SDK tests (self-hosted) 步骤失败,耗时 1 小时 36 分。该步骤根本没走到 Maven:它以队列票据 93 的身份等满了 90 分钟的宿主机锁预算,超时时自己就是队首。job 于 17:38 到达、19:15 失败;MariaDB 集成通道从 15:54 到 18:57 一直持有同一把宿主机锁(run 39628985638),覆盖了它的 install 和完整 MySQL verify。

票据队列本身按设计工作 —— 队首是被持有者挡住,而不是被排队顺序挡住。缺陷在持锁范围:只有 90 分钟预算的 fast 通道等待者,不可能熬过一次数小时的集成持锁。

修复

MariaDB 通道的两次 Maven 运行本来就都从 job 本地仓库解析,因此只有缓存写回会修改宿主机共享状态。该通道现在:

  • 在取锁之前先把宿主机仓库种子复制到 job 本地仓库(只读、尽力而为);
  • 不持锁地运行 SDK install 和 MySQL verify;
  • 只对缓存写回取一次宿主机锁;
  • 让写回的 mkdir/tar 子进程关闭 fd 9,避免幸存进程继续持有锁;
  • 把该通道的上限从 200 分钟提高到 300 分钟,覆盖约 3 小时的 verify 加一次等锁和 setup。

fast 通道和发布回退路径不变:同一个辅助脚本、同一个 5400 秒预算、同一个 lock not acquired within 错误签名。

验证

  • npm run build — 通过(沙箱的 corepack 缓存只读,首次尝试被挡住后改用 COREPACK_HOME=/tmp/corepack-home 重跑通过)。
  • npm run typecheck — 通过。
  • npm run lint — 通过。
  • npx vitest run --config ./scripts/tests/vitest.config.ts sdk-java-workflow workflow-size sdk-java-lock — 3 个文件,338 通过,1 跳过。
  • npx vitest run --config ./scripts/tests/vitest.config.ts workflow-size managed-agent-gate-workflow hosted-process-ci main-ci-failure-issue-workflow managed-agent-server-e2e qwen-triage-workflow spotbugs-gate event-transport-broker-ci event-transport-multi-node-ci — 9 个文件,686 通过,1 跳过。
  • 变异探针(临时修改后还原):
    • 把取锁移到 Maven 之前 → holds the per-host sdk-java lock once around the cache write-back 失败(1 失败 / 95 通过);还原后通过。
    • 去掉写回 mkdir 的 9>&- → keeps the host lock out of every locked child 失败;还原后通过。
    • 去掉第一个写回 tar 的 9>&- → 同一个测试失败;还原后通过。
  • Java 测试套件 — 未运行:本沙箱没有 JDK/Maven。改动只涉及工作流;CI 的 Java 通道是最终验证。
--- 🧠 Handled by **Qwen Code** · model/模型 `deepseek-v4.1-flash-1017` · CLI `0.25.3`

@github-actions github-actions Bot added the review/self-reported The linked issue was opened by the PR author (self-reported) label Oct 21, 2026
@qwen-code-review-bot

qwen-code-review-bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator

⚠️ Deferred approval withheld — 1 PR CI workflow run(s) on 2769100 did not finish green; see the updated table in the Stage 2 comment. Re-run @qwen-code /triage after fixes. finalize run

⚠️ 延迟审批已搁置 —— 2769100 有 1 个 PR CI workflow 未以绿色完成,详见 Stage 2 评论中已更新的表格。修复后可重新运行 @qwen-code /triage。查看 finalize 运行

@qwen-code-review-bot

Copy link
Copy Markdown
Collaborator

Thanks for the PR!

Template looks good ✓

Problem: observed, not theoretical. Run 39627609266's fast-tier job arrived at 17:38 and died at 19:15 in Run Java SDK tests (self-hosted) — 97 minutes, which lines up with the 5400 s lock-wait budget plus setup — while run 39628985638 held the same sdk-java-tests.lock from 15:54 to 18:57. That is a 183-minute hold against a 90-minute wait, and it is tracked as a main CI failure in #15117. The diagnosis holds up against the workflow as it currently stands.

Direction: aligned. The fair-lock-queue design already rejected the obvious alternative — raising the budget — on the grounds that "a wait budget bounds one attempt, not the wait", so shortening the hold is the structurally correct move rather than another step up from 5400 s. It also lands on the criterion the rest of this file already uses for pool-safety: event-transport-broker and event-transport-multi-node both run mvn ... clean verify unlocked against a -Dmaven.repo.local under runner.temp, and their comments justify pool-safety by where the state lives, not by holding a lock. Bringing mysql-integration in line with that is consistency, not a new stance. No CHANGELOG signal to cite for a CI-only change, but the area is well documented in docs/design/.

Size: not applicable — no core paths are touched. 5 files, +136/−74, entirely CI workflow, design docs, and meta-tests; zero production logic lines.

Approach: the scope feels right and I could not find anything worth cutting. It updates the meta-test that parses the changed step, as the repo's workflow-edit rule requires — I checked the other eight files under scripts/tests/ that reference sdk-java.yml, and none of them pin the moved ordering or the old 200-minute ceiling (hosted-process-ci.test.js slices from -Pmysql-integration, which still sits before the write-back; the event-transport-* tests only assert runs-on parity). The .size-baseline bump is exact: 113617 is the real byte count of the workflow at this commit. Both design-doc languages are updated in step.

Risk: no elevated risk signals — nothing here matches the high-risk paths. The one thing a reviewer should look at deliberately is that this widens concurrency on the pool: the fast tier's Maven runs and the MariaDB lane's can now overlap on one host, where the lock used to make them mutually exclusive. I have written up in the code review why I do not think that blocks it.

Moving on to code review. 🔍

中文说明

感谢贡献!

模板完整 ✓

问题: 是已观测到的故障,不是理论性加固。run 39627609266 的 fast 通道 job 17:38 到达,19:15 在 Run Java SDK tests (self-hosted) 步骤失败 —— 97 分钟,正好对上 5400 秒等锁预算加 setup;同期 run 39628985638 从 15:54 到 18:57 持有同一把 sdk-java-tests.lock。183 分钟的持锁对上 90 分钟的等待预算,#15117 已把它作为 main CI 失败记录在案。对照当前工作流,这个诊断成立。

方向: 对齐。公平锁队列的设计文档已经否掉了最直接的替代方案 —— 提高预算 —— 理由是"等待预算只能约束一次尝试,约束不了排队本身",所以收窄持锁范围才是结构上正确的做法,而不是把 5400 秒再往上调一档。它也落在这个文件里既有的 pool-safety 判据上:event-transport-broker 和 event-transport-multi-node 本来就用 runner.temp 下的 -Dmaven.repo.local 不持锁跑 mvn ... clean verify,它们的注释是按"状态放在哪"而不是"有没有持锁"来论证 pool-safe 的。让 mysql-integration 与之一致,是保持一致,不是新立场。纯 CI 改动没有 CHANGELOG 信号可引,但 docs/design/ 里对这块有完整记录。

规模: 不适用 —— 未触及核心路径。5 个文件,+136/−74,全部是 CI 工作流、设计文档和元测试,生产逻辑行数为零。

方案: 范围合理,没找到可以砍的部分。按仓库对改动工作流的要求,它更新了会解析该步骤的元测试 —— 我核对了 scripts/tests/ 下另外八个引用 sdk-java.yml 的文件,没有一个 pin 住被移动的顺序或旧的 200 分钟上限(hosted-process-ci.test.js 是从 -Pmysql-integration 往后切片,写回仍在其后;event-transport-* 只断言 runs-on 一致)。.size-baseline 的调整是精确的:113617 正是该 commit 上工作流文件的真实字节数。两份设计文档语言同步更新。

风险: 无升级风险信号 —— 没有命中高风险路径。唯一需要评审者刻意看一眼的是它放宽了池上的并发:fast 通道的 Maven 运行和 MariaDB 通道的现在可以在同一台宿主机上重叠,而过去这把锁让它们互斥。我在代码审查里写了为什么我认为这不构成阻塞。

进入代码审查 🔍

— Qwen Code · qwen3.8-max-2026-09-02

Reviewed at 2769100e12280f96804c517ee5162c8562e519e2 · re-run with @qwen-code /triage

@qwen-code-review-bot

qwen-code-review-bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator

Code review

Before reading the diff I wrote down what I would do from the title and the "why" alone: the lock exists to protect host-shared state, so find the lane's only host-shared mutation and hold the lock across just that, leaving the Maven runs outside it. That is what this PR does, so I have no materially simpler alternative to offer. The checks below are the things I confirmed rather than assumed.

What holds up:

  • fd 9 really is the lock fd in both arms — exec 9>"${lock}" in the sourced helper and exec 9>"${HOME}/.cache/qwen-code-ci/sdk-java-tests.lock" in the bare-lock fallback — so the new 9>&- on mkdir and both tars closes the right descriptor, and those three are the only children spawned after the acquisition.
  • cd packages/managed-agent-server now sits above the lock acquisition, which looked wrong at a glance. It is inert: everything after the cd uses an absolute path (the helper path via ${GITHUB_WORKSPACE}, the lock file via ${HOME}, the write-back via ${HOME} and tar -C). The cd only matters for the relative -f packages/sdk-java/qwencode/pom.xml, and the meta-test still pins it between the two Maven runs.
  • Keeping the write-back locked is genuinely necessary, not just conservative. The sibling test job has no -Dmaven.repo.local in its MAVEN_ARGS, so it reads and writes the host-shared ~/.m2/repository directly under the lock. The write-back is the only step in this lane that mutates that same directory, and the PR leaves it inside the lock.
  • The concurrency this introduces is not a new stance for this file. event-transport-broker and event-transport-multi-node already do exactly this on the same pool — cp -aln seed, then mvn ... clean verify unlocked against -Dmaven.repo.local=${{ runner.temp }}/m2-repo, with no lock anywhere in either job. Their pool-safety comments argue from where the state lives, which is the same argument this PR now makes for mysql-integration. The lock never serialized this lane against those two anyway, only against the test job.

Non-blocking observations:

  1. A lock timeout now fails the job after a passing verify. The write-back is deliberately best-effort, and its own comment says the warning arm "keeps a copy failure from reddening the gate after mvn has already passed". Acquiring the lock for it is still fatal, though: the helper returns 1 and the step has no shell: key, so Actions' Linux default (bash -eo pipefail) aborts it, and the fallback arm does an explicit exit 1. Previously that abort cost nothing because it happened before Maven; now it discards up to three hours of green clean verify. Two things soften it — the lock not acquired within signature is preserved, so qwen-autofix.yml still classifies it as infra and reruns, and with every hold now short a 90-minute wait for a tar copy should be close to unreachable. But if you want the stated invariant to actually hold, skipping the write-back with a warning on acquisition failure would do it. Skipping, not proceeding unlocked — the write-back does mutate ~/.m2, which is precisely what the lock guards.
  2. The 300-minute ceiling passes its own assertion at exact equality. timeout-minutes * 60 >= 5400 + 3*3600 + 30*60 is 18000 >= 18000. The previous comment carried "plus 20 minutes of margin"; the new one drops the margin concept entirely. And the one measured data point in the description — 15:54 to 18:57 — is 183 minutes, not the 180 the model uses. In practice this should be fine, because the whole point is that the 90-minute wait no longer occurs; but the arithmetic only balances if the wait really does collapse. Worth either saying that explicitly in the comment or leaving a few minutes of headroom.
  3. Minor doc inconsistency: the design doc's non-goal now reads "changing the 5400 s budget or the 150/300-minute job ceilings" in the same document whose new section raises a ceiling from 200 to 300. The number is now accurate against the workflow; the "non-goal" framing is what reads oddly next to the addendum.

House rules: the workflow-edit rule is satisfied — I checked all nine scripts/tests/ files that reference sdk-java.yml, and only sdk-java-workflow.test.js pins anything this diff moves. .size-baseline records 113617, which is the exact byte count of the workflow at this commit (verified through the raw contents API). English and Chinese design docs are both updated, with matching structure and content.

Testing

This run carries the PR's own CI check results, read through the API — no PR code was built or executed here.

The load-bearing fact: this PR's CI cannot exercise the step it changes. mysql-integration is gated if: "${{ inputs.tier != 'fast' }}", and the tier selector chose fast, so SDK Java (fast) / Runtime Broker and Managed Agent MySQL / Java 21 is skipped. The reordered seed → install → verify → lock → write-back block never runs. The only coverage the change gets is the static workflow pins in scripts/tests/sdk-java-workflow.test.js, which execute inside Test (ubuntu-latest, Node 22.x) — still in progress at the time of writing.

Nothing is red. There are no failures to excerpt; 8 checks are green, 45 skipped (the tier-gated lanes plus the profile-gated bot jobs), and 7 still running. The author's "338 passed, 1 skipped" is their own local claim, not something I re-ran. Not verified: the Java suites and any real ECS host behaviour — the author says so too, and no JDK or Maven is available in this environment.

Final CI results for 2769100 (auto-updated by the triage finalize job after CI completed):

Check Conclusion
web-shell E2E Smoke (ubuntu-latest, Node 22.x) ❌ failure
Classify PR ✅ success
Integration Tests (no-AK, No Sandbox) ✅ success
Lint & Static (ubuntu-latest, Node 22.x) ✅ success
Managed Agent fast tier ✅ success
SDK Java (fast) / Flyway migration version uniqueness ✅ success
SDK Java (fast) / Real daemon E2E / Java 11 ✅ success
SDK Java (fast) / ubuntu-latest / Java 21 ✅ success
Select the Managed Agent tier ✅ success
Test (ubuntu-latest, Node 22.x) ✅ success

One row per check name (latest run); skipped checks omitted; failures sort first. / 每个检查名一行(取最新一次运行),省略 skipped,失败项排在最前。

Sandboxed verification would settle part of this: @qwen-code /verify — whether the new ordering and fd-9 assertions are actually load-bearing is not observable from the diff, and a green Test lane does not prove the pins bite. The mutation check the description proposes (move the lock above Maven, or drop a 9>&- from a write-back child, and confirm the pins go red) is exactly an A/B pin test, and it is the part that can be settled without a pool. The starvation fix itself cannot be: it needs a full-tier run on the real ECS pool, which is why the design doc already names post-merge CI as the soak.

Real-scenario tmux testing: N/A — this is an unattended CI run, and a workflow-file change has no user-visible surface to drive in a terminal.

中文说明

代码审查

在读 diff 之前,我先只看标题和"为什么需要"写下了自己的做法:这把锁是为了保护宿主机共享状态,那就找出该通道唯一会修改共享状态的动作,只用锁包住它,把 Maven 运行放在锁外。这个 PR 做的正是这件事,所以我没有更简单的替代方案可提。下面这些是我实际核对过、而不是默认成立的点。

站得住的部分:

  • fd 9 在两个分支里确实都是锁的描述符 —— 辅助脚本里是 exec 9>"${lock}",裸锁回退分支里是 exec 9>"${HOME}/.cache/qwen-code-ci/sdk-java-tests.lock" —— 所以新加在 mkdir 和两个 tar 上的 9>&- 关的是对的描述符,而且取锁之后只派生这三个子进程。
  • cd packages/managed-agent-server 现在位于取锁之前,乍看像是错的。实际无影响:cd 之后的一切都用绝对路径(辅助脚本走 ${GITHUB_WORKSPACE},锁文件走 ${HOME},写回走 ${HOME} 和 tar -C)。这个 cd 只对相对路径 -f packages/sdk-java/qwencode/pom.xml 有意义,而元测试仍然 pin 住它位于两次 Maven 运行之间。
  • 写回继续持锁是确实必要的,不只是保守。同文件的 test job 的 MAVEN_ARGS 里没有 -Dmaven.repo.local,所以它是在锁内直接读写宿主机共享的 ~/.m2/repository。写回是本通道唯一修改同一目录的动作,PR 把它留在了锁内。
  • 这里引入的并发对这个文件来说不是新立场。event-transport-broker 和 event-transport-multi-node 在同一个池上本来就是这么做的 —— cp -aln 种子复制,然后不持锁跑 mvn ... clean verify,用 -Dmaven.repo.local=${{ runner.temp }}/m2-repo,两个 job 里完全没有锁。它们的 pool-safety 注释是按"状态放在哪"论证的,和本 PR 现在为 mysql-integration 给出的论证一致。这把锁本来也从未让本通道与那两个通道互斥,只让它与 test job 互斥。

不阻塞的观察:

  1. 等锁超时现在会在 verify 已经通过之后才让 job 失败。 写回是刻意设计成尽力而为的,它自己的注释就写着警告分支"让复制失败不会在 mvn 已经通过之后把门禁弄红"。但为它取锁这一步仍然是致命的:辅助脚本返回 1,而这个 step 没有 shell: 键,所以用的是 Actions 在 Linux 上的默认值(bash -eo pipefail),会直接中止;回退分支里也有显式的 exit 1。以前这个中止不损失什么,因为它发生在 Maven 之前;现在它会丢掉最多三小时已经跑绿的 clean verify。有两点缓解 —— lock not acquired within 签名保留了,所以 qwen-autofix.yml 仍会把它归类为基础设施失败并重跑;而且在所有持锁都变短之后,为一次 tar 复制等满 90 分钟应该几乎不可能。但如果想让上面那条不变量真正成立,可以在取锁失败时跳过写回并给出警告。是跳过,不是不持锁继续 —— 写回确实会修改 ~/.m2,而那正是这把锁要保护的东西。
  2. 300 分钟上限恰好卡在自己的断言边界上。 timeout-minutes * 60 >= 5400 + 3*3600 + 30*60 就是 18000 >= 18000。原来的注释里有"再加 20 分钟余量",新注释把余量这个概念整个去掉了。而描述里唯一的实测数据点 —— 15:54 到 18:57 —— 是 183 分钟,不是模型用的 180。实际上应该没问题,因为这次改动的全部意义就是让 90 分钟等待不再发生;但这个算术能平衡的前提是等待确实塌缩了。建议在注释里把这一点写明,或者留几分钟余量。
  3. 文档小不一致: 设计文档的非目标现在写着"不修改 5400 秒预算或 150/300 分钟的 job 上限",而同一份文档新增的章节把某个上限从 200 提到了 300。数字现在与工作流一致了;读起来别扭的是"非目标"这个说法与新增章节并列。

仓库规则: 改动工作流的要求已满足 —— 我核对了 scripts/tests/ 下全部九个引用 sdk-java.yml 的文件,只有 sdk-java-workflow.test.js pin 住了本 diff 移动的内容。.size-baseline 记录的是 113617,正好是该 commit 上工作流文件的字节数(通过 raw contents API 核对)。中英文设计文档都已更新,结构与内容一致。

测试

本次运行携带的是该 PR 自己的 CI 检查结果,通过 API 读取 —— 这里没有构建或执行任何 PR 代码。

最关键的一点:这个 PR 的 CI 无法执行它改动的那个步骤。 mysql-integration 的条件是 if: "${{ inputs.tier != 'fast' }}",而 tier 选择器选出的是 fast,所以 SDK Java (fast) / Runtime Broker and Managed Agent MySQL / Java 21 是 skipped。重排后的"种子复制 → install → verify → 取锁 → 写回"这一段一次都没有运行。该改动能得到的唯一覆盖,是 scripts/tests/sdk-java-workflow.test.js 里的静态工作流 pin,它们在 Test (ubuntu-latest, Node 22.x) 里执行 —— 写这段时仍在运行中。

没有红色项。没有失败日志可摘:8 个检查通过,45 个跳过(tier 门控的通道加上 profile 门控的机器人 job),7 个仍在运行。作者所说的"338 通过,1 跳过"是其本地声明,不是我重跑的结果。未验证:Java 套件以及任何真实 ECS 宿主机上的行为 —— 作者也这么说明,本环境没有 JDK 和 Maven。

(CI 表格见上方区域标记内。)

沙箱验证可以解决其中一部分:@qwen-code /verify —— 新加的顺序断言和 fd 9 断言是否真的有约束力,从 diff 上看不出来,Test 通道跑绿也不能证明这些 pin 会咬住。描述里提出的变异检查(把锁移到 Maven 之前,或去掉写回子进程的一个 9>&-,确认 pin 转红)正是一个 A/B pin 测试,而且这是不需要资源池就能验证的部分。饥饿修复本身则不行:它需要在真实 ECS 池上跑一次完整 tier,这也是设计文档已经把合并后的 CI 指定为浸泡验证的原因。

真实场景 tmux 测试:N/A —— 这是无人值守的 CI 运行,而且工作流文件改动没有可在终端里驱动的用户可见界面。

— Qwen Code · qwen3.8-max-2026-09-02

Reviewed at 2769100e12280f96804c517ee5162c8562e519e2 · re-run with @qwen-code /triage

@qwen-code-review-bot

Copy link
Copy Markdown
Collaborator

Confidence: 4/5 — the diagnosis is arithmetically evidenced and the fix is the structurally right one; the two reservations below are real but neither blocks.

Stepping back: this PR found the actual reason the lock existed in that lane and then removed only the part that was not doing work. The mysql-integration job's MAVEN_ARGS already carried -Dmaven.repo.local=${{ runner.temp }}/m2-repo, so its Maven runs were never touching host-shared state — the lock was spanning three hours of work to protect a tar copy that takes seconds. Once you see that, narrowing the hold is the obvious move, and it is what I would have proposed from the title alone.

What makes me comfortable approving an infrastructure change I cannot run: the two run IDs in the description reconcile. 15:54→18:57 is a 183-minute hold; 17:38→19:15 is 97 minutes, which is the 5400 s wait budget plus setup. Those are not round numbers chosen to fit a story, and #15117 independently records the same job failing in the same step. The repo's own design doc had already ruled out the lazier fix — raising the budget — with "a wait budget bounds one attempt, not the wait", so this is the second-order fix rather than another nudge upward. And the concurrency it introduces is not new to this file: two sibling lanes already run unlocked mvn clean verify against job-local repos on the same pool, with pool-safety argued from where the state lives.

My two reservations, neither blocking:

  • The 300-minute ceiling balances only if the wait really does collapse. Its assertion passes at exact equality (18000 >= 18000), the old "20 minutes of margin" is gone, and the modelled Maven time (180 min) is under the one measured data point (183 min). If holds stay short — which is the entire thesis — this never bites. If they do not, the failure mode changes from "fast tier times out at the lock" to "this lane times out at 300 minutes having done the work", which is a worse place to fail. Worth the maintainer knowing that the headroom is exactly zero.
  • A lock timeout now discards a green verify. The write-back is best-effort by design and says so in its own comment, but failing to acquire the lock for it is still fatal, and it is now fatal after up to three hours of passing Maven. The infra signature is preserved so the rerun classifier still catches it, and the path should be near-unreachable now — but skipping the write-back with a warning would make the stated invariant actually true.

Would I curse the author in six months? No. The comments carry the run IDs that justify the ordering, the meta-tests pin the order and the fd-9 closes so a future edit that moves the lock back above Maven goes red, and both design-doc languages were updated in the same commit. That is better maintained than most CI changes here.

One thing the maintainer should weigh that I cannot settle: this PR's own CI never runs the changed step. The tier selector picked fast, and mysql-integration is gated to non-fast, so the reordered block is skipped and only the static workflow pins cover it. That is inherent to a full-tier lane — it cannot be exercised from a PR — and the design doc already names post-merge CI as the soak. I am not holding it against the PR, but the first full-tier pool run after merge is the real test, and it is worth watching.

CI is still running (Managed Agent gate, Qwen Code CI), so I am not posting an approval in this run — approval deferred until CI lands green on 2769100e12280f96804c517ee5162c8562e519e2.

中文说明

信心:4/5 —— 诊断有可核对的算术证据,修复也是结构上正确的那个;下面两条保留意见是真实的,但都不构成阻塞。

退一步看:这个 PR 找到了这把锁在该通道里存在的真正理由,然后只去掉了没在起作用的那部分。mysql-integration job 的 MAVEN_ARGS 本来就带着 -Dmaven.repo.local=${{ runner.temp }}/m2-repo,所以它的 Maven 运行从来没有触碰宿主机共享状态 —— 这把锁横跨三小时的工作,保护的只是一次几秒钟的 tar 复制。看清这一点之后,收窄持锁范围就是显而易见的做法,也正是我只看标题就会提出的方案。

让我能安心通过一个自己无法运行的基础设施改动的是:描述里的两个 run ID 能对上。15:54→18:57 是 183 分钟的持锁;17:38→19:15 是 97 分钟,正好是 5400 秒等待预算加 setup。这些不是为凑故事挑的整数,而 #15117 也独立记录了同一个 job 在同一个步骤失败。仓库自己的设计文档早就排除了那个更省事的修法 —— 提高预算 —— 理由是"等待预算只能约束一次尝试,约束不了排队本身",所以这是二阶修复,而不是把数字再往上调一档。它引入的并发对这个文件也不新:两个同级通道本来就在同一个池上不持锁跑 mvn clean verify,用 job 本地仓库,pool-safety 是按"状态放在哪"论证的。

我的两条保留意见,都不阻塞:

  • 300 分钟上限能平衡的前提是等待确实塌缩了。 它的断言恰好相等通过(18000 >= 18000),原来的"20 分钟余量"没有了,而模型用的 Maven 时间(180 分钟)低于唯一的实测数据点(183 分钟)。如果持锁都很短 —— 这正是全部论点 —— 它永远不会被触发。如果并非如此,失败模式就从"fast 通道在锁上超时"变成"本通道在 300 分钟超时,而活已经干完了",那是更糟的失败位置。维护者应该知道这里的余量正好是零。
  • 等锁超时现在会丢掉一次已经跑绿的 verify。 写回按设计是尽力而为的,它自己的注释也这么写;但为它取锁失败仍然是致命的,而且现在是在最多三小时 Maven 通过之后才致命。基础设施错误签名保留了,所以重跑分类器仍能识别,而且这条路径现在应该几乎不可达 —— 但在取锁失败时跳过写回并给出警告,才能让上面那条不变量真正成立。

六个月后我会骂作者吗?不会。注释里带着证明这个顺序合理的 run ID,元测试 pin 住了顺序和 fd 9 的关闭,所以将来谁把锁移回 Maven 之前会直接跑红,而且两种语言的设计文档是在同一个 commit 里更新的。这比这里大多数 CI 改动维护得更好。

有一件我无法定论、需要维护者权衡的事:这个 PR 自己的 CI 从未运行被改动的步骤。 tier 选择器选了 fast,而 mysql-integration 门控为非 fast,所以重排后的那一段被跳过,只有静态工作流 pin 覆盖它。这对一个完整 tier 的通道来说是固有情况 —— 从 PR 里无法执行它 —— 设计文档也已经把合并后的 CI 指定为浸泡验证。我不因此扣分,但合并后第一次完整 tier 的池上运行才是真正的检验,值得关注。

CI 仍在运行(Managed Agent gate、Qwen Code CI),所以本次运行不提交批准 —— 批准推迟到 CI 在 2769100e12280f96804c517ee5162c8562e519e2 上全绿之后。

— Qwen Code · qwen3.8-max-2026-09-02

Reviewed at 2769100e12280f96804c517ee5162c8562e519e2 · re-run with @qwen-code /triage

@github-actions

Copy link
Copy Markdown
Contributor

Managed Agent gate failed

Run 39676626070 · commit 2769100e1228

No test identifier was available; follow the run link for job logs and annotations.
GitHub runner infrastructure failed; this result does not identify a Java test regression.

@qwen-code-dev-bot

Copy link
Copy Markdown
Collaborator Author

🔀 Base updated: red check(s) [web-shell E2E Smoke (ubuntu-latest, Node 22.x)] pass on current main — merged current main via update-branch; CI will re-run.

中文说明

🔀 已更新 base:红色检查 [web-shell E2E Smoke (ubuntu-latest, Node 22.x)] 在当前 main 上通过 —— 已通过 update-branch 合入当前 main,CI 将重新运行。

chiga0
chiga0 previously approved these changes Oct 22, 2026

@chiga0 chiga0 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tier: Standard (elevated for concurrency/fd-lifecycle concern).

No blocking findings.
Approval blockers: none.


Checked:

  • Lock scope and fd 9 management (sdk-java.yml, mysql-integration step): Maven install and verify run before any lock acquisition in both the primary path (source sdk-java-lock.sh) and the fallback (exec 9>). The write-back children (mkdir, both tar sides) carry 9>&-, so a cancelled job's survivor cannot retain the host lock. Ordering is correctly mirrored in the test's indexOf assertions.
  • Unlocked seed safety (cp -aln "${HOME}/.m2/repository" "${RUNNER_TEMP}/m2-repo" 2>/dev/null || true): read-only hard-link copy; any race with a concurrent write-back produces a cache miss at worst, not corrupted state. Best-effort is documented and || true-guarded.
  • Timeout 200 → 300 min: absorbs one full 90-min lock wait (5400 s) + ~3-hour MySQL verify + ~30 min setup — consistent with the cited run timestamps.
  • Test updates (scripts/tests/sdk-java-workflow.test.js): lockAt derived from the source line; installAt and verifyAt asserted < lockAt; writeBackAt asserted > lockAt; mkdir 9>&- and tar … 9>&- | tar … 9>&- pattern assertions present; retains serialization test scoped to jobs.test only, consistent with the behavioral change.
  • Size baseline (.size-baseline): 109610 → 113617, consistent with the net +62 lines in the workflow.
  • Design doc sync (EN + 中文): non-goal ceiling updated 280 → 300 min; new "Hold scope" section matches the code. No EN/中文 divergence found.
  • Cross-check: no other reviewer findings at this head.

Unreviewed dimensions:

  • Working tree unavailable; npx vitest run --config ./scripts/tests/vitest.config.ts sdk-java-workflow workflow-size sdk-java-lock not executed locally. PR states 338 passed, 1 skipped — no execution witness from this review.
  • Java/Maven not available locally; real ECS host behaviour not exercised. Both acknowledged in the PR's test plan.

Reviewed with AI assistance.

@qwen-code-dev-bot

qwen-code-dev-bot commented Oct 22, 2026 •

Copy link
Copy Markdown
Collaborator Author

✅ AutoFix round 2 finished — view run. See this round's report below.

中文说明

✅ AutoFix 第 2 轮已完成 —— 查看运行。本轮报告见下方。

The main merge brought three interactive suites from #14964. The no-AK gate requires every runnable integration test to be selected or explicitly exempted, so the Test job fails until they are listed.

Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
@qwen-code-dev-bot

Copy link
Copy Markdown
Collaborator Author

🤖 Addressed the latest review feedback (round 1/10). What changed, and what I pushed back on: · 已处理最新评审反馈(第 1/10 轮)。改动内容与我反驳保留之处如下:

Autofix round summary — PR #15128

Feedback triaged

No review bodies, inline comments, or issue-level comments were newer than the evaluation cutoff. Two failed checks were listed:

  • Test (ubuntu-latest, Node 22.x)
  • web-shell E2E Smoke (ubuntu-latest, Node 22.x)

Test failure — fixed

scripts/tests/no-ak-integration-ci.test.js fails on the merged tree: the three interactive suites interactive/statusline-parity.test.ts, interactive/tui-editor-parity.test.ts, and interactive/tui-input-ownership.test.ts are runnable candidates, but the no-AK gate's explicit exemption table does not list them. Those files came from main's #14964; origin/main has the same stale table, so the failure is pre-existing rather than introduced by this PR, but it reddens the merged tree's Test job.

Fix: add the three entries to the exemption table with the same Runtime budget reason the neighbouring interactive suites use (3 lines).

web-shell E2E Smoke — not locally reproducible

The PR touches only the sdk-java workflow, its size baseline, its design docs, and its workflow test; no web-shell runtime or test path changed. The browser suite needs Playwright Chromium/WebKit plus system libraries, which are unavailable in this sandbox, so it cannot be reproduced locally. The job will re-run on the next CI pass; no code change was made for it.

Verification

  • npm run build — passed.
  • npm run typecheck — passed.
  • npm run lint — passed.
  • npx vitest run --config ./scripts/tests/vitest.config.ts scripts/tests/no-ak-integration-ci.test.js — 16/16 passed after the fix; the same file failed 1/16 before it.
  • npx vitest run --config ./scripts/tests/vitest.config.ts scripts/tests/sdk-java-workflow.test.js — 96/96 passed.
  • Full scripts Vitest suite — passed except sandbox-only failures: npm pack steps fail with EACCES on the read-only /home/github-runner/.npm cache, and the TTY-colour test cannot preserve bold in this non-interactive sandbox. no-ak-integration-ci.test.js no longer appears among the failures.
  • Mutation probe — deleting the statusline-parity exemption fails the guard (1 failed / 15 passed); restoring it passes (16/16).
  • npm run test:ci:workspaces — completed with sandbox-only failures (read-only HOME/~/.qwen, Docker detected by the footer fixture, npm pack cache EACCES, and parallel-load timeouts); the failing main-boot tests pass in isolation.
  • npm run test:scripts — its standalone-runtime prefetch could not run here because the registry returned 502 for the Bun download; the Vitest scripts suite was run directly instead.

Changed files

  • scripts/tests/no-ak-integration-ci.test.js (+3)
中文说明

Autofix 轮次总结 — PR #15128

已分诊的反馈

评估截止时间之后没有新的评审正文、行内评论或 issue 级评论。列出了两个失败的检查:

  • Test (ubuntu-latest, Node 22.x)
  • web-shell E2E Smoke (ubuntu-latest, Node 22.x)

Test 失败 — 已修复

scripts/tests/no-ak-integration-ci.test.js 在合并后的树上失败:三个交互式套件 interactive/statusline-parity.test.ts、interactive/tui-editor-parity.test.ts 和 interactive/tui-input-ownership.test.ts 是可运行的候选测试,但 no-AK 门禁的显式豁免表没有列出它们。这些文件来自 main 的 #14964;origin/main 上的豁免表同样陈旧,因此这是既有失败,而不是本 PR 引入的,但它会让合并后树的 Test 任务变红。

修复:把这三个条目加入豁免表,并使用与相邻交互式套件相同的 Runtime budget 理由(共 3 行)。

web-shell E2E Smoke — 本地无法复现

本 PR 只改动了 sdk-java 工作流、其大小基线、其设计文档和工作流测试;没有改动任何 web-shell 运行时或测试路径。该浏览器套件需要 Playwright Chromium/WebKit 以及系统库,而此沙箱中不可用,因此无法在本地复现。该任务会在下一次 CI 运行时重跑;没有为它做代码改动。

验证

  • npm run build — 通过。
  • npm run typecheck — 通过。
  • npm run lint — 通过。
  • npx vitest run --config ./scripts/tests/vitest.config.ts scripts/tests/no-ak-integration-ci.test.js — 修复后 16/16 通过;修复前同一文件为 1/16 失败。
  • npx vitest run --config ./scripts/tests/vitest.config.ts scripts/tests/sdk-java-workflow.test.js — 96/96 通过。
  • 完整 scripts Vitest 套件 — 除沙箱特有的失败外通过:npm pack 步骤因只读的 /home/github-runner/.npm 缓存报 EACCES,TTY 颜色测试在非交互式沙箱中无法保留粗体。no-ak-integration-ci.test.js 已不再出现在失败列表中。
  • 变异探针 — 删除 statusline-parity 豁免会让门禁失败(1 失败 / 15 通过);恢复后通过(16/16)。
  • npm run test:ci:workspaces — 已完成,仅出现沙箱特有的失败(只读 HOME/~/.qwen、页脚 fixture 检测到 Docker、npm pack 缓存 EACCES、并行负载超时);失败的 main-boot 测试在单独运行时通过。
  • npm run test:scripts — 其 standalone 运行时预取在此处无法运行,因为注册表下载 Bun 返回 502;改为直接运行 Vitest scripts 套件。

改动文件

  • scripts/tests/no-ak-integration-ci.test.js(+3)
Base-conflict check · 基分支冲突检查: no conflict with main. · 与 main 无冲突。

Re-review when you have a moment. After round 10 this bot stops and leaves the PR for a human. · 有空请复审;第 10 轮后本 bot 停止并将 PR 交给人工。


🧠 Handled by Qwen Code · model/模型 deepseek-v4.1-flash-1017 · CLI 0.25.3

@github-actions

Copy link
Copy Markdown
Contributor

Managed Agent gate failed

Run 39716652687 · commit 3e0d7e16462f

No test identifier was available; follow the run link for job logs and annotations.

@yiliang114

Copy link
Copy Markdown
Collaborator

Consolidating the sdk-java host-lock repair family into this PR. The rest are being closed:

One salvage from the closed set: #15121's behavioural surviving-child harness — a runtime test that spawns a child outliving its step and asserts the lock actually releases — was called load-bearing in review and pins something the string-match guards cannot. Worth carrying into this PR or a follow-up so the write-back section's 9>&- closures get the same runtime pin.

Not urgent: the Managed Agent gate on main is green again as of ~03:54Z today. One design assumption worth watching before merge: with the lock gone from the ~3h Maven runs, multiple integration lanes may execute concurrently per pool host — sound in principle via per-job containers on random ports plus job-local ~/.m2, but host-level contention (fs.aio-nr, CPU) is unmeasured. The 300-minute ceiling gives headroom in the meantime.

qwen-code-dev-bot pushed a commit that referenced this pull request Oct 22, 2026
The identical fix is already on main (cfd95f6). Resolving all four conflicting files in favor of main's reviewed version leaves no functional or documentation delta on this branch; the PR is a duplicate of #15128 and should be closed rather than merged.

Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
qwen-code-dev-bot added a commit that referenced this pull request Oct 23, 2026
…15243)

The bare-lock fallbacks hand-write their timeout line, so raising
`flock --wait` could leave `160 minutes`/`90 minutes` stale with every
test green; pin both literals. The sizing comments also cited a
pre-#15128 run as evidence for a sibling-hold chain and repeated an
uncited 925 s hold; frame that run as the history it is, state that
9600 s is headroom sized from the hold length rather than an observed
chain, and keep the derivation in the design doc's Goals.
wenshao pushed a commit that referenced this pull request Oct 24, 2026
#15257)

* fix(ci): raise the sdk-java test-lane lock wait above the longest hold (#15243)

Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>

* docs(ci): restate sdk-java lock wait sizing and lane budgets (#15243)

* fix(ci): pin the sdk-java lock fallback minutes and restate sizing (#15243)

The bare-lock fallbacks hand-write their timeout line, so raising
`flock --wait` could leave `160 minutes`/`90 minutes` stale with every
test green; pin both literals. The sizing comments also cited a
pre-#15128 run as evidence for a sibling-hold chain and repeated an
uncited 925 s hold; frame that run as the history it is, state that
9600 s is headroom sized from the hold length rather than an observed
chain, and keep the derivation in the design doc's Goals.

---------

Co-authored-by: Qwen Autofix <qwen-code-assistant@users.noreply.github.com>
Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
wenshao pushed a commit that referenced this pull request Oct 24, 2026
* fix(ci): skip stale main CI failure issues after main advances (#15259)

Run 39777472391's push run on 06e5966 failed on the sdk-java host-lock
timeout that #15128 had already fixed. The failure-issue analyzer then ran
after main had advanced to 3d65e23, whose own push run passed, and filed
#15259 asking for a repair main already contained.

Skip a push run whose head is no longer the tip of main: the newer push run
owns that result, exactly as a later attempt does. The tip read fails open,
and scheduled/manual runs keep reporting because they are the only signal
some lanes have.

* test(ci): pin the empty main-tip fail-open path (#15259)

Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>

* fix(ci): require a successor run before suppressing stale main failures (#15259)

Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>

* fix(ci): cover retried runs in the stale-head guard (#15259)

Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>

* fix(ci): suppress only successor runs that can report (#15259)

---------

Co-authored-by: Qwen Autofix <qwen-autofix@users.noreply.github.com>
Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
Co-authored-by: qwen-code-dev-bot <qwen-code-dev-bot@users.noreply.github.com>
@qwen-code-review-bot

Copy link
Copy Markdown
Collaborator

Released in v0.26.1.

wenshao pushed a commit that referenced this pull request Oct 25, 2026
* fix(ci): run the sdk-java suite before taking the host lock (#15242)

Run 39771949573's fast-tier job spent both attempts' full 5400 s lock
budget waiting: the full-tier Java 21 leg of run 39754414693 ran for 89
minutes on the same host, and the fast job failed five seconds after that
leg ended. The MariaDB lane's #15128 fix scoped the lock to its cache
write-back, but the test job still held the host lock across the SDK and
Managed Agent Maven suites.

Seed the test job's Maven repository from the host cache, run its suite
from that job-local copy, and take the ticketed host lock only for the
cache write-back. The write-back children close fd 9 so a cancelled job's
survivor cannot retain the lock, and the workflow pins move with the new
order.

* fix(ci): keep the SDK Java gate green when the write-back lock expires (#15242)

The write-back lock acquisition could redden a job whose suite had already passed (CI run 39814549427). Treat it like the best-effort copy it guards: warn and skip on timeout in both the helper and fallback arms, while leaving the helper's pinned lock-not-acquired signature alone.

Tighten the workflow pins: assert the region below the lock instead of first-occurrence Maven indices, pin both write-back copies byte-equal, bind the job-local MAVEN_ARGS to its step, and include the write-back bound in the test job's ceiling model. Restate the hold-length derivations and record the concurrency blast radius in both design languages.

Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>

* fix(ci): keep the lock helper outside this PR's footprint (#15242)

Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>

* fix(ci): describe the sdk-java 89-minute leg as the leg, not the suite (#15242)

Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>

* fix(ci): keep tolerated sdk-java lock timeouts out of rerun classification (#15242)

The warn-and-skip downgrade left the helper's `lock not acquired within`
annotation on a step that now exits 0, so a later genuine failure in the
same job could be classified as infrastructure. Both call sites rewrite
the timeout line before it reaches the log, and probe flock plus the cache
directory first: bash suppresses errexit for a function on the left of
`||`, so without the probes a setup fault spun out the full wait green.

The skip test now plants a publishable delta, asserts nothing was
published, checks the captured annotations against the live
INFRA_FAILURE_SIGNATURES list, and adds a setup-fault arm. The workflow
size baseline is re-measured, and the design docs' classifier, promotion,
and budget claims are corrected in both languages.

---------

Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
Co-authored-by: Qwen Autofix <qwen-autofix@users.noreply.github.com>
wenshao pushed a commit that referenced this pull request Oct 25, 2026
)

* fix(ci): skip main-failure issues for post-merge reruns (#15239)

A rerun re-executes the PR's pre-merge workflow. When it starts after the
PR merged, its failure describes a head that main has already replaced,
and the post-merge push run owns that signal. Compare the attempt's start
time with the merge time and skip filing for the stale head.

Run 39768204855's attempt 2 started after PR #15094 merged, hit the
pre-#15128 suite-wide host-lock hold, and filed #15239.

* fix(ci): discriminate post-merge reruns by attempt (#15239)

The clock proxy suppressed fresh manual dispatches and post-merge
approval runs while missing reruns dispatched before the merge. Use
run_attempt, which states rerun identity directly: attempt 1 still
files, later attempts on a merged PR keep the comment but open no
issue. Announce the empty-attempt fail-open path.

Preserve an existing tracking-issue link when a suppressed attempt
updates the PR comment, and sync the design docs and the contract
comments with the narrower scope.

* fix(ci): pin rerun guard scope and tracking-link recovery (#15239)

Drop the pre-loop body write that no publish path could read, and cover the
publish-retry truncate with a row that fails the first publish. Pin the
writer's tracking line against the recovery grep's anchored pattern, and pin
that the attempt guard never suppresses a main run. Scope the design doc's
exclusivity sentence to the infrastructure path in both languages.

The workflow size ratchet records the file's new size; the growth is the
attempt guard and tracking-link recovery this PR adds.

Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>

* test(ci): pin tracking-link precedence for PR failure comment (#15239)

Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>

* test(ci): pin tracking-link wiring and POST coverage (#15239)

* fix(ci): file dispatch reruns on merged PR branches (#15239)

* test(ci): pin merged-PR rerun event and lookup-failure publish guard (#15239)

Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>

* test(ci): pin approval-run suppression and merged-PR comment guard (#15239)

Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>

---------

Co-authored-by: qwen-code-dev-bot <qwen-code-dev-bot@users.noreply.github.com>
Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
yiliang114 added a commit that referenced this pull request Nov 7, 2026
#15274)

* fix(ci): scope the sdk-java test lock to the cache write-back (#15262)

The Managed Agent fast tier can fail without reporting a test result after waiting out the per-host SDK Java lock. #15128 scoped the MariaDB lane's lock to its cache write-back, but the test job still held the same lock across the SDK and Managed Agent Maven suites, so a long full-tier suite could starve the fast tier on that host.

Seed the test job's Maven repository from the host cache, run both suites unlocked from that job-local copy, and take the ticketed host lock only for the cache write-back. The write-back children close fd 9 so a cancelled job's survivor cannot retain the lock. Pin the new order and the write-back fd closures in the workflow tests, and update the fair-lock design docs and the workflow size baseline.

Fixes #15262

Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>

* fix(ci): keep sdk-java cache write-back best-effort (#15262)

* fix(ci): keep the sdk-java gate result across the cache write-back (#15262)

The write-back runs after the suite, so the step's exit code must come from
the suite: both lanes now track the Maven status, demote only a lock-wait
timeout with a step-summary record, and redden when the queue itself is
unusable. The wait drops to 1800 s because it guards only the bounded copy;
the seed stops hiding cp's diagnosis while staying quiet on a fresh host;
the design docs record the head-of-queue stall the shorter hold does not
bound.

* fix(ci): revert the sdk-java lock helper to the PR footprint (#15262)

The previous round changed .github/scripts/sdk-java-lock.sh, which this PR never touched; the deterministic gate rejects that expansion. Restore the helper and keep both call sites on the pre-round `if !` shape, retaining the 1800 s wait and the step-summary record. Drop the status-2 pins and tests that only existed for the reverted helper, and record the unusable-queue gap as a follow-up in the design docs. The timeout-returns test stays.

Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>

* fix(ci): de-alias every resolver tracking file and guard an unusable lock queue (#15262)

* fix(ci): de-alias Maven install metadata in the sdk-java seed (#15262)

Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>

* test(ci): restore the per-job Maven seed binding (#15262)

* fix(ci): keep sdk-java resolution metadata out of the host cache (#15262)

The seed drops maven-metadata-*.xml hardlinks, but the write-back still
published job-local metadata with cp -n, where the first publisher's copy
could never be refreshed. Exclude it at both write-back sites and pin the
seed and write-back name sets to each other.

Also witness each suite-status exit with an exit-1 case, restore the lock
test's combined-stream assertion, and bring the design docs' acceptance
criteria and fault arms up to date.

* test(ci): carry the deferred-status helper into the sdk-java slices

The tolerated-skip arms now call mvn_report_failed_suite, which the step
defines above the lock block. Every slice that starts at lock_setup_fault()
or at the write-back therefore ran bash -e without the definition and died
with 127, which read as a code failure rather than a missing prelude.

* docs(ci): clarify remaining sdk-java cache hardening scope

* fix(ci): bound failed MySQL cache waits and exclude resolver residue

Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>

* fix(ci): clear inherited resolver state in the Maven cache seed prune

The seed de-alias prunes carried -links +1, which restricted deletion to
the hardlinks cp -aln created in this job. RUNNER_TEMP survives across
jobs on the pool, so a previous job's mutable resolver state (nlink 1
*.lastUpdated, _remote.repositories, resolver-status.properties) survived
in the job-local repository and could redden an otherwise green Maven
lane. Drop -links +1 from all four seed prunes so inherited state is
cleared again; the write-back's -links 1 delta filter is unchanged and
the seed stays best-effort (2>/dev/null || true).

Move the deAlias test pin with the fix and add a regression witness that
plants single-link markers in the job-local repository before the seed
body runs and asserts they are gone (red with -links +1 restored, exit 1;
green with the fix, on macOS and Linux). Re-record the sdk-java.yml
.size-baseline entry (153970 bytes) and synchronize both design-doc
languages, including the nine-coordinate set the main merge brought in.

---------

Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
Co-authored-by: Qwen Autofix <qwen-autofix@users.noreply.github.com>
Co-authored-by: qwen-code-dev-bot <qwen-code-dev-bot@users.noreply.github.com>
Co-authored-by: Shaojin Wen <shaojin.wensj@alibaba-inc.com>
Co-authored-by: yiliang114 <1204183885@qq.com>
Co-authored-by: Shaojin Wen <szujobs@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

review/self-reported The linked issue was opened by the PR author (self-reported) scope/ci-cd Continuous integration/deployment type/bug Something isn't working as expected

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Main CI failed: Managed Agent gate on e00767c8e3cb

5 participants