Skip to content

fix(serve): serialize overlapping cleanup retries - #15482

Merged
wenshao merged 17 commits into
QwenLM:mainfrom
jabrailkhalil:fix/hosted-cleanup-retry-serialization
Nov 7, 2026
Merged

wenshao merged 17 commits into
QwenLM:mainfrom
jabrailkhalil:fix/hosted-cleanup-retry-serialization

Conversation

@jabrailkhalil

@jabrailkhalil jabrailkhalil commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

What this PR does

I serialize background and resident-load cleanup retries within each attached Session. A queued request rechecks unsettled work and durable cleanup debt after its predecessor, using its own authorization options. A failed attempt remains an error for its caller and leaves the next request free to retry. An older caller cannot erase a successor's queue slot. Java unit validation also bounds JVM-held H2 catalogs to a test class instead of retaining every class's catalog in one process.

Why it's needed

I reproduced a wake retry and passive reload observing the same unpaid owner before either confirms its release. This produces three release calls where two are expected. A late background refusal can also leave recovery blocked after successful cleanup. This is the cleanup-debt case in #14101, encountered while checking #14937.

Reviewer Test Plan

How to verify

  • Hold the background original-owner release while admitting a passive load. Success must produce exactly two total release calls, one terminal turn, one subsequent cleanup confirmation, an unblocked Session, and no repeated model turn.
  • Refuse the held release. The queued load must make its own retry, with exactly three calls after two refusals and the same final state. Every call must retain the original binding, generation and Runtime owner.
  • Admit a real later prompt while a cleanup is queued. After the predecessor refuses, queued cleanup must recheck the prompt before touching its owner.
  • For three callers, hold the second release after the first fails. The first caller must preserve the successor's slot, and the third must wait. Once all callers settle, the retry promise must clear.

Evidence (Before & After)

Current head: 888fc1fd01ada97b7eb16dcf38d70b52df0b641d. I merged upstream main 4cd3372b to include the current lint gate. The cleanup implementation and regression-test file are byte-identical to 0967b44b, and the PR retains its five-file scope, including the class-fork setting in the verification module. All nine gate-freshness tests pass locally. Fresh upstream CI is pending; the successful checks previously recorded at 3fd698a2 remain historical evidence for that revision. Configured skipped lanes remain coverage limits.

The recorded original-source control fails the duplicate-release and stale-recovery assertions. This does not claim that both baseline branches fail deterministically on every host. Prior guard mutations were caught by the unsettled-prompt, settled-promise and successor-slot cases; those mutation receipts belong to the earlier recorded candidate.

Root build, bundle, complete TypeScript typecheck, scoped ESLint/Prettier, nine gate-freshness tests and 44 lint-scope tests pass at 2470302a0a. The selected cleanup suite passes 29/29 with no retry or timeout override. These are earlier revision-specific receipts; the H2 increment changed only Java unit-fork configuration and its two regression classes, and the latest main merge awaits fresh CI. Other Harness cases were filtered; this is not the complete Harness suite.

The Java run at 2470302a0a reports 5,573 tests, zero assertion failures, four errors and 19 skips. Three fresh-schema errors terminate in H2 heap exhaustion; a separate provisioner happy-path case exceeds its one-second startup deadline. The earlier run exhausted heap in two different schema cases. The relation between GC pressure and the deadline error remains unproven.

A bounded probe of the actual schema, its 38 canonical tables and H2 2.4.240 confirms one retention mechanism: after four databases are initialized, all connections are closed and all four data sources are collected, H2 still retains all four catalogs after GC. Explicit shutdown removes them. The default reused test JVM lets such fixture state accumulate across 375 classes. Ordinary Surefire now starts a fresh JVM per class, with its default single fork keeping classes sequential. Failsafe MySQL/Hosted settings and the overriding fault-gate profile remain unchanged; JaCoCo continues appending fork execution data.

Two real H2 regression classes deliberately reuse one catalog name. With the old reused fork, the second class sees the first class's table and fails; with class forks, both pass. The local Maven harness uses the unmodified test sources, actual default plugin configuration, Surefire 3.5.6, JUnit 5.14.1, H2 2.4.240 and JDK 21.0.7: before one passed/one failed; after two passed, zero skips. It is an isolated harness, not a full server-module compile. Direct JUnit controls also confirm both build-oracle tests skip outside Maven and both fail when a synthetic CI run lacks the Surefire marker; restored fork reuse still fails normally under Maven.

I verified that the required Java 21 fast gate, including the module compile and unit/H2 verification, passed on 3fd698a2; the job took 59m35s including build, tests and static checks. The Java 11 real-daemon E2E job also passed. The MySQL-backed Hosted/EventTransport profiles were not selected by this CI tier. I retain the earlier failures and their qualifications; the focused local harness alone was not the acceptance evidence.

An earlier cleanup run also had two 15-second timeouts, followed by a 29/29 pass on an unchanged candidate; their cause remains unresolved and the failed logs are retained. No source timeout increase is included.

Tested on

OS Status
macOS No local current-head run
Windows No local current-head cleanup run
Linux Current-head upstream lint/static, unit/integration/Web Shell/Serve A/B checks and the Managed Agent fast tier pass. The earlier 29-case and H2 controls remain revision-specific receipts.

Environment

Node 22.21.1, frozen pnpm dependencies, 4 CPU / 8 GiB isolated Linux container, 6 GiB build/typecheck heap. Public HTTP/journal tests use mocked model and Runtime Broker responses; no live provider or TUI was exercised.

Risk & Scope

Linked Issues

Related to #14101 and #15483; encountered while checking #14937.

中文说明

此 PR 的改动

我在每个已附加 Session 内串行执行后台和驻留加载触发的清理重试。排队请求等待前序完成后,使用自身授权选项重新检查未结束工作和持久清理债务。失败仍返回给对应调用者,同时允许后续请求重试。较早调用者不能清除后继请求的队列槽位。Java 单元验证还将 JVM 保留的 H2 目录限定在一个测试类的生命周期内,而不是在一个进程中累积所有类的目录。

为什么需要

我复现了唤醒重试与被动重新加载在任一方确认释放之前同时发现同一未偿还所有者的情况。这会产生三次释放,而预期为两次。后台较晚的拒绝还可能在成功清理后重新阻塞恢复。这对应 #14101 的清理债务场景,在检查 #14937 时再次遇到。

审阅者测试计划

如何验证

  • 阻塞后台对原始所有者的释放,同时允许被动加载。成功时必须恰好有两次释放调用、一个终态回合、其后一个清理确认、解除阻塞的 Session,且不重复模型回合。
  • 拒绝被阻塞的释放。排队加载必须自行重试,两次拒绝后恰好有三次调用,并得到相同最终状态。每次调用都必须保持原始绑定、代次和 Runtime 所有者。
  • 清理排队时接受一个真实的后续提示。前序拒绝后,排队清理必须在操作所有者之前重新检查该提示。
  • 三个调用者场景中,第一次失败后阻塞第二次释放。第一个调用者必须保留后继槽位,第三个必须等待。所有调用结束后,重试 promise 必须清除。

前后对比证据

当前提交:888fc1fd01ada97b7eb16dcf38d70b52df0b641d。我合并了上游 main 4cd3372b,纳入当前 lint 门槛。清理实现及回归测试文件与 0967b44b 的字节完全一致;PR 仍保持五文件范围,包括 verification 模块中的按类分叉设置。本地九个门槛新鲜度测试全部通过。新的上游 CI 尚待完成;此前 3fd698a2 的成功检查仍仅作为该版本的历史证据。配置中跳过的车道仍列为覆盖限制。

已记录的原始源码对照会触发重复释放和过时恢复状态的断言失败。这不意味着两个基线分支在每个主机上都确定失败。此前的守卫变异被未结束提示、已结束 promise 和后继槽位用例捕获;这些变异记录对应先前记录的候选提交。

根目录构建、打包、完整 TypeScript 类型检查、限定范围 ESLint/Prettier、九个门槛新鲜度测试及 44 个 lint 范围测试在 2470302a0a 通过。选定的清理套件 29/29 通过,没有修改重试或超时。这些证据对应前述提交;H2 增量仅修改 Java 单元分叉配置及两个回归类,最新 main 合并仍待新的 CI 验证。其他 Harness 用例被过滤;这不是完整 Harness 套件。

2470302a0a 的 Java 运行报告 5,573 个测试、零断言失败、四个错误和 19 个跳过。三个新建模式错误终止于 H2 堆耗尽;另一个提供器正常路径用例超过其一秒启动期限。更早的运行在另外两个模式用例中耗尽堆。GC 压力与启动期限错误之间的关系仍未证实。

对真实模式、其 38 张规范表及 H2 2.4.240 的有界探针确认了一种保留机制:初始化四个数据库后,全部连接关闭且四个数据源均被回收,GC 后 H2 仍保留全部四个目录。显式关闭可将其移除。默认复用测试 JVM 会让此类夹具状态在 375 个类之间累积。普通 Surefire 现为每个类启动新 JVM,其默认单分叉保持类串行执行。Failsafe 的 MySQL/Hosted 设置及覆盖配置的 fault-gate profile 不变;JaCoCo 继续追加各分叉执行数据。

两个真实 H2 回归类有意复用同一目录名称。旧的复用分叉让第二个类看到第一个类的表并失败;按类分叉时两者均通过。本地 Maven 夹具使用未修改的测试源、真实默认插件配置、Surefire 3.5.6、JUnit 5.14.1、H2 2.4.240 和 JDK 21.0.7:修改前一个通过、一个失败;修改后两个通过、零跳过。这是独立夹具,不是完整服务端模块编译。直接 JUnit 对照还确认两个构建检查在 Maven 之外跳过,模拟 CI 缺少 Surefire 标记时两个均失败;在 Maven 中恢复复用分叉仍正常触发回归。

我确认 3fd698a2 上必需的 Java 21 fast gate 已通过,包括模块编译和单元/H2 验证;该任务包含构建、测试和静态检查,共用时 59 分 35 秒。Java 11 真实 daemon E2E 任务也已通过。本 CI 层级未选择 MySQL 支撑的 Hosted/EventTransport profiles。我保留此前失败及其限定说明;本地定向夹具本身不是验收证据。

此前一次清理运行还出现两个 15 秒超时,之后未修改的候选提交 29/29 通过;原因仍未确定,失败日志已保留。本改动不增加源码超时上限。

已测试平台

操作系统 状态
macOS 当前提交未在本地运行
Windows 当前提交的清理用例未在本地运行
Linux 当前提交的上游 lint/静态检查、单元/集成/Web Shell/Serve A/B 检查及 Managed Agent fast tier 已通过。此前 29 用例和 H2 对照仍对应各自列出的提交。

环境

Node 22.21.1,冻结 pnpm 依赖,4 CPU / 8 GiB 独立 Linux 容器,构建和类型检查使用 6 GiB 堆。公开 HTTP/日志测试使用模拟模型与 Runtime Broker 响应;未测试真实提供商或 TUI。

风险与范围

关联问题

关联 #14101 和 #15483;在检查 #14937 时遇到。

@jabrailkhalil

jabrailkhalil commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor Author

Precision after the independent macOS check: the base failures below describe my recorded Linux / Node 22 run. The duplicate-release branch is the deterministic witness in the reviewer’s three controls; the refused-retry base branch is scheduler-dependent and was green on their first run. See the review.

I verified the cleanup retry overlap with the local HTTP/journal test-script fallback on Linux, Node 22.21.1 and pnpm 11.24.0. No external model/provider or live Runtime Broker was used.

  • Before: the same permanent tests with the production source restored to main 1eee20a7 fail both branches: three releases instead of two, and a stale recovery block after cleanup has succeeded.
  • After: bc689ee8 passes both branches, including original owner/binding/generation, one terminal event, one later cleanup confirmation, no repeated model turn and final deletion. Promise barriers force the overlap; assertions, test budgets and retry settings are retained.
  • Final checks: 706/706 Harness/approval cases, 298/298 cases across 12 related recovery/wake files, root typecheck, full ESLint, full Prettier, bundled serve startup and CLI version. No tracing, delay preload, external Vitest config or retries was used in these final suites.
  • Build scope: frozen install, root workspace build and bundle passed at 448bdea; final bc689ee8 changes only the test fixture scheduler capture, with identical production/dependency blobs and its own final checks. No full-monorepo preflight claim.

The five historical full-run timeouts are a separate I/O investigation. This PR addresses the proven cleanup race. Live provider/Broker behavior, TUI and macOS/Windows were not tested.

中文说明

我在 Linux、Node 22.21.1、pnpm 11.24.0 上通过本地 HTTP/journal test-script fallback 验证 cleanup 重试竞态。未调用外部模型/provider 或真实 Runtime Broker。

  • 修改前:保持相同的永久测试,仅将 production 恢复为 main 1eee20a7,两个分支都失败:预期两次的 release 实际为三次;cleanup 已成功后仍错误保留 recovery block。
  • 修改后:bc689ee8 的两个分支均通过,包括原 owner/binding/generation、单个终结事件、之后单个 cleanup 确认、不重复模型 turn,以及最终删除。Promise 屏障强制并发重叠;保留断言、时间预算和 retry 设置。
  • 最终验证:706/706 项 Harness/approval、12 个相关 recovery/wake 文件共 298/298 项测试、根目录 typecheck、完整 ESLint/Prettier、bundle serve 启动和 CLI version。最终测试不含 tracing、延迟 preload、外部 Vitest 配置或重试。
  • 构建范围:冻结依赖安装、根 workspace build 和 bundle 在 448bdea 上通过;最终 bc689ee8 仅改变测试 fixture 获取 scheduler 的写法,production/dependency blob 完全相同,并执行最终检查。不声称完整 monorepo preflight。

此前五项完整运行超时属于独立的 I/O 调查。本 PR 仅修复已证明的 cleanup 竞态;未测试真实 provider/Broker、TUI、macOS 或 Windows。

@qwen-code-review-bot

qwen-code-review-bot commented Oct 24, 2026 •

Copy link
Copy Markdown
Collaborator

⏳ Approval still deferred — 1 PR CI workflow run(s) still in progress for bc689ee; the finalize job approves automatically once every run lands green. finalize run

⏳ 审批仍在延迟中 —— bc689ee 还有 1 个 PR CI workflow 在运行,全部通过后 finalize 任务会自动提交审批。查看 finalize 运行

@qwen-code-review-bot

Copy link
Copy Markdown
Collaborator

Thanks for the PR!

Template looks good ✓ — every required section is filled in, including the Tested-on matrix and a complete Chinese translation.

Problem: observed, and I confirmed it from the code rather than taking the description's word for it. On main, the passive resident-load path calls retryHostedCleanup only when resident.active is undefined, while the wake pump's recoverBlocked calls it only when wakeBusy() is false — and session.active !== undefined is one of wakeBusy()'s own terms. So the load that reaches its retry is precisely the load that lets recoverBlocked through. Neither site is wrapped in monitorWake.exclusive() (only two call sites in the whole file are), and both read hasHostedCleanupDebt before either commits confirmed. Two concurrent reconciles against one owed owner is therefore reachable without any test scaffolding — the harness only makes the window deterministic. The commit order backs this up too: the reproduction landed two commits before the fix.

Direction: aligned. This is qwen serve daemon recovery correctness — a duplicate Runtime-owner release and a Session left recovery-blocked after its cleanup was already confirmed are both real operational faults, not code hygiene. Nothing here touches auth, sandboxing, model selection, telemetry, release, or a public contract; the queue is in-memory only and the PR says so explicitly.

Size: not a core path — packages/cli/src/serve/** matches none of the protected patterns and the change stays inside a single package. For the record: 36 production lines (26+/10−) against 247 test lines (157+/90−), roughly 1:7. Well under every threshold; no maintainer-awareness escalation and no large-PR advisory apply.

Approach: the scope feels right, and it is close to what I proposed from the title alone before reading the diff — one optional per-session promise field, chain onto the predecessor, re-read the debt inside the handler, keep each caller's own authorization options. I did check whether the existing monitorWake.exclusive() primitive could have been reused instead of adding a field, since "modify what already exists" is the obvious simpler path. It cannot: one of the two racers is the pump, and exclusive() called from inside an in-flight pass would wait on that same pass. A separate per-session chain is justified here, not a parallel utility. The diff carries no drive-by refactors or unrelated edits.

Risk: no elevated risk signals — neither changed file matches the high-risk path set.

One thing to think about, not a blocker: a queued caller now waits for its predecessor's broker round-trip, so a slow or hung release() adds latency to the passive-load and lifecycle-close paths that previously fired independently. Your Risk & Scope section already names this tradeoff, which is the right place for it — worth just confirming there is a bound on that call.

Moving on to code review. 🔍

中文说明

感谢贡献!

模板完整 ✓ —— 所有必填章节都填写了,包括测试平台表格和完整的中文说明。

问题: 属于已观测到的 bug,而且我是从代码本身独立确认的,不是只采信 PR 描述。在 main 上,passive resident-load 路径只在 resident.active 为 undefined 时调用 retryHostedCleanup,而 wake pump 的 recoverBlocked 只在 wakeBusy() 为 false 时调用它 —— 而 session.active !== undefined 正是 wakeBusy() 的判断项之一。也就是说,能走到重试的 load,恰好就是会放行 recoverBlocked 的 load。两处都没有包在 monitorWake.exclusive() 里(全文件只有两处调用),并且都在任一方提交 confirmed 之前读取 hasHostedCleanupDebt。因此针对同一笔未偿还 owner 的两次并发 reconcile 在没有任何测试脚手架的情况下也可达 —— harness 只是把这个窗口变成确定性的。提交顺序也印证了这一点:复现比修复早两个提交。

方向: 对齐。这是 qwen serve 守护进程的恢复正确性问题 —— 重复释放 Runtime owner、以及在 cleanup 已确认后 Session 仍被 recovery-blocked,都是真实的运行故障,不是代码整洁度问题。改动不涉及 auth、sandbox、模型选择、telemetry、发布或对外契约;队列仅在内存中,PR 也明确说明了这一点。

规模: 不属于核心路径 —— packages/cli/src/serve/** 不匹配任何受保护模式,且改动局限在单个 package 内。记录一下:生产代码 36 行(26+/10−),测试 247 行(157+/90−),约 1:7。远低于所有阈值,不触发维护者知会升级,也不触发大 PR 提示。

方案: 范围合理,而且和我在读 diff 之前仅凭标题给出的设想很接近 —— 每个 Session 增加一个可选的 promise 字段,挂到前一次尝试之后,在 handler 内重新读取债务,每个调用方保留自己的授权选项。我确实检查过能否复用已有的 monitorWake.exclusive() 而不是新增字段,因为"改动已有机制"通常是更简的路径。答案是不能:两个竞争方之一就是 pump 本身,而在一个正在执行的 pass 内部调用 exclusive() 会等待这个 pass 自己。所以这里新增一个 per-session 串行链是合理的,不是重复造轮子。diff 中没有顺手重构或无关改动。

风险: 无升级风险信号 —— 两个改动文件都不匹配高风险路径集合。

有一点值得考虑,但不是阻塞项:排队的调用方现在要等前一次尝试的 broker 往返完成,所以一次缓慢或挂住的 release() 会给 passive-load 和 lifecycle-close 路径带来额外延迟,而这两条路径原本是各自独立发起的。你的"风险与范围"已经点出了这个取舍,位置很恰当 —— 建议再确认一下这个调用是否有超时上限。

进入代码审查 🔍

— Qwen Code · qwen3.8-max-2026-09-02

Reviewed at bc689ee8f557ce96f00ad45fecb6cebf15230dc0 · re-run with @qwen-code /triage

@qwen-code-review-bot

qwen-code-review-bot commented Oct 24, 2026 •

Copy link
Copy Markdown
Collaborator

Code review

I wrote down my own approach from the title and the "Why it's needed" section before opening the diff: hold a per-session in-flight promise, let a second caller chain onto it, have the queued caller re-read the debt after its predecessor settles so it skips work already confirmed, and keep each caller's own authorization options. That is what this does, so I have no simpler alternative to argue for.

I looked hardest for a self-deadlock, since a promise chain that re-enters itself is the obvious way this design goes wrong. It does not:

  • The only thing awaited inside the new handler is reconcileHostedTurnCleanup, which lives in hosted-turn-cleanup.ts and has no reference to retryHostedCleanup or to the wake scheduler — I grepped for monitorWake, exclusive, and recoverBlocked there and got zero hits. So the handler cannot reach back into its own chain.
  • The catch path calls session.monitorWake?.kick() without awaiting it, and kick() is fire-and-forget (void this.pump(), with its own .catch/.finally). The handler then rethrows immediately, so the current link settles and any successor chained onto it proceeds. The pump's recoverBlocked calling retryHostedCleanup from the other side is a new link, not a cycle.
  • The finally clears session.cleanupRetry only when it still is this caller's promise. That identity guard is what keeps an earlier caller from clobbering a newer link — with three overlapping callers, only the last one clears. Correct.

Error semantics match the description. A rejection reaches the caller that produced it (that caller awaits its own link directly), while a queued successor swallows its predecessor's rejection via .catch(() => undefined) and re-reads the debt with its own options. Every link is awaited by the caller that created it, so no rejection is left unhandled.

No correctness blockers and no AGENTS.md violations. Three non-blocking observations:

  1. Latency on the queued path. A caller that previously fired concurrently now waits out its predecessor's broker.release() round-trip. That is inherent to the fix and your Risk & Scope names it; I'd just like to see that release() has a timeout, because a hung broker would now stall the passive-load route (4963) and the lifecycle-close path (7594) rather than only the background retry.
  2. cleanupRetry is not cleared on detach/close. The monitorWake.close() sites don't touch the field, so a serialized retry can still finish a broker release after close begins. This was equally true before the change (the retry was simply in flight), and the session object is discarded either way, so I don't consider it a regression — just noting the field outlives the pump.
  3. The new test mocks kick() to a no-op after the first call. That is a reasonable way to build a deterministic overlap, but it means the catch-path re-arm (monitorWake?.kick() on failure) is suppressed in the retryRefused: true arm, so the third release comes from the queued load's own retry rather than from a kick-driven pump pass. The assertion still pins the right observable; I'd just be aware that the kick/chain interaction itself is not what this test exercises.

The test rewrite is sound. Converting the existing case to it.each([false, true]) keeps the original assertions intact and adds the refused-retry arm, and replacing the timing sleep with wake.exclusive() plus a hasHostedCleanupDebt spy as barriers is the right call — those are real synchronization points, not a guess at a duration. The added per-call assertions (every release names the original bindingId/generation, every broker context carries the right runtimeSessionId) are a genuine strengthening over what was there. capturedWakeSession() is declared later in the same describe block and hoists, so the forward reference is fine.

I skipped the sequence diagram and the changed-files table here — two files and 36 production lines don't need either, and Stage 1 already names the two racing call sites.

Testing

This is an unattended CI run, so per the review rules I executed nothing from this PR — no build, no test run, no gh pr checkout. The evidence below is the PR's own CI, read through the API for the reviewed commit. Nothing here is my re-run of your results, and the before/after A/B numbers in your description remain your claim, not independently verified by me.

Final CI results for bc689ee (auto-updated by the triage finalize job after CI completed):

Check Conclusion
Check TUI impact ✅ success
Classify PR ✅ success
Integration Tests (no-AK, No Sandbox) ✅ success
Lint & Static (ubuntu-latest, Node 22.x) ✅ success
Managed Agent fast tier ✅ success
OpenTUI no-flicker gate ✅ success
SDK Java (fast) / Flyway migration version uniqueness ✅ success
SDK Java (fast) / Real daemon E2E / Java 11 ✅ success
SDK Java (fast) / ubuntu-latest / Java 21 ✅ success
Select the Managed Agent tier ✅ success
Serve A/B (ubuntu-latest, Node 22.x) ✅ success
Test (ubuntu-latest, Node 22.x) ✅ success
TUI parity snapshots (ink vs opentui) ✅ success
web-shell E2E Smoke (ubuntu-latest, Node 22.x) ✅ success

One row per check name (latest run); skipped checks omitted; failures sort first. / 每个检查名一行(取最新一次运行),省略 skipped,失败项排在最前。

Read at review time: 14 success, 7 in progress, 1 queued, 16 skipped, 0 failures. The gates that matter most for this change — Test, Serve A/B, and Integration Tests — were all still running, and I did not poll or sleep-wait on them. The Qwen Triage Finalize job rewrites the table above in place once CI settles. Lint & Static passing green is consistent with your typecheck/ESLint/Prettier claim, but the unit suite is the one that actually covers this file and it had not reported yet.

Not verified: whether the two regression assertions are load-bearing. Your description says they fail with only hosted-harness-session.ts reverted to main, and that is the single most important fact in this PR — but a green suite proves the tests pass, not that they would fail without the fix, and I cannot check that statically.

Sandboxed verification would settle this: @qwen-code /verify — specifically the A/B claim that expect(release).toHaveBeenCalledTimes(retryRefused ? 3 : 2) and the final recoveryBlocked === false assertion both fail when the production file is restored to main and pass with it. If that A/B does not hold, the serialization is decoration and the race is still open. Since you have read access only, this would be a sponsored run: a maintainer's @qwen-code /verify comment approves the head it was written against, and that run carries a pre-execution risk screen and a full workspace wipe before any of this PR's code executes. Maintainers: read the resulting report with the same skepticism you'd give a fork's own CI logs — the code under verification is adversarial input, and a crafted PR can shape what a report says even though the sandbox bounds what it can do. /tmux is not the right lane here (and is unavailable to a read-only author); this is internal daemon recovery with no TUI surface, so the A/B proof is the instrument that applies.

中文说明

代码审查

在读 diff 之前,我先只根据标题和"为什么需要"写下了自己的方案:在每个 Session 上保存一个进行中的 promise,第二个调用方挂到它后面,排队的调用方在前一次结束后重新读取债务以跳过已确认的工作,并且每个调用方保留自己的授权选项。这个 PR 做的正是这件事,所以我没有更简的替代方案可提。

我最重点排查的是自死锁 —— 一个会重入自身的 promise 链是这种设计最典型的翻车方式。结论是不会:

  • 新 handler 内唯一被 await 的是 reconcileHostedTurnCleanup,它位于 hosted-turn-cleanup.ts,其中没有任何对 retryHostedCleanup 或 wake 调度器的引用 —— 我 grep 了 monitorWake、exclusive、recoverBlocked,零命中。所以 handler 不可能回到自己的链上。
  • catch 分支调用 session.monitorWake?.kick() 且不 await,而 kick() 本身是发后即忘(void this.pump(),自带 .catch/.finally)。handler 随后立即 rethrow,当前环节随即 settle,挂在它后面的后继环节得以继续。另一侧由 pump 的 recoverBlocked 发起的 retryHostedCleanup 是一个新环节,不是环。
  • finally 只在 session.cleanupRetry 仍然是本调用方自己的 promise 时才清空它。正是这个身份判断避免了较早的调用方清掉更新的环节 —— 三个调用方重叠时,只有最后一个会清空。正确。

错误语义与描述一致。拒绝会传回产生它的那个调用方(该调用方直接 await 自己的环节),而排队的后继者通过 .catch(() => undefined) 吞掉前一次的拒绝,并用自己的 options 重新读取债务。每个环节都被创建它的调用方 await,因此不存在未处理的 rejection。

没有正确性阻塞项,也没有违反 AGENTS.md 的地方。三点非阻塞观察:

  1. 排队路径的延迟。 原本并发发起的调用方,现在要等前一次的 broker.release() 往返结束。这是修复本身的固有代价,你的"风险与范围"也点明了;我只希望确认 release() 有超时上限,因为一个挂住的 broker 现在会拖住 passive-load 路由(4963)和 lifecycle-close 路径(7594),而不只是后台重试。
  2. detach/close 时不会清空 cleanupRetry。 monitorWake.close() 的几处调用都没有处理这个字段,所以一个串行中的重试可能在 close 开始之后才完成 broker release。改动之前同样如此(那时重试只是单纯在进行中),而且 Session 对象两种情况下都会被丢弃,所以我不认为这是回归 —— 只是记录这个字段的生命周期比 pump 更长。
  3. 新测试把首次之后的 kick() mock 成了 no-op。 这是构造确定性重叠的合理做法,但意味着 retryRefused: true 分支里 catch 路径的重新武装(失败时的 monitorWake?.kick())被抑制了,所以第三次 release 来自排队 load 自己的重试,而不是 kick 驱动的 pump pass。断言仍然钉住了正确的可观测量;只是要清楚这个测试所验证的并不是 kick 与链的交互本身。

测试改写是扎实的。把原有用例改成 it.each([false, true]) 保留了原有断言并新增了"重试被拒绝"分支;用 wake.exclusive() 加 hasHostedCleanupDebt spy 作为屏障替代原来的定时 sleep 也是正确选择 —— 它们是真实的同步点,而不是对时长的猜测。新增的逐次调用断言(每次 release 都带原始 bindingId/generation,每个 broker context 都带正确的 runtimeSessionId)相比原来确实是加强。capturedWakeSession() 声明在同一个 describe 块的后面并且会提升,所以这个前向引用没有问题。

这里我省略了时序图和改动文件总览表 —— 两个文件、36 行生产代码不需要它们,而且 Stage 1 已经点名了两个竞争调用点。

测试

这是一次无人值守的 CI 运行,因此按照审查规则我没有执行本 PR 的任何代码 —— 没有构建、没有跑测试、没有 gh pr checkout。下面的证据是 PR 自己的 CI,通过 API 读取被审查提交的结果。这里没有一项是我对你结果的复跑,你描述中的 before/after A/B 数据仍然是你的声明,未经我独立验证。

审查时刻的读数:14 项成功、7 项进行中、1 项排队、16 项跳过、0 项失败。对本次改动最关键的几个门禁 —— Test、Serve A/B、Integration Tests —— 当时都还在运行,我没有轮询或 sleep 等待。CI 结束后,Qwen Triage Finalize 任务会就地重写上面的表格。Lint & Static 通过与你声称的 typecheck/ESLint/Prettier 结果一致,但真正覆盖这个文件的单元测试当时还没有出结果。

未验证:两条回归断言是否真正 load-bearing。你的描述说,只把 hosted-harness-session.ts 还原为 main 版本时它们会失败 —— 这是本 PR 中最关键的一个事实,但绿色的测试套件只能证明测试通过,不能证明缺少这个修复时它们会失败,而这一点我无法静态确认。

沙箱验证可以解决这一点:@qwen-code /verify —— 具体验证 A/B 声明,即 expect(release).toHaveBeenCalledTimes(retryRefused ? 3 : 2) 和最终的 recoveryBlocked === false 断言在把生产文件还原为 main 时都失败、应用修复后都通过。如果这个 A/B 不成立,那么串行化就是装饰品,竞态仍然存在。由于你只有 read 权限,这将是一次受助运行(sponsored run):由维护者发出 @qwen-code /verify 评论来批准其所针对的 head,且该运行在本 PR 的任何代码执行前带有执行前风险筛查和完整的工作区擦除。请维护者以对待 fork 自己 CI 日志的同等怀疑态度阅读产出的报告 —— 被验证的代码是对抗性输入,一个精心构造的 PR 可以影响报告说什么,尽管沙箱限定了它能做什么。/tmux 在这里不是合适的通道(且对只读作者不可用);这是内部守护进程恢复逻辑,没有 TUI 界面,因此 A/B 证明才是适用的手段。

— Qwen Code · qwen3.8-max-2026-09-02

Reviewed at bc689ee8f557ce96f00ad45fecb6cebf15230dc0 · re-run with @qwen-code /triage

@qwen-code-review-bot

Copy link
Copy Markdown
Collaborator

Confidence: 4/5 — the fix is correct and I verified the race independently; the one thing I could not check is whether the new tests actually fail without it.

Stepping back: I went into this expecting to push back. A fork PR with 247 test lines wrapped around 36 production lines, in a 12,000-line daemon file, describing a race that only appears when two async paths interleave — that is the shape of a PR where the test scaffolding does the convincing and the underlying bug turns out to be unreachable. So I checked reachability from the code first, before reading the diff's own framing, and it holds up on its own terms: the passive-load retry is gated on resident.active being undefined, recoverBlocked is gated on wakeBusy() being false, and session.active !== undefined is one of the terms inside wakeBusy(). The two gates are not merely compatible, they are satisfied by the same state. Neither call site takes monitorWake.exclusive(), and both read the cleanup debt before either can commit it confirmed. That is a real window in shipped code, not an artifact of a test harness.

Given that, the fix is the obvious one and it is implemented carefully. My own proposal from the title alone — a per-session in-flight promise, re-read the debt after the predecessor settles, keep each caller's options — is what landed, so I have no simpler path to argue for. The parts I would normally expect a first attempt to get wrong are right: the finally clears the field only when it still owns it, so a slow earlier caller cannot clobber a newer link; a predecessor's rejection is swallowed for successors but still propagates to the caller that produced it; and every link is awaited by its creator, so nothing rejects unhandled. I specifically hunted for a self-deadlock, which is the failure mode this design invites, and there isn't one — the only awaited call inside the handler lives in a module with no path back to it, and the kick() in the catch path is fire-and-forget followed by an immediate rethrow.

On whether this deserves to merge rather than just whether it is correct: a Session left recovery-blocked after its cleanup was already confirmed is a wedge that no user can clear without a reload, and a duplicate owner release pins a workspace lease. Both are the kind of fault that produces a confused bug report a month later with no reproduction attached. Fixing it in 36 lines is proportionate. Six months from now the chain idiom will read as slightly subtle, but the inline comment states the actual constraint, which is what makes it maintainable.

What keeps this at 4 rather than 5, all non-blocking:

  • The A/B claim is the author's, not mine. That the two regressions fail with only the production file reverted to main is the strongest evidence in the PR and I could not verify it — this was an unattended CI run, so I executed nothing from the PR. Worth being precise about what that leaves open: my approval rests on the fix being correct, which I established statically from the gating logic above. If the tests turned out not to be load-bearing, the fix would still be right and merely under-tested, which is a follow-up rather than a blocker. Stage 2 names the sponsored @qwen-code /verify run that would settle it.
  • No confirmed bound on release(). Serialization means a hung broker now stalls the passive-load and lifecycle-close paths too, not just the background retry. The PR names the tradeoff; I'd like the timeout confirmed.
  • The test mocks kick() to a no-op after the first call, which suppresses the catch-path re-arm in the refused-retry arm. The assertion still pins the right observable, but the kick/chain interaction is not what that test exercises.

CI was still running at review time — 14 green, 0 red, with Test, Serve A/B, and Integration Tests in flight — so I am not approving against a result that does not exist yet. Approval is deferred until CI lands green on bc689ee8f557ce96f00ad45fecb6cebf15230dc0; the finalize job posts the commit-pinned approval and withholds it if anything lands red or the head moves.

One note on process, since this author has four open PRs: I evaluated this one on its own merits and it is the strongest of them — reproduction committed before the fix, an honest Risk & Scope section that lists what was not validated (live broker, macOS/Windows, TUI, the five separate full-run timeouts) rather than claiming a clean preflight it didn't run. That candour made the review easier, not harder.

中文说明

信心度:4/5 —— 修复本身是正确的,而且竞态是我独立验证的;唯一没能确认的是新测试在缺少该修复时是否真的会失败。

退一步看:我本来预期会提出反对意见。一个 fork PR,用 247 行测试包住 36 行生产代码,改的是一个 12000 行的守护进程文件,描述的竞态只在两条异步路径交错时才出现 —— 这正是那种"测试脚手架负责说服你、而底层 bug 其实不可达"的 PR 形态。所以我在读 diff 自己的叙述之前,先从代码确认了可达性,而它靠自己就站得住:passive-load 的重试以 resident.active 为 undefined 为前提,recoverBlocked 以 wakeBusy() 为 false 为前提,而 session.active !== undefined 正是 wakeBusy() 内部的判断项之一。这两个门不只是兼容,它们是被同一个状态同时满足的。两处调用都没有进入 monitorWake.exclusive(),并且都在任一方能够提交 confirmed 之前读取 cleanup 债务。这是已发布代码中真实存在的窗口,不是测试 harness 造出来的假象。

在此前提下,修复方案是显而易见的那一个,而且实现得很细致。我仅凭标题给出的设想 —— 每个 Session 一个进行中的 promise、前一次结束后重新读取债务、每个调用方保留自己的 options —— 正是最终落地的形态,所以我没有更简的路径可提。通常第一版会写错的地方这里都是对的:finally 只在字段仍属于自己时才清空,所以较慢的早期调用方不会清掉更新的环节;前一次的拒绝对后继者被吞掉,但仍会传回产生它的那个调用方;每个环节都被其创建者 await,因此不存在未处理的 rejection。我专门去找了这种设计最容易引发的失效模式 —— 自死锁 —— 结论是没有:handler 内唯一被 await 的调用所在模块没有任何回到它的路径,而 catch 分支里的 kick() 是发后即忘,紧接着就 rethrow。

至于它是否值得合入,而不只是是否正确:一个在 cleanup 已确认之后仍被 recovery-blocked 的 Session,是用户不重新 load 就无法解开的卡死;而重复的 owner release 会钉住 workspace lease。两者都属于一个月后会产生一份没有复现步骤的困惑 bug 报告的那类故障。用 36 行修掉它是相称的。六个月后这个链式写法读起来会有点微妙,但内联注释说明了真正的约束,这正是它可维护的原因。

让它停在 4 分而不是 5 分的原因,均为非阻塞项:

  • A/B 声明来自作者,不是我。 "只把生产文件还原为 main 时两条回归会失败"是本 PR 中最有力的证据,而我无法验证 —— 这是一次无人值守的 CI 运行,所以我没有执行 PR 的任何代码。需要精确说明这留下了什么缺口:我的批准建立在这个修复是正确的之上,而这一点我是从上面的门控逻辑静态确认的。如果测试最终并非 load-bearing,修复依然是对的,只是测试覆盖不足 —— 那属于后续跟进,不是阻塞项。Stage 2 已点名可以解决这一点的受助 @qwen-code /verify 运行。
  • release() 的上限未确认。 串行化意味着一个挂住的 broker 现在也会拖住 passive-load 和 lifecycle-close 路径,而不只是后台重试。PR 点明了这个取舍;我希望确认存在超时。
  • 测试把首次之后的 kick() mock 成了 no-op,这在"重试被拒绝"分支里抑制了 catch 路径的重新武装。断言仍钉住了正确的可观测量,但那个测试所验证的并不是 kick 与链的交互。

审查时 CI 仍在运行 —— 14 项绿、0 项红,Test、Serve A/B、Integration Tests 还在进行中 —— 所以我不会对着一个尚不存在的结果批准。批准已延迟到 CI 在 bc689ee8f557ce96f00ad45fecb6cebf15230dc0 上全绿之后;finalize 任务会发出绑定该提交的批准,若有任一项变红或 head 发生移动则会撤回。

关于流程的一点说明,因为这位作者有四个开着的 PR:我是按本 PR 自身的价值评估的,而它是这几个里最扎实的一个 —— 复现比修复更早提交,"风险与范围"一节诚实地列出了未验证的内容(真实 broker、macOS/Windows、TUI,以及另外五次完整运行超时),而不是声称跑过它其实没跑的完整 preflight。这种坦率让审查更容易,而不是更难。

— Qwen Code · qwen3.8-max-2026-09-02

Reviewed at bc689ee8f557ce96f00ad45fecb6cebf15230dc0 · re-run with @qwen-code /triage

@qwen-code-review-bot

qwen-code-review-bot commented Oct 24, 2026 •

Copy link
Copy Markdown
Collaborator

🩺 serve daemon A/B

Built the PR base vs this PR head 890429b, drove a fixed endpoint set against each, and diffed the JSON responses. Only fields that changed are shown.

health-deep-with-session

field PR base (before) this PR (after)
activeWorkStaleMs 128 125

— Qwen Code · serve A/B

@jabrailkhalil

jabrailkhalil commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor Author

Precision after the independent macOS check: the base failures below describe my recorded Linux / Node 22 run. The duplicate-release branch is the deterministic witness in the reviewer’s three controls; the refused-retry base branch is scheduler-dependent and was green on their first run. See the review.

Confirmed: release() makes one broker request, and that request uses AbortSignal.timeout(30_000). That bounds the broker RPC, not the entire queue or local journal work.

The local A/B uses identical test blob 9c261052: old production fails both cases (extra release; stale recovery block), while bc689ee passes both. The test report records that separately from CI. Could a maintainer sponsor the Verify run suggested in stage 2 on this exact head to check those two existing regressions independently?

The kick mock deliberately forces the queued-load retry; I am not claiming coverage of catch-path pump re-arming in these two cases.

@wenshao

wenshao commented Oct 25, 2026

Copy link
Copy Markdown
Collaborator

Verdict: merge-ready — 9/9 scripted assertions passed. Verified head bc689ee8f557ce96f00ad45fecb6cebf15230dc0; base 1eee20a7 (merge-base; the changed production file is byte-identical at GitHub's base tip 87735a9a). Environment: macOS arm64, Node 24.18.1, Vitest 3.2.7. Round: 2026-10-25 16:38–17:20 (UTC+8).

中文摘要

结论:可以合并(merge-ready),9/9 条脚本断言通过。 验证 head 为 bc689ee8,base 为 1eee20a7;本机 macOS arm64 / Node 24.18.1 / Vitest 3.2.7(作者测的是 Linux / Node 22.21.1)。

  • 中心主张成立:只把生产文件换回 base、保留 PR 的新测试,retry refused: false 分支稳定红:expected "release" to be called 2 times, but got 3 times——同一个原 owner 被并发 release 两次。head 2/2 通过;base 原生旧测试 1/1 通过(夹具对照);base 重复两次都在两个分支红。
  • 变异矩阵:删掉 session.cleanupRetry = retry(序列化本身)和删掉 .catch(() => undefined)(前一次失败不污染后一次)分别被对应分支钉死;另有 3 个变异存活——内层 unsettled 复查、finally 清理、条件清理——都是测试覆盖缺口,不是中心主张的缺陷,按仓库规则属于建议而非阻塞。
  • 门禁:head 上名称含 cleanup 的 27 个用例 27/27 通过。完整 678 例文件超过本机 10 分钟后台上限被终止(未产生 JSON),见「未覆盖」。
  • 未覆盖:完整 678 例文件;CLI 全量 typecheck(Node 24 下 V8 崩溃,非 TS 诊断);真实 Runtime Broker/模型(测试内 mock);Linux/Windows;整仓 web-shell / generated-ui 构建(环境既有失败)。
  • 对描述的更正:作者说 base 上「两个分支都失败」。本机首次 base 跑时 retry refused: true 分支是绿的,随后两次重复才两分支都红——false 分支是确定性见证,true 分支在 base 上是时序相关的;不是代码问题,但测试计划的确定性表述偏强。

Central claim and A/B

The PR serializes cleanup retries per attached Session: a queued caller waits for its predecessor, re-reads unsettledPromptId and durable cleanup debt with its own broker options after the predecessor settles, and a failed attempt remains an error for its caller while leaving the next request free to retry. The one behavior it exists to change is that a wake retry and a passive load must not both release the same original owner.

The A/B keeps one tree and one build: the head tree at bc689ee8, with only packages/cli/src/serve/hosted-harness-session.ts restored to base. The new test file is held constant in every cell except the base-native fixture control.

Cell Production source Test file Tests run Result
head head bc689ee8 head 2 2 passed
base base 1eee20a7 head 2 1 failed: retry refused: false — expected "release" to be called 2 times, but got 3 times
base native base 1eee20a7 base 1 1 passed (fixture control)
base repeat 2 base 1eee20a7 head 2 both branches failed
base repeat 3 base 1eee20a7 head 2 both branches failed
head cleanup gate head bc689ee8 head 27 27 passed

The deterministic witness is the duplicate release: the false branch failed on all three base runs (3/3) and passed on head (2/2 in the focused cell, plus the 27-test cleanup gate). The retry refused: true branch passed the first base run and failed the next two; on this host the base-side stale-recovery-block symptom is timing-dependent, so it is not the branch to cite as deterministic proof.

A/B cells and mutation matrix

Base arm failure from the vitest run

Corrections

The PR body and the author's verification comment say the same two regressions fail on base in both branches. On this host they do not fail both deterministically: the first base run passed retry refused: true (expected true to be false did not fire), and only the two repeats failed both branches. This does not change the conclusion — the duplicate-release branch is deterministic and is the PR's central claim — but "fail both branches" should be read as "fail both branches under the reviewer's run", not as a property of the test on every host. The author tested Linux / Node 22.21.1; this round is macOS / Node 24.18.1, so a scheduler difference is a plausible cause.

Findings

All three are coverage gaps, not merge conditions. Each survivor is a mutation of a guard the PR adds that no committed test distinguishes from the unmutated head.

  1. Inner unsettled re-check is unpinned. Removing if (unsettledPromptId(session)) return; inside the queued then leaves both new tests green. The scenario never admits a prompt between the predecessor settling and the queued caller's re-check. Reachable in production: the queued caller can wait on a long release while a user prompt is accepted. Suggested witness: a variant that submits a prompt (or marks an unsettled prompt) before releasing the barrier, then asserts the queued caller does not reconcile.
  2. finally clear is unpinned. Removing the if (session.cleanupRetry === retry) session.cleanupRetry = undefined; block leaves both tests green. Behavior stays correct in this scenario, but the Session retains a settled promise chain; a white-box assertion on session.cleanupRetry after the caller settles would pin it.
  3. Conditional clear is unpinned. Replacing the identity check with an unconditional clear also survives. With only two callers there is no third arrival to observe the lost chain; a three-caller variant is the missing fixture. This is the same three-caller gap as (2), and it is why the identity check cannot be shown load-bearing with the current test.

Mutation matrix:

Mutation Outcome Witness
remove session.cleanupRetry = retry KILLED retry refused: false: release 3, expected 2
remove .catch(() => undefined) KILLED retry refused: true: release 2, expected 3
remove inner unsettled re-check SURVIVED 2/2 green
remove finally clear SURVIVED 2/2 green
clear unconditionally SURVIVED 2/2 green

Vacuity checks ran as live cells: the head cell is green, the base+new-test cell is red on the expected assertion (not a setup or import failure), and the base-native control is green. Both mutation kills were attributed to the assertion message quoted above, not to a TypeError, fixture error, or prerequisite guard.

Not covered

  • The full 678-test hosted-harness-session.test.ts file. It exceeded the local 10-minute background-task cap (stopped at ~612 s, no JSON reporter output), so it is not a gate result. The 27 tests whose names contain cleanup passed on head.
  • CLI typecheck. npx tsc -p tsconfig.json --noEmit aborted inside Node 24.18.1's V8 with a stack trace; no typecheck result was obtained. This is an environment crash, not a typecheck pass. The author reports typecheck on Node 22.21.1.
  • Real Runtime Broker, model, and provider. The tests mock the Broker, as the PR's own harness does; no live broker or model call was made.
  • Linux and Windows. This round ran macOS arm64 / Node 24.18.1. The author tested Linux / Node 22.21.1; the retry refused: true base-side nondeterminism above is the one place the platform difference may matter.
  • Full workspace build. The dependency build stops in unrelated workspaces — integrations/generated-ui (missing @jitl/quickjs-singlefile-browser-release-sync) and packages/web-shell (Rollup failed to resolve import "docx-preview"). CLI test prerequisites (core, acp-bridge, web-templates, all channels, generated git-commit.ts) were built directly; the failure is environmental and symmetric, not a PR regression.

Methodology

Worktree: git worktree add --detach /Users/wenshao/pr15482-rig/head origin/pr15482-head; the base source comes from merge-base 1eee20a7 via git show. git diff --stat 1eee20a7 87735a9a -- packages/cli/src/serve/hosted-harness-session.ts is empty, so the base tip and merge-base are the same control for the changed file. readlink -f /Users/wenshao/pr15482-rig/head/node_modules/@qwen-code/qwen-code-core points at /Users/wenshao/pr15482-rig/head/packages/core, so the arm resolves its own workspace rather than the parent checkout. Each cell restores the two PR files with git checkout HEAD -- before mutating.

Harness: tmp/pr15482-verify-20261025-163802/harness/run-cells.mjs runs npx vitest run src/serve/hosted-harness-session.test.ts -t '<literal>' --reporter=json --outputFile=<abs> --reporter=default from head/packages/cli, with all QWEN* variables removed from the child environment. Counts come from numPassedTests + numFailedTests; a cell with zero executed tests is recorded as HARNESS_ERROR, never as a survivor. Raw logs and per-cell JSON are under tmp/pr15482-verify-20261025-163802/logs/; the script asserts the nine outcomes in harness/assertions.mjs and writes assertions.json ({"pass": 9, "fail": 0, "total": 9}).

@jabrailkhalil

Copy link
Copy Markdown
Contributor Author

Thanks for the independent macOS check, @wenshao. I corrected the body and verification notes: both failures were observed in my Linux run; the duplicate-release branch is the deterministic witness in your controls, while the refused-retry base branch is scheduler-dependent. I also recorded the three non-blocking coverage gaps. The reviewed head stays bc689ee8.

@jabrailkhalil

Copy link
Copy Markdown
Contributor Author

Thanks for the careful review. I added two HTTP/journal regression scenarios for the three coverage gaps: the inner unsettled-input recheck, settled-promise clearing and successor queue ownership. All three previously surviving guard mutants now fail the intended assertions. The restored cleanup suite passes 29 cases, and final-head typecheck/style checks pass. The retry function is unchanged from the reviewed version; the body separates the earlier build and macOS results from this test follow-up. Current head: ef4703a24. Upstream CI is pending.

@github-actions

Copy link
Copy Markdown
Contributor

Managed Agent gate failed

Run 40190722511 · commit ef4703a241c2

Failing tests

  • com.alibaba.qwen.code.runtimebroker.JdbcPlacementGuardTest.incompleteSeedAndLeaseColumnsConservativelyBlockWithoutMappingHistoricalRows
  • com.alibaba.qwen.code.runtimebroker.WorkspaceMigrationRepositoryTest.historicalLookupKeepsOriginalScopeAndRejectsAmbiguity

…ed-cleanup-retry-serialization

# Conflicts:
#	packages/cli/src/serve/hosted-harness-session.test.ts
@jabrailkhalil

Copy link
Copy Markdown
Contributor Author

I pushed 2470302a0a with the current gate and resolved the journal-API test conflict while retaining the queue regressions. Fresh build, bundle, typecheck and all 29 selected cleanup cases pass. I traced the previous Java errors to heap exhaustion; the seven focused cases pass at that previous head without overrides. That does not clear the full Java gate—the new upstream run still needs to validate this head.

@jabrailkhalil

Copy link
Copy Markdown
Contributor Author

I pushed a6f26ee3be to bound ordinary Java test fixtures to a class-scoped JVM. The actual-schema probe confirmed H2 catalogs survive connection close and data-source GC. The two regression classes fail with fork reuse and pass without it; the IDE/CI-marker controls also pass. Production cleanup code is unchanged. Full Java CI, its runtime cost, and the separate one-second provider deadline still need validation; heap, timeouts and coverage were not relaxed.

@github-actions

Copy link
Copy Markdown
Contributor

Managed Agent gate failed

Run 40451552417 · commit 2470302a0a91

Failing tests

  • com.alibaba.qwen.code.runtimebroker.provisioning.ExternalRuntimeProvisionerTest.startingPollsUntilRunning
  • com.alibaba.qwen.code.runtimebroker.DispatchAuthorizationEvidenceTest.persistedMalformedEvidenceFailsDecode
  • com.alibaba.qwen.code.runtimebroker.DispatchAuthorizationEvidenceTest.alteredSchemaIsRefusedWithoutBackfillingAuthorizationEvidence
  • com.alibaba.qwen.code.runtimebroker.RuntimeMaintenanceRecoveryTest.maintenanceRequiresHolderCleanupAndNeverResolvesOrReprovisions(boolean)[2]

@wenshao
wenshao dismissed stale reviews from yiliang114 and themself via 0967b44 October 31, 2026 02:47
@qwen-code-dev-bot

Copy link
Copy Markdown
Collaborator

Qwen Code attempted to resolve merge conflicts but the run did not complete successfully.

address-summary.md

Root cause

#15871 (Maven module split) reduced managed-agent-server/pom.xml to an aggregator (<build> now only pluginManagement); the surefire block this PR edited moved to verification/pom.xml. It also moved src/test/** → verification/src/test/**, hence the location conflicts on the new H2ForkIsolation*Test.java. #14752 + #15935 deleted const SETTLE_TIMEOUT_MS from hosted-harness-session.test.ts for a per-test settleTimeoutMs().

Textual or semantic

Test file: both. The conflict was textual — the PR re-indented it('retains cleanup debt …') into it.each([false, true])(…); main changed one token inside it. Kept the PR's structure plus main's rename:

{ timeout: settleTimeoutMs() }

The rename reached past the conflict: the PR's two new tests auto-merged at ~L22836 still naming the deleted constant. All 11 sites renamed, else TS2304.

pom.xml: semantic — the PR's 3 lines sat in a <build> that no longer exists, so I took main's side verbatim. The .ts merged cleanly; main's #15881/#15857 edits there are E2E fault hooks, disjoint.

Load-bearing

  • settleTimeoutMs is set in beforeEach and returns the current test's remaining budget; calls must stay inside a test body.
  • retryHostedCleanup must read the prior promise before session.cleanupRetry = retry, and finally may clear the slot only if it still holds that promise, else a newer queued retry is dropped.

Not verified — action needed

The PR's <reuseForks>false</reuseForks> is dropped; its two tests fail until it is re-applied in verification/pom.xml's surefire <configuration> (~L128) — a non-conflicted file, so out of scope. Surefire defaults to reuseForks=true/forkCount=1, so both tests share a JVM: First creates class_owner in the fixed-name jdbc:h2:mem:test-class-isolation, then Second's assertFalse(tables.next(), …) fails. No build or tests were run.

中文说明

根因:#15871(Maven 模块拆分)把 managed-agent-server/pom.xml 变成聚合 POM,<build> 只剩 pluginManagement,本 PR 改的 surefire 段落迁到 verification/pom.xml;src/test/** 也移到 verification/src/test/**,故新增 H2ForkIsolation*Test.java 触发位置冲突。#14752 + #15935 删除了测试文件的 const SETTLE_TIMEOUT_MS,改为按用例的 settleTimeoutMs()。

文本还是语义:测试文件两者都有。冲突是文本性的 —— PR 把 it('retains cleanup debt …') 重缩进为 it.each([false, true])(…),main 只改了其中同一标识符。保留 PR 结构并套用 main 的重命名:

{ timeout: settleTimeoutMs() }

重命名不止于冲突区:PR 新增的两个用例在约 L22836 自动合并,仍引用已删除常量。共 11 处全部改名,否则报 TS2304。

pom.xml 是语义冲突 —— PR 的 3 行位于 main 上已不存在的 <build> 中,故按 main 一侧原样解决。.ts 干净合并:main 的 #15881/#15857 是 E2E 故障注入钩子,与之无交集。

关键约束:settleTimeoutMs 在 beforeEach 赋值、返回当前用例剩余预算,调用须留在用例体内。retryHostedCleanup 须先读上一个 promise 再赋值 session.cleanupRetry = retry,且 finally 只在槽位仍是同一 promise 时清空,否则丢弃后续排队的重试。

待办 / 未能验证:本次合并丢失了 PR 的 <reuseForks>false</reuseForks>,补回前它的两个测试会失败,应加在 verification/pom.xml 的 surefire <configuration>(约 L128);该文件无冲突,不在可改范围。surefire 默认 reuseForks=true/forkCount=1,两测试共用一个 JVM:First 在固定名 jdbc:h2:mem:test-class-isolation 建出 class_owner,Second 的 assertFalse(tables.next(), …) 随即失败。本次未跑构建或测试。

Check the workflow run for full logs.

@qwen-code-dev-bot

Copy link
Copy Markdown
Collaborator

Qwen Code did not run conflict resolution for this request.

PR #15482 does not currently have merge conflicts with main.

@qwen-code-review-bot

Copy link
Copy Markdown
Collaborator

Qwen Code review did not complete successfully. Qwen review exited with status 1. A transient error is retried automatically; if you are seeing this, retry with @qwen-code /review. See workflow logs.

@jabrailkhalil

jabrailkhalil commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor Author

I merged current main in 14d9a019, retaining 699ccfab. Author verification passes 689 Hosted Harness tests, both H2 isolation tests without skips, build/bundle, typecheck, scoped lint and reactor Checkstyle. Upstream lint and browser smoke pass (426 passed, 1 flaky). Maintainer review and remaining CI are pending.

@qwen-code-review-bot

Copy link
Copy Markdown
Collaborator

Qwen Code review timed out. Qwen review timed out after 21600 seconds (of the 360-minute budget). This run already used the maximum 360 minute timeout. See workflow logs.

@jabrailkhalil

Copy link
Copy Markdown
Contributor Author

I checked the remaining failure on 14d9a019: the review job hit the 360-minute limit after provider rate-limit retries. The build, lint, unit, Java and browser-smoke jobs passed. I have no completed review verdict from that run and am waiting for maintainer re-review. Review run.

wenshao added a commit to wenshao/qwen-code that referenced this pull request Nov 6, 2026
wenshao added a commit to wenshao/qwen-code that referenced this pull request Nov 6, 2026
@wenshao

wenshao commented Nov 6, 2026

Copy link
Copy Markdown
Collaborator

Verdict: merge-ready — 25/25 scripted assertions passed (0 failed). Verified head 14d9a01994bdc30318489a0a552e2f2b712788b8; base 7b4b35375071183b155089b111a6b3c887c3fe03 (GitHub base tip = merge-base); trial merge into current main 208beb2b3b clean at edba2eb7 (tree 28181480). Round 3 (follow-up to bc689ee8 / 3fd698a2), 2026-11-06 18:03–19:05 UTC. Environment: credential-free node:22-bookworm container (--network none), Node 24.22.1, Vitest 3.2.7, JDK 21.0.9, Maven 3.9.11, Surefire 3.5.6, JUnit 5.14.1, H2 2.4.240.

中文摘要

结论:merge-ready,25/25 条脚本断言通过。 验证 head 14d9a019,base 7b4b3537;对当前 main 208beb2b 的试合并无冲突(edba2eb7)。本轮是 track 1(bc689ee8)与 track 2(3fd698a2)报告的跟进轮。

  • 上一轮结论全部在新 head 上重新测量,无回退:三个“测试覆盖缺口已修复”的断言在新 head 上仍被各自变异击杀,且失败文案与上一轮一致(expected undefined to be Promise{…}、expected "release" to be called once, but got 2 times、expected Promise{…} to be undefined)。Java 的 <reuseForks> 仍承重:同一对类在 reuseForks=false 下 2/2 绿,在 true 下后运行者以 Another test class retained its H2 catalog in this JVM 失败。
  • 中心主张在最终落地树上成立:head 5/5 绿、merge 树 4/4 绿;base(只换回生产文件、保留 PR 测试)两轮均 4/4 红,确定性见证仍是重复 release:expected "release" to be called 2 times, but got 3 times。
  • 变异矩阵 5/5 全杀(序列化本身、.catch 隔离、内层 unsettled 复查、finally 清理、条件清理),另加 Java reuseForks 对照。
  • 代价已实测:8 个平凡测试类在 per-class fork 下 8,086 ms vs 复用 fork 3,620 ms,约 638 ms/类;模块里 431 个含 @Test 的类意味着全量单测多出约 4.6 分钟 JVM 启动下限(Spring context 重建未计)。作者已在正文承认该代价。
  • 门禁:完整 hosted-harness-session.test.ts 在本 head 上 690/690 通过(含本轮新增的四调用者格,297.6 s),上一轮因本地时限未跑完的项已补齐。
  • 未覆盖:真实 Runtime Broker/模型、真实 verification 模块(Java 用同版本 Surefire/JUnit/H2 的等价夹具)、本地 typecheck(引用 CI)、macOS/Windows 与 MySQL lane。
  • 更正上一轮描述的一处位置:<reuseForks> 在 3fd698a2 时位于 packages/managed-agent-server/pom.xml;main 之后的模块拆分把它移到了 packages/managed-agent-server/verification/pom.xml,父 pom 已无 Surefire 配置。本轮按新位置验证。

Previous-finding status

Prior rounds on this PR: comment #6395780548 (round 1, head bc689ee8, 9/9) and comment #6484615548 (round 2, head 3fd698a2, 20/20). Both ended merge-ready; round 1 carried three non-blocking coverage gaps, round 2 reported them fixed. All rows below were re-measured at 14d9a019, not carried over.

# Previous finding Severity Status at 14d9a019 Re-measurement
R1-1 Inner unsettledPromptId re-check in the queued link unpinned coverage gap fixed, still pinned Mutant M3 (delete the inner re-check) killed by rechecks an admitted prompt…: expected "release" to be called once, but got 2 times — same witness as round 2
R1-2 finally clear of session.cleanupRetry unpinned coverage gap fixed, still pinned Mutant M5b (session.cleanupRetry = undefined → void …) killed by two cells: expected Promise{…} to be undefined — same witness as round 2's M4
R1-3 Conditional (identity) clear unpinned coverage gap fixed, still pinned Mutant M2 (=== retry → true) killed by the three-caller and four-caller cells: expected undefined to be Promise{…} — same witness as round 2's M5
R2-J1 <reuseForks>false</reuseForks> is load-bearing config pin holds at the new pom location Same two PR classes, Surefire 3.5.6: false → 2 passed; true → 1 failure with Another test class retained its H2 catalog in this JVM ==> expected: <false> but was: <true>
R2-obs Base retry refused: true branch was timing-dependent in round 1 (macOS) observation not reproduced here At this head on Linux it failed in both base runs (2/2); the deterministic witness remains the false branch's 3-vs-2 release count

No previous finding stands or worsened.

What changed since round 2, and why the matrix was re-run

The delta 3fd698a2..14d9a019 is not test-only: hosted-harness-session.ts differs by 72 lines and its test file by 914 lines, because the branch merged main twice (7b4b3537 latest) after round 2 and main itself rewrote the same file (+83 lines since the PR base). The PR's own contribution is still the single 36-line hunk around retryHostedCleanup; the surrounding file changed. Per the follow-up rule, production changed, so every cell of the scenario matrix was re-run at the new head, including a fresh trial merge.

The Java change also moved: at 3fd698a2 the <reuseForks>false</reuseForks> line lived in packages/managed-agent-server/pom.xml; main has since split the verification module out, so at this head it lives in packages/managed-agent-server/verification/pom.xml alongside the two H2ForkIsolation*Test classes. The parent pom no longer configures Surefire.

Central claim and A/B

The PR serializes cleanup retries per attached Session: a queued caller chains onto session.cleanupRetry, re-reads unsettledPromptId and durable cleanup debt with its own broker options after the predecessor settles, and a failed attempt stays an error for its caller while leaving the next request free to retry. The behaviour it exists to change: a wake retry and a passive load must not both release the same original owner (three release calls where two are expected), and a late background refusal must not leave recovery blocked after a successful cleanup.

Each arm is a separate worktree at the exact revision; packages/cli resolves @qwen-code/* to that arm's sources (one generated alias list, 49 entries, from each package's exports), so no built dist/ sits between a mutant and the test.

Cell Production source Test file Executed Result
head focus head 14d9a019 head 5 5 passed (it.each both branches + prompt re-check + queue ownership + the round's 4-caller variant)
base focus (run 1) base 7b4b3537 head 5 5 failed as predicted: retry refused: false — expected "release" to be called 2 times, but got 3 times
base focus (run 2) base 7b4b3537 head 4 4 failed again; the true branch also red (2/2 here)
trial-merge focus merge edba2eb7 (main 208beb2b + head) merged 4 4 passed
Java head verification pom setting reproduced PR classes 2 2 passed, 0 skipped
Java control same, reuseForks=true PR classes 2 1 failed as predicted, retained-catalog assertion

A/B: cleanup-retry serialization on base vs head vs trial merge

The deterministic witness is the duplicate release of the same original owner on base. The four-caller cell (two refusals, four overlapping loads, three release calls, queue slot never lost) is this round's added setting; it fails on base (expected undefined to be defined — the mechanism is absent) and passes on head and on the trial merge.

Mutation matrix

Five production mutants at this head, each applied to the arm's source with a unique-anchor assertion and restored byte-for-byte from an in-memory copy (never git checkout). The unmutated head run (5/5) is the positive control; base focus is an independent red control.

Mutation Site Outcome Killing witness
M1 replace the hunk with the base implementation serialization itself KILLED all 5 cells; expected "release" to be called 2 times, but got 3 times
M2 === retry → true successor queue ownership KILLED queue-ownership + 4-caller: expected undefined to be Promise{…}
M3 delete inner if (unsettledPromptId(session)) return queued-caller re-check KILLED prompt re-check: expected "release" to be called once, but got 2 times
M4 delete .catch(() => undefined) predecessor failure must not poison the chain KILLED retry refused: true: expected "release" to be called 3 times, but got 2 times
M5b cleanupRetry = undefined → void … settled-promise clear KILLED prompt re-check + queue ownership: expected Promise{…} to be undefined
J1 <reuseForks>false</reuseForks> → true verification/pom.xml Surefire KILLED second-running H2 class: Another test class retained its H2 catalog in this JVM

Every kill names an intended assertion message, not a TypeError or fixture error. M1 is a known kill from both prior rounds and doubles as the harness control. No mutant regressed from killed to survived.

Mutation matrix and Java fork cost

The mutation matrix is per-guard, not per-commit: this round verifies the aggregate 7b4b3537..14d9a019 diff plus the trial merge, and makes no per-commit coverage claim.

Java fork isolation

The two PR classes reuse one H2 catalog name; under the old default (reuseForks=true, one JVM for the whole suite) the class that runs second sees the first class's table. Reproduced with the classes copied verbatim into a standalone Maven project that mirrors the module's Surefire shape (Spring Boot parent 3.5.16, Surefire 3.5.6, JUnit 5.14.1, H2 2.4.240, JDK 21): reuseForks=false → 2 passed; -Dreuse.forks=true → H2ForkIsolationFirstTest.startsWithoutAnotherTestClassCatalog:23 Another test class retained its H2 catalog in this JVM ==> expected: <false> but was: <true> (run order was Second→First; the pair is the witness, which class fails is order-dependent). The assumeTrue(surefire.test.class.path) gate opened in both runs, so the check really executed; under GITHUB_ACTIONS the same gate is a hard assertion.

Scope check on the sibling profile: fault-gates uses combine.self="override" and therefore drops reuseForks=false, but a grep enumeration of the 8 @Tag("fault-gate") classes finds none touching H2/DriverManager/jdbc:h2, so that lane has no H2-catalog exposure from this change. The mysql-integration Failsafe profile's explicit reuseForks=true runs MySQL, not H2, and is unchanged.

Cost, measured: 8 trivial test classes take 8,086 ms with per-class forks vs 3,620 ms with reuse (≈638 ms per class of JVM start). The verification module has 431 @Test-bearing *Test.java classes (441 files minus 10 helpers), so the JVM-start floor added to a full unit run is ≈4.6 min; Spring context reboots — the larger part the author names — were not measured here. The author's reported 59m35s fast-gate duration includes build and static checks.

Findings

None new, and no blocking finding carried forward. Two non-blocking observations:

  1. Queued callers inherit the predecessor's latency (bounded, by design). A passive load that used to start its own attempt now waits for the predecessor link. The Broker request carries AbortSignal.timeout(30_000) and reconcileHostedTurnCleanup releases owed cleanups in a loop, so a hung predecessor can hold a load for 30 s per owed cleanup. This is inherent to the serialization the PR chooses and the description does not claim otherwise — it is not a defect, but a reviewer comparing the two behaviours should know the load latency is now coupled to the wake path's.
  2. The "own authorization" half of the design is not distinguished by the tests (inferred from reading the fixtures, not measured). The tests assert each release keeps the original binding, generation and Runtime owner, but both callers in the fixture share the same broker authority options, so a mutant that reused the predecessor's options for the queued attempt would not be caught. The code passes each link's own options, so the property holds by construction; there is simply no cell that separates the two authorities.

Not covered

  • Real Runtime Broker, model, and provider — the TS tests mock the Broker, as the PR's own harness does; the Java proof uses H2 in-memory. No live broker or model call was made.
  • The real verification Maven module (full compile + Spring) was not built; the Java proof is a faithful Surefire/JUnit/H2 reproduction of the same classes and settings, not the module's own compile. No qwencode-sdk/sibling installation was performed.
  • Local typecheck / lint — not run; CI owns those. The round's TypeScript evidence is behavioural.
  • macOS, Windows, and the MySQL-backed Hosted/EventTransport lanes — not run; this round is Linux only.
  • Per-commit attribution — the clone's history is shallow around 14d9a019's merges; the round verifies the aggregate effective diff and the trial-merge tree, not each commit.

Methodology

Isolation: every arm command ran inside docker run --rm --init --network none --user <uid>:<gid> with env -i, a round-local HOME/TMPDIR, the repository mounted read-only except the round directory, and the host JDK 21/Maven mounted read-only; scripts/verify-pr-prepare.mjs --require-isolation printed /.dockerenv; credential-free per attestation, no known credential signal (exit 0). Three worktrees: head-tree at 14d9a019, base-tree at 7b4b3537 (with the head test file copied in for the control), and merge-tree at edba2eb7, a commit created from git merge-tree --write-tree --merge-base=7b4b3537 208beb2b 14d9a019 (tree 28181480, no conflicts) with parents 208beb2b + 14d9a019. TypeScript cells: node node_modules/vitest/vitest.mjs run src/serve/hosted-harness-session.test.ts --testNamePattern <pattern> from each arm's packages/cli, with a generated vitest.config.ts that drops the dist global-setup guard, disables coverage, points cacheDir into the round, and aliases 49 @qwen-code/* specifiers to that arm's sources (mapping taken from each package's exports; channelControlTimeouts → src/channel-control-timeouts.ts). The merge arm's test file is the merged one (main's edits included), so the trial-merge cell is the landing state, not a copy. Mutation runner: harness/mutate-ts.mjs (in-memory pristine bytes, single-anchor assertion, restore + byte comparison); M5b ran on the merge tree because the head tree was busy with the full-file run, and the function is byte-identical there (checked by diff of retryHostedCleanup, with the merged test file differing at byte 3343 as the positive control that the comparison can see a difference). Java: harness/java-run.sh with -o -Dmaven.repo.local=<round>/tmp/m2 (hardlink seed of ~/.m2/repository, 1.5 GB, 0.2 s) and -Dtest='H2ForkIsolation*Test'; classes copied verbatim from the head tree. Raw logs, per-cell output, and harnesses: this directory's logs/ and harness/; assertions.json and mutations.json are generated from those logs.

— DeepSeek-V4.1-Flash-1017@fda99a87 via Qwen Code /verify-pr (v0.26.8; skill c5273cc80df5)

@wenshao

wenshao commented Nov 6, 2026

Copy link
Copy Markdown
Collaborator

Update — mergeability moved after the report above. Head is unchanged (14d9a019), so every measurement in the round-3 report still describes this head. What changed is main: it advanced from 208beb2b (clean trial merge, reported above) to 11c83aa21efc2c103927bdb34e321f3382c61308, which rewrote the same production file again (+213 lines since 208beb2b).

A fresh three-way merge against the new tip now conflicts:

git merge-tree --write-tree --name-only --merge-base=7b4b3537 11c83aa2 14d9a019
exit=1
CONFLICT (content): Merge conflict in packages/cli/src/serve/hosted-harness-session.ts

The test file and verification/pom.xml auto-merge. So the merge-ready verdict stands for the code as verified, but the PR needs a re-sync with main before it can land, and the resolution touches the file the A/B is about — after the re-sync I will re-run the cleanup-retry cells and the trial merge at the new head rather than carry this round's numbers over.

中文:head 未变(14d9a019),上轮报告的证据仍然只对这一个 head 成立;main 已前进到 11c83aa2,与生产文件 hosted-harness-session.ts 产生真实冲突(测试文件与 pom 可自动合并)。结论对代码本身不变,但需要先与 main 重新同步;同步后我会在新 head 上重跑 A/B 与试合并,不沿用本轮数字。

— DeepSeek-V4.1-Flash-1017@fda99a87 via Qwen Code /verify-pr (v0.26.8; skill c5273cc80df5)

# Conflicts:
#	packages/cli/src/serve/hosted-harness-session.ts
wenshao added a commit to wenshao/qwen-code that referenced this pull request Nov 7, 2026
wenshao added a commit to wenshao/qwen-code that referenced this pull request Nov 7, 2026
@wenshao

wenshao commented Nov 7, 2026 •

Copy link
Copy Markdown
Collaborator

Verdict: merge-ready — 20/20 scripted assertions passed (0 failed). Verified head 890429b2ac3f16cac473440163c4c19f7da66345; base 0a562773f76140e6c62a2c48a7e2eb806c120baf. The head is the landing tree: its second parent is the current main tip 0a562773, and base..head is exactly the five PR files (419+/116−, no conflict markers). Round 4 (follow-up to 14d9a019, which superseded the merge-conflict report). Environment: credential-free node:22-bookworm container (--network none), Node 24.22.1, Vitest 3.2.7, JDK 21.0.9, Maven 3.9.11, Surefire 3.5.6, JUnit 5.14.1, H2 2.4.240.

Post-merge check — squash fd3324b7: the PR merged with squash fd3324b7cd48b4a8616d71915ff179f73fa7f262 (2026-11-07 14:28 UTC). All five PR files are byte-identical between the verified head 890429b2 and that squash; a file main added after the verified head reports differ in the same comparison, so the match is meaningful. The squash tree also carries later main drift in the verification module (journal-snapshot tests, class-size ratchet) that is outside this PR's files. Nothing is left to follow up on this round.

中文摘要

结论:merge-ready,20/20 条脚本断言通过。 验证 head 890429b2,base 0a562773;head 本身就是落地树——它的第二父提交正是当前 main tip 0a562773,base..head 恰好是 PR 的 5 个文件(419+/116−,无冲突标记),因此不再单独构造试合并臂。

  • 此前的冲突已由作者解决且解冲突保留了双方语义:main 已把 retryHostedCleanup 改成带 drain 参数、cleanupHolds 与 retainHostedCleanupHolds 的版本;新 head 的 PR hunk 把整个新函数体包进 session.cleanupRetry 链(内层 unsettled 复查也带上 !drain),不是旧版本的机械回放。
  • 中心主张在新落地树上重新成立:base 5/5 红(expected "release" to be called 2 times, but got 3 times),head 5/5 绿;四调用者格(两拒后成功、3 次 release、槽位不丢)仍承重。
  • 变异 5/5 全杀:序列化本身、条件清理、内层 unsettled 复查、.catch 隔离、finally 清理;失败文案与上一轮一致。
  • Java <reuseForks> 仍承重:同两个 H2 类 false 下 2/2 绿、true 下后运行者以 Another test class retained its H2 catalog in this JVM 失败;8 个平凡类 per-class fork 代价约 598 ms/类。
  • 门禁:完整 hosted-harness-session.test.ts 在本 head 712/712 通过(main 合并后新增了用例)。
  • 落地:PR 已以 squash fd3324b7 合并;5 个 PR 文件与已验证 head 890429b2 逐字节一致(正对照能看到 main 新增文件的差异),合并树多出的只是 main 之后的 drift,不在 PR 文件内。
  • 未覆盖:真实 Runtime Broker/模型、真实 verification 模块编译(Java 为等价夹具)、本地 typecheck/CI、macOS/Windows 与 MySQL lane。

Previous-finding status

Newest prior verdict comment: round 3 at 14d9a019 (25/25), plus the later merge-conflict addendum. All rows re-measured at 890429b2; nothing is carried over.

# Previous finding Severity Status at 890429b2 Re-measurement
R3-1 Trial merge was clean at 208beb2b; then main 11c83aa2 conflicted in hosted-harness-session.ts merge condition fixed (author re-synced) Head's second parent is main 0a562773; git diff 0a562773 890429b2 is 5 files, no conflict markers, and the head tree is the landing state
R1-1 Inner unsettledPromptId re-check unpinned coverage gap fixed, still pinned M3 (delete if (!drain && unsettledPromptId(session)) return;) killed by the prompt re-check cell: expected "release" to be called once, but got 2 times
R1-2 finally clear of session.cleanupRetry unpinned coverage gap fixed, still pinned M5b (= undefined → void …) killed by 3 cells: expected Promise{…} to be undefined
R1-3 Conditional (identity) clear unpinned coverage gap fixed, still pinned M2 (=== retry → true) killed by the queue-ownership and 4-caller cells: expected undefined to be Promise{…}
R2-J1 <reuseForks>false</reuseForks> is load-bearing config pin holds false → 2 passed; true → 1 failure with the retained-catalog assertion
R3-2 Queued callers inherit the predecessor's latency (30 s per owed cleanup) observation unchanged (by design) Same code shape; no new evidence, still non-blocking

What changed since round 3

14d9a019 → 890429b2 is a merge of main 0a562773; head's parents are exactly 14d9a019 and 0a562773. The PR's own hunk is now applied to a newer retryHostedCleanup that main had extended:

  • the function takes drain = false, and both unsettled-prompt checks are !drain && …;
  • the body clears session.cleanupHolds and calls retainHostedCleanupHolds(await reconcileHostedTurnCleanup(…, drain)).

The head wraps that new body in the serialization chain ((session.cleanupRetry ?? Promise.resolve()).catch(...).then(...), plus the identity-guarded finally), so the conflict resolution kept both sides instead of reverting main's additions. That is why the whole scenario matrix was re-run rather than carried over: production source changed even though the PR hunk text is the same idea.

Central claim and A/B

The PR serializes cleanup retries per attached Session: a queued caller chains onto session.cleanupRetry, re-reads unsettledPromptId and durable cleanup debt with its own broker options after the predecessor settles, and a failed attempt stays an error for its caller while leaving the next request free to retry. The behaviour it exists to change: a wake retry and a passive load must not both release the same original owner (three release calls where two are expected), and a late background refusal must not leave recovery blocked after a successful cleanup.

Cell Production source Test file Executed Result
head focus head 890429b2 (= main + PR) head 5 5 passed (it.each both branches + prompt re-check + queue ownership + 4-caller variant)
base focus base 0a562773 head 5 5 failed as predicted: retry refused: false — expected "release" to be called 2 times, but got 3 times
whole file head head 712 712 passed, 0 failed
Java head verification pom setting reproduced PR classes 2 2 passed, 0 skipped
Java control same, reuseForks=true PR classes 2 1 failed as predicted, retained-catalog assertion

A/B on the landing tree: base vs head

The deterministic witness is the duplicate release of the same original owner on base. The 4-caller cell (two refusals, four overlapping loads, three release calls, queue slot never lost) remains a setting the author's three tests do not cover; it fails on base (expected undefined to be defined — the mechanism is absent) and passes on the landing tree.

Mutation matrix

Five production mutants, each applied with a unique-anchor assertion and restored byte-for-byte from an in-memory copy. The unmutated landing-tree run (5/5) is the positive control; the base cell is an independent red control.

Mutation Site Outcome Killing witness
M1 revert the hunk to base serialization itself KILLED all 5 cells; duplicate release 3 vs 2
M2 === retry → true successor queue ownership KILLED queue ownership + 4-caller: expected undefined to be Promise{…}
M3 delete the inner !drain && unsettledPromptId re-check queued-caller re-check KILLED prompt re-check: expected "release" to be called once, but got 2 times
M4 delete .catch(() => undefined) predecessor failure must not poison the chain KILLED retry refused: true: expected "release" to be called 3 times, but got 2 times
M5b cleanupRetry = undefined → void … settled-promise clear KILLED 3 cells: expected Promise{…} to be undefined
J1 <reuseForks>false</reuseForks> → true verification/pom.xml Surefire KILLED second-running H2 class: Another test class retained its H2 catalog in this JVM

Every kill names an intended assertion message, not a TypeError or fixture error. The matrix is per-guard, not per-commit: this round verifies the aggregate 0a562773..890429b2 diff, and makes no per-commit coverage claim.

Mutation matrix and Java fork cost

Java fork isolation

The two PR classes are byte-identical to the previous round; the verification/pom.xml hunk is still present at line 180, with main's new argLine/plugin-management changes merged around it. Reproduced with the classes copied verbatim into a standalone Maven project mirroring the module's Surefire shape (Spring Boot parent 3.5.16, Surefire 3.5.6, JUnit 5.14.1, H2 2.4.240, JDK 21): reuseForks=false → 2 passed; -Dreuse.forks=true → H2ForkIsolationFirstTest.startsWithoutAnotherTestClassCatalog:23 Another test class retained its H2 catalog in this JVM (run order was Second→First; which class fails is order-dependent). The assumeTrue(surefire.test.class.path) gate opened in both runs, so the check executed.

Cost, re-measured at this head: 8 trivial classes take 7,917 ms with per-class forks vs 3,733 ms with reuse (≈598 ms per class of JVM start); the module's 431 @Test-bearing classes put the JVM-start floor around 4.3 min. Spring context reboots are not measured here.

Findings

None new, and no blocking finding carried forward. The round-3 observation about queued-caller latency still holds: a passive load can now wait behind the wake path for up to 30 s per owed cleanup (AbortSignal.timeout(30_000) in the Broker request). It is inherent to the serialization and bounded; not a merge condition.

One correctness note on the resolution itself: it is a manual conflict resolution, not a mechanical merge, so there is no tree-hash equality to cite. The evidence is behavioural — the head keeps main's drain/cleanupHolds body inside the PR's chain, and all five cells plus the whole 712-test file pass on that tree.

Not covered

  • Real Runtime Broker, model, and provider — the TS tests mock the Broker, as the PR's own harness does; the Java proof uses H2 in-memory. No live broker or model call was made.
  • The real verification Maven module (full compile + Spring) was not built; the Java proof is a faithful Surefire/JUnit/H2 reproduction of the same classes and settings, not the module's own compile.
  • Local typecheck / lint — not run; CI owns those.
  • macOS, Windows, and the MySQL-backed Hosted/EventTransport lanes — not run; this round is Linux only.
  • Per-commit attribution — the round verifies the aggregate effective diff and the landing tree, not each of the PR's commits.

Methodology

Isolation: every arm command ran inside docker run --rm --init --network none --user <uid>:<gid> with env -i, a round-local HOME/TMPDIR, the repository mounted read-only except the round directory, and the host JDK 21/Maven mounted read-only. Two worktrees: head-tree at 890429b2 (the landing tree) and base-tree at 0a562773 with the head test file copied in for the control. TypeScript cells: node node_modules/vitest/vitest.mjs run src/serve/hosted-harness-session.test.ts --testNamePattern <pattern> from each arm's packages/cli, with a generated vitest.config.ts that drops the dist global-setup guard, disables coverage, points cacheDir into the round, and aliases 50 @qwen-code/* specifiers to that arm's sources (mapping taken from each package's exports). The 4-caller cell is the same harness-added test as round 3, inserted into both arms' test files. Mutation runner: harness/mutate-ts.mjs (in-memory pristine bytes, single-anchor assertion, restore + byte comparison); M5b was run separately in the head tree with the same discipline. Java: harness/java-run.sh with -o -Dmaven.repo.local=<round>/tmp/m2 (hardlink seed) and -Dtest='H2ForkIsolation*Test'. Raw logs and harnesses: logs/ and harness/; assertions.json and mutations.json are generated from those logs.

— DeepSeek-V4.1-Flash-1017@fda99a87 via Qwen Code /verify-pr (v0.26.8; skill c5273cc80df5)

@qqqys qqqys left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at head 890429b2ac3f16cac473440163c4c19f7da66345. No historical blocker stands: there are zero inline review threads, the only two reviews are both dismissed LGTMs, and no sev:"C" was ever filed. The maintainer's round-4 verification comment at this exact head reports merge-ready with 20/20 assertions and "no blocking finding carried forward". I read the production diff and censused the new state myself. No blocking finding.

The serialization, verified

The fix moves retryHostedCleanup's body into a promise chained onto the previous caller's, so overlapping callers queue instead of racing (hosted-harness-session.ts:2582-2625). Four properties matter and each holds:

  • Each caller re-reads state with its own authority. The unsettledPromptId(session) guard is repeated inside the chained callback, not only at entry, and the body closes over that caller's options. So a caller that queued behind a predecessor does not act on a debt check that was true when it arrived, and does not release using someone else's broker authority. That is the defect the change exists to remove — the base tree produces three release calls where two are expected.
  • A predecessor's failure cannot poison the chain, but is still an error for its own caller. .catch(() => undefined) is applied to the predecessor promise only; the inner catch rethrows cause, so await retry rejects for the caller that actually failed. Successors proceed, the failure is not swallowed for whoever it belongs to.
  • Slot ownership is compare-and-clear, not blind. The finally clears the slot only when session.cleanupRetry === retry (:2621-2622), so an older caller settling late cannot erase a newer caller's queue entry. This also makes the re-entrant path safe: the failure branch calls session.monitorWake?.kick() before rethrowing, and if that leads to another retryHostedCleanup while this one is still rejecting, the new call overwrites the slot, the old finally then sees a different promise and leaves it alone, and the new caller proceeds once the old rejects. No lost entry and no deadlock.
  • The wait is bounded. A queued caller can wait behind the wake path for up to the Broker request's AbortSignal.timeout(30_000) per owed cleanup. The maintainer records this as inherent to serialization and bounded, not a merge condition, and I agree — it is latency, not a hang, and it replaces a duplicate release.

The new field has no other reader. cleanupRetry appears at exactly five places in the 13,410-line file: the HostedSession declaration (:765), the chain read (:2587), the slot claim (:2617) and the compare-and-clear (:2621-2622). Nothing else in the module observes it, so there is no unmigrated consumer reading a queue slot under an unchanged name. retryHostedCleanup itself has ten call sites (:2546, :3855, :5437, :6753, :7054, :7802, :8231, :8278, :9850, :12999 — the last with drain: true), all of which await it, so the serialization covers every path that can retry and each caller still sees its own rejection.

The Maven change, verified

<reuseForks>false</reuseForks> (verification/pom.xml:180) sits in the default maven-surefire-plugin configuration (:172-180), not behind a -D user property, so it is authoritative rather than a flag a POM element could silently override. Two adjacent details are right: <argLine>@{argLine} -XX:MaxRAMPercentage=40</argLine> is untouched, so the late-bound JaCoCo agent still reaches every fork; and no <forkCount> is added, leaving the default of 1, which with reuseForks=false gives exactly one fresh JVM per test class — the isolation the two new H2 witnesses exist to prove. The mysql-integration profile's own <forkCount>${mysql.integration.forks}</forkCount> and <reuseForks>true</reuseForks> (:273-274) belong to a different plugin configuration and are unaffected.

CI

Fully settled and clean at this head: 37 pass, 18 skipping, zero failing and zero pending. That is worth stating because a per-class JVM fork is the kind of change that shows up as a timeout rather than a failure — the lane that runs the verification module completed inside its budget on a green run, so the cost is measured rather than estimated.

Non-blocking

  • The reuseForks=false applies to the verification module's whole default surefire run, not only to the two H2 fixture classes, so every test class there now pays JVM startup. CI absorbs it today; if that module's suite grows, this is the first place a timeout will appear. A scoped alternative would be isolating the H2 catalogs by name per test class instead of per JVM.
  • The dismissed LGTM's two P2s both still hold and neither blocks: there is no fixes/closes keyword, so the related issues will not auto-close on merge, and the Java fork-isolation work is bundled with a TypeScript service fix. The bundling is defensible — the H2 retention is what made the module's suite order-dependent — but it does mean two unrelated revert scopes share one commit.
  • The head is a merge whose resolution the maintainer describes as manual rather than mechanical, so there is no tree-hash equality to cite for it. The behavioural evidence is that main's drain/cleanupHolds body survives inside the PR's chain and the whole 712-test file passes on this tree; I confirmed the same by reading the merged body above rather than by comparing hashes.

Coverage

Read directly at head: the whole retryHostedCleanup rewrite and its surrounding function, the HostedSession field addition, a full census of cleanupRetry and of retryHostedCleanup's call sites, and the verification pom's surefire and failsafe configuration blocks. Not read line by line: the 318 changed lines of hosted-harness-session.test.ts and the two 30-line Java witnesses, whose behaviour I take from the maintainer's reproduced A/B and mutation runs rather than from re-executing them. The production surface of this PR is 41 changed lines plus 3 in the pom, and I read all of them.

@wenshao
wenshao merged commit fd3324b into QwenLM:main Nov 7, 2026
85 of 86 checks passed
wenshao added a commit that referenced this pull request Nov 7, 2026
One conflict, in the capturedWakeSession() structural type: both sides added a
field, ours `hooks?: HostedHookSession` for the R22-1 witness and main's
`cleanupRetry?: Promise<void>` from #15482. They are orthogonal, so both stay.

hosted-harness-session.ts auto-merged, which is the shape that produced R22-1,
so it was audited rather than assumed: #15482 only serializes the inside of
retryHostedCleanup and moves no call site, while R22-1 moves a call site. The
v2 order is intact (retireOwedFileHistory -> drain -> hooks.close) and the
retry body is byte-identical to main's.

Verified: build rc0 (type-checks the resolved test file), Prettier clean,
hosted-harness-session 715/715, hosted-hook-session 139, hosted-mcp-session 69.
@wenshao

wenshao commented Nov 7, 2026

Copy link
Copy Markdown
Collaborator

CI attribution — both failed checks are inherited from main, not from this PR. Head 890429b2; run 42375266747 (pull_request_review). Its checkout is a synthetic merge c15da3d of this head into main 0807f5d3 (from the job's own checkout log), so the lanes below ran against that main, not against the head tree alone.

  1. SDK Java (full) / Runtime Broker and Managed Agent MySQL / Java 21 — the only failure is ClassSizeRatchetTest.productionClassesStayWithinTheirSizeRatchet:
    RuntimeBindingProvisioning grew to 1327 lines, above its ratchet ceiling 1326 and RuntimeBrokerService grew to 2947 lines, above its ratchet ceiling 2943.
    At the checkout's main 0807f5d3 the ratchet file pins 1326/2943 and the two classes are 1327/2947 — the violation lives on main. This PR's five files (serve TS, its test, the verification pom and two H2 tests) touch neither. Corroboration: main itself failed at 0b54b9be (the squash parent, run 42381475547, "The fast tier failure"); a dedicated branch fix/runtime-binding-provisioning-ratchet is in flight; current main d0b80d4a has both classes at the ceilings (1326/2943), so main has since fixed it.

  2. SDK Java (full) / Hosted process fault gates / MySQL 8.4 / Java 21 — three failures: HostedWorkspaceToolTurnIT.lostBrokerRepliesNeverReplayEffectsOnMySql (the driver's cleanupRetries accepts only a plain release; the observed retry is release with drain: true after release-mount returns runtime_reconciliation_required), HostedWorkspaceToolTurnIT.recordsLatencyWithDelayedRuntimeProvisioning (Timed out after 30000ms), and HostedPublicWorkspaceIT.publicChildQuestionsAllowTheSecondChildToAnswerFirst (Hosted Workspace Shell requires Workspace files and default or auto-edit approval mode).
    The same job on the unrelated PR fix/a2a-approval-test-unhandled-rejection (head ca34c569, run 42397627763, job 128490944575) fails the same three tests verbatim, including the same drain: true release sequence. None of those fixtures is in this PR's diff.

The route check is a CANCELLED job from the Qwen Autofix workflow with no steps run; it is not a test. So there is nothing to fix in this PR: both reds come from the main revision the CI merged into (ratchet violation) and from the Hosted fixture drift already failing on unrelated PRs. Re-running the two lanes on current main, after the main-side ratchet and fixture fixes, is the path to green.

中文:这两个失败检查都不是本 PR 引入的。MySQL lane 只红在 ClassSizeRatchetTest(RuntimeBindingProvisioning 1327 > 1326、RuntimeBrokerService 2947 > 2943),而 CI 实际签出的是本 head 合入 main 0807f5d3 的合成合并提交;该 main 上 ratchet 上限就是 1326/2943、类也是 1327/2947,本 PR 的 5 个文件完全不碰它们。旁证:0b54b9be(squash 父提交)上 main 自身的 fast tier 就红,仓库里另有 fix/runtime-binding-provisioning-ratchet 分支在修,当前 main d0b80d4a 已把两个类降回 1326/2943。Hosted fault gates 的三条失败在无关 PR ca34c569 的同名 job 上逐字复现(含同样的 drain: true release 序列),属于 Hosted 夹具漂移。route 是 Qwen Autofix 里被取消、一步都没跑的路由 job,不是测试。结论:本 PR 无需改动,等 main 侧 ratchet 与夹具修好后重跑两条 lane 即可。

— DeepSeek-V4.1-Flash-1017@fda99a87 via Qwen Code /verify-pr (v0.26.8; skill c5273cc80df5)

@qwen-code-review-bot

Copy link
Copy Markdown
Collaborator

Released in v0.27.0.

@github-actions

github-actions Bot commented Nov 8, 2026

Copy link
Copy Markdown
Contributor

Managed Agent gate failed

Run 42375266747 · commit 890429b2ac3f

Failing tests

  • com.alibaba.qwen.code.managedagent.HostedWorkspaceToolTurnIT.lostBrokerRepliesNeverReplayEffectsOnMySql
  • com.alibaba.qwen.code.managedagent.HostedWorkspaceToolTurnIT.recordsLatencyWithDelayedRuntimeProvisioning
  • com.alibaba.qwen.code.managedagent.HostedPublicWorkspaceIT.publicChildQuestionsAllowTheSecondChildToAnswerFirst
  • com.alibaba.qwen.code.managedagent.architecture.ClassSizeRatchetTest.productionClassesStayWithinTheirSizeRatchet

Tracking issue

@github-actions

github-actions Bot commented Nov 8, 2026

Copy link
Copy Markdown
Contributor

Managed Agent gate failed

Run 42381726935 · commit 890429b2ac3f

Failing tests

  • com.alibaba.qwen.code.managedagent.HostedWorkspaceToolTurnIT.lostBrokerRepliesNeverReplayEffectsOnMySql
  • com.alibaba.qwen.code.managedagent.HostedWorkspaceToolTurnIT.recordsLatencyWithDelayedRuntimeProvisioning
  • com.alibaba.qwen.code.managedagent.HostedPublicWorkspaceIT.publicChildQuestionsAllowTheSecondChildToAnswerFirst
  • com.alibaba.qwen.code.managedagent.architecture.ClassSizeRatchetTest.productionClassesStayWithinTheirSizeRatchet

Tracking issue

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants