Skip to content

fix(managed-agent): bound retry loops with terminal states - #13219

Merged
wenshao merged 94 commits into
mainfrom
fix/13182-retry-terminal-states
Oct 16, 2026
Merged

wenshao merged 94 commits into
mainfrom
fix/13182-retry-terminal-states

Conversation

@wenshao

@wenshao wenshao commented Oct 2, 2026 •

Copy link
Copy Markdown
Collaborator

What this PR does

Across the managed-agent stack, asynchronous retry loops had no terminal state, and in one case a permanently wedged projection. This change gives every loop a budget and a terminal state: the message projection heals past a provably permanent sequence gap instead of throwing on every scan — settling as failed any items whose turn's terminal event was in the skipped range — and the scanner itself backs off per session (exponential to a 60s cap, surfaced loudly once) so a deterministic failure can no longer flood the log at 10Hz; admitted turns converge to a bounded terminal failure once their retry budget is spent; lifecycle operations whose settle can never succeed terminate failed with the settle's failure code kept (the Session keeps its pending status — nothing certifies a completion that never happened) — except while a live journal writer means the close can still succeed, which keeps waiting; a blocked workspace close keeps its failure code through every attempt and through the terminal record; a failed action response whose Action is still waiting is re-admitted by the caller's retry under the same idempotency key instead of returning a stale failure receipt forever; a memory-blocked embedded scheduler arms its own recheck wake so a missed capacity notification can no longer starve queued work; transient store IO failures are retried within the lease window (renewals) or retried bounded before abandoning (releases), a consecutive-failure streak halts the worker loudly when a "transient" outage is really persistent, and a per-activation budget records a terminal outcome instead of re-running forever; session close seals the writer in a finally; and a failed HTTP seal throws loudly out of that finally while the grant lapses with the lease, so a takeover can proceed. There is deliberately no seal retry (issue #13182 item 6b): keeping the grant renewed was found in review to let a stale client seal the journal out from under a freshly re-attached session, so the bounded-retry cut was withdrawn and 6b is answered by the loud throw plus the grant lapse.

Why it's needed

Audit of the retry loops (issue #13182) found: a single event-sequence gap wedged a session's projection forever while the scanner logged the same stack at 10Hz; an admitted turn failing transiently was rescheduled without bound; a close/delete whose settle can never succeed retried forever; a missed capacity notification starved queued activations silently; one transient renewal error bricked the embedded scheduler; and a failed close could leave a writer unsealed, blocking both clean resume and takeover. The fail-closed posture of the session authority is correct — the layers around it treated transient IO failures as indistinguishable from consistency damage. Review of the first cut found the remaining gaps this wave closes: the exhaustion arm used the success-path completion write (erasing the blocked failure code and certifying an unsettled close), the gap heal left orphaned items streaming forever, the materializer's other permanent failure modes still spun at 10Hz, the scheduler's transient retry had no escalation bound, the first cut's renewal-keeping seal retry let a stale client seal the journal under a re-attached session (withdrawn in favor of the loud throw), and a budget-terminated action answer could never be re-delivered.

Reviewer Test Plan

How to verify

The affected paths are not reachable from the CLI binary (the Java control plane needs a deployed broker; the embedded scheduler is hardened ahead of its first production wiring), so the evidence is the new regression suites, each verified to fail on the pre-fix code and pass after:

  • packages/sdk-java/managed-agent-server: mvn test — 2096 tests; the full suite is green except two pre-existing ToolPublicationStoreTest timing errors that reproduce identically on base, and one ChildWorktreeGitTest non-UTF8-name case that fails only on macOS's sh (identical on main). Key suites: MessageProjectionGapHealTest (real H2 + real transaction boundary: a gapped projection heals, the gap is logged once naming only the truly missing sequences, a lost turn.completed settles its orphaned items as failed, and a second-connection FOR UPDATE witness pins that the scan holds the session row lock the permanence premise rests on), MessageMaterializerTest (backoff + surface-once + reset on success), AdmittedTurnRetryTerminalStateTest (budget boundary 9 retries / 10 terminates, configured budgets honored, both the daemon-503 and broker arms), OperationRetryTerminalStateTest (budget boundary, configured budgets, the committed-decision arm, the live-writer gate, the blocked-code-preserving terminal, the bound-session workspace-close routing, the terminal-record-failure fallback), ManagedActionStoreTest (stage invariants + re-admission of a FAILED response while the Action is still requested), SessionLifecycleCoordinatorTest (a blocked close stays blocked below the budget).
  • packages/core: npx vitest run src/managed-runtime — 3281 tests (one known under-load-only flake that passes in isolation). Coverage includes journal tail repair fault injection (torn write repaired, unrepairable write halts, first-write ENOENT is transient, vanished journal is fatal), the scheduler's claim/renewal/release retry policies, the transient-streak halt, per-activation budget convergence, dispose-race wake guards, the memory-blocked poll's decelerating cadence (one store scan per tick, doubling from 1s to a 30s cap, with a cancelled-queue tick clearing the block), and the seal failure path's loud throw with the grant left to lapse.
  • npm run build && npm run bundle and npm run typecheck pass; packages/web-shell OpenAPI contract sync test passes (the generated client was regenerated after the contract description update).

Evidence (Before & After)

N/A — no user-visible UI change.

Tested on

OS Status
🍏 macOS ✅
🪟 Windows ⚠️
🐧 Linux ⚠️

Environment (optional)

Unit/integration-level verification only (H2 in-memory for the Java broker, fault-injection fs mocks for the TS store); the hosted paths are covered by the suites above rather than a live broker.

Risk & Scope

  • Main risk or tradeoff: a turn/operation/action that keeps failing transiently now reaches a terminal failure instead of retrying forever — that is the intended change; budgets are configurable (dispatch.max-post-admission-retries, dispatch.max-operation-retries, both default 10) and validated at startup. A lifecycle operation that hits the budget now reads failed with a failure code instead of completed — the more honest answer, and the one the contract text now describes.
  • Not validated / out of scope: the embedded scheduler has no production caller yet (hardened ahead of wiring, disclosed in test comments); the MySQL-backed *IT integration tests run in CI's Hosted process fault gates lane (locally they need Docker/MySQL); the materializer backoff is in-memory (a persisted terminal consumer-progress state is a possible follow-up if operators need it to survive restarts).
  • Breaking changes / migration notes: the startup validation now rejects max-pre-admission-retries greater than max-post-admission-retries — a combination that was accepted before (and silently stripped late-admitted turns of retries). Deployments with that combination in external config must align the two keys before upgrading; the error message names both values. Otherwise: two new optional config keys with safe defaults; one new turn failure code; new lifecycle/action failure codes (free-form string fields, no enum change).

Linked Issues

Fixes #13182

中文说明

本 PR 做了什么

managed-agent 栈中的异步重试环路此前没有终态,其中一处会让投影永久卡死。本次改为给每个环路预算与终态:消息投影在遇到可证永久的事件序号缺口时自愈越过——并把终结事件落在缺口区间的 turn 的在途 item 置为 failed——不再每次扫描都抛错,扫描器自身也按 session 退避(指数退避封顶 60 秒,超预算后响亮上报一次),确定性失败不再以 10Hz 刷日志;已受理的 turn 在重试预算耗尽后收敛到有界终态;settle 永远无法成功的生命周期操作以 failed 终结并保留 settle 的失败码(Session 保持待定状态——不再为从未发生的完成出具凭证)——除非 journal writer 仍存活,此时 close 仍可能成功,operation 继续等待;被阻塞的 workspace close 在每次尝试与终态记录中都保留其失败码;Action 仍在等待应答的失败 action response 可被调用方按同一幂等键重试重新受理,而不是永远返回陈旧的失败回执;内存不足的嵌入式调度器会自行武装复检唤醒,错过容量通知不再让排队激活永久饥饿;瞬时存储 IO 失败在租约窗口内重试(续约)或有界重试后再放弃(释放),连续失败计数会在"瞬时"故障实为持续故障时响亮地 halt worker,按激活的预算则记录终态而不是无限重跑;会话关闭在 finally 中保证封存 writer;HTTP 封存失败会从该 finally 中响亮抛出,并让授权随租约失效,使接管得以进行。此处刻意不做封存重试(issue #13182 第 6b 项):评审发现保持续约会让陈旧客户端在新会话重新挂载后从其手中封存 journal,因此有界重试方案被撤回,6b 由响亮抛错加授权失效来回答。

为什么需要

对重试环路的审计(issue #13182)发现:一个事件序号缺口会让该会话的投影永久停摆,扫描器以 10Hz 重复打印同一堆栈;瞬时失败的已受理 turn 被无上限重排;永远无法 settle 的 close/delete 无限重试;一次遗漏的容量通知会让排队激活静默饥饿;一次瞬时续约错误就让嵌入式调度器报废到进程重启;失败的 close 可能留下未封存的 writer,既不能干净恢复也不能被接管。会话 authority 的 fail-closed 姿态本身是对的——问题是外围各层把瞬时传输/IO 失败与一致性损坏混为一谈。对第一版实现的评审发现了本轮修复的剩余缺口:预算耗尽分支此前复用成功路径的完成写入(抹掉被阻塞的失败码,并为从未 settle 的关闭出具凭证);gap 自愈让失去 turn 终结事件的 item 永远停在"流式"状态;materializer 的其他永久失败模式仍以 10Hz 空转;调度器的瞬态重试没有升级上限;第一版保持续约的封存重试会让陈旧客户端在新会话重新挂载后被其封存 journal(已撤回,改为响亮抛错);预算耗尽的 action 应答永远无法再投递。

审查者测试计划

如何验证

受影响路径无法从 CLI 二进制触达(Java 控制面需要部署的 broker;嵌入式调度器在首次生产接线前完成加固),因此证据是新增的回归套件,每个都已在修复前代码上验证为红、修复后转绿:

  • packages/sdk-java/managed-agent-server:mvn test —— 2096 个测试;除两个在基线上同样复现的既有 ToolPublicationStoreTest 时序错误,以及一个仅在 macOS sh 上失败的 ChildWorktreeGitTest 非 UTF8 命名用例(在 main 上同样失败)外全绿。关键套件:MessageProjectionGapHealTest(真实 H2 + 真实事务边界:缺口投影自愈、缺口只记录一次且只命名真正缺失的序号、丢失 turn.completed 时其孤立 item 被置为 failed、第二连接 FOR UPDATE 见证钉住"扫描持有 Session 行锁"这一自愈前提)、MessageMaterializerTest(退避 + 一次性上报 + 成功后重置)、AdmittedTurnRetryTerminalStateTest(预算边界 9 重试 / 10 终结、配置预算生效、daemon-503 与 broker 两臂)、OperationRetryTerminalStateTest(预算边界、配置预算、已提交决定分支、存活 writer 闸门、保留失败码的终态、绑定 Session 的 workspace 关闭路由、终态写入失败的回退)、ManagedActionStoreTest(stage 不变式 + Action 仍在等待时 FAILED 行的重新受理)、SessionLifecycleCoordinatorTest(预算内被阻塞的 close 保持阻塞)。
  • packages/core:npx vitest run src/managed-runtime —— 3281 个测试(一个仅在负载下出现、单独运行通过的既有 flake)。覆盖含 journal 尾部修复故障注入(撕裂写入可修复、不可修复则熔断、首写 ENOENT 为瞬时、日志消失为致命)、调度器 claim/续约/释放重试策略、瞬态连续失败熔断、按激活预算收敛、dispose 竞态唤醒保护、内存阻塞轮询的减速节拍(每 tick 一次存储扫描,从 1 秒翻倍至 30 秒封顶,队列被清空的那一拍解除阻塞),以及封存失败路径的响亮抛错与授权失效。
  • npm run build && npm run bundle 与 npm run typecheck 通过;packages/web-shell 的 OpenAPI 契约同步测试通过(契约描述更新后已重新生成客户端)。

前后对比证据

N/A —— 无用户可见的 UI 变更。

测试平台

OS 状态
🍏 macOS ✅
🪟 Windows ⚠️
🐧 Linux ⚠️

环境(可选)

仅单元/集成级验证(Java broker 用 H2 内存库,TS store 用故障注入 fs mock);hosted 路径由上述套件覆盖,未跑真实 broker。

风险与范围

  • 主要风险或取舍:持续瞬时失败的 turn/operation/action 现在会到达终态而不是无限重试——这正是预期变更;预算可配置(dispatch.max-post-admission-retries、dispatch.max-operation-retries,默认均为 10)并在启动时校验。命中预算的生命周期操作现在读作 failed 并带失败码,而不是 completed——这是更诚实的回答,也是契约文本现在描述的回答。
  • 未验证/超出范围:嵌入式调度器尚无生产调用方(在接线前完成加固,已在测试注释中声明);MySQL 版 *IT 集成测试由 CI 的 Hosted process fault gates 通道运行(本地需要 Docker/MySQL);materializer 的退避在内存中(若运维需要跨重启的终态 consumer-progress,可作为后续工作)。
  • 破坏性变更/迁移说明:启动校验现在会拒绝 max-pre-admission-retries 大于 max-post-admission-retries 的配置——该组合在以前是合法的(并且会静默地剥夺迟受理 turn 的重试)。外部配置中存在该组合的部署需要在升级前对齐两个键;报错信息会指明两个值。除此之外:新增两个带安全默认值的可选配置键;turn 侧新增一个失败码;lifecycle/action 侧新增失败码(自由字符串字段,无枚举变更)。

关联 Issue

Fixes #13182

Asynchronous retry loops across the managed-agent stack had no terminal
state, and the message projection wedged permanently on a sequence gap.
Issue #13182.

- Projection: a sequence gap observed under the session row lock is
  provably permanent, so materializeNextBatch now logs once per gap and
  skips ahead instead of throwing on every scan at 10Hz forever.
- Turns: a new maxPostAdmissionRetries budget (default 10) terminates an
  admitted turn as hosted_harness_unavailable_after_admission once spent,
  distinct from the pre-admission code because the turn may already have
  run.
- Operations: a new maxOperationRetries budget (default 10) completes an
  unsettleable close/delete unconfirmed (java_durable) after a best-effort
  drain, and completes an undeliverable action response FAILED with
  action_response_delivery_failed instead of retrying forever. A committed
  decision seen on the projection re-check still wins over the budget.
- Scheduler: a memory-blocked pump arms a 1s recheck wake so a missed
  capacity notification no longer starves queued activations; transient
  store failures on claim/renew/release abandon only the affected
  activation instead of halting the worker, which still halts when the
  store reports consistency damage.
- Store: persist() repairs a torn journal tail (truncate to the last
  synced byte, like the open() loader) and only fails closed when the
  repair itself fails.
- Session close seals the writer in a finally; a failed HTTP seal keeps
  the grant renewed and retries the seal on the renewal cadence until
  sealed, bounded so a broken seal endpoint eventually lets the grant
  lapse for takeover instead of wedging the session.
@github-actions github-actions Bot added the review/self-reported The linked issue was opened by the PR author (self-reported) label Oct 2, 2026
@wenshao

wenshao commented Oct 2, 2026

Copy link
Copy Markdown
Collaborator Author

Test report (reproduction → verification)

All six findings from #13182 were reproduced against the pre-fix code with failing tests, then verified fixed after the change. These paths are not reachable from the CLI binary (the Java control plane needs a deployed broker; the embedded scheduler is hardened ahead of its first production wiring), so reproduction and verification ran at the store/coordinator level with real stores (H2 for Java, the real file store for TS) and fault injection instead of CLI E2E.

Finding Reproduction (pre-fix) After fix
Projection gap wedge 10/10 scans threw sequence has a gap; covered_sequence stuck at 1 Heals in one scan; post-gap event still materialized; session leaves the scanner target list
Admitted turn retries forever retryCount 42 still rescheduled; failTurn never called Terminates as hosted_harness_unavailable_after_admission at the budget (9 retries / 10 terminates; configured budgets honored; daemon-503 and broker arms both covered)
Lifecycle ops / action responses retry forever attempt 41 scheduled with a permanently unsettleable close; completeOperation/actions.complete never called CLOSE completes unconfirmed after a best-effort drain; action response completes FAILED with action_response_delivery_failed and java_durable; a committed decision seen on the projection re-check still wins over the budget
Memory-blocked starvation queued activation never ran without notifyCapacityChanged() The 1s recheck wake runs it within one poll cycle
Transient renewal error halts worker one disk busy bricked the scheduler and aborted a healthy sibling The store repairs the journal tail and the failure is transient; only the affected activation is abandoned and re-queued after lease expiry; an unrepairable write still halts
Non-atomic close / seal half-state seal never attempted when release failed; a 500 seal left the writer unsealed AND unrenewed Seal runs in a finally; a failed seal keeps renewing and retries the seal until SEALED, bounded so a permanently broken seal endpoint lets the grant lapse for takeover

Verification runs: mvn test on packages/sdk-java/managed-agent-server — 399 tests, green except two pre-existing ToolPublicationStoreTest timing errors that reproduce identically on base HEAD de2612434a; npx vitest run src/managed-runtime on packages/core — 1937 green; npm run build && npm run bundle, npm run typecheck, and the web-shell OpenAPI contract sync test all pass; mvn checkstyle:check clean. The review also ran repeated undirected and adversarial audit rounds over the diff until two consecutive clean rounds; every finding they produced is fixed in this branch, except one declined as out of scope (a supervision channel for the not-yet-wired embedded scheduler) and one adjudicated not-a-defect (fsync-after-repair-truncate: truncate is journaled metadata, and the claimed resurrection outcome is the designed at-least-once replay, not a defect).

…nal-states

# Conflicts:
#	packages/sdk-java/managed-agent-server/src/main/resources/openapi/managed-agent-public-api.openapi.json
#	packages/web-shell/client/components/managed/generated/managed-agent-api.ts
@wenshao
wenshao dismissed a stale review via 194472a October 2, 2026 11:15
@wenshao

wenshao commented Oct 2, 2026

Copy link
Copy Markdown
Collaborator Author

Merge note (main @ 9478f28 merged, conflicts resolved)

Merged current main into this branch; the PR is mergeable again. Two textual conflicts, both in the failure_code description this PR extends — resolved by keeping main's new Workspace close recovery_blocked sentence and this PR's action_response_delivery_failed clause (both survive); the generated web-shell client was regenerated and its contract sync test passes.

Verification on the merged tree:

  • packages/core: npx vitest run src/managed-runtime — 1943/1944 (the one failure is a hook-scale test timeout under concurrent load; 6/6 in isolation).
  • packages/sdk-java/managed-agent-server: a full mvn test on the merged tree currently fails to boot Flyway — main itself ships two migrations both numbered V31 (V31__workspace_session_close.sql from feat(managed-agent): reliably close workspace-bound sessions #13135 and V31__workspace_recovery_bundle.sql from feat(managed-agent): Add offline W1b recovery bundles #13138), tracked as Main CI failed: SDK Java — flyway duplicate version 31 in packages/sdk-java/managed-agent-server (+291 more) #13222 with an autofix in progress. This is main-side, not this PR.
  • With that incident worked around locally (throwaway worktree, one file renumbered V32, never committed): the suite runs 469 tests with the only remaining failures being the known pre-existing ToolPublicationStoreTest timing flake plus WorkspaceRecoveryStoreTest failures that reproduce identically on pure main with the same workaround — i.e. nothing attributable to this PR. All of this PR's suites (MessageProjectionGapHealTest, AdmittedTurnRetryTerminalStateTest, OperationRetryTerminalStateTest, ManagedActionStoreTest, HarnessCoordinatorTest, ManagedSessionLifecycleTest) are green on the merged tree.
  • The merge also required rebuilt sibling artifacts (qwencode, runtime-broker) — main's feat(managed-agent): reliably close workspace-bound sessions #13135 added RuntimeBrokerService/RuntimeProvisioner methods that the stale ~/.m2 copies lacked. Local-only concern.

Once #13222 lands on main, this branch should be re-merged/rebased and CI re-run.

…y budgets

Address the PR review: the lifecycle exhaustion arm no longer reuses the
success-path completion write — a new fenced failOperation terminates the
operation FAILED with the settle's failure code kept, leaving the Session
in its pending status (a live journal writer keeps the operation waiting
instead). The gap heal settles orphaned items as failed, guarded to never
touch a turn whose terminal event is in the journal or whose turn row is
still live. The materializer backs off per session and surfaces a stuck
projection once. The embedded scheduler retries transient renewal/release
failures within the lease, halts on a consecutive-failure streak, and
converges per-activation via a transient-failure budget. Pending HTTP
seals now abandon after two lease durations, and a FAILED action response
whose Action is still requested is re-admitted on the caller's retry.
Docs, the OpenAPI descriptions, and the generated client match.
@wenshao

wenshao commented Oct 3, 2026

Copy link
Copy Markdown
Collaborator Author

Addressed all 21 review findings (R1-1..R1-21) in c434cb8; per-thread details are in each reply. Summary of the semantics changes:

  • Lifecycle exhaustion is now an honest terminal failure. A new fenced failOperation write terminates the operation FAILED + CONFIRMED with the settle's failure code kept (blocked codes preserved) — no Session flip, no session.closed, no epoch wipe. The receipt is minted only because the contract's allOf requires one on every confirmed row. A live journal writer gates termination: the close can still succeed once it stops, so the operation keeps waiting. Bound Sessions' bindings release through requestWorkspaceClose + closeWorkspace, never the unbound drain.
  • The gap heal settles orphaned items, conservatively. Items below the gap fail only when their turn is provably over (no terminal event in the journal AND no live turn row); a mid-turn gap or a committed-but-unmaterialized terminal event leaves them to their own settle. Turn recovery's turn.failed covers the live-turn case.
  • The materializer backs off. Per-session consecutive-failure counter, 100ms→60s exponential backoff, one loud ERROR past a 10-failure budget, reset on success, entries swept when a session leaves the target list.
  • Scheduler transient handling converges. Renewals retry within lease headroom; releases retry bounded (3×250ms) within the lease; a 10-streak of consecutive transient store failures halts the worker loudly; a per-activation budget (3) records a terminal outcome instead of re-running forever — and failures absorbed by stale-ended runs count toward it.
  • Seals bound by wall clock. Pending seals abandon after 2 lease durations, not just 10 attempts (~10min at the contract-max lease instead of ~27min).
  • Action answers can be re-delivered. A replayed FAILED response whose Action is still requested re-admits on the same row (same key+digest; a second vote stays impossible). The client error now carries code; action_response_delivery_failed deliberately stays out of ENDED_ACTION_CODES because re-admission makes a retry meaningful.
  • Docs/contract: both OpenAPI description spots, the regenerated client, and both design-doc languages now describe the FAILED-terminal + live-writer semantics, the ~5-minute attempt-denominated budget reality, and the operator reconciliation path. Migration notes now disclose the new max-pre-admission-retries ≤ max-post-admission-retries startup constraint (its message names both values).

Verification: Java suite 486 tests (the only 2 errors are the pre-existing ToolPublicationStoreTest timing pair that reproduces identically on base); TS managed-runtime 1951/1952 (the one failure is the known hook-scale under-load timeout, green in isolation); web-shell 198/198 incl. the contract sync test; npm run build + typecheck + lint clean. Every new guard was additionally verified red under its mutant (gap-log expected++, settle removed, session-row lock removed, journal guard removed, live-turn guard removed, stale-budget merge removed), and the whole wave went through three review rounds (undirected + adversarial), the last one clean.

wenshao added 6 commits October 3, 2026 11:09
…nal-states

# Conflicts:
#	packages/sdk-java/managed-agent-server/src/test/java/com/alibaba/qwen/code/managedagent/ManagedSessionOperationStoreTest.java
#	packages/sdk-java/managed-agent-server/src/test/java/com/alibaba/qwen/code/managedagent/service/HarnessCoordinatorTest.java
…ed status

Round-4 audit fixes. The exhaustion gate now checks hasLiveWriter for every
delivered shape — a delete of a closed Session whose retention retirement
keeps refusing on a residual live writer waits instead of terminating into
an unrecoverable DELETING row. The terminal arm mirrors settle()'s routing
exactly: a closed-Session delete makes zero Runtime calls. A retried action
answer re-admits only while the Session still accepts responses; the
post-admission turn terminal keeps the Runtime Broker's own failure code;
the scheduler prunes budget entries for activations that can never re-queue;
and the seal bounds, blocked-fallback and fencing comments now state what
the code does.
…the round-4 gates

Round-5 audit follow-ups: the design docs' writer exception now covers every
admitted status (matching the gate that keys on the writer itself, in both
languages); the blocked-fallback test comment states the scanner's real
re-drive condition; and the two round-4 gates that lacked witnesses gain
them — a closed-Session delete terminating without any Runtime call (red
when the terminal arm's short-circuit is removed) and a replayed FAILED
action response staying terminal on an inactive Session.
…inal routing

The budget-exhausted paragraph claimed the terminal arm always releases the
binding through the workspace close pair; for a delete of an already closed
bound Session settle() — and now the terminal arm — deliberately makes no
Runtime call at all. Both languages.
@wenshao

wenshao commented Oct 3, 2026

Copy link
Copy Markdown
Collaborator Author

Audit closure for the review-fix wave (rounds 1–5, undirected + adversarial agents over each evolving delta):

  • Round 1 (3 Critical, all fixed + mutant-verified): renewal failing while the handler finishes no longer abandons the run; the per-activation budget survives stale terminal releases; the gap-heal settle stopped failing every in-progress item below the gap.
  • Round 2 (2 Critical, all fixed + mutant-verified): the gap-heal settle gained the live-turn guard (a mid-turn gap can no longer fail a running turn's items seconds before its own terminal event); stale-ended runs now merge their failures into the activation budget. Plus: CLOSE-only blocked fallback, materializer map sweep, unref'd retry sleep, operator-reconciliation doc paragraph.
  • Round 3: C=0 APPROVE; its 6 Suggestions were addressed before pushing (test timing that actually pins the finishing race, the stale-budget convergence test, the guard-1 discriminating fixture, the provider code assertion, doc wording, the OpenAPI failure_code field description).
  • Round 4 (post-merge; 1 Critical + 6 Suggestions, all fixed + mutant-verified): the writer gate now keys on hasLiveWriter for every delivered shape instead of the admitted status — a delete of a closed Session with a residual live writer waits instead of terminating into an unrecoverable DELETING row; the terminal arm shares settle()'s closedSessionDeletion shape (zero Runtime calls); re-admission gained the Session-ACTIVE gate; the post-admission turn terminal keeps the broker's own code; the scheduler prunes budget entries; fencing/comment/test gaps closed.
  • Round 5: C=0 APPROVE (both agents). It also re-verified all six R1 Criticals against the code and confirmed the merge had no evil-merge drift (git merge-tree byte-identical), SpotBugs High gate 0 findings, and the twice-merged contract/generated-client sync.

Deferred (recorded per the 5-round rule — Suggestions only, no Criticals outstanding):

  1. A hook-level witness that action_response_delivery_failed stays out of ENDED_ACTION_CODES (the invariant is currently pinned by the code comment plus the provider-side code assertion; a mutant adding it to the set survives the web-shell suite). Follow-up.
  2. A client "this failure is final" signal for the narrow shape where the Session left ACTIVE after a delivery-failed answer (today the card stays retryable and each retry replays the recorded FAILED — a manual loop with a visible error, not an automatic one; the contract behavior itself is correct and store-tested by aReplayedFailedResponseIsNotReadmittedOnAnInactiveSession). Follow-up.

Final verification on head 1fadcd5: Java mvn clean test 604 tests — only the 2 pre-existing ToolPublicationStoreTest timing errors (reproduce on base; the main-side HarnessCoordinatorTest cancellation flake also reproduced on clean origin/main and passes most runs); TS managed-runtime 1955/1956 (known hook-scale under-load timeout, green isolated); web-shell 207/207; typecheck/eslint/prettier/checkstyle/SpotBugs clean; every behavioral guard in this wave has a test verified red under its mutant.

Round-5 bot review (R2) Critical fixes. The lifecycle terminal arm now fires
only when settle() never returned — a completion-write failure after a
successful settle retries instead of recording session_lifecycle_delivery_failed
for a close that actually settled. The action terminal exempts the shape
where the Harness answered 200 and only Java's projection lags (a dedicated
DecisionNotYetProjected failure), so a committed decision is never recorded
as an undelivered one. Action re-admission narrows to the delivery-budget
terminal: a definitive Harness 400 keeps replaying its recorded row instead
of re-sending the refused body. The scheduler's release retry re-checks
liveness after its sleep, so a dispose racing the retry no longer writes a
terminal release on a stopped worker. Each fix is pinned by a test verified
red under its mutant.
@wenshao

wenshao commented Oct 3, 2026

Copy link
Copy Markdown
Collaborator Author

R2 review round disposition (head 32b68a5).

Criticals — 4 fixed, 1 deferred with reasons (all per-thread):

  • R1-3 (terminal certifying a settle that succeeded): the arm now requires harnessConfirmed == null — a completion-write failure after a successful settle retries instead of recording session_lifecycle_delivery_failed. Pinned + mutant-verified.
  • R1-5 (action terminal certifying "never answered" for an answered 200): the projection-lag shape throws a dedicated DecisionNotYetProjected and is exempt from the budget terminal — it retries until the projection heals or the Action's end state settles it. Pinned + mutant-verified.
  • R2-3 (re-admission re-opening a definitive Harness 400): narrowed to failureCode == action_response_delivery_failed; a Harness-produced terminal keeps replaying its recorded row. Pinned + mutant-verified.
  • R1-18 (dispose racing the release-retry sleep writing a terminal release): liveness re-checked after the sleep, before the write. Pinned + mutant-verified.
  • R2-32 (async terminal leaves the approval card hidden for the mount): adjudicated valid, deferred to a follow-up — the sound fix is a server→client terminal signal (contract addition to the pending-action payload or an event + hook pruning), feature-scale at this point in the PR's life. Mitigations today: answered is mount-scoped so any navigation/reload re-shows the still-requested Action and the re-click re-admits; the operation row stays queryable with its failure_code; Action expiry clears the pending list. Details in the thread reply.

Suggestions (17) — deferred per the 5-round rule (land Criticals only; record deferrals): R2-1 (live-writer shape never terminates — the documented writer-gate trade-off), R2-2 (runtime_broker_recovery_blocked terminal publishes the mapped workspace_close_identity_unverified code — deliberate mapping, will document), R2-5/R2-6 (§4.6 status row omits recovery_blocked; §8 follow-up stale), R2-7 (aSuccessfulBatchResetsTheBackoff weak witness — the sweep backstops), R2-8 (backoff cap/shift constants unexercised), R2-10 (hook-side ended-vs-retryable witness), R2-11 (renewal counter incremented inside the polled predicate), R2-14 (run.transientFailures reset only on renewal success), R2-15 (eviction loop placement in scheduleRecoveryWake), R2-16 (spent-budget terminal release issued on a lease already proved expired — the stale is handled, write is wasted), R2-18 (sweep infers from a LIMIT-truncated page — failing sessions sort first by updated_at ASC, argued safe in-thread earlier), R2-19 (single-target fixtures), R2-23 (retryable-vs-ended split for the only client-observable shape — tied to R2-32's follow-up), R2-24 (re-admission reset completeness in the doc), R2-26 (release path cannot exercise cross-run re-queue semantics), R2-31 (budget fixtures model the outage as a bare IllegalStateException), plus the R1-9 fix-induced fixture note. These will be triaged into the follow-up issue together with R2-32.

Verification on 32b68a5: Java mvn clean test 610 tests — only the 2 pre-existing ToolPublicationStoreTest timing errors and the main-side HarnessCoordinatorTest.runningOwnerObservesCancellationAfterStreamingStarts flake (reproduced on clean origin/main); TS managed-runtime 1956/1957 (known hook-scale under-load timeout, green isolated); typecheck/eslint/prettier clean. Every new gate is pinned by a test verified red under its mutant.

… clock fails

R3-1: store.getCurrentTime() validates its clock, and the renewal/release
error paths called it outside any try — a throwing injected clock escaped as
an unhandled rejection and leaked the active slot forever (pump skips keys
in active, so the activation could neither run nor re-queue, and the worker
silently stopped claiming once every slot leaked). Read the clock through a
helper that treats a throw as no-headroom/lease-expired, move the slot
cleanup and trailing pump into execute()'s finally, and turn any escaping
execute/renewal rejection into a loud halt. Pinned by a broken-clock test
verified red (timeout + unhandled rejection) against the pre-fix source.
@wenshao

wenshao commented Oct 4, 2026

Copy link
Copy Markdown
Collaborator Author

CI note: Hosted process fault gates / MySQL 8.4 / Java 21 failed on a6d1a90f07 — upstream flake, re-run triggered

What failed. One failsafe integration test, not a compile/style/unit problem (Checkstyle 0 violations, SpotBugs 0 bugs, unit phase green):

com.alibaba.qwen.code.managedagent.HostedWorkspaceToolTurnIT
  .packagedHarnessUsesSavedWorkspacesThroughRealBrokerWorkerAndSqlStore
  -- Time elapsed: 18.9 s <<< FAILURE!
Tests run: 18, Failures: 1, Errors: 0, Skipped: 0

Signature: qwen serve: Hosted Harness turn … failed: Error: Hosted Workspace profile refused a tool call. → driver AssertionError: {"error":"hosted_turn_recovery_required"} / 409 !== 200 at integration-tests/helpers/hosted-workspace-tool-turn-driver.ts:513 (javaLoad).

Why this is not a regression from this PR.

  1. The identical failure — same test, same ~18.9 s elapsed, same refusal → 409 hosted_turn_recovery_required → javaLoad chain, same Tests run: 18, Failures: 1 — occurred on docs/h3-shell-monitor-design (run 37170924797, job 111343549412), a documentation-only branch that cannot change harness behavior, and on fix/13333-admission-gap-lock (run 37178358786).
  2. On main, the last 15 runs of this workflow are 14 successes and 1 still in progress — no failures — over roughly the last 24 hours (2026-10-03T05:01Z … 2026-10-04T05:20Z), so the flake is recent and branch-independent rather than something main carries.
  3. Every link in the failure chain lives outside this diff. The refusal is thrown by the harness's tool-call validation in packages/cli/src/serve/hosted-workspace-tool-turn.ts, which fires on model-stream artifacts only — an undeclared tool name, a duplicate callId, wasOutputTruncated, or hadIncompleteArguments — before any broker write; the 409 hosted_turn_recovery_required is emitted by the Node session handler in packages/cli/src/serve/hosted-harness-session.ts. git diff --name-only against the merge base matches none of those files, the driver, or the IT, and the diff contains zero occurrences of hosted_turn_recovery_required. This PR's TypeScript surface is packages/core/src/managed-runtime/{embedded-harness-scheduler,http-managed-session-store,managed-activation-store,managed-session-assembly}.ts plus the web-shell managed components, none of which participates in model output, tool declarations, or stream parsing; the Java surface is broker retry/terminal semantics (SessionLifecycleCoordinator, ManagedAgentStore.failOperation and the projection gap heal, ManagedActionStore re-admission, HarnessCoordinator error-code propagation, ActionResponseCoordinator, MessageMaterializer), which only runs after the harness has already refused the call.

Action taken. Re-ran the failed jobs of run 37165371496 (gh run rerun --failed); no code change, since there is nothing in this diff to fix. If it fails again on the re-run, the same evidence applies and it should be treated as an infrastructure flake rather than a blocker for this PR.

wenshao added 2 commits October 4, 2026 14:01
…n added

Merging main brought #13301, which adds a toolProfile component to
SessionRecord and turns a null profile on a bound Session into an invalid
state: QwenHostedHarnessConnector throws "Hosted Workspace Session tool
profile is missing". The bound fixtures in the two retry-terminal suites
still used the 16-argument shape, so the module stopped compiling after
the merge.

The value is inert for these tests - both suites mock HarnessConnector, and
toolProfile() is read only inside the real connector - but a bound fixture
without a profile now describes a state production rejects, so it follows
main's own HarnessCoordinatorTest fixtures.
@wenshao
wenshao requested a balanced review from Copilot October 16, 2026 02:59

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.

@wenshao

wenshao commented Oct 16, 2026

Copy link
Copy Markdown
Collaborator Author

@qqqys — thanks for the bounded pass. Here are the three Criticals you could not confirm, located by symbol and checked by mutation on the new head 1cccd2860d. That head is 25d131fe17 plus a main merge whose only conflict was the OpenAPI info block, now 1.49.0 with main's v1.48 sentence kept. Paths are under packages/managed-agent-server/ (the round-14 anchors predate #14277's move).

R14-1: terminal-arm drain-route guard (service/SessionLifecycleCoordinator.java:390)

  • Fix: the bound terminal arm now gates the legacy requestWorkspaceClose on !operationIsLifecycle(claimed). operationIsLifecycle (:562) is WorkspaceToolProfiles.lifecycleProtocol(version), which is true for protocols 1 and 2. A v2 operation therefore never takes the v0 drain request.
  • Mutant: reverting the guard to claimed.lifecycleProtocolVersion() != 1 turns OperationRetryTerminalStateTest.boundV2SessionExhaustionSkipsTheV0DrainRequest red (69 run, 1 failure).

R14-2: recovered-receipt branch (service/SessionLifecycleCoordinator.java:689-690, store/WorkspaceLifecycleStore.java:168,173)

  • Fix: both synthesis branches of WorkspaceLifecycleStore.recoverEffects stamp storeSynthesized: true. The coordinator's latch is harnessSettled.set(!neverInitialized && !storeSynthesized), so a receipt built from the journal and store alone never earns the settled budget.
  • Mutant: dropping the storeSynthesized term turns both cases of aStoreSynthesizedReceiptDoesNotReachTheSettledBudget(int) red (69 run, 2 failures).

R14-3: the answered watermark (service/ActionResponseCoordinator.java:166-172, store/ManagedAgentStore.markOperationAnswered at :3147)

  • Fix: the first statement of the catch (RuntimeException) block is if (harnessAnswered) sessions.markOperationAnswered(...). It runs before the actions.response(...) / settled() / decisionExpired() recovery reads, so a throw from any of them can no longer lose the watermark.
  • Fencing: the write is fenced on state = 'RUNNING', delivery_state = 'LEASED', claim owner, claim generation and claim_until > SQL_NOW. It only bumps budget_exempt_attempt.
  • Mutant: skipping that call turns anAnsweredAttemptPersistsTheWatermarkBeforeTheCatchBlocksReads red (69 run, 1 failure).

The unmutated baseline is 69/69 green, and each mutant fails only its own witness.

The same head also carries the #14185 port described in the merge note above. That work matters for R14-3: without budget_exempt_attempt in ManagedSchemaProjection.OPERATION, the mapper's additiveInt would read the watermark back as 0.

Wider verification (Linux aarch64, JDK 21, MariaDB 10.11; Java CI does not run on pull_request):

  • Unit tests: 3997 pass.
  • ITs: 673/674 pass. The remaining one is the environmental ManagedAgentMySqlIT Hook-admission case, which fails identically on main.
  • Checkstyle: both executions are clean.
  • Since the new main merge: 791 targeted server tests and 80 HostedHarnessClientTest cases pass, along with the overlapping TS suites and the cli typecheck.

The independent scan of the full diff is still open on your side; nothing above substitutes for it.

@qqqys qqqys left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed at head 1cccd2860dd038ef07423e7bd4db1a107850a516 (base main), Critical-only, inside a bounded budget. Comment rather than Approve — but the reason is coverage, not a finding: all four standing Criticals are verifiably closed at this head, and what I cannot certify is the other ~9700 lines of a 58-file diff.

What moved since the last review

The head advanced from 25d131fe17 by seven commits, six of them main's (#14220, #14057, #14071, #14298, #14319, #14169) plus a merge whose own change is the contract description going 1.48 → 1.49. SessionLifecycleCoordinator.java, ActionResponseCoordinator.java and managed-session-assembly.ts are byte-identical between the two heads — I compared the delta's file list rather than assuming — and the PR's own diff is still the same 58 files. So this is the same change re-based, and the four findings could be re-read at their existing sites.

All four round-14 Criticals, verified in the code at this head

R14-1 — closed. The finding was that the terminal arm's drain-route guard excluded only protocol v1, so a v2 bound operation called requestWorkspaceClose → requestHarnessDrain → an unconditional ma_session_head upsert to phase='DRAINING', claim_lease_until = NULL, minting the fence authorization itself and then passing closeWorkspace's isHarnessDraining gate on the strength of it. At this head the guard is if (!operationIsLifecycle(claimed)) around the requestWorkspaceClose call, and operationIsLifecycle delegates to WorkspaceToolProfiles.lifecycleProtocol, which is version == 1 || version == 2 (WorkspaceToolProfiles.java:31-33). So for v1 and v2 the call is skipped, and only a non-lifecycle operation takes the v0 drain route. The comment above closeWorkspace states the invariant the finding asked for: the close proceeds only when the settle path already flipped the fence, and failOperation's terminal write clears the dead operation's claim without minting that authorization.

R14-2 — closed. The finding was that the recovered-receipt branch treated every non-neverInitialized receipt as proof the settle reached the Harness, while WorkspaceLifecycleStore.recoverEffects synthesizes and persists receipts itself — zero Harness contact — for a booted Session whose definition has no hookCatalog. At this head the assignment is harnessSettled.set(!effects.path("neverInitialized").asBoolean(false) && !effects.path("storeSynthesized").asBoolean(false)), with a comment citing R14-2, and WorkspaceLifecycleStore now stamps receipt.put("storeSynthesized", true) at both of its synthesis sites. The two halves meet: a store-synthesized receipt can no longer set the latch, so settleReachedHarness stays false and budgetSpent keeps the 10-attempt bound instead of the 40-attempt settled-completion one.

R14-3 — closed. The finding was that the answered watermark was per-attempt state whose durable half (budgetExemptAttempt > 0) was written only by retryOperation(..., true), the last statement in the catch block, so a throw anywhere earlier escaped with no DB write and the next lease-driven attempt recorded a terminal certifying the Harness never answered for a decision it had already committed. At this head the patch persists the watermark first — "Persist this attempt's answered watermark before any … must not read budgetExemptAttempt == 0 for a decision the Harness already committed (review round 14, R14-3)" — and reads it back as durable rather than local: boolean answered = harnessAnswered || op.budgetExemptAttempt() > 0;, with the accompanying comment stating that an answered attempt never records action_response_delivery_failed. The write now precedes the failure window the finding named instead of sitting after it.

R14-4 — still closed. managed-session-assembly.ts is unchanged between the heads, and I verified it last round: close() stops the renewal, returns early for an adopted lease, and its catch logs through debugLogger and rethrows the release error before the finally runs the seal — the comment names the masking this prevents. The seal-in-finally is deliberate and documented.

Round 14's ledger carried those four as its only sev:"C" findings at floor:"c", and no other round filed a Critical, so the standing blocking set is now empty as far as I can verify.

Why this is still a Comment

The independent Critical-only scan of the current diff did not run. This is 58 files at +9775/−317 across packages/core/managed-runtime, packages/cli/serve, the server's stores and coordinators, packages/web-shell's managed provider, the OpenAPI contract, V1__managed_schema.sql, application.yml and ManagedAgentProperties. I read the four finding sites and their load-bearing dependencies — WorkspaceToolProfiles.lifecycleProtocol, the two storeSynthesized writes, the watermark's new position, the assembly's close() — and the file list. Everything else, including ManagedAgentStore.java (+351/−30), ManagedActionStore.java (+143/−1), embedded-harness-scheduler.ts (+469/−34) and session-writer-lease.ts, is unread by me. Fourteen prior automated rounds are evidence about what those rounds read, not a substitute for this one, and a diff this size has produced a fresh Critical in every one of its last rounds.

CI has not finished at this head. Six lanes are pending — Lint & Static, Test (ubuntu-latest, Node 22.x), Integration Tests (no-AK, No Sandbox), Serve A/B, Capture web-shell visuals and Prepare reference distribution — against six passing and no failures. I did not wait on them. Pending is not a gate, but it does mean this head has no completed Node test or typecheck evidence yet, and no Java lane runs on a pull_request event at all, so the 137 targeted Java tests in the description remain local evidence.

So the actionable next step is a re-run with a budget sized to this diff rather than to a single-file change — or, what would end the loop faster, the split round 14's own convergence note recommended, since the finding rate across 14 rounds has not fallen. If a re-run reads the four coordinators and stores above and finds nothing, the standing set is already closed and this becomes an Approve.

Verification boundary

Static reading at the exact head inside a 15-minute budget — I ran no build, test or typecheck, and no JRE exists on this machine, so nothing on the Java side was executed. Every claim above comes from contents-API or per-file patch fetches at 1cccd2860d, and the "unchanged between heads" claim comes from the compare endpoint's file list rather than from an assumption that a merge touched nothing.

@yiliang114 yiliang114 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at head 1cccd2860dd038ef07423e7bd4db1a107850a516 (base main @ 17d5f774ea5e5c49c6461078efc53a2d91e59d1a). Approving — no blocking finding stands at this head.

The CHANGES_REQUESTED on this PR is stale

reviewDecision still reads CHANGES_REQUESTED, but nothing at this head carries that verdict. All twelve CHANGES_REQUESTED are from qwen-code-review-bot, and the most recent is pinned at 7366f9d37b (2026-10-14) — two heads behind. That round's own body recommended land-with-residual-risk, i.e. the exit was a maintainer risk-acceptance call rather than another review round. The Copilot review at head is a billing-quota notice, not findings. Worth naming the SHA so nobody goes chasing a red light that isn't pointed at this code.

What I checked myself rather than taking on faith

The head advanced twice while I was reading (the Oct-16 port to the consolidated packages/managed-agent-server plus a main sync), so I re-baselined and re-verified at 1cccd2860d instead of publishing against the older snapshot.

  • All 48 [Critical] threads read back isResolved=true, including the round-14 four (R14-1..R14-4). The 18 still-unresolved threads are all [Suggestion] plus two explicitly recorded deferrals.
  • The port's flagged silent break is genuinely closed. The description calls out that additiveInt answers 0 for a column missing from the result set, so main's explicit projections would have read every Turn's failure count and every operation's answered watermark as zero — budgets never closing, nothing failing to compile. At head: ManagedSchemaProjection.TURN carries retry_count, consecutive_failures, budget_exempt_failures, retry_after; OPERATION carries budget_exempt_attempt; V1__managed_schema.sql declares all three. And ManagedSchemaProjectionTest asserts each projection's column list equals the columns H2 actually reports for ma_<name>, so this specific regression is now guarded rather than merely fixed.
  • The fail-closed shapes hold where they matter. writerStillLive() returns true when the writer check itself throws, so an unavailable check extends the wait instead of certifying a terminal failure. retryable() whitelists only the two writer-active codes among 4xx and returns false for HostedHarnessCapabilityMismatchException / DaemonProtocolException. noteStoreOutcome() now short-circuits on disposed || fatalError before halt() (the R2-3 concern).
  • No schema-slot collision. The new V1 columns don't overlap what V2__private_csi_authority.sql and V3__session_list_created_at.sql splice in via ManagedSchema.ddl().

Coverage, stated plainly

I did not re-derive all 9,775 added lines across 58 files. My pass was targeted at the terminal-state machinery, the schema/projection port, and the four round-14 Criticals. qqqys APPROVED at this exact head after separately confirming the three files carrying those Criticals are byte-identical across the two heads. Test and Serve A/B are still in progress here; pending CI isn't a code finding and doesn't change this verdict.

The remaining Suggestions are real but non-blocking, and they're already recorded inline. Keeping them as follow-ups matches this PR's standing convention of Critical-only closeout after round five.

@yiliang114 yiliang114 added category/core Core engine and logic daemon type/bug Something isn't working as expected labels Oct 16, 2026
@wenshao
wenshao dismissed stale reviews from qwen-code-review-bot and ghost October 16, 2026 03:32

FIXED

auto-merge was automatically disabled October 16, 2026 03:32

Merge commits are not allowed on this repository

@wenshao
wenshao merged commit 7643735 into main Oct 16, 2026
110 checks passed
wenshao added a commit that referenced this pull request Oct 16, 2026
Resolve the submitTurn conflict with #13219's busy-turn status probe:
the probe now runs on top of this branch's bin-locked registration
(registerAdmission), so a vetoing marker is cleared only by an idle
status read taken while no send attempt is open on it. Tests that
assert a local veto after a stale or racing status read now have the
Harness report the admitted prompt running for the veto check, so the
probe reads a live answer instead of healing the marker.
wenshao added a commit that referenced this pull request Oct 16, 2026
…ons (#14358)

* fix(managed-agent): close the deferred #13182 terminal-state suggestions

PR #13219 deferred its Suggestion-level review findings under the
post-round-5 Critical-only rule. This lands the ones still valid on main.

Behavior:
- Bound the generation-change retry exemption (R3-6). A bound Session's
  HostedHarnessGenerationException stays exempt from the Turn retry
  budgets, but only until the Turn has gone four budgets without journaled
  progress; past that, it fails with hosted_harness_unavailable or
  hosted_harness_unavailable_after_admission instead of retrying forever.
- Read the Action once per catch-block inspection in
  ActionResponseCoordinator; the expiry check reuses that read (R2-11).
- Keep the release failure as the cause when the close-time seal fails
  too (R1-4).
- Treat action_response_decision_expired as an ended Action in the
  WebShell approval hook (R2-18).

Contract 1.51.0: failure_code now lists the codes a lifecycle operation
can end with once its budget runs out, maps runtime_broker_recovery_blocked
to workspace_close_identity_unverified, and notes that
session_close_writer_live also covers a failed writer check (R2-2, R2-16,
R2-17, R3-4).

Docs: the recovery-scan population includes CWD_CHANGE (R2-15), the
recovery_blocked publication trigger is the raw attempt count (R2-9), the
stale section 8 bullet is gone (R1-26), and the generation cap is
documented in the replaceable-Harness design.

Tests, each checked red against its mutant:
- R3-3: the generation exemption under the cap.
- R3-6: the cap before and after admission.
- R2-11: one Action read per inspection.
- R3-2: atLeast(3) renewals.
- R4-2: exempt-count rows on the writer-live and stale-boot publish arms.
- R2-10: no renewal survives a failed close.
- R3-5: a created enqueue resets the streak.
- R3-7: a dispose race that reaches the halt bound itself.
- R1-4 and R2-18: one test each for the fixes above.

* fix(managed-agent): address review round 1 on the #13182 follow-ups

- Floor the generation churn cap at a budget of one, so a zero budget
  still re-attaches across a single Harness change (R1-1).
- Derive the generation wait from the exception instead of a positional
  flag and drop the delegating overload (R1-5).
- State that the cap bounds the whole run without journaled progress,
  whatever wait filled it, in the code and in D9a (R1-8), and pin the
  counter with a mixed-history row (R1-6).
- Reject a close whose seal and activation release both fail with an
  AggregateError instead of annotating the seal error, which the lease
  memoizes and hands back to later seals (R1-3, R1-4).
- Leave action_response_decision_expired out of the WebShell hook's
  ended codes: the service keeps listing the Action, so hiding the card
  only dropped its reason (R1-2); the test uses a fixture the service can
  produce (R1-9).
- Stub lease renewals to succeed and raise the renewal floor so the
  period mutant fails (R1-7).

* fix(managed-agent): keep a fenced writer's 409 when a refused load's close fails twice

Review round 1 made a close whose activation release and writer seal both
fail reject with a plain AggregateError. The Hosted load route awaits that
close on its refusal branches and classifies the rejection by class, so a
fenced writer whose cleanup also hit the writer conflict answered the
generic 503 managed_session_open_failed instead of the 409
managed_session_writer_conflict the coordinator exempts from the Turn
retry budget.

The double failure now rejects with ManagedSessionCloseError, still an
AggregateError holding both failures, which exposes the seal failure as
sealError. The load route classifies on that seal error and logs the
wrapper, so neither failure is lost and the memoized seal rejection stays
untouched. A route test covers both a seal-only and a double failure.
yiliang114 added a commit that referenced this pull request Oct 17, 2026
…4394)

main went red on a84dbff because the idle-status poll in the Hosted
Harness model-selection case gave the turn's settle only the default 1s
`vi.waitFor` window, and the runner was carrying a load average of 15-19
while the file ran. The identical test and serve code passed in 391ms on
the next push (20ddb52), so the failure was the budget, not the change.

The suite already moved two neighbouring cases off that window for the
same reason (#14101, #13219): the settle crosses fsync-backed journal
commits before the active flag clears. Give this poll the same 10s
budget.
@qwen-code-review-bot

Copy link
Copy Markdown
Collaborator

Released in v0.25.3.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

category/core Core engine and logic daemon review/self-reported The linked issue was opened by the PR author (self-reported) type/bug Something isn't working as expected

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(managed-agent): retry loops without terminal states and a permanently wedged projection

8 participants