Repository navigation
fix(managed-agent): bound retry loops with terminal states - #13219
Conversation
Asynchronous retry loops across the managed-agent stack had no terminal state, and the message projection wedged permanently on a sequence gap. Issue #13182. - Projection: a sequence gap observed under the session row lock is provably permanent, so materializeNextBatch now logs once per gap and skips ahead instead of throwing on every scan at 10Hz forever. - Turns: a new maxPostAdmissionRetries budget (default 10) terminates an admitted turn as hosted_harness_unavailable_after_admission once spent, distinct from the pre-admission code because the turn may already have run. - Operations: a new maxOperationRetries budget (default 10) completes an unsettleable close/delete unconfirmed (java_durable) after a best-effort drain, and completes an undeliverable action response FAILED with action_response_delivery_failed instead of retrying forever. A committed decision seen on the projection re-check still wins over the budget. - Scheduler: a memory-blocked pump arms a 1s recheck wake so a missed capacity notification no longer starves queued activations; transient store failures on claim/renew/release abandon only the affected activation instead of halting the worker, which still halts when the store reports consistency damage. - Store: persist() repairs a torn journal tail (truncate to the last synced byte, like the open() loader) and only fails closed when the repair itself fails. - Session close seals the writer in a finally; a failed HTTP seal keeps the grant renewed and retries the seal on the renewal cadence until sealed, bounded so a broken seal endpoint eventually lets the grant lapse for takeover instead of wedging the session.
Test report (reproduction → verification)All six findings from #13182 were reproduced against the pre-fix code with failing tests, then verified fixed after the change. These paths are not reachable from the CLI binary (the Java control plane needs a deployed broker; the embedded scheduler is hardened ahead of its first production wiring), so reproduction and verification ran at the store/coordinator level with real stores (H2 for Java, the real file store for TS) and fault injection instead of CLI E2E.
Verification runs: |
…nal-states # Conflicts: # packages/sdk-java/managed-agent-server/src/main/resources/openapi/managed-agent-public-api.openapi.json # packages/web-shell/client/components/managed/generated/managed-agent-api.ts
Merge note (main @ 9478f28 merged, conflicts resolved)Merged current main into this branch; the PR is mergeable again. Two textual conflicts, both in the Verification on the merged tree:
Once #13222 lands on main, this branch should be re-merged/rebased and CI re-run. |
…y budgets Address the PR review: the lifecycle exhaustion arm no longer reuses the success-path completion write — a new fenced failOperation terminates the operation FAILED with the settle's failure code kept, leaving the Session in its pending status (a live journal writer keeps the operation waiting instead). The gap heal settles orphaned items as failed, guarded to never touch a turn whose terminal event is in the journal or whose turn row is still live. The materializer backs off per session and surfaces a stuck projection once. The embedded scheduler retries transient renewal/release failures within the lease, halts on a consecutive-failure streak, and converges per-activation via a transient-failure budget. Pending HTTP seals now abandon after two lease durations, and a FAILED action response whose Action is still requested is re-admitted on the caller's retry. Docs, the OpenAPI descriptions, and the generated client match.
|
Addressed all 21 review findings (R1-1..R1-21) in c434cb8; per-thread details are in each reply. Summary of the semantics changes:
Verification: Java suite 486 tests (the only 2 errors are the pre-existing |
…nal-states # Conflicts: # packages/sdk-java/managed-agent-server/src/test/java/com/alibaba/qwen/code/managedagent/ManagedSessionOperationStoreTest.java # packages/sdk-java/managed-agent-server/src/test/java/com/alibaba/qwen/code/managedagent/service/HarnessCoordinatorTest.java
…ed status Round-4 audit fixes. The exhaustion gate now checks hasLiveWriter for every delivered shape — a delete of a closed Session whose retention retirement keeps refusing on a residual live writer waits instead of terminating into an unrecoverable DELETING row. The terminal arm mirrors settle()'s routing exactly: a closed-Session delete makes zero Runtime calls. A retried action answer re-admits only while the Session still accepts responses; the post-admission turn terminal keeps the Runtime Broker's own failure code; the scheduler prunes budget entries for activations that can never re-queue; and the seal bounds, blocked-fallback and fencing comments now state what the code does.
…the round-4 gates Round-5 audit follow-ups: the design docs' writer exception now covers every admitted status (matching the gate that keys on the writer itself, in both languages); the blocked-fallback test comment states the scanner's real re-drive condition; and the two round-4 gates that lacked witnesses gain them — a closed-Session delete terminating without any Runtime call (red when the terminal arm's short-circuit is removed) and a replayed FAILED action response staying terminal on an inactive Session.
…inal routing The budget-exhausted paragraph claimed the terminal arm always releases the binding through the workspace close pair; for a delete of an already closed bound Session settle() — and now the terminal arm — deliberately makes no Runtime call at all. Both languages.
|
Audit closure for the review-fix wave (rounds 1–5, undirected + adversarial agents over each evolving delta):
Deferred (recorded per the 5-round rule — Suggestions only, no Criticals outstanding):
Final verification on head 1fadcd5: Java |
Round-5 bot review (R2) Critical fixes. The lifecycle terminal arm now fires only when settle() never returned — a completion-write failure after a successful settle retries instead of recording session_lifecycle_delivery_failed for a close that actually settled. The action terminal exempts the shape where the Harness answered 200 and only Java's projection lags (a dedicated DecisionNotYetProjected failure), so a committed decision is never recorded as an undelivered one. Action re-admission narrows to the delivery-budget terminal: a definitive Harness 400 keeps replaying its recorded row instead of re-sending the refused body. The scheduler's release retry re-checks liveness after its sleep, so a dispose racing the retry no longer writes a terminal release on a stopped worker. Each fix is pinned by a test verified red under its mutant.
|
R2 review round disposition (head 32b68a5). Criticals — 4 fixed, 1 deferred with reasons (all per-thread):
Suggestions (17) — deferred per the 5-round rule (land Criticals only; record deferrals): R2-1 (live-writer shape never terminates — the documented writer-gate trade-off), R2-2 ( Verification on 32b68a5: Java |
… clock fails R3-1: store.getCurrentTime() validates its clock, and the renewal/release error paths called it outside any try — a throwing injected clock escaped as an unhandled rejection and leaked the active slot forever (pump skips keys in active, so the activation could neither run nor re-queue, and the worker silently stopped claiming once every slot leaked). Read the clock through a helper that treats a throw as no-headroom/lease-expired, move the slot cleanup and trailing pump into execute()'s finally, and turn any escaping execute/renewal rejection into a loud halt. Pinned by a broken-clock test verified red (timeout + unhandled rejection) against the pre-fix source.
CI note:
|
…n added Merging main brought #13301, which adds a toolProfile component to SessionRecord and turns a null profile on a bound Session into an invalid state: QwenHostedHarnessConnector throws "Hosted Workspace Session tool profile is missing". The bound fixtures in the two retry-terminal suites still used the 16-argument shape, so the module stopped compiling after the merge. The value is inert for these tests - both suites mock HarnessConnector, and toolProfile() is read only inside the real connector - but a bound fixture without a profile now describes a state production rejects, so it follows main's own HarnessCoordinatorTest fixtures.
There was a problem hiding this comment.
Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.
An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.
|
@qqqys — thanks for the bounded pass. Here are the three Criticals you could not confirm, located by symbol and checked by mutation on the new head R14-1: terminal-arm drain-route guard (
R14-2: recovered-receipt branch (
R14-3: the
The unmutated baseline is 69/69 green, and each mutant fails only its own witness. The same head also carries the #14185 port described in the merge note above. That work matters for R14-3: without Wider verification (Linux aarch64, JDK 21, MariaDB 10.11; Java CI does not run on
The independent scan of the full diff is still open on your side; nothing above substitutes for it. |
qqqys
left a comment
There was a problem hiding this comment.
Re-reviewed at head 1cccd2860dd038ef07423e7bd4db1a107850a516 (base main), Critical-only, inside a bounded budget. Comment rather than Approve — but the reason is coverage, not a finding: all four standing Criticals are verifiably closed at this head, and what I cannot certify is the other ~9700 lines of a 58-file diff.
What moved since the last review
The head advanced from 25d131fe17 by seven commits, six of them main's (#14220, #14057, #14071, #14298, #14319, #14169) plus a merge whose own change is the contract description going 1.48 → 1.49. SessionLifecycleCoordinator.java, ActionResponseCoordinator.java and managed-session-assembly.ts are byte-identical between the two heads — I compared the delta's file list rather than assuming — and the PR's own diff is still the same 58 files. So this is the same change re-based, and the four findings could be re-read at their existing sites.
All four round-14 Criticals, verified in the code at this head
R14-1 — closed. The finding was that the terminal arm's drain-route guard excluded only protocol v1, so a v2 bound operation called requestWorkspaceClose → requestHarnessDrain → an unconditional ma_session_head upsert to phase='DRAINING', claim_lease_until = NULL, minting the fence authorization itself and then passing closeWorkspace's isHarnessDraining gate on the strength of it. At this head the guard is if (!operationIsLifecycle(claimed)) around the requestWorkspaceClose call, and operationIsLifecycle delegates to WorkspaceToolProfiles.lifecycleProtocol, which is version == 1 || version == 2 (WorkspaceToolProfiles.java:31-33). So for v1 and v2 the call is skipped, and only a non-lifecycle operation takes the v0 drain route. The comment above closeWorkspace states the invariant the finding asked for: the close proceeds only when the settle path already flipped the fence, and failOperation's terminal write clears the dead operation's claim without minting that authorization.
R14-2 — closed. The finding was that the recovered-receipt branch treated every non-neverInitialized receipt as proof the settle reached the Harness, while WorkspaceLifecycleStore.recoverEffects synthesizes and persists receipts itself — zero Harness contact — for a booted Session whose definition has no hookCatalog. At this head the assignment is harnessSettled.set(!effects.path("neverInitialized").asBoolean(false) && !effects.path("storeSynthesized").asBoolean(false)), with a comment citing R14-2, and WorkspaceLifecycleStore now stamps receipt.put("storeSynthesized", true) at both of its synthesis sites. The two halves meet: a store-synthesized receipt can no longer set the latch, so settleReachedHarness stays false and budgetSpent keeps the 10-attempt bound instead of the 40-attempt settled-completion one.
R14-3 — closed. The finding was that the answered watermark was per-attempt state whose durable half (budgetExemptAttempt > 0) was written only by retryOperation(..., true), the last statement in the catch block, so a throw anywhere earlier escaped with no DB write and the next lease-driven attempt recorded a terminal certifying the Harness never answered for a decision it had already committed. At this head the patch persists the watermark first — "Persist this attempt's answered watermark before any … must not read budgetExemptAttempt == 0 for a decision the Harness already committed (review round 14, R14-3)" — and reads it back as durable rather than local: boolean answered = harnessAnswered || op.budgetExemptAttempt() > 0;, with the accompanying comment stating that an answered attempt never records action_response_delivery_failed. The write now precedes the failure window the finding named instead of sitting after it.
R14-4 — still closed. managed-session-assembly.ts is unchanged between the heads, and I verified it last round: close() stops the renewal, returns early for an adopted lease, and its catch logs through debugLogger and rethrows the release error before the finally runs the seal — the comment names the masking this prevents. The seal-in-finally is deliberate and documented.
Round 14's ledger carried those four as its only sev:"C" findings at floor:"c", and no other round filed a Critical, so the standing blocking set is now empty as far as I can verify.
Why this is still a Comment
The independent Critical-only scan of the current diff did not run. This is 58 files at +9775/−317 across packages/core/managed-runtime, packages/cli/serve, the server's stores and coordinators, packages/web-shell's managed provider, the OpenAPI contract, V1__managed_schema.sql, application.yml and ManagedAgentProperties. I read the four finding sites and their load-bearing dependencies — WorkspaceToolProfiles.lifecycleProtocol, the two storeSynthesized writes, the watermark's new position, the assembly's close() — and the file list. Everything else, including ManagedAgentStore.java (+351/−30), ManagedActionStore.java (+143/−1), embedded-harness-scheduler.ts (+469/−34) and session-writer-lease.ts, is unread by me. Fourteen prior automated rounds are evidence about what those rounds read, not a substitute for this one, and a diff this size has produced a fresh Critical in every one of its last rounds.
CI has not finished at this head. Six lanes are pending — Lint & Static, Test (ubuntu-latest, Node 22.x), Integration Tests (no-AK, No Sandbox), Serve A/B, Capture web-shell visuals and Prepare reference distribution — against six passing and no failures. I did not wait on them. Pending is not a gate, but it does mean this head has no completed Node test or typecheck evidence yet, and no Java lane runs on a pull_request event at all, so the 137 targeted Java tests in the description remain local evidence.
So the actionable next step is a re-run with a budget sized to this diff rather than to a single-file change — or, what would end the loop faster, the split round 14's own convergence note recommended, since the finding rate across 14 rounds has not fallen. If a re-run reads the four coordinators and stores above and finds nothing, the standing set is already closed and this becomes an Approve.
Verification boundary
Static reading at the exact head inside a 15-minute budget — I ran no build, test or typecheck, and no JRE exists on this machine, so nothing on the Java side was executed. Every claim above comes from contents-API or per-file patch fetches at 1cccd2860d, and the "unchanged between heads" claim comes from the compare endpoint's file list rather than from an assumption that a merge touched nothing.
yiliang114
left a comment
There was a problem hiding this comment.
Reviewed at head 1cccd2860dd038ef07423e7bd4db1a107850a516 (base main @ 17d5f774ea5e5c49c6461078efc53a2d91e59d1a). Approving — no blocking finding stands at this head.
The CHANGES_REQUESTED on this PR is stale
reviewDecision still reads CHANGES_REQUESTED, but nothing at this head carries that verdict. All twelve CHANGES_REQUESTED are from qwen-code-review-bot, and the most recent is pinned at 7366f9d37b (2026-10-14) — two heads behind. That round's own body recommended land-with-residual-risk, i.e. the exit was a maintainer risk-acceptance call rather than another review round. The Copilot review at head is a billing-quota notice, not findings. Worth naming the SHA so nobody goes chasing a red light that isn't pointed at this code.
What I checked myself rather than taking on faith
The head advanced twice while I was reading (the Oct-16 port to the consolidated packages/managed-agent-server plus a main sync), so I re-baselined and re-verified at 1cccd2860d instead of publishing against the older snapshot.
- All 48
[Critical]threads read backisResolved=true, including the round-14 four (R14-1..R14-4). The 18 still-unresolved threads are all[Suggestion]plus two explicitly recorded deferrals. - The port's flagged silent break is genuinely closed. The description calls out that
additiveIntanswers 0 for a column missing from the result set, so main's explicit projections would have read every Turn's failure count and every operation's answered watermark as zero — budgets never closing, nothing failing to compile. At head:ManagedSchemaProjection.TURNcarriesretry_count, consecutive_failures, budget_exempt_failures, retry_after;OPERATIONcarriesbudget_exempt_attempt;V1__managed_schema.sqldeclares all three. AndManagedSchemaProjectionTestasserts each projection's column list equals the columns H2 actually reports forma_<name>, so this specific regression is now guarded rather than merely fixed. - The fail-closed shapes hold where they matter.
writerStillLive()returnstruewhen the writer check itself throws, so an unavailable check extends the wait instead of certifying a terminal failure.retryable()whitelists only the two writer-active codes among 4xx and returnsfalseforHostedHarnessCapabilityMismatchException/DaemonProtocolException.noteStoreOutcome()now short-circuits ondisposed || fatalErrorbeforehalt()(the R2-3 concern). - No schema-slot collision. The new V1 columns don't overlap what
V2__private_csi_authority.sqlandV3__session_list_created_at.sqlsplice in viaManagedSchema.ddl().
Coverage, stated plainly
I did not re-derive all 9,775 added lines across 58 files. My pass was targeted at the terminal-state machinery, the schema/projection port, and the four round-14 Criticals. qqqys APPROVED at this exact head after separately confirming the three files carrying those Criticals are byte-identical across the two heads. Test and Serve A/B are still in progress here; pending CI isn't a code finding and doesn't change this verdict.
The remaining Suggestions are real but non-blocking, and they're already recorded inline. Keeping them as follow-ups matches this PR's standing convention of Critical-only closeout after round five.
Merge commits are not allowed on this repository
Resolve the submitTurn conflict with #13219's busy-turn status probe: the probe now runs on top of this branch's bin-locked registration (registerAdmission), so a vetoing marker is cleared only by an idle status read taken while no send attempt is open on it. Tests that assert a local veto after a stale or racing status read now have the Harness report the admitted prompt running for the veto check, so the probe reads a live answer instead of healing the marker.
…ons (#14358) * fix(managed-agent): close the deferred #13182 terminal-state suggestions PR #13219 deferred its Suggestion-level review findings under the post-round-5 Critical-only rule. This lands the ones still valid on main. Behavior: - Bound the generation-change retry exemption (R3-6). A bound Session's HostedHarnessGenerationException stays exempt from the Turn retry budgets, but only until the Turn has gone four budgets without journaled progress; past that, it fails with hosted_harness_unavailable or hosted_harness_unavailable_after_admission instead of retrying forever. - Read the Action once per catch-block inspection in ActionResponseCoordinator; the expiry check reuses that read (R2-11). - Keep the release failure as the cause when the close-time seal fails too (R1-4). - Treat action_response_decision_expired as an ended Action in the WebShell approval hook (R2-18). Contract 1.51.0: failure_code now lists the codes a lifecycle operation can end with once its budget runs out, maps runtime_broker_recovery_blocked to workspace_close_identity_unverified, and notes that session_close_writer_live also covers a failed writer check (R2-2, R2-16, R2-17, R3-4). Docs: the recovery-scan population includes CWD_CHANGE (R2-15), the recovery_blocked publication trigger is the raw attempt count (R2-9), the stale section 8 bullet is gone (R1-26), and the generation cap is documented in the replaceable-Harness design. Tests, each checked red against its mutant: - R3-3: the generation exemption under the cap. - R3-6: the cap before and after admission. - R2-11: one Action read per inspection. - R3-2: atLeast(3) renewals. - R4-2: exempt-count rows on the writer-live and stale-boot publish arms. - R2-10: no renewal survives a failed close. - R3-5: a created enqueue resets the streak. - R3-7: a dispose race that reaches the halt bound itself. - R1-4 and R2-18: one test each for the fixes above. * fix(managed-agent): address review round 1 on the #13182 follow-ups - Floor the generation churn cap at a budget of one, so a zero budget still re-attaches across a single Harness change (R1-1). - Derive the generation wait from the exception instead of a positional flag and drop the delegating overload (R1-5). - State that the cap bounds the whole run without journaled progress, whatever wait filled it, in the code and in D9a (R1-8), and pin the counter with a mixed-history row (R1-6). - Reject a close whose seal and activation release both fail with an AggregateError instead of annotating the seal error, which the lease memoizes and hands back to later seals (R1-3, R1-4). - Leave action_response_decision_expired out of the WebShell hook's ended codes: the service keeps listing the Action, so hiding the card only dropped its reason (R1-2); the test uses a fixture the service can produce (R1-9). - Stub lease renewals to succeed and raise the renewal floor so the period mutant fails (R1-7). * fix(managed-agent): keep a fenced writer's 409 when a refused load's close fails twice Review round 1 made a close whose activation release and writer seal both fail reject with a plain AggregateError. The Hosted load route awaits that close on its refusal branches and classifies the rejection by class, so a fenced writer whose cleanup also hit the writer conflict answered the generic 503 managed_session_open_failed instead of the 409 managed_session_writer_conflict the coordinator exempts from the Turn retry budget. The double failure now rejects with ManagedSessionCloseError, still an AggregateError holding both failures, which exposes the seal failure as sealError. The load route classifies on that seal error and logs the wrapper, so neither failure is lost and the memoized seal rejection stays untouched. A route test covers both a seal-only and a double failure.
…4394) main went red on a84dbff because the idle-status poll in the Hosted Harness model-selection case gave the turn's settle only the default 1s `vi.waitFor` window, and the runner was carrying a load average of 15-19 while the file ran. The identical test and serve code passed in 391ms on the next push (20ddb52), so the failure was the budget, not the change. The suite already moved two neighbouring cases off that window for the same reason (#14101, #13219): the settle crosses fsync-backed journal commits before the active flag clears. Give this poll the same 10s budget.
|
Released in v0.25.3. |
What this PR does
Across the managed-agent stack, asynchronous retry loops had no terminal state, and in one case a permanently wedged projection. This change gives every loop a budget and a terminal state: the message projection heals past a provably permanent sequence gap instead of throwing on every scan — settling as failed any items whose turn's terminal event was in the skipped range — and the scanner itself backs off per session (exponential to a 60s cap, surfaced loudly once) so a deterministic failure can no longer flood the log at 10Hz; admitted turns converge to a bounded terminal failure once their retry budget is spent; lifecycle operations whose settle can never succeed terminate
failedwith the settle's failure code kept (the Session keeps its pending status — nothing certifies a completion that never happened) — except while a live journal writer means the close can still succeed, which keeps waiting; a blocked workspace close keeps its failure code through every attempt and through the terminal record; a failed action response whose Action is still waiting is re-admitted by the caller's retry under the same idempotency key instead of returning a stale failure receipt forever; a memory-blocked embedded scheduler arms its own recheck wake so a missed capacity notification can no longer starve queued work; transient store IO failures are retried within the lease window (renewals) or retried bounded before abandoning (releases), a consecutive-failure streak halts the worker loudly when a "transient" outage is really persistent, and a per-activation budget records a terminal outcome instead of re-running forever; session close seals the writer in a finally; and a failed HTTP seal throws loudly out of that finally while the grant lapses with the lease, so a takeover can proceed. There is deliberately no seal retry (issue #13182 item 6b): keeping the grant renewed was found in review to let a stale client seal the journal out from under a freshly re-attached session, so the bounded-retry cut was withdrawn and 6b is answered by the loud throw plus the grant lapse.Why it's needed
Audit of the retry loops (issue #13182) found: a single event-sequence gap wedged a session's projection forever while the scanner logged the same stack at 10Hz; an admitted turn failing transiently was rescheduled without bound; a close/delete whose settle can never succeed retried forever; a missed capacity notification starved queued activations silently; one transient renewal error bricked the embedded scheduler; and a failed close could leave a writer unsealed, blocking both clean resume and takeover. The fail-closed posture of the session authority is correct — the layers around it treated transient IO failures as indistinguishable from consistency damage. Review of the first cut found the remaining gaps this wave closes: the exhaustion arm used the success-path completion write (erasing the blocked failure code and certifying an unsettled close), the gap heal left orphaned items streaming forever, the materializer's other permanent failure modes still spun at 10Hz, the scheduler's transient retry had no escalation bound, the first cut's renewal-keeping seal retry let a stale client seal the journal under a re-attached session (withdrawn in favor of the loud throw), and a budget-terminated action answer could never be re-delivered.
Reviewer Test Plan
How to verify
The affected paths are not reachable from the CLI binary (the Java control plane needs a deployed broker; the embedded scheduler is hardened ahead of its first production wiring), so the evidence is the new regression suites, each verified to fail on the pre-fix code and pass after:
packages/sdk-java/managed-agent-server:mvn test— 2096 tests; the full suite is green except two pre-existingToolPublicationStoreTesttiming errors that reproduce identically on base, and oneChildWorktreeGitTestnon-UTF8-name case that fails only on macOS's sh (identical on main). Key suites:MessageProjectionGapHealTest(real H2 + real transaction boundary: a gapped projection heals, the gap is logged once naming only the truly missing sequences, a lostturn.completedsettles its orphaned items as failed, and a second-connectionFOR UPDATEwitness pins that the scan holds the session row lock the permanence premise rests on),MessageMaterializerTest(backoff + surface-once + reset on success),AdmittedTurnRetryTerminalStateTest(budget boundary 9 retries / 10 terminates, configured budgets honored, both the daemon-503 and broker arms),OperationRetryTerminalStateTest(budget boundary, configured budgets, the committed-decision arm, the live-writer gate, the blocked-code-preserving terminal, the bound-session workspace-close routing, the terminal-record-failure fallback),ManagedActionStoreTest(stage invariants + re-admission of a FAILED response while the Action is still requested),SessionLifecycleCoordinatorTest(a blocked close stays blocked below the budget).packages/core:npx vitest run src/managed-runtime— 3281 tests (one known under-load-only flake that passes in isolation). Coverage includes journal tail repair fault injection (torn write repaired, unrepairable write halts, first-write ENOENT is transient, vanished journal is fatal), the scheduler's claim/renewal/release retry policies, the transient-streak halt, per-activation budget convergence, dispose-race wake guards, the memory-blocked poll's decelerating cadence (one store scan per tick, doubling from 1s to a 30s cap, with a cancelled-queue tick clearing the block), and the seal failure path's loud throw with the grant left to lapse.npm run build && npm run bundleandnpm run typecheckpass;packages/web-shellOpenAPI contract sync test passes (the generated client was regenerated after the contract description update).Evidence (Before & After)
N/A — no user-visible UI change.
Tested on
Environment (optional)
Unit/integration-level verification only (H2 in-memory for the Java broker, fault-injection fs mocks for the TS store); the hosted paths are covered by the suites above rather than a live broker.
Risk & Scope
dispatch.max-post-admission-retries,dispatch.max-operation-retries, both default 10) and validated at startup. A lifecycle operation that hits the budget now readsfailedwith a failure code instead ofcompleted— the more honest answer, and the one the contract text now describes.*ITintegration tests run in CI's Hosted process fault gates lane (locally they need Docker/MySQL); the materializer backoff is in-memory (a persisted terminal consumer-progress state is a possible follow-up if operators need it to survive restarts).max-pre-admission-retriesgreater thanmax-post-admission-retries— a combination that was accepted before (and silently stripped late-admitted turns of retries). Deployments with that combination in external config must align the two keys before upgrading; the error message names both values. Otherwise: two new optional config keys with safe defaults; one new turn failure code; new lifecycle/action failure codes (free-form string fields, no enum change).Linked Issues
Fixes #13182
中文说明
本 PR 做了什么
managed-agent 栈中的异步重试环路此前没有终态,其中一处会让投影永久卡死。本次改为给每个环路预算与终态:消息投影在遇到可证永久的事件序号缺口时自愈越过——并把终结事件落在缺口区间的 turn 的在途 item 置为 failed——不再每次扫描都抛错,扫描器自身也按 session 退避(指数退避封顶 60 秒,超预算后响亮上报一次),确定性失败不再以 10Hz 刷日志;已受理的 turn 在重试预算耗尽后收敛到有界终态;settle 永远无法成功的生命周期操作以
failed终结并保留 settle 的失败码(Session 保持待定状态——不再为从未发生的完成出具凭证)——除非 journal writer 仍存活,此时 close 仍可能成功,operation 继续等待;被阻塞的 workspace close 在每次尝试与终态记录中都保留其失败码;Action 仍在等待应答的失败 action response 可被调用方按同一幂等键重试重新受理,而不是永远返回陈旧的失败回执;内存不足的嵌入式调度器会自行武装复检唤醒,错过容量通知不再让排队激活永久饥饿;瞬时存储 IO 失败在租约窗口内重试(续约)或有界重试后再放弃(释放),连续失败计数会在"瞬时"故障实为持续故障时响亮地 halt worker,按激活的预算则记录终态而不是无限重跑;会话关闭在 finally 中保证封存 writer;HTTP 封存失败会从该 finally 中响亮抛出,并让授权随租约失效,使接管得以进行。此处刻意不做封存重试(issue #13182 第 6b 项):评审发现保持续约会让陈旧客户端在新会话重新挂载后从其手中封存 journal,因此有界重试方案被撤回,6b 由响亮抛错加授权失效来回答。为什么需要
对重试环路的审计(issue #13182)发现:一个事件序号缺口会让该会话的投影永久停摆,扫描器以 10Hz 重复打印同一堆栈;瞬时失败的已受理 turn 被无上限重排;永远无法 settle 的 close/delete 无限重试;一次遗漏的容量通知会让排队激活静默饥饿;一次瞬时续约错误就让嵌入式调度器报废到进程重启;失败的 close 可能留下未封存的 writer,既不能干净恢复也不能被接管。会话 authority 的 fail-closed 姿态本身是对的——问题是外围各层把瞬时传输/IO 失败与一致性损坏混为一谈。对第一版实现的评审发现了本轮修复的剩余缺口:预算耗尽分支此前复用成功路径的完成写入(抹掉被阻塞的失败码,并为从未 settle 的关闭出具凭证);gap 自愈让失去 turn 终结事件的 item 永远停在"流式"状态;materializer 的其他永久失败模式仍以 10Hz 空转;调度器的瞬态重试没有升级上限;第一版保持续约的封存重试会让陈旧客户端在新会话重新挂载后被其封存 journal(已撤回,改为响亮抛错);预算耗尽的 action 应答永远无法再投递。
审查者测试计划
如何验证
受影响路径无法从 CLI 二进制触达(Java 控制面需要部署的 broker;嵌入式调度器在首次生产接线前完成加固),因此证据是新增的回归套件,每个都已在修复前代码上验证为红、修复后转绿:
packages/sdk-java/managed-agent-server:mvn test—— 2096 个测试;除两个在基线上同样复现的既有ToolPublicationStoreTest时序错误,以及一个仅在 macOS sh 上失败的ChildWorktreeGitTest非 UTF8 命名用例(在 main 上同样失败)外全绿。关键套件:MessageProjectionGapHealTest(真实 H2 + 真实事务边界:缺口投影自愈、缺口只记录一次且只命名真正缺失的序号、丢失turn.completed时其孤立 item 被置为 failed、第二连接FOR UPDATE见证钉住"扫描持有 Session 行锁"这一自愈前提)、MessageMaterializerTest(退避 + 一次性上报 + 成功后重置)、AdmittedTurnRetryTerminalStateTest(预算边界 9 重试 / 10 终结、配置预算生效、daemon-503 与 broker 两臂)、OperationRetryTerminalStateTest(预算边界、配置预算、已提交决定分支、存活 writer 闸门、保留失败码的终态、绑定 Session 的 workspace 关闭路由、终态写入失败的回退)、ManagedActionStoreTest(stage 不变式 + Action 仍在等待时 FAILED 行的重新受理)、SessionLifecycleCoordinatorTest(预算内被阻塞的 close 保持阻塞)。packages/core:npx vitest run src/managed-runtime—— 3281 个测试(一个仅在负载下出现、单独运行通过的既有 flake)。覆盖含 journal 尾部修复故障注入(撕裂写入可修复、不可修复则熔断、首写 ENOENT 为瞬时、日志消失为致命)、调度器 claim/续约/释放重试策略、瞬态连续失败熔断、按激活预算收敛、dispose 竞态唤醒保护、内存阻塞轮询的减速节拍(每 tick 一次存储扫描,从 1 秒翻倍至 30 秒封顶,队列被清空的那一拍解除阻塞),以及封存失败路径的响亮抛错与授权失效。npm run build && npm run bundle与npm run typecheck通过;packages/web-shell的 OpenAPI 契约同步测试通过(契约描述更新后已重新生成客户端)。前后对比证据
N/A —— 无用户可见的 UI 变更。
测试平台
环境(可选)
仅单元/集成级验证(Java broker 用 H2 内存库,TS store 用故障注入 fs mock);hosted 路径由上述套件覆盖,未跑真实 broker。
风险与范围
dispatch.max-post-admission-retries、dispatch.max-operation-retries,默认均为 10)并在启动时校验。命中预算的生命周期操作现在读作failed并带失败码,而不是completed——这是更诚实的回答,也是契约文本现在描述的回答。*IT集成测试由 CI 的 Hosted process fault gates 通道运行(本地需要 Docker/MySQL);materializer 的退避在内存中(若运维需要跨重启的终态 consumer-progress,可作为后续工作)。max-pre-admission-retries大于max-post-admission-retries的配置——该组合在以前是合法的(并且会静默地剥夺迟受理 turn 的重试)。外部配置中存在该组合的部署需要在升级前对齐两个键;报错信息会指明两个值。除此之外:新增两个带安全默认值的可选配置键;turn 侧新增一个失败码;lifecycle/action 侧新增失败码(自由字符串字段,无枚举变更)。关联 Issue
Fixes #13182