Skip to content

ci: diversify RunsOn capacity and bound retained test tails - #156158

Closed
steipete wants to merge 32 commits into
mainfrom
ci-burden/s2-runson
Closed

steipete wants to merge 32 commits into
mainfrom
ci-burden/s2-runson

Conversation

@steipete

@steipete steipete commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

Related: #155403. Head: bc5cdbe0a7dd6607b005c912d93ac74092ea993b.

Problem and behavior

Blacksmith's requested 32/16 classes deliver eight/four CPUs; measured workloads use roughly two to five busy cores. This opt-in RunsOn profile moves suitable complete workloads to appropriately sized AWS allocations, retains measured Blacksmith advantages, and splits the observed 1,142-second CLI tail through the existing packing owner. Production remains hybrid; this PR requests neither a backend switch nor auto-merge.

AWS requests offer M8azn/M8a/C8a/M7a/C7a alternatives across every AZ exposed by the existing us-east-1 stack, preserving memory floors, on-demand capacity fallback and retry=false. Actual instance/market/CPU/memory/AZ receipts determine the comparison; no unmeasured single-thread speed guarantee is claimed for older alternatives. Unknown or long work uses on-demand. Memory32 also uses on-demand after four explicit interruptions among 17 diversified Spot allocations in run 36050763810. Every AWS row currently emitted by this candidate is on-demand, so the campaign does not qualify Spot reliability.

A positive forecast plus the existing 320-second market reserve must fit within 480 seconds for Spot. This is admission policy, not a raised execution deadline or a fitted interruption hazard. The original 19/36 pooled interruption rate mixes correlated AMD losses with unaffected pools. Captured-rate break-even probabilities were 64.26% for repeated full losses and 81.97% for a hypothetical eight-minute attempt followed by on-demand replacement at the observed 189.84-second mean lost allocation. No interruption retry is enabled.

The intact 134-file/three-child tooling cohort uses its two native observations before the existing long-tooling route, selecting general16/on-demand with a 634-second forecast. Packing preserves complete inventories, builds, worker policies and the 90/70/130 caps. The RunsOn Node plans contain 123 PR rows and 65 main-shaped rows. Hosted lint/type routing stays unchanged.

Critical-path gates

Ordinary merge CI 36064878896 passed all 174 executed jobs but took 2,020 seconds raw / 2,019 seconds to ci-gate. Baseline job 107852905814 waited 568 seconds and ran 187; final gate 107860989914 waited 300 seconds and ran four. These waits motivate the narrow paid-control route: existing admitted first-attempt hybrid/RunsOn contexts use Blacksmith16 for the baseline prerequisite (four actual CPUs, matching the hosted sample) and Blacksmith4 for the Bash-only final gate, adopting R4 #156129's control-job direction.

One preflight decision owns routing and hosted counts. The existing admission reservation preserves every optional hosted offload decision. Checks, dependencies, permissions, release runner-group wrappers, deadlines and protected lint/type stripes remain intact. Missing/failed preflight, untrusted contexts, GitHub override, retries and ordinary manual dispatches retain their existing behavior. Native timing from the following ordinary merge run is recorded below.

Qualification

First exact-head PR sample failed: run 36072751716, attempt one at cfdd4c1446b3870463e8493f400d655c779136f6, finished in 1,280 seconds raw and through ci-gate. It passed 176 jobs, skipped 13 and failed three owner rows plus the aggregate gate. Blacksmith consumed 387.433333 machine-minutes / $9.862800; AWS reference compute was $2.896327; combined $12.759127. All 62 strict AWS allocation receipts, source receipts and market checks validate; all allocations were on-demand across two AZs with zero interruptions. Failed child coverage is incomplete. No qualification or savings credit is claimed, and no later slot or rerun has been dispatched.

The three failures are a retained Blacksmith CLI transport EPIPE race (107879799307), a tooling fixture assuming a committed Git object is loose (107879798618), and the previously bounded compiler cancellation-noise mismatch (107879798821). The transport and Git-fixture repairs are now proven below, without weakened assertions or deadlines. Hosted preflight waited 384 seconds and ran for 103; the existing live permission/PR checks must precede paid allocation, so that admission policy remains intact. Four green exact-head samples within 900 seconds and successful same-head hybrid controls are still required before the 40% main / 30% PR Blacksmith-minute targets can be claimed.

The preceding c4b2f4c787ad candidate's PR sample 36050763810 failed at 885 seconds: four explicit Spot interruptions, two unresolved shutdown cancellations, a Blacksmith ratchet-fixture failure and the failed aggregate gate. All 61 allocation/source receipts validated, but interrupted work lacks completed coverage. Blacksmith was 421.783333 machine-minutes / $10.634400; AWS reference compute was $2.337276; total $12.971676. This failed sample earns no qualification or saving credit and was not rerun.

Ordinary merge 36064878896 used 755.716667 Blacksmith machine-minutes / $29.241067 and no AWS. Its 86-selector/two-child tooling tail took 822 seconds; this differs from the frozen candidate's 134-file/three-child cohort and is not a direct speed comparison or same-head control. Reference prices exclude storage, network, control-plane, teardown and invoice reconciliation; GitHub-hosted compute is treated as free.

Owner repairs and proof

The routing campaign exposed fixture and ownership defects. Completed proof is retained against its exact source; changed integration inputs receive current-head CI rather than duplicate stress campaigns.

Owner Mechanism and proof
CI routing/packing Intended before/after runner regressions; 19 focused cases; all eight planner/market profiles and 43 plugin payloads; workflow validation, changed checks, formatting and P2 review. The original 1,142-second cohort is split without dropping work or raising workers/deadlines.
Gateway fork cohorts Main-thread-only database broker was reached from thread pools. Boot and relay reuse the existing fork owner. Mapping regressions fail before the repair; 288 routing cases and 47 native phases pass, including 20 runs per file and three canonical children per owner.
Gateway resident refresh Wait only for the captured oldest row's unreceived facts; complete accepted/clean/removed rows and join the actual producer. Controlled negatives, 20 worker plus 20 accepted-facts rounds, three children per owner and latency integration pass. Bulk interference is demonstrated; the original 101.604ms whole-CI excess remains unattributed.
UI session fixture Register active run identity and preserve matching early terminal receipts. Original-shape regression, focused endurance, canonical shards and matched native providers pass.
Host-hook and delivery fixtures Drain canonical session/database owners before unchanged directory removal. Each repair passed 20 standalone runs, three actual children and owner siblings. Diagnostics show live SQLite descriptors before cleanup and none after. Original late-created-file races were not reproduced; passing replays alone are not claimed as attribution.
Copied runner lifecycle fixture Missing registerSourceRunnerServiceFixture let runtime-publication admission exit before the fake child. Native proof: 20/20 standalone runs, three exact 80-file/2,414-case children (212.26/217.96/182.47s), 71 siblings. Same-platform cold control was three failures/two passes, then five passes after repair. Run 36044826845.
Release writer fixtures Supply rejecting synthetic I/O and verified artifact/manifest/SDK/npm facts through the real writer. Twenty standalone runs and three complete 43-file children passed. Adopt #157486's producer-only policy and authoritative state/summary owners; publication gates/waivers stay unchanged. Final integration passes 631 cases.
Landed Android/failover/wrapper owners Reuse #157397's canonical Android fixture, #157406's behavior-preserving 73/19 failover layout, and #157394's wrapper dependency inventory. Final layout: 128 focused cases and the complete 358-selector child pass; wrapper: 106 cases pass/one platform skip. Earlier 20×/3× evidence retains its original binding.

The historical UI RPC timeout (35859004781 / 107174968410) remains unexplained and its spec stays in a disjoint Blacksmith row. The matched five-case native pair passed on both providers with identical selectors/workers/runtime: complete jobs 150→94 seconds, UI steps 30→23 seconds. Those successes do not prove an environment cause. The Gateway retained-history pilot passed on RunsOn (35887809735 / 107273123947), 9.132 seconds for the case and 424 seconds for its job, without an NVMe/cgroup workaround.

Two later failures received bounded diagnosis without speculative repairs: the original 112-file compiler-ratchet replay passed 3,248 cases but emitted cancellation noise; the exact CLI mismatch remained unreproduced. The ordinary merge UI deadline failure's original 79-file/375-case shard passed, and a focused trace captured successful +3ms request/+4ms reply ordering inside the existing 100ms advance. No provider-cause or flake-fix claim is made. All diagnostic leases are stopped.

Measured one-worker fixture costs include UI session creation 3.848s, host-hook 21.672s, delivery completion 16.054s median in its endurance policy, Gateway worker/accepted files 29.954/43.650s, lifecycle 31.60s, Android 8.983s and release artifact contract 19.072s. Existing native child timings are retained above; current CI provides candidate integration/performance evidence.

Final control-gate proof: six original runner regressions reproduced; 29 admission cases passed; 741 owner/sibling cases passed with 12 existing skips. P2 found stale preflight outputs could retain paid routing on partial reruns. The selectors now also require the current attempt to be one; retained-output regressions failed before the correction and all 49 focused cases pass afterward (11.54 seconds). Final P2 review is scoped-clean with no actionable P0–P2 findings. The actual changed-file gate, owning typecheck, typed lint/canary, full workflow validation, formatting and whitespace checks pass. Thirty other job definitions and both control jobs’ non-routing fields are unchanged.

The test-only main overlap is resolved without importing Android behavior: the control cases moved byte-for-byte within their existing describe, and equivalent reservation arithmetic preserves main’s manifest-call extension. Exact three-way merge proof is clean, all 29 affected cases pass (13.47 seconds), the actual changed-file gate passes, and its separate P2 review is scoped-clean. Production, dependency and routing inputs are unchanged by that reconciliation.

Ordinary merge run 36071199008 finished failed in 925 seconds: 156 successes, 18 failures and 14 skips. Sixteen checks inherited main’s stale UpdateStepResult import, independently repaired by #157659 (b5cfc8dd9221); that importer is absent from the frozen candidate. The separate question-guest UI fixture timeout is repaired on main as 229ab8a1fe957d3d4822309a39ddf9c7ec616395; the aggregate gate also failed. The paid baseline waited eight seconds and ran 92; the final gate waited six and ran six. Their combined reference compute was $0.049867. The longest remaining path was hosted core lint (444 seconds waiting, 367 running), whose routing is unchanged. Total ordinary Blacksmith usage was 712.183333 machine-minutes / $28.254133, with zero AWS. Failed, different-merge evidence earns no qualification, control or savings credit.

The next candidate bc5cdbe0a7dd6607b005c912d93ac74092ea993b repairs two of the three failed frozen-owner rows. The failed cfdd4c1 sample is retained unchanged; no unchanged-head rerun or later qualification slot was dispatched.

Repair Mechanism and proof
Anthropic CLI transport Stdin write callbacks and error events could expose raw EPIPE before native startup diagnostics. A single failure reconciliation preserves genuine live-child pipe failures, protocol errors, explicit abort identity, and immediate credential disposal. Four deterministic real-child ordering/precedence regressions fail on original production; the live-child control passes. Final source passes 20/20 owner runs, 37 siblings, and 3/3 original 74-file/1,356-case provider children at their two-worker policy (49.742/42.158/38.140 seconds). One-worker file wall: 24.136 seconds. The uninstrumented original full-row replay passed once, which is non-reproduction, not proof of a fix. Native run 36075111400.
Frozen-target Git fixture The test assumed a committed blob occupied a loose object path and could fail with ENOENT before admission. A synthetic commit now references an absent blob independently of loose/packed storage and preserves working bytes. Forced packing makes all three original cases fail before product execution; the fixed cases reach real rejection without hydration. Twenty owner runs pass 104 cases each; three original children pass 42 files/1,291 cases plus six existing skips (179.547/153.420/141.548 seconds); four sibling files pass 218 cases. One-worker file wall: 43.771 seconds. Native run 36077041209. The historical runner's exact storage mechanism remains unknown.

Both owner patches have scoped-clean P2 review. The combined actual changed-file gate, formatting and whitespace checks pass. All eight saved Node profiles, 43 plugin payloads, workers, coverage and market requests are identical to the prior candidate; this is topology proof, not a runtime claim. No lockfile, deadline, original assertion, backend setting or protected hosted route changed.

Compiler cancellation noise remains unresolved. The upstream native-server shutdown fix in microsoft/TypeScript#64276 is merged, but the currently published stable dependency is still 7.0.2. No stderr filter, dependency patch/upgrade, private transport access, or broadened shutdown budget is included. Hosted qualification admission still performs its existing live authorization checks before paid allocation, and R4 #156129 remains open. Qualification and the savings targets remain unproven.

The main-only guest-question fixture failure was deterministic: manual admitted authority lacked the live role-assignment reader added by main, so question registration threw before responding and its deferred never settled. The existing Gateway capture owner now constructs the fixture authority and cleanup releases it. The original native shard failed; the unchanged test passes 20 standalone runs, three original 78-file/353-case children (370.956/356.626/371.884 seconds), and 57 sibling cases. P2 review is scoped-clean. Fresh integration also passes the focused E2E, owning UI type graph, actual changed-file gate and source checks. The one-file repair landed directly on main as 229ab8a1fe957d3d4822309a39ddf9c7ec616395, Main push CI 36080144666 completed successfully with only security-fast executed; the current hourly-CI policy skipped preflight, tests and ci-gate. That push is not UI execution proof; the native campaign/current-main integration above supply the executed evidence. It does not alter the frozen S2 source or routing.

Ordinary merge run 36079187703 at bc5cdbe completed in 802 seconds raw and to ci-gate: 173 success, three failures and 14 skips. Blacksmith consumed 686.116667 machine-minutes / $27.272800 reference compute; no AWS. Paid baseline waited six seconds and ran 84; final gate waited 30 and ran six. The final compute row took 594 seconds. The run's two owner failures were the inherited main guest-question fixture (now landed above) and update/restart's VM fixture: main added a mixed default/named builtin import and real filesystem authority/removal dependencies that the fixture did not supply. The latter fixture repair is now proven and landed below without changing production or its 19 assertions. Aggregate gate failed. This ordinary merge run meets the wall bound but is red and is neither an exact-head qualification sample nor a valid hybrid control.

The inherited update/restart VM fixture repair landed directly on main as 691d2e5cd48c0ddda650123b8b9ee0089fa39004. The fixture now exposes real builtin namespaces and loads the actual pinned filesystem authority/removal closure; production, all 19 native cases, assertions and deadlines are unchanged. Native Linux before/after: 16/19 pass with three default-export link failures → 19/19 pass. Twenty owner runs pass at eight workers; three saved 48-file children pass 1,500 tests each at two workers (219.595/214.022/215.836 seconds); four siblings pass 137 cases with nine existing skips. One-worker file wall is 2.559 seconds. These main-source children omit only S2's three additional planner cases, recorded separately from the failing merge's 1,503-case count. P2, typed lint, formatting and final source/runtime/lock guards pass. Fresh-main Knip input changes received a new actual changed-file gate and 39 passing dead-code contract cases, without repeating the native campaign. All associated test leases are stopped.

Qualification remains incomplete: zero green qualifying samples on bc5cdbe, no successful same-head hybrid controls, and no claim that the Blacksmith-minute targets are achieved. The compiler shutdown dependency repair awaits explicit approval under repository policy; no dependency, stderr assertion, hosted lint/type route, or production backend was changed. R4 remains open. The failed samples are retained, and no unchanged CI run was retried.

The second main push run 36082149605 also completed successfully with only security-fast executed and 33 jobs skipped under the hourly policy. Neither main push supplies test-execution or routing-qualification credit; the recorded native campaigns are the executed owner proof.

@clawsweeper

clawsweeper Bot commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

ClawSweeper review complete

ClawSweeper finished reviewing this revision. The review result is being finalized.

View the workflow run.

@openclaw-barnacle openclaw-barnacle Bot added docs Improvements or additions to documentation scripts Repository scripts size: M maintainer Maintainer-authored PR labels Sep 23, 2026
@clawsweeper clawsweeper Bot added P2 Normal backlog priority with limited blast radius. merge-risk: 🚨 automation 🚨 May affect CI, automerge, proof capture, label sync, or maintainer automation. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. labels Sep 23, 2026
@clawsweeper

clawsweeper Bot commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

Codex review: blocked before merge. Reviewed October 7, 2026, 4:20 PM ET / 20:20 UTC (Revision 46).

ClawSweeper review

What this changes

The branch expands opt-in AWS runner placement, splits measured CI test tails, moves admitted control jobs to Blacksmith, and repairs subprocess diagnostics, Gateway scheduling, and test fixtures.

Merge readiness

⛔ Blocked before merge - 5 items remain

The central routing expansion remains absent from main, and this protected, maintainer-owned branch retains distinct, measured work. It is not a cleanup-close candidate.

Priority: P2
Reviewed head: bc5cdbe0a7dd6607b005c912d93ac74092ea993b

Review scores

Measure Result What it means
Overall readiness 🦐 gold shrimp (3/6) Useful implementation and substantial native evidence remain, with a persistent documentation defect and unresolved whole-run qualification.
Proof confidence 🐚 platinum hermit (4/6) Sufficient (terminal): The complete captured body records native AWS allocation execution, production Anthropic transport exercised with real child failures and aborts, and Gateway worker-boundary recovery. These establish changed-path behavior, while final-head whole-run performance qualification remains a separate unresolved risk; no stored-data model changes were introduced.
Patch quality 🦐 gold shrimp (3/6) 1 actionable review finding remain.

Verification

Check Result Evidence
Real behavior Verified Sufficient (terminal): The complete captured body records native AWS allocation execution, production Anthropic transport exercised with real child failures and aborts, and Gateway worker-boundary recovery. These establish changed-path behavior, while final-head whole-run performance qualification remains a separate unresolved risk; no stored-data model changes were introduced.
Evidence reviewed 11 items Pinned introduction reviewed: Reviewed the merge-base-to-head introduced hunks, separating them from base-branch drift. The checkout remains clean, and the introduced diff passes the read-only whitespace check.
Prior fragment finding remains: The introduced heading rename removes the published runson-remains-unqualified fragment; docs/ci/runners.md:285 still targets it, and no explicit replacement anchor exists. The previously reviewed head is identical to this checkout.
Current main retains narrower routing: Current main's routing owner extracts cron into runson-c8i-8xlarge; it does not contain this branch's general16/memory32 placement or measured serial repricing. This establishes a meaningful remaining contribution without treating endpoint drift as PR-owned changes.
Findings 1 actionable finding [P3] Restore the published RunsOn cost fragment
Security None None.

How this fits together

OpenClaw's CI planner combines selected tests, workload measurements, backend settings, and admission checks to choose runner jobs. Those jobs execute validation and feed the aggregate merge gate.

flowchart TD
  A[Selected tests and measurements] --> C[CI planner]
  B[Backend and admission checks] --> C
  C --> D[Runner placement and packing]
  D --> E[Hosted and Blacksmith jobs]
  D --> F[Opt-in AWS jobs]
  E --> G[Validation results and merge gate]
  F --> G
Loading

Before merge

  • Restore the published RunsOn cost fragment (P3) - Renaming this heading removes #runson-remains-unqualified, but docs/ci/runners.md:285 still links to that fragment. Readers following the qualification link cannot reach its intended section. Preserve the published fragment with an explicit anchor; the prior finding remains unfixed on the identical reviewed head.
  • Resolve merge risk (P1) - The expanded runner placement and packing remain unqualified against the documented whole-run targets: the final head has no green qualifying samples or successful same-head hybrid controls.
  • Resolve merge risk (P1) - GitHub reports merge conflicts; the refreshed candidate needs review against current-main owners, including the later merged Anthropic transport repair.
  • Complete next step (P2) - Restore the published fragment, resolve current-main merge conflicts while preserving newer owner repairs, and complete the documented routing qualification.
  • Add data-model compatibility proof - Confirm migration or upgrade compatibility proof before merge.

Findings

  • [P3] Restore the published RunsOn cost fragment — docs/ci/routing-costs.md:56
Agent review details

Security

None.

PR surface

Source +62, Tests +3793, Docs +345, Config +75, Other +389. Total +4664 across 41 files.

View PR surface stats
Area Files Added Removed Net
Source 3 81 19 +62
Tests 25 4282 489 +3793
Docs 4 419 74 +345
Config 1 139 64 +75
Generated 0 0 0 0
Other 8 416 27 +389
Total 41 5337 673 +4664

Review metrics

Metric Value Why it matters
Production and test growth Production/workflow net +525 LOC; tests, support, and fixtures net +3,793 LOC Production growth is explained by expanded routing, measured packing, transport reconciliation, and scheduling; most test growth consists of retained workload fixtures.
Qualification coverage 0 green qualifying samples on final head; 4 required Native execution evidence does not yet establish the documented whole-run performance target.

Stored data model

Persistent data-model change detected: migration/backfill/repair: scripts/lib/vitest-shard-metadata.mts. Confirm migration or upgrade compatibility proof before merge.

Merge-risk options

Maintainer options:

  1. Finish native qualification (recommended)
    Refresh the integration and complete the documented same-head main/PR qualification and hybrid controls before accepting the expanded routing campaign.
  2. Pause the routing expansion
    Retain current hybrid operation while separating already-landed owner repairs from the remaining capacity experiment.

Technical review

Best possible solution:

Keep hybrid as the operating default and qualify the expanded route through the existing planner, preserving complete coverage, resource limits, fallback behavior, and newer owner repairs.

Do we have a high-confidence way to reproduce the issue?

Yes for the remaining documentation defect: the pinned branch removes a heading fragment that the runners page still references. The broader routing goals were assessed from recorded native runs, without executing target code.

Is this the best way to solve the issue?

The existing planner and lifecycle owners are appropriate boundaries, but acceptance remains incomplete until integration is refreshed and whole-run qualification establishes the intended result.

Full review comments:

  • [P3] Restore the published RunsOn cost fragment — docs/ci/routing-costs.md:56
    Renaming this heading removes #runson-remains-unqualified, but docs/ci/runners.md:285 still links to that fragment. Readers following the qualification link cannot reach its intended section. Preserve the published fragment with an explicit anchor; the prior finding remains unfixed on the identical reviewed head.
    Confidence: 0.99

Overall correctness: patch is correct
Overall confidence: 0.8

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning medium; reviewed against e73028f81cda.

Labels

Label changes:

No label changes.

Label justifications:

  • P2: This is a bounded CI efficiency and reliability improvement without evidence of an urgent user-facing regression.
  • merge-risk: 🚨 automation: The introduced changes alter runner allocation and test packing, whose complete workflow latency and reliability remain unqualified.
  • rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🐚 platinum hermit and patch quality is 🦐 gold shrimp.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (terminal): The complete captured body records native AWS allocation execution, production Anthropic transport exercised with real child failures and aborts, and Gateway worker-boundary recovery. These establish changed-path behavior, while final-head whole-run performance qualification remains a separate unresolved risk; no stored-data model changes were introduced.
  • proof: sufficient: Contributor real behavior proof is sufficient. The complete captured body records native AWS allocation execution, production Anthropic transport exercised with real child failures and aborts, and Gateway worker-boundary recovery. These establish changed-path behavior, while final-head whole-run performance qualification remains a separate unresolved risk; no stored-data model changes were introduced.

Evidence

What I checked:

  • Pinned introduction reviewed: Reviewed the merge-base-to-head introduced hunks, separating them from base-branch drift. The checkout remains clean, and the introduced diff passes the read-only whitespace check. (bc5cdbe0a7dd)
  • Prior fragment finding remains: The introduced heading rename removes the published runson-remains-unqualified fragment; docs/ci/runners.md:285 still targets it, and no explicit replacement anchor exists. The previously reviewed head is identical to this checkout. Public docs: docs/ci/routing-costs.md. (docs/ci/routing-costs.md:56, bc5cdbe0a7dd)
  • Current main retains narrower routing: Current main's routing owner extracts cron into runson-c8i-8xlarge; it does not contain this branch's general16/memory32 placement or measured serial repricing. This establishes a meaningful remaining contribution without treating endpoint drift as PR-owned changes. (scripts/lib/ci-node-test-plan.mts:4590, e73028f81cda)
  • Latest-release boundary checked: The v2026.9.8 source also retains the narrower runson-c8i-8xlarge cron route. No release implementation of the central expansion was established. (scripts/lib/ci-node-test-plan.mts:4501, fc23bc864e45)
  • Complete captured proof body inspected: Read the full 17,943-unit body and verified its SHA-256 matches the supplied snapshot: d35fc5710dbe70825f176e2ebd076eba60d4d88336a75be07131be3916e74554. It records real AWS allocation receipts, native subprocess fault-ordering proof, Gateway worker proof, and complete child executions. It explicitly reports zero green qualifying samples on the final head and no successful same-head hybrid controls. (bc5cdbe0a7dd)
  • Qualification contract: The branch documents two green main-shaped and two green PR-shaped workflows at the same head within 900 seconds. Recorded successful children and failed workflows demonstrate execution but do not satisfy whole-run performance acceptance. The maintainer-authored scope keeps production hybrid and requests neither a backend switch nor auto-merge. Public docs: docs/ci/runners.md. (docs/ci/runners.md:287, bc5cdbe0a7dd)

Likely related people:

  • steipete: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)
  • vincentkoc: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)

Rank-up moves

Optional improvements that raise the rating; they are not merge blockers.

  • Restore the published RunsOn fragment.
  • Complete the documented same-head qualification and hybrid controls after refreshing integration.

Rating scale

Score Internal tier Crab rank Meaning
6/6 S 🦀 challenger crab Exceptional readiness
5/6 A 🦞 diamond lobster Very strong readiness
4/6 B 🐚 platinum hermit Good normal PR; ordinary maintainer review
3/6 C 🦐 gold shrimp Useful, but confidence is limited
2/6 D 🦪 silver shellfish Proof or implementation needs work
1/6 F 🧂 unranked krab Not merge-ready
N/A NA 🌊 off-meta tidepool Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (45 earlier review cycles; latest 8 shown)
  • reviewed 2026-09-24T21:34:01.319Z sha 9893da3 :: blocked before merge. :: [P3] Preserve the published RunsOn cost fragment | [P3] Align the release manual with the performance gate
  • reviewed 2026-09-24T22:08:09.799Z sha ea0f0d6 :: blocked before merge. :: [P3] Restore the published RunsOn cost fragment
  • reviewed 2026-09-24T22:18:54.372Z sha ea0f0d6 :: blocked before merge. :: [P3] Preserve the published RunsOn cost fragment
  • reviewed 2026-09-24T23:19:32.217Z sha cfdd4c1 :: blocked before merge. :: [P3] Preserve the published RunsOn cost fragment
  • reviewed 2026-09-25T00:07:57.189Z sha cfdd4c1 :: blocked before merge. :: [P3] Restore the published RunsOn cost fragment
  • reviewed 2026-09-25T00:57:36.066Z sha bc5cdbe :: blocked before merge. :: [P3] Restore the published RunsOn cost fragment
  • reviewed 2026-09-25T01:13:35.343Z sha bc5cdbe :: blocked before merge. :: [P3] Restore the published RunsOn cost fragment
  • reviewed 2026-09-25T01:42:52.582Z sha bc5cdbe :: blocked before merge. :: [P3] Restore the published RunsOn cost fragment

@clawsweeper clawsweeper Bot added proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. and removed rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. labels Sep 23, 2026
@clawsweeper clawsweeper Bot added status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. and removed proof: sufficient ClawSweeper judged the real behavior proof convincing. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. labels Sep 23, 2026
@steipete

steipete commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor Author

TP-G1 coordination from #156270: the real-Gateway inventory now has two planner-owned rows (6/17 files, 7/32 cases, desktop proof once), with unchanged worker limits and runner labels. Exact-head main qualification 35851552173 at acd29809b9cc4fb3bd2fb79af5f7087d9229b717 passed all 39 UI cases, but complete rows took 492/500 seconds, over the required 450-second budget. Both used the existing 32-class, reporting eight CPUs and about 32 GiB.

The split is balanced: two standalone companions run together after the desktop row's serial phase, without a bundled preview build. The build still costs 87/81 seconds (bootstrap validation 37.1/31.1); UI steps 216/350 seconds, with desktop 122 seconds only in row 1. SDK declarations are already removed from these runtime-only consumers and remain validated in build-artifacts. No timeout, retry, coverage, runner-label, variable or ruleset changes were used to obtain these results.

Your current routing head deliberately retains real-Gateway on Blacksmith. Please qualify a route with higher clock speeds for these two rows through S2's runner owner, using the S2b m8azn.3xlarge evidence or the cheapest measured class that fits. TP-G1 will keep the split in planner data and leave label/backend policy with S2. The two-worker bound and exact file inventory must remain unchanged, and the final native jobs must each pass within 450 seconds. No auto-merge is armed on TP-G1.

@clawsweeper clawsweeper Bot added proof: sufficient ClawSweeper judged the real behavior proof convincing. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. and removed status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. labels Sep 23, 2026
@openclaw-barnacle openclaw-barnacle Bot added app: web-ui App: web-ui and removed size: M labels Sep 23, 2026
Replace the single AMD capacity pool with prioritized instance alternatives across the configured RunsOn subnets. Admit Spot only when a known forecast plus measured startup allowance fits eight minutes; keep interruption retries disabled and use on-demand otherwise.

Split the underpriced CLI runtime cohort without dropping coverage or changing workers, deadlines, caps, or hosted stripe routing. Retain the unresolved GitHub-projects UI spec in a disjoint Blacksmith row and pin AWS Node for comparable runtime evidence.
Creation responses marked sessions active without registering their run IDs, so terminal events could leave stale running state. Materialize identity and run receipts through the shared fixture owner, preserve early terminal or abort receipts, and discard unrelated state when the session generation changes.

Extend serialized RPC coverage for creation, replay, replacement, and early settlement. Include the awaited RPC method in existing timeout errors without logging request payloads or changing deadlines. This repairs proven fixture defects; the historical AWS timeout remains isolated rather than declared fixed.
The RunsOn section rewrite accidentally removed the following hosted-assignment guidance and Related links. Restore that existing tail byte-for-byte so the assignment-guard anchor resolves and the separately owned hosted routing documentation remains intact.

Validation: markdownlint scanned 1306 files with zero issues; docs listing, formatting, diff checks, and P2 review pass.
Direct AWS proof used a different operating-system image and could not qualify the unexplained native RunsOn RPC timeout. Add a main-qualification-only counterpart with the retained spec's exact selector and worker policy, and record both providers' actual toolchains. Production routing and retry policy remain unchanged.
The measured 85-file tooling envelope took 786 seconds on Blacksmith's two visible CPUs and retained that placement in the RunsOn PR plan. Move only long serial two-worker ordinary tooling to the existing diversified general pool, preserving all selectors, packing, workers and deadlines. Its forecast keeps the allocation on-demand.
Four successful Spot rows exceeded eight minutes because the planner underpriced their work. The observed forecast-to-allocation gap reached 307 seconds. Use a 320-second market-only reserve to prefer on-demand for those rows while preserving packing, workers, deadlines and interruption policy.
Host-hook session writes schedule maintenance after returning, leaving agent databases open when the fixture deletes its state directory. Drain the existing session cleanup owner and close those handles before removal; extract the fixture to preserve the contract file limit.
Resident row refresh can consume synchronous render slices while an exact
worker read is pending. Capture one existing exact completion and yield
before admitting the next resident slice, preserving readiness checks and
caller-owned errors without letting later arrivals starve bulk progress.

A controlled real-worker regression fails on the original scheduling and
proves both exact response priority and bounded resident progress. The
historical CI latency excess remains unproven as a whole; the unchanged
latency budget still gates the qualification campaign.
Waiting for an exact caller after its database facts were accepted blocked
resident materialization of its pinned archive suffix. Use the oldest
row's existing dirty, cold and accepted-facts state to wait only for facts
that still need to arrive, preserving the supported bulk completion path.

The handoff fixture also treated fake timer admission as asynchronous
refresh completion. Join the actual suffix render before asserting its
order, keeping the later exact request held and every assertion intact.
Matched whole-job observations show six serial workloads exceeding the
25% RunsOn slowdown threshold in both samples. Recognize their co-resident
owners after packing and retain all execution policies. Constrained native
fits support five 8-class candidates; the SDK declaration and planner pair
needs the 16-class for memory headroom. Actual-label qualification follows.

The cron comparator lacked its peer's admission priority and appeared last
in a 118-row matrix. Admit both providers together without changing row
payloads, other ordering, the cap, worker limits, or deadlines. Preserve the
unexplained residual scheduling delay in the evidence.
Main repacked the unified-declaration envelope from the measured 80-file
workload to 47 files with 37 different companions. Its earlier constrained
fit cannot qualify this new job on the 8-class runner. Preserve the original
32-class allocation while retaining its execution contract and row count.

Record the expired workload bindings and current planner inventory separately
from historical fit evidence. The remaining guarded retention shapes no
longer match the current broad plans.
Boot and Talk relay cleanup can reach the main-thread-only shared-state
broker after an unsettled native worker exit. Their ordinary Gateway
thread pool cannot own that cleanup authority. Classify both files with
the existing database-worker fork owner and pin the routing boundary in
the existing regression table.

Keep production admission guards, assertions and deadlines unchanged.
The mapping regression fails on the old owner; the native exit trigger
was not reproduced by either valid original-child control.
The delivery-completion fixture removes its private store while background
session maintenance can still own database resources. Native observation
found writable SQLite descriptors at every old removal boundary; the
canonical scoped drain closes them before removal.

Await the existing session cleanup owner before the unchanged zero-retry
fs.rm. Keep assertions, mocks, environment and timeouts unchanged. The
historical ENOTEMPTY failure was not reproduced by the original row replay.
Bind current row counts and retained compiler capacity to the rebased
inventory. Record the new baseline-ratchet prerequisite so older workflow
walls and small-host probes are not presented as current qualification.
The obsolete-only scenario inherited unrelated drift when main added native
strings before the generated-resource refresh. Seed its existing in-memory
resource overrides from the canonical catalog, then inject the obsolete row.
Keep the strict validator, all negative assertions, and production data intact.
Publish/all manifest writing now seals qualified npm inputs, but the workflow
command fixtures still cleared the required token and artifact facts. Supply
rejecting synthetic GitHub, archive, SDK, and npm inputs through that real
writer boundary and assert its sealed result.

Restore the documented stable/full blocking performance policy shared by
fresh manifest facts and failure classification. Keep beta advisory behavior,
reused manifest evidence, artifact integrity checks, and waiver authority.
Runtime-publication regressions pushed the failover classification file past
the existing 1,000-line lint limit. Move its eight unchanged remediation-hint
cases into the existing diagnostics owner, with the required command import.

Keep every assertion and both files in their existing CI child. Production
behavior, test inventory, worker policy, deadlines, and lint limits stay intact.
Adopt the canonical fixture from #157397. Build the synthetic obsolete rows
directly from generated catalog entries while preserving strict validation,
live-reference checks, and cleanup.
Adopt the canonical grouping from #157406. Keep coordination-error cases in
the existing diagnostics owner and retain all 92 case bodies, assertions,
parameter tables, describe paths, and provider-hook isolation unchanged.
The extracted PR wrapper reaches openclaw-state.worker, whose eager plugin
state contract import was absent from the closed source inventory. Native
main-refresh checks then exited with a missing-module error before their
intended boundary.

Carry Vincent Koc's one-entry fix from 9a1a6f7 (#157394). The eager
closure, native provisioning and main-refresh owner tests pass 106 cases
with one platform skip; changed checks and P2 review pass.
The copied runner fixtures reached runtime-publication admission before
spawning their fake child. Unlike the original-wrapper controls, they did
not register the existing source-runner service fixture, so host service
state could reject the command and surface as a child.pid readiness failure.

Install the existing fixture in the child preload before applying prepared
module mappings. Keep real runner behavior, assertions and deadlines intact.

The matching cold replay changes from three failures to five passing cases.
Native proof passes 20 standalone runs at eight workers, three exact
80-file CI children at two workers, and 71 owner/sibling cases. Changed
checks, formatting and P2 review pass.
Successful children in run 36038423993 exposed serial work hidden by old
packing estimates. Two Gateway children exceeded 370 seconds before their
siblings, two other serial cohorts exceeded 500 seconds against 360-second
forecasts, and a retained compiler cohort was admitted late at 48 seconds.

Apply exact provider and child observations in the existing placement owner
after routing. Separate intact oversized children and complete measured
serial cohorts, keep conservative sibling forecasts and worker policies,
and correct the compiler admission cost. Exact extension selector costs
also separate the measured slow plugin pair under its existing envelope.

RunsOn broad PR plans grow from 118 to 123 rows within the unchanged cap.
All push plans and complete child execution contracts remain unchanged.
Regressions fail on the original packing; final focused checks, all eight
profile replays, changed checks, formatting and P2 review pass. Native
exact-head qualification remains required for wall time and cost claims.
The diversified memory32 request still concentrated all 17 Spot allocations
on one family in run 36050763810. Four workers were explicitly interrupted,
including a short job whose forecast could not predict capacity loss.
Select on-demand for this class while preserving family and AZ eligibility,
worker limits, remaining Spot fallback and disabled interruption retries.

The intact tooling cohort completed twice on two-core Blacksmith allocations
in 502 and 557 seconds while retaining a 461-second forecast. Consume its
complete source-cohort price before the existing long-tooling route, moving
one row to general16 with a 634-second admission forecast. No shards,
workers, thresholds or deadlines are added.

Both new regressions fail on the prior head. Nineteen focused checks, all
eight profile comparisons, type-aware lint, full workflow and changed
checks, formatting and P2 review pass. Native AWS performance and final
exact-head qualification remain to be measured; no Spot reliability or
completed-run savings are claimed.
The branch's manifest repair overlaps the producer-only fix in #157486.
Use the landed producer declarations and advisory execution policy, while
preserving publication gates and explicit waivers. Remove the branch-only
execution-policy tightening and align its state, summary and docs owners.

Keep the rejecting synthetic artifact fixture and extend its writer table
to the full profile. The focused integration passes 631 cases; source
compositions, changed checks, formatting and P2 review pass. Prior native
fixture proof remains bound to its tested source and is not repeated.
Ordinary CI 36064878896 passed its test jobs, but the baseline
prerequisite waited 568 seconds for a hosted runner and the final
gate waited another 300 seconds. The workflow took 33m40s.

Reuse the existing trusted first-attempt preflight admission for
Blacksmith16 baseline ratchets and Blacksmith4 result aggregation.
The baseline class supplies four actual CPUs, matching its hosted
sample. Missing or failed preflight keeps the existing fallback.

Count the actual hosted jobs while reserving the former control
slots for admission, preserving every optional offload decision.
Retain runner-group wrappers, permissions, checks and deadlines.
The paid-control cases and a new Android budget table on main touched
the same test insertion and manifest call, blocking the PR merge ref.

Move the unchanged control cases within their existing describe and
account for the two reserved slots in the local reference base. This
preserves the assertions and emitted row count while leaving the
manifest call compatible with main's Android selection argument.
Reconcile stdin callback, pipe error, and native exit through one transport failure operation so a startup EPIPE cannot replace its diagnostic. Preserve live-child pipe errors, protocol failures, abort identity, and immediate credential disposal without changing retirement budgets.
Deleting a presumed loose object can fail before frozen-target admission when Git stores the blob in a pack. Commit a tree referencing an absent blob and bind that commit to the fixture, preserving working bytes and proving real rejection without hydration.
@steipete

Copy link
Copy Markdown
Contributor Author

Closing this unqualified runner-capacity experiment. The recorded samples did not satisfy the whole-run time and reliability gates, and the branch combines routing changes with separate runtime and fixture repairs that should be evaluated independently.

Closed in a sweep of stale maintainer PRs (no activity for 7+ days). Reopen with a current reproduction if this still matters.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agents Agent runtime and tooling app: web-ui App: web-ui docs Improvements or additions to documentation extensions: anthropic gateway Gateway runtime maintainer Maintainer-authored PR merge-risk: 🚨 automation 🚨 May affect CI, automerge, proof capture, label sync, or maintainer automation. P2 Normal backlog priority with limited blast radius. proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. scripts Repository scripts size: XL status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant