Skip to content

fix(core): match one sed character for . and bracket expressions - #15701

Merged
wenshao merged 25 commits into
QwenLM:mainfrom
2SinceJuly:fix/15675-sed-regex-flags
Nov 7, 2026
Merged

wenshao merged 25 commits into
QwenLM:mainfrom
2SinceJuly:fix/15675-sed-regex-flags

Conversation

@2SinceJuly

@2SinceJuly 2SinceJuly commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

What this PR does

The shell tool's in-process sed simulation now compiles substitution patterns with the s and u flags. This makes . match a carriage return and makes character classes and quantifiers operate on Unicode characters instead of UTF-16 code units. A literal ] outside a bracket expression is escaped during translation so the stricter Unicode mode still accepts patterns such as [\[\]].

Why it's needed

On CRLF files, s/=.*$/=new/ matched nothing and the tool reported that the edit made no changes. On files containing astral characters, s/^.// removed one half of an emoji and wrote the lone surrogate back as U+FFFD, silently corrupting the file. Real sed treats both cases as one character.

Reviewer Test Plan

How to verify

Run the sed parser unit tests. The new regression case covers the six rows from the issue: astral characters with ., a negated class, a zero-width x* match, and a CRLF substitution. Before the change it fails on the first astral row; after the change all cases pass. Also confirm that ordinary bracket patterns such as [abc], [^abc], and [\[\]] still take the simulation path rather than falling back.

Evidence (Before & After)

N/A for UI. Unit tests: 53 passed, including new pins for negated, mixed and equivalence-class POSIX forms ([^[:space:]], [[:alpha:]abc], [[=a=]]) and for the GNU/BSD multibyte-backreference split. The full core suite passed 39,351 tests; the single bun-pty.test.ts failure also reproduces on unmodified main. Typecheck, lint, and formatting pass.

Tested on

OS Status
🍏 macOS ✅
🪟 Windows ⚠️
🐧 Linux ✅

Linux was verified in a Docker node:22-bookworm arm64 container.

Environment (optional)

Node 22.23.4; unit tests only.

Risk & Scope

  • Main risk or tradeoff: the u flag can reject patterns that previously compiled, so those commands now fall back to real sed. That is the existing fail-safe path and never writes a wrong file.
  • Not validated / out of scope: Windows; GNU sed's zero-width x* iteration and multibyte backreferences are bytewise, while BSD sed and this simulation are characterwise. The POSIX-class guard now declines any pattern containing [:, [. or [=, so mixed and negated character classes also fall back to real sed instead of being simulated wrongly.
  • Breaking changes / migration notes: none.

Linked Issues

Closes #15675

中文说明

这个 PR 做了什么

shell 工具的进程内 sed 模拟现在会用 s 和 u 标志编译替换模式。这样 . 能匹配回车符,字符类和量词也按 Unicode 字符而不是 UTF-16 code unit 工作。翻译时会把字符类外的字面量 ] 转义,保证更严格的 Unicode 模式仍能接受 [\[\]] 这类模式。

为什么需要

在 CRLF 文件上,s/=.*$/=new/ 匹配不到任何内容,工具会报告没有产生修改。在包含星平面字符的文件上,s/^.// 只删掉 emoji 的一半,并把孤立代理项写成 U+FFFD,静默损坏文件。真实 sed 在两种情况下都按一个字符处理。

评审测试计划

如何验证

运行 sed 解析器单测。新增回归用例覆盖 issue 里的六行:星平面字符配合 .、否定字符类、零宽 x* 匹配,以及 CRLF 替换。修改前第一行星平面用例失败,修改后全部通过。同时确认 [abc]、[^abc]、[\[\]] 这类普通字符类仍然走模拟路径,而不是回退。

证据(修改前后)

UI 变化:N/A。单测 53 通过,新增取反/混合/等价类 POSIX 形态([^[:space:]]、[[:alpha:]abc]、[[=a=]])以及 GNU/BSD 多字节反向引用分歧的钉子。core 全量 39,351 通过;唯一失败的 bun-pty.test.ts 在未修改的 main 上也能复现。类型检查、lint、格式化均通过。

测试环境

系统 状态
🍏 macOS ✅
🪟 Windows ⚠️
🐧 Linux ✅

Linux 在 Docker node:22-bookworm arm64 容器中验证。

运行环境(可选)

Node 22.23.4;仅单测。

风险与范围

  • 主要风险/取舍:u 标志会拒绝一些以前能编译的模式,这些命令会回退到真实 sed。这是既有的 fail-safe 路径,不会写错文件。
  • 未验证/超出范围:Windows;GNU sed 的零宽 x* 与多字节反向引用按字节推进,而 BSD sed 和本模拟按字符推进。POSIX 字符类 guard 现在对含 [:、[.、[= 的模式一律回退,混合与取反写法也不再被错误模拟。
  • 破坏性变更/迁移说明:无。

关联 issue

Closes #15675

The simulated sed path compiled patterns without the s and u flags, so `.`
did not match `\r` in CRLF files and astral characters were matched as two
UTF-16 code units, leaving a lone surrogate that was written as U+FFFD.

Compile with su/gsu and escape a literal ] outside bracket expressions so
the stricter u mode still accepts patterns such as [\[\]].

Closes QwenLM#15675
@wenshao

wenshao commented Oct 27, 2026 •

Copy link
Copy Markdown
Collaborator

Verdict: merge-ready — 77/77 scripted assertions passed at head 45edd5b72e405f90465c1b337146b2b9c388359e. Against real GNU sed 4.5, the base build simulates 16 of 59 comparable cells wrongly; the PR head simulates 0 wrongly. No regressions, no blocking findings. Three Suggestions below are non-blocking.

中文摘要
  • 结论:merge-ready(77/77 断言通过,head 45edd5b7)。
  • A/B(用真 GNU sed 4.5 做 oracle,59 格可比):base 有 16 格模拟结果与真 sed 不一致,head 0 格。CRLF 下 .、emoji(星芒字符)按一个字符、字面 ] 三类全部翻转为一致;base 在没有守恒的单元格上尖峰集中在 A/B/E 组(见下表)。
  • 变异矩阵:7 杀 2 活。新测试确实钉住核心改动(把 gsu 退回 g,新测试立即红),] 转义由既有 -E 括号测试钉住。两个 survivor 都不是缺陷:M1(gate 不带 su)是会真实抛错的覆盖缺口,M8(gate 只带 u)在 59 格上与 head 逐字节相同,说明 gate 里的 s 不可观测。
  • Findings(均为 Suggestion,不阻塞):S1 GNU/BSD 在「零宽全局替换 + emoji」上仍有分歧(head 选 BSD,GNU 给 -<byte>-…);S2 gate 与 apply 的 flag 一致性没有测试钉住;S3 gate 里的 s 标志是死代码。
  • 未覆盖:整包 core 套件、typecheck、lint 未跑(只跑了改动文件套件 + shell 的 sed 子集);没有跑完整 CLI 工具调用;本机无 BSD sed,BSD 那一格用的是 PR 自己测试声明的期望值;Windows/macOS 未测。

A/B against real GNU sed

Central claim

The simulated sed -i fast path (which writes files itself instead of running sed) must treat ., bracket expressions and astral characters the way sed does: . matches a carriage return, one astral character is one character, and a literal ] outside a bracket expression stays literal now that the translated pattern is compiled with the strict u flag.

The oracle here is not hand-written expectations — it is the real sed binary. For every cell the harness builds the same command, runs it through parseSedEditCommand and applySedSubstitution in each arm (the exact two calls ShellTool makes), and runs the identical expression through GNU sed -i on a real temp file. A cell is unsafe when the simulation accepts a pattern and writes bytes sed would not; a declined pattern is safe because production then runs the real sed.

A/B result

group cells base wrong head wrong
A — . must match \r (CRLF) 6 5 0
B — astral character is one character 10 8 0
C — literal ] outside a class 11 0 0
D — regression corpus (previously supported forms) 28 0 0
E — command spellings (-i -E, --in-place, double quotes) 4 3 0
total 59 16 0

The cells are byte comparisons, not visual checks. Examples from the run (base → real sed):

cell input base wrote real sed wrote
sed -i 's/=.*$/=new/' key=old\r\n unchanged (.* cannot cross \r) key=new\n
sed -i 's/^.//' 😀 Title\n lone low surrogate + Title (U+FFFD on disk) Title\n
sed -i 's/[^a-z]/_/g' a😀b\n a__b (two surrogates) a_b
sed -i -E 's/^.//' (also -i -E, --in-place) 😀x\n \uDE00x x

Both arms were bundled separately from their own worktree; the base bundle contains no gsu and the head bundle contains exactly one, so the arms really did load different code.

Mutation matrix

Mutation matrix

mutant change suite outcome
C1 (control) replaceAll forced false head 52 killed (15 failures)
M2 apply gsu → g (full revert of the central change) head 52 killed by the new test (matches one character for . and bracket expressions)
M3 apply gsu → gu (drop s) head 52 killed by the new test
M4 apply gsu → gs (drop u) head 52 killed by the new test
M5 remove the literal-] escaping head 52 killed by keeps backslashes literal inside bracket expressions under -E (existing test)
M6 POSIX-class check back on jsPattern head 52 killed by rejects POSIX bracket expressions before simulated edits
M7 M5 + M6 together head 52 killed by the same -E bracket test
M1 gate new RegExp(jsPattern) without flags head 52 survived → coverage gap (S2)
M8 gate new RegExp(jsPattern, 'u') (drop s only) head 52 survived → dead flag (S3)

The new test is not vacuous: reverting the whole apply-side change (M2) turns it red with the assertion it was written for, and each half of the bundled s/u change is pinned separately (M3, M4). M6 shows the hasPosixBracketExpression(sedInfo.pattern) change is load-bearing rather than cosmetic: that mutant compiles the check against the translated pattern, accepts s/[[:digit:]]/X/g, and writes a1b unchanged where real sed writes aXb — a silent wrong edit (the translated pattern no longer contains the literal [[:digit:]] once the trailing ] is escaped). The existing rejects POSIX bracket expressions before simulated edits test is what catches it.

Survivors were adjudicated by building the mutant, not by reading it:

  • M1 — its build accepts s/\q/X/g and s/a\-b/X/g (identity escapes that u mode rejects) and then throws Invalid escape inside applySedSubstitution. With auto-approve there is no preview pass, so that surfaces to the user as a tool error instead of falling back to real sed. Head is consistent (su gate / gsu apply) and declines both, so this is a missing test, not a defect.
  • M8 — byte-identical to head on all 59 cells. The s flag cannot change whether a pattern compiles, so only the u in that gate is observable.

Findings

S1 — a zero-width global match over an astral character follows BSD, not GNU. sed -i 's/x*/-/g' on 😀\n gives -😀- at head (the new test pins this and the comment documents it as BSD behaviour), while GNU sed 4.5 on this host gives -\xf0-\x9f-\x98-\x80-, i.e. - between each byte. This is not a regression — base wrote -<lone high surrogate>-<lone low surrogate>-, which lands on disk as U+FFFD — and the divergence is deliberate. It is worth an explicit acceptance note because on Linux the simulation still is not "what sed would do" for this shape. Reproduce: node harness.mjs base-tree/sed-bundle.mjs head-tree/sed-bundle.mjs, cell 16. No fix proposed; declining this shape so real sed runs would also be defensible, but that is a product decision.

S2 — nothing pins the gate's flags to the substitution's flags (M1). A one-line test would close it: expect(parseSedEditCommand("sed -i 's/\\q/X/' f.txt")).toBeNull(). Today the two are written independently, and a future edit to either can reintroduce the throw-with-no-fallback path M1 demonstrates.

S3 — the s in the gate's new RegExp(jsPattern, 'su') is unobservable (M8). Not harmful, and arguably kept for symmetry with gsu; noted only so a future reader does not attribute meaning to it.

Observation (not a finding): head declines two patterns base simulated — s/\q/X/g and s/a\-b/X/g — because those escapes are invalid under u. Both fall back to real sed, so they are correct but leave the fast path (51 accepted cells at head vs 52 at base). No output drift was measured on any cell that base simulated correctly.

Corrections

None. There was no earlier verification round or bot review on this PR describing the code, so there is nothing to correct; the two author claims I tested (the ] escaping and the CRLF/astral behaviour) both hold.

Not covered

  • Full packages/core suite, npm run typecheck, and lint were not run. Only the changed file's suite (head 52/52, base 51/51) and the 31 sed-related tests in shell.test.ts (31 passed | 343 skipped) ran.
  • No end-to-end CLI tool invocation. The harness calls the same two functions ShellTool.prepareSedEdit calls, but the preview/approval plumbing was read, not driven. The auto-approve consequence in S2 is inferred from shell.ts, while the throw itself is measured.
  • BSD sed is not installed here. The BSD cell uses the expected value the PR's own test asserts, not a live BSD sed; GNU 4.5 under en_SG.UTF-8 is the only oracle that ran.
  • Windows/macOS, and the interaction with -i.bak/combined flags, are untouched by the diff and were not exercised.
  • Real CI was still in flight (precheck passed, other jobs pending/skipping) when the evidence was captured.

Methodology

Two detached worktrees, one at baseRefOid 7b09c0a9 and one at headRefOid 45edd5b7. Each tree's sedEditParser.ts was bundled with esbuild into a self-contained ESM module, so no workspace symlink can leak one arm's code into the other; the base bundle has zero gsu occurrences and the head bundle has one, as a positive control that the arms differ. The harness (harness.mjs) runs 61 cases, of which 59 are comparable (two are excluded because real sed itself rejects the expression, e.g. the empty pattern). Mutation rows run the PR's own test file via the JSON reporter and attribute every kill to a named failing test; the tree is restored with git checkout after each row and the restore is verified by sha256. Tests were run through a minimal vitest config because a bare worktree lacks the built workspace dist/ the package's global setup requires. Environment: Node v24.22.1, GNU sed 4.5, locale en_SG.UTF-8. Raw logs, bundles, and harnesses live in tmp/pr15701-verify-20261027-153859/.

wenshao
wenshao previously approved these changes Oct 27, 2026
@wenshao

wenshao commented Oct 27, 2026

Copy link
Copy Markdown
Collaborator

@qwen-code /triage

@wenshao
wenshao enabled auto-merge (squash) October 27, 2026 09:22
@2SinceJuly

2SinceJuly commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor Author

Head is f51590bb — ci: rerun the checks on the unchanged tree — same tree e00b2c97 as the previously reviewed 45edd5b7, so no code change. It was pushed only because the earlier runs on that head ended cancelled and the required checks never completed. The fresh Qwen Code CI run is action_required (GitHub is holding the fork's runs), so it needs a maintainer approval before it starts.

@qwen-code-review-bot qwen-code-review-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not explored to full depth (tool budget reached): "agent 6b": packages/core/src/tools/shell.test.ts sed-integration tests were not run — the file needs packages/core/dist/index.js , and this worktree is unbuilt; the uni…; "agent 6c": did not run the full packages/core suite (only the parser file plus targeted greps of shell.test.ts ), and did not run the mutation experiment by deleting th….

Test Plan (not a blocker): tests: 52 passed — this review observed 18646, 2035, 38776, 2278, 306, 13338 passed.

中文说明

未探索到全部深度(达到工具调用预算):"agent 6b":packages/core/src/tools/shell.test.ts sed-integration tests were not run — the file needs packages/core/dist/index.js , and this worktree is unbuilt; the uni…;"agent 6c":did not run the full packages/core suite (only the parser file plus targeted greps of shell.test.ts ), and did not run the mutation experiment by deleting th…。

Test Plan(非阻断):tests: 52 passed — this review observed 18646, 2035, 38776, 2278, 306, 13338 passed。

— deepseek-v4.1-flash-1017 via Qwen Code /review (v0.26.2)

@wenshao

wenshao commented Oct 28, 2026

Copy link
Copy Markdown
Collaborator

Verdict: blocked — 289/297 scripted assertions passed at head f51590bb9e1c95ebc7e76d754baf39ce1b4c096b (base 9df10f873b016f2db6b32cfb1a42823bcd8020d1). The 8 failures are one reproduced regression, R1-1: with -E, the new u flag turns sed-literal escapes into JavaScript Unicode metacharacters, and the simulation writes bytes that neither real GNU sed 4.10 nor real BSD sed writes (4 cells × 2 oracles). The central CRLF/astral fix itself is confirmed load-bearing: all six issue #15675 rows flip from base-wrong to head-match against both real seds. The author has acknowledged R1-1 and promised a fix commit; none has landed — the head tree is byte-identical to the one reviewed in round 1.

中文摘要
  • 结论:blocked(297 条脚本断言,289 通过,8 失败;head f51590bb)。8 条失败全部是同一个已复现的回归 R1-1(4 格 × GNU/BSD 双 oracle),归因 PR。
  • 中心修复成立:issue sed edit simulation: . does not match \r or a whole emoji, so results differ from real sed #15675 的六行在 base 上全部与真 sed 不一致,head 上全部一致(GNU 4.10 与 macOS BSD sed 双 oracle,en_US.UTF-8)。A/B 见 01-sed-oracle-ab.png。
  • R1-1(Critical,阻塞):sed -E -i 's/\p{L}/X/g' 这类模式在 head 被模拟成 JS Unicode 属性转义,把每个字母都替换掉(ab1 p{L} A → XX1 X{X} X);真 GNU sed 直接报错退出、BSD sed 按字面 p{L} 替换。\P{L}、\u{41}、\p{Lu} 同类。base 对 BSD 全部正确,故为 PR 引入的回归。字节证据见 02-r11-regression-bytes.png。作者已确认并承诺修复,但截至本轮 head 未推送。
  • 建议修复已实测:把 hasSedJavascriptDivergentEscape 扩为 [<>dDwWsSpPu] 后,5 个 hostile 格全部回退真 sed,59 个非 F 格逐字节零附带损伤,套件 54/54 绿;新增断言在 head 上变红(M11),证明断言钉得住修复。
  • 变异矩阵:12 行全部符合预期(10 杀 2 活),r1 的 S2/S3 缺口复现;R1-2 的两条验收断言能杀掉 M1(M9)。见 03-mutation-matrix.png。
  • 后续轮分类:新 head 是空 CI 重跑提交,树哈希与 r1 完全相同(e00b2c97,含阳性对照)。r1 的 merge-ready 结论被 bot 的 Critical 推翻——r1 语料未覆盖 u-only 接受类。
  • 姊妹 PR:fix(core): match sed text semantics in unicode mode #15694 与本 PR 是同一 issue 的竞争修复,trial merge 直接冲突(需维护者二选一);fix(core): keep sed anchors inside groups and alternations #14428 干净合并且合并树套件 60/60 绿。
  • 未覆盖:core 全量/typecheck/lint(CI 仍 queued);shell.test.ts sed 子集本轮环境缺新依赖无法收集(A/A 证明 base 同样失败,承接 r1 的 31/31);Windows;完整 CLI 工具调用。

Follow-up round: previous-finding status

Round 1 (2026-10-27, head 45edd5b7, tree e00b2c97) reported merge-ready, 77/77. This round's head f51590bb is an empty CI-rerun commit: both heads resolve to tree e00b2c970a49b23d4a673e4960f6a29df9b7f10b, git diff 45edd5b7 f51590bb is empty, and the comparator's positive control (git diff 9df10f87 f51590bb --stat) reports the real 2-file diff. Every carried-forward measurement below was nevertheless re-run on this machine (Node 22.23.1, GNU sed 4.10 + BSD sed, en_US.UTF-8) because the round-1 rig differed (Node 24, GNU sed 4.5 only).

Correction to round 1 (my own earlier report): its merge-ready verdict is superseded. The r1 corpus (59 cells, groups A–E) did not include the u-only acceptance class — patterns that compile under u with a different meaning than sed's. The bot review of 2026-10-28 12:50 UTC found exactly that class (R1-1), and this round reproduces it with byte evidence. r1's "0 wrong of 59" remains true for those 59 cells; the verdict was wrong because the corpus was incomplete, not because any r1 cell was miscounted.

# finding severity status at f51590bb
R1-1 (bot, 2026-10-28) u flag simulates \p{L} / \P{L} / \u{41} as JS Unicode metacharacters under -E; wrong bytes written Critical stands — reproduced on both real seds (F1/F2/F3/F8); author acknowledged 2026-10-28 13:03, fix promised, not pushed
R1-2 (bot, 2026-10-28) nothing pins the gate's u-only rejection class Suggestion stands — M1 survives 52/52; the two proposed decline assertions kill it (M9)
S1 (r1) zero-width global + astral follows BSD, not GNU Suggestion stands — B5 re-measured: head -😀- = BSD; GNU 4.10 inserts - between each byte; deliberate, test-pinned
S2 (r1) gate/apply flag consistency untested Suggestion stands — same gap as R1-2, re-measured (M1)
S3 (r1) the s in the gate's 'su' is unobservable Suggestion stands — M8 survives; gate 'u'-only bundle byte-identical to head on all 68 cells

Central claim + A/B

The simulated sed -i fast path must treat ., bracket expressions and astral characters the way real sed does. The oracle is the real binary, not hand-written expectations: each cell runs parseSedEditCommand + applySedSubstitution (the exact two calls ShellTool.prepareSedEdit makes) in each arm, and the identical expression through GNU sed -i and BSD sed -i on a real temp file, comparing bytes. Arms are esbuild-bundled in isolation; positive control: the base bundle contains zero gsu occurrences, the head bundle exactly one.

group cells base wrong head wrong
A — . must match \r (CRLF) 6 5 0
B — astral character is one character 10 8 0
C — literal ] outside a class 11 0 0
D — regression corpus 28 0 0
E — command spellings (-i -E, --in-place, double quotes, -Ei) 4 4 0
F — u-flag acceptance class (R1-1 + siblings) 9 2 5
total 68 19 5

"Wrong" counts a cell against both oracles unless the cell is marked as deliberately following one (B5 follows BSD; A6/D4/D12–D14 follow GNU — all pre-existing choices, identical in both arms). Of the 5 head-wrong cells, F1/F2/F3/F8 are the PR regression and F5 is pre-existing (both arms wrong). All six issue rows flip base-wrong → head-match on both oracles. The corpus is a reconstruction: r1's harness no longer exists locally, so groups A–E were rebuilt from r1's group descriptions, the issue's six rows verbatim, and existing-test shapes (r1's A/B counts 5/6 and 8/10 reproduce exactly; E is 4/4 here vs r1's 3/4 because this round's fourth spelling is -Ei).

A/B against real GNU and BSD sed

R1-1 regression (Critical, blocking)

Under -E/-r, toJavascriptPattern copies \p{L}, \P{L}, \u{41} through verbatim; none of the four guards matches them (hasSedJavascriptDivergentEscape covers only \<>dDwWsS); the gate's new RegExp(jsPattern, 'su') compiles them as Unicode property / codepoint escapes; applySedSubstitution then writes the simulated text, which prepareSedEdit hands to writeTextFile. There is no fallback because nothing throws.

Measured bytes (input ab1 p{L} A\n = 61623120707b4c7d20410a):

cell base sim head sim GNU sed 4.10 BSD sed
F1 s/\p{L}/X/g ab1 X A (literal p{L}) XX1 X{X} X exit 1 Invalid content of \{\}, file unchanged ab1 X A
F2 s/\P{L}/X/g unchanged abXXpXLXXA exit 1, unchanged unchanged
F3 s/\u{41}/X/g unchanged (u{41} = 41 u's, as in sed) rewrites A→X exit 0, unchanged unchanged
F8 s/\p{Lu}/X/g on AB c unchanged XX c exit 1, unchanged unchanged

Against BSD (the issue's own platform) every row is a clean match→wrong regression. Against GNU, F1 was already wrong at base (base simulated a pattern GNU rejects — pre-existing), and head changes the wrong bytes to a worse shape. BRE is safe: the translator emits \p\{L\}, which u rejects, so the command declines and real sed runs (F4 measured). V8 probe confirms the mechanism: non-u \u{41} compiles as the quantifier u{41} (sed's reading), u-mode \u{41} is the codepoint A.

Blast radius: every -E/-r simulated sed -i whose pattern contains \p{…}, \P{…} or \u{…} — the simulation silently writes wrong content with no error and no fallback. \u0041-style 4-hex escapes (F5) are wrong in both arms (pre-existing; JS hex escape vs sed literal) and are covered by the same proposed fix.

Reproduce: node tmp/pr15701-verify-20261029-020933/harness.mjs <base-bundle> <head-bundle> out.json, cells F1–F3, F8; raw bytes in harness-results.json and the capture below.

R1-1 regression bytes

The proposed fix is measured, not eyeballed. Widening the guard to /\\[<>dDwWsSpPu]/u (the bot's suggestion, which the author accepted) in a scratch copy:

  1. hostile fixtures clean: F1/F2/F3/F5/F8 all decline on both oracles → real sed runs;
  2. benign fixtures byte-identical: 0 of 59 non-F cells change sim output or classification. One documented over-decline: F9 [\p{L}] (class-member \p, previously simulated correctly) now declines too — safe, costs only the fast path, consistent with the module's stated policy;
  3. suite: 54/54 green with the R1-1/R1-2 decline assertions added (M10); removing the widening while keeping the assertions reds exactly the R1-1 acceptance test (M11), so the assertions pin the fix.

The fix should ship with those assertions; the author's promised commit has not landed at this head.

Mutation matrix

12 rows, head's 52-test suite via JSON reporter, fresh scratch copy per row, every anchor asserted unique, kills attributed to named failing tests. Runner positive control: C1 (replaceAll forced false) kills with 15 failures; unmutated baseline 52/52.

row change outcome
C1 control: replaceAll forced false killed (15 failures)
M2 apply gsu→g (central revert) killed by the new test
M3 apply gsu→gu (drop s) killed by the new test (CRLF row)
M4 apply gsu→gs (drop u) killed by the new test (astral row)
M5 remove literal-] escaping killed by keeps backslashes literal inside bracket expressions under -E
M6 POSIX check back on jsPattern killed by rejects POSIX bracket expressions before simulated edits
M7 M5+M6 combined killed by the same -E test
M1 gate without flags survived → coverage gap (S2/R1-2)
M8 gate 'u' only survived → dead s flag (S3); byte-identical to head on all 68 cells (measured this round)
M9 M1 + R1-2's two decline assertions killed by the R1-2 acceptance test
M10 widened guard + R1-1/R1-2 assertions green 54/54 (the measured fix)
M11 head source + R1-1/R1-2 assertions killed by the R1-1 acceptance test (assertions are non-vacuous)

The new test is not vacuous: transplanted onto base source it fails exactly once, on its first astral assertion (expected ' Title\n' to be ' Title\n', U+FFFD vs space) — the author's "fails on the first astral row" claim holds verbatim.

Mutation matrix

Trial merge and sibling PRs

Findings

  1. R1-1 — Critical, blocking. Reproduced above; author acknowledged, fix promised but not pushed. The PR's own risk note ("the u flag can reject patterns that previously compiled, so those commands now fall back to real sed") covers only the reject direction; the accept-with-different-meaning direction is the regression. Suggested fix and its acceptance assertions are measured in the previous section.
  2. R1-2 / S2 — Suggestion. Gate/apply flag consistency is unpinned (M1 survives). The two toBeNull() assertions the author agreed to add kill M1 (M9 measured). Ship them with the R1-1 fix.
  3. S1 — Suggestion (carried, re-measured). Zero-width global + astral follows BSD (-😀-), GNU 4.10 gives - between each byte. Deliberate and test-pinned; on Linux the simulation still is not "what sed would do" for this shape. Acceptance note, not a blocker.
  4. S3 — Suggestion (carried, re-measured). The s in the gate's 'su' is unobservable (M8 byte-identical on 68 cells). Kept for symmetry; noted so no future reader attributes meaning to it.
  5. Pre-existing observations (not PR regressions, both arms identical). F5 \u0041 under -E: JS hex escape vs sed literal — wrong in both arms, covered by the proposed widened guard. D4 2g flag and D12–D14 BRE \+ \? \|: the simulation follows GNU semantics; macOS BSD sed rejects 2g outright and reads those BRE escapes literally. A6 \r: GNU extension, both arms match GNU. These are the module's existing GNU-leaning choices, unchanged by this PR.

Not covered

  • Full packages/core suite, typecheck, lint. The author claims 39,351 passed with one bun-pty.test.ts failure that also reproduces on main; the re-run CI (Qwen Code CI 40619669842) was still queued (Test/Lint jobs queued) at capture time, so the claim is unverified this round.
  • shell.test.ts sed-integration subset. Attempted on a head worktree with cloned node_modules; collection fails on @modelcontextprotocol/client — a dependency main added after this machine's install (head's packages/core/package.json requires @modelcontextprotocol/client@2.0.0; only sdk@1.29.0 is installed). A/A attribution: base's mcp-client.ts imports the same package and the PR touches no manifest, so both arms fail collection identically — environmental. Round 1's 31/31 on the identical tree is carried forward with that proof, not re-measured here.
  • No end-to-end CLI tool invocation. The harness drives the same two functions prepareSedEdit calls; the write path (writeTextFile) is read, not driven. R1-1's file-write consequence is inferred from that reading plus the measured sim bytes.
  • Windows (author marks it untested), BSD sed on Linux, -i.bak/backup-suffix interactions, and the CRLF line-ending restoration layer in shell.ts (the issue notes it restores \r\n on write; that layer is untouched by this diff and unexercised here).
  • Per-commit attribution beyond the aggregate: the PR has one production commit (45edd5b7) plus an empty CI-rerun commit (f51590bb, tree-identical, proven); the mutation matrix is per-guard over that one production commit's hunks.

Methodology

Local maintainer round on macOS (Node v22.23.1, GNU sed 4.10 at /opt/homebrew/bin/gsed, BSD sed at /usr/bin/sed, locale en_US.UTF-8 for both oracles). Two arm sources exported via git show at head f51590bb and merge-base 9df10f87 (the PR's parent; sedEditParser.ts/shell.ts are byte-identical across merge-base, baseRefOid 7b09c0a9, and current main, verified by empty git log over those paths), each bundled with esbuild into a self-contained ESM module so no workspace link can leak one arm into the other (gsu census: base 0, head 1). The 68-cell harness compares sim bytes against real sed -i bytes per oracle; divergent-oracle cells are scored against the oracle the simulation deliberately follows, and the delta is reported. Unit gates run through a minimal vitest config in scratch dirs (the package globalSetup requires built workspace dist); the mutation runner uses a fresh copy per row with unique-anchor assertions and JSON-reporter tallies that exclude pending tests. Trial merges run in a scratch worktree at main's tip, local-only, never pushed. Guard snapshots taken at round start and re-checked before posting. Raw logs, bundles, harnesses, ledgers: tmp/pr15701-verify-20261029-020933/ (harness-results.json, fix-results.json, mutations-results.json, assertion-ledger.json, test-*/results.json, tree-identity.txt, sibling-*.log).


Verification round 2 by @wenshao's local /verify-pr rig, 2026-10-29. Advisory evidence for human reviewers; this round does not approve, merge, or close anything.

@qwen-code-review-bot qwen-code-review-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Test Plan (not a blocker): tests: 52 passed — this review observed 19060, 2067, 39336, 2340, 306, 13620 passed.

Deferred under the convergence posture (round 2, not a blocker) — recorded, not requested in this round:

  • packages/core/src/utils/sedEditParser.test.ts:264 (+1 locations) — [review] the new s/x*/-/g assertion pins a result GNU sed does not produce for zero-width global matches over an astral character
中文说明

Test Plan(非阻断):tests: 52 passed — this review observed 19060, 2067, 39336, 2340, 306, 13620 passed。

收敛姿态下延后(第 2 轮,非阻断)——已记录,本轮不要求修改:共 1 条(原文未翻译,列表见上方英文部分)。

— deepseek-v4.1-flash-1017 via Qwen Code /review (v0.26.3)

@2SinceJuly

Copy link
Copy Markdown
Contributor Author

Two Criticals at 3be40510, both valid. Note the head moved only by a merge of main; the R1-1 fix I agreed to earlier has not landed, so R1-1 reproduces exactly as described.

R1-1 (still standing). Confirmed on 3be40510: hasSedJavascriptDivergentEscape is still /\\[<>dDwWsS]/u, so \p{…}, \P{…} and \u{…} pass every guard under -E, new RegExp(jsPattern, 'su') compiles them as JavaScript Unicode property / code-point escapes, and the simulated text is what writeTextFile receives. The u flag stays — your ^. / .$ astral cases depend on it. Fix is the narrow decline: widen the class to [<>dDwWsSpPu], and add toBeNull() cases beside rejects sed escapes that diverge in JavaScript regexes for -E 's/\p{L}/X/g', -E 's/\P{L}/X/g', -E 's/\u{41}/X/g'; deleting p/P/u from the class must red those three.

R1-2. Taking the decline assertions as written, a{ included, so removing u from the matcher reds the suite instead of flipping the pattern to "accepted, then Invalid escape at apply time".

R2-1 (new, accepted). Confirmed at shell.ts:2068: the gate compares the parser's string before writeTextFile normalises LF to CRLF for lineEnding: 'crlf'. abc\r\n + s/.$// therefore produces abc\n, passes the gate, is rewritten to abc\r\n, and reports sed edit applied with the bytes unchanged. I will compare what will actually be written — normalise the new content the way the write path does, and fall through to the existing made no changes branch when the result equals originalContent — without changing the CRLF write policy, plus the abc\r\n + _meta: { lineEnding: 'crlf' } case asserting sed edit made no changes that reds if the comparison is deleted.

Both fixes and their assertions go in one commit on top of 3be40510, no rebase.

肖洪进 added 2 commits October 29, 2026 09:24
`\p{…}`, `\P{…}` and `\u{…}` are literals to sed but Unicode property and
code-point escapes to a `u`-flagged regex, so the simulator accepted patterns
real sed rejects or leaves alone and wrote the simulated bytes to disk. Widen
the divergent-escape guard and pin the declines, plus the `u`-only rejection
class the flag introduced.
On a CRLF file, `s/.$//` consumes the trailing carriage return, but the write
path restores CRLF, so the edit reported success while the bytes on disk were
unchanged. Compare the prepared write content against the original instead of
the parser's intermediate string.
auto-merge was automatically disabled October 29, 2026 01:25

Head branch was pushed to by a user without write access

@2SinceJuly

Copy link
Copy Markdown
Contributor Author

All three are fixed.

R1-1 — ec8cb95ce. hasSedJavascriptDivergentEscape now covers \p, \P and \u (/\\[<>dDwWsSpPu]/u), so -E 's/\p{L}/X/g', -E 's/\P{L}/X/g' and -E 's/\u{41}/X/g' decline and real sed runs. The u flag stays; the decline is the fix. Three toBeNull() cases sit beside rejects sed escapes that diverge in JavaScript regexes, and removing p/P/u from the class reds them (verified).

R1-2 — same commit. Added declines u-only invalid escapes so the real sed runs with s/a\-b/X/g, s/a\ b/X/g and -E 's/a{/X/g'.

R2-1 — 28b5d7a37. prepareSedEdit now normalises the new content through the same prepareTextFileContentAsync path the write uses and compares that against the original, so a CRLF edit whose only effect is the trailing \r falls through to sed edit made no changes. The write policy is untouched: s/=.*$/=new/ still writes key=new\r\nk2=v\r\n. New case does not report a CRLF edit whose only effect is the carriage return (abc\r\n + _meta: { lineEnding: 'crlf' }, s/.$//) reds when the comparison is removed (verified).

Evidence: sedEditParser.test.ts 53/53, shell.test.ts 375/375, both on 28b5d7a37. Two commits on top of 3be40510, no rebase.

@qwen-code-review-bot qwen-code-review-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not explored to full depth (tool budget reached): "agent 6c": I did not run the two touched test files ( npx vitest run for shell.test.ts / sedEditParser.test.ts ), so the new assertions are reviewed by inspection only…; "agent 3a": none — no check was cut short..

Convergence: round 3 posted 4 inline comment(s), 4 of them reported for the first time; the previous round posted 3 (1 new). Findings keep coming back to the same files: packages/core/src/utils/sedEditParser.ts (findings in rounds 1, 2; 1 more now). The rate of new findings is not falling. A cluster that keeps producing siblings usually means the fixes are treating instances of a shared root cause — triaging that cause before the next round, or splitting an independent cluster into its own pull request, tends to end the loop faster than fixing them one at a time. Batching the remaining fixes and verifying them before the next push, or dropping this PR's reviews to --severity-floor critical, keeps the loop from re-deriving the same set. (Observation only — nothing was withheld from this review because of this observation.)

中文说明

未探索到全部深度(达到工具调用预算):"agent 6c":I did not run the two touched test files ( npx vitest run for shell.test.ts / sedEditParser.test.ts ), so the new assertions are reviewed by inspection only…;"agent 3a":none — no check was cut short.。

收敛情况:第 3 轮发布了 4 条行内评论,其中 4 条是首次提出;上一轮发布了 3 条(其中 1 条首次提出)。发现反复回到同一批文件:packages/core/src/utils/sedEditParser.ts(第 1、2 轮已出过发现,本轮又有 1 条)。新发现的产出速度没有下降。一个不断再生兄弟发现的簇,通常意味着逐条修复只在处理同一根因的实例——先定位并处理该根因,或把独立的簇拆成单独的 PR,通常比逐条修复更快结束循环。把剩余修复攒成一批、验证后再推送,或将本 PR 的评审降到 --severity-floor critical,可以避免循环反复推导同一组发现。(仅为观察——本轮评审未因此扣留任何内容。)

— deepseek-v4.1-flash-1017 via Qwen Code /review (v0.26.3)

Comment thread packages/core/src/tools/shell.ts Resolved
Comment thread packages/core/src/tools/shell.ts Resolved
Comment thread packages/core/src/tools/shell.test.ts Resolved
@wenshao

wenshao commented Oct 29, 2026

Copy link
Copy Markdown
Collaborator

Verdict: findings — 62/62 scripted assertions passed at head 28b5d7a374ffda1fb4a2384fb37aaee57ec6c9da (base 35fd1f44ff58b29d4e9eaf1b6f1e5472b6d3de20). R1-1, R2-1 and R1-2 are fixed and each is pinned by its own witness; the central CRLF/astral fix remains load-bearing. Two blocking problems stand at this head: on BOM files the new byte guard still certifies sed edit applied while the bytes on disk are unchanged (R3-1), and \0 is still simulated as a NUL where sed reads a literal 0 (R3-3). One further consequence of the guard is stated with its bound below: it now reports "made no changes" for CRLF→LF edits that neither build ever wrote (N-1).

中文摘要
  • 结论:findings(62/62 脚本断言通过;head 28b5d7a3)。阻塞项不是断言失败,而是本轮实测出的行为问题,见 Findings 一节。
  • A/B(真 GNU sed 4.5 为 oracle,三臂:base 35fd1f44 / 第二轮 head f51590bb / head 28b5d7a3):. 匹配 \r(issue sed edit simulation: . does not match \r or a whole emoji, so results differ from real sed #15675)与星芒字符按一个字符在 base 全错、head 全对;R2-1 的 \p{L}/\u{41} 在第二轮 head 被改写成错误内容,head 已改为回退真 sed。见 01-ab-three-arms.png 与 Central claim 表。
  • 上一轮三条全部修复并钉住:R1-1(变 M2/M4/M5 全杀)、R2-1(M3/M7/M8 全杀)、R1-2(M6 杀掉——第二轮同一变异是存活)。
  • 本轮新发现:
    • R3-1(Critical,复现):带 BOM 的 CRLF 文件仍报 sed edit applied,磁盘字节未变。机器人建议改为比较 encodeTextFileContentAsync 的两侧字节,实测可修,且 10 个端点格零附带变化。
    • R3-3(Suggestion,复现,双向都错):\0 在 JS 里是 NUL、在 sed 里是字面 0;加进字符类后三个用例全部回退真 sed。
    • N-1(关于写入路径的边界说明):s/\r$// 在 CRLF 文件上,base 与 head 都没有真正改变磁盘字节(行尾还原层把模拟器产出的 LF 又变回 CRLF),但 head 现在如实说“没有变化”,base 之前会谎报“已应用”。我起初把它写成“head 丢弃了真实编辑”,实测后已改正。
    • R3-2 / R3-4(Suggestion):写入服务不还原行尾时(ACP/IDE)同样不写入;作者测试里没有“CRLF 正向写入”用例,套件对这个轴不设防。
  • 一处更正(我自己的第二轮报告):第二轮“补丁在 59 格零附带损伤”建立在 |sim| 与 |sed| 锁步的前提上,本轮实测该前提不成立,见 Correction 一节。
  • 未覆盖:Windows、BSD sed(本轮只有 GNU 4.5)、真实 ACP 会话(只用按原样写入的服务替身)、\0 之外的兄弟形状、core 全量套件(作者声称 39,351 通过,该 head 的 CI 为绿)。

A/B: base, round-2 head and head against GNU sed

Previous-finding status

Round 2 (2026-10-28, head f51590bb, tree e00b2c97) reported blocked. This head adds two production commits on top of that tree. Nothing was carried forward by assumption — every row was re-run in this round (Node 24.22.1, GNU sed 4.5, this checkout).

# finding severity status at 28b5d7a3
R1-1 -E turns sed's \p{…} / \P{…} / \u{…} into JS Unicode escapes and writes wrong bytes Critical fixed — F1/F2/F3/F8 now match sed; M2/M4/M5 killed
R2-1 CRLF edit whose only effect is the \r reports "applied" with unchanged bytes Critical fixed — M3/M7/M8 killed; the round-2 head is kept as the failing control
R1-2 gate's u-only rejection class unpinned Suggestion fixed — M6 killed (round 2 measured the same edit as a survivor)
S1 (r1) zero-width global over an astral character follows BSD, not GNU Suggestion stands — cell B5 identical at all three heads
S2/S3 (r1) gate/apply flag consistency; s in the gate unobservable Suggestion superseded — dropping u from the gate now reds the suite
R3-1 (bot, 2026-10-29) BOM / non-UTF-8 file: the guard decodes prepared bytes, never fires, false "applied" Critical stands — reproduced; the bot's own fix is measured below
R3-2 (bot) a write service that does not restore line endings (ACP/IDE) gets an edit skipped Suggestion stands — reproduced
R3-3 (bot) \0 is NUL to JS, a literal 0 to sed Suggestion stands — reproduced in both directions
R3-4 (bot) no positive-write CRLF test: the guard's writing direction is unpinned Suggestion stands — an added verbatim-writer test fails; candidate fixes leave the suite green

The bot's three corrected rulings (R1-1, R1-2, R2-1 "fixed by …") are accurate: each is confirmed by that fix's own witness going red when the fix is reverted.

Central claim + A/B

Claim unchanged: the simulated sed -i fast path must treat ., bracket expressions and astral characters the way sed does. Oracle: the real binary, never hand-written expectations. Each arm is its own bundle of sedEditParser.ts / tools/shell.ts, executed in a working copy holding only that arm's originals. The arm census is green with a negative control (arm-census.txt): per-bundle banner, exactly one parser and one shell copy, "no other arm's source present", and a runtime reach proof (\p{L}, \u{41}, ^.) that differs per arm.

cell base 35fd1f44 round-2 head f51590bb head 28b5d7a3 claim
A1 s/=.*$/=new/ on CRLF wrong (key=old… untouched) matches sed matches sed . spans the \r (issue #15675)
B1 s/^.// on 😀1 wrong (lone surrogate) matches sed matches sed one astral character is one sed character
F1 -E s/\p{L}/X/g wrong (ab1 X A) wrong (XX1 X{X} X) declines → real sed R2-1
F3 -E s/\u{41}/X/g matches wrong (XB) declines → real sed R2-1
L1 s/.$// on abc\r\n (pattern space) wrong matches sed matches sed dotAll + per-line processing
L3 s/\r$// on a\r\nb\r\n (pattern space) matches matches matches a greedy sticky tail must not cross \r
shell S1 s/.$// (verified R2-1) "no changes" "applied", bytes unchanged "no changes" the endpoint
shell S2 s/=.*$/=new/ on CRLF "no changes" applied key=new\nk2=new\n applied key=new\nk2=new\n issue #15675 end to end
shell S7 s/\r$// on CRLF "applied" (bytes unchanged) "applied" (bytes unchanged) "no changes" (bytes unchanged) N-1

Derived from those rows: head differs from base on four parser cells (A1, B1, F1, L1), all in the fixed direction; the shell matrix moves S2 into the fixed column and changes S7's report only.

Mutation matrix

Runner: the author's suites through a throwaway copy of the head tree, baseline green (53/53, 375/375) before any row, each anchor asserted unique, each revert replayed as that commit's production diff. The coverage claim is per-commit: git log --no-merges f51590bb..28b5d7a3 --not 4196cf81d1 over the two production files lists exactly ec8cb95c and 28b5d7a3, and both have rows (mutations.json).

row commit change outcome
M2 ec8cb95 whole commit reverted killed — rejects sed escapes that diverge in JavaScript regexes
M4 / M5 ec8cb95 drop u / drop p from the class killed — same witness
M6 ec8cb95 gate compiled 's' instead of 'su' killed — declines u-only invalid escapes so the real sed runs
M3 28b5d7a whole commit reverted killed — does not report a CRLF edit whose only effect is the carriage return
M7 / M8 28b5d7a gate disabled / gate compares pre-write text killed — same witness
B0 — baseline green
M1 — control: one assertion falsified red as expected (vitest names the describe block, not the it() title)

Both survivors were adjudicated by running their builds. Removing s from applySedSubstitution is caught by this round's corpus (A1, B5), not by the author's suite. The bracket-expression backslash escape (toJavascriptPattern) shows no behavioural difference: its variant scored 32/33, the same as head, declining on the backslash cells instead of simulating them — redundant defence today, pinned by nothing.

Mutation matrix

Correction to round 2 (my own earlier report)

Round 2 said the widened guard left "0 of 59 non-F cells" with collateral damage, reasoning from |sim| against |sed| treated as lockstep. That premise does not hold: this product restores LF→CRLF on write when the read reported lineEnding: 'crlf', while GNU sed on the same file drops the \r terminator (probe lockstep-noproduct: a plain rewrite round-trips abc\r\n). Round 2 was not miscounting its cells; it was measuring a model the write path does not implement. I also corrected one of my own claims mid-round: an earlier draft of this report said the guard "suppresses real CRLF edits". Measured on disk (probe/disk-crlf.mjs), s/\r$// never changed the bytes at base either — the write path turns the simulated abc\n back into abc\r\n. What the guard changes is the report: base claimed sed edit applied, head says made no changes, and both leave the file as it was. The pre-existing write-path behaviour is documented here, not blamed on this PR; only the BOM case below is a newly false success.

Findings

R3-1 — Critical, blocking, reproduced. On a UTF-8-BOM CRLF file the read returns BOM-stripped abc\r\n; prepareTextFileContent's BOM branch returns a Buffer with no encoding; the guard decodes it as UTF-8 ("\uFEFFabc\r\n"), the comparison never matches, and the tool reports sed edit applied while the bytes on disk are unchanged (probe r31-shell: head report="sed edit applied", bytesUnchanged=true). At the merge base . could not match \r, so the same input answered made no changes — the false success is new to this line of changes. The bot's suggested fix — compare encodeTextFileContentAsync(newContent) with encodeTextFileContentAsync(content) — was applied in a scratch copy and measured: the same probe reports sed edit made no changes, and all 10 endpoint cells are byte-identical to head (shell-fix_combo.json, shared library filesystem.test.ts 34/34). It should ship with a bom: true fixture beside the new CRLF test.

R3-3 — Suggestion, reproduced in both directions. sed -i 's/\0/X/' reads a literal 0 to sed (a0b → aXb); head's simulation treats \0 as U+0000 (a0b unchanged, a NUL byte rewritten). Widening the class to [<>0dDwWsSpPu] makes all three cells decline to real sed (probe r33). Same class as R1-1 one character over — the bot's point that the fixes have been treating instances of one root cause is fair.

R3-2 — Suggestion, reproduced. The guard assumes StandardFileSystemService's line-ending preparation; AcpFileSystemService.writeTextFile forwards content verbatim (packages/cli/src/acp-integration/service/filesystem.ts:214), so in an IDE session the guard's decision is about bytes the service never writes. Same root cause as R3-1: the decision belongs to the service that writes.

R3-4 — Suggestion. Every positive-write assertion in the simulated sed edit block uses an LF fixture; the only CRLF fixture asserts a skip. Candidate fixes (endpoint, BOM, \0) applied in scratch leave the suite at 428/428 with nothing changing colour — the axis has no fixture, and I would add the one above.

N-1 — note on the write path (pre-existing, not a regression of this PR). A CRLF→LF conversion simulated in-process never reaches the file: the write path restores CRLF (s/\r$// on abc\r\n leaves abc\r\n at base, at the round-2 head and at head — probe/disk-crlf.mjs). Head's report is now truthful ("made no changes"); base's was not ("applied"). This is the line-ending preservation issue #15675 explicitly leaves in place, so I record it as a known limitation rather than a defect to fix here — R3-4's positive fixture is the guard against making it worse.

Pre-existing, unchanged at every arm: B5 zero-width global over an astral character follows BSD (round 1's S1, still standing).

Not covered

  • Windows (author marks it untested) and BSD sed — this rig has GNU sed 4.5 only, so every "matches sed" verdict is GNU-only.
  • A real ACP session. R3-2 is proven by a verbatim-writing FileSystemService stand-in following the ACP service's own code contract; no ACP connection was driven.
  • \0 siblings outside the four probe cells (e.g. \cX, \xHH agreement) were not enumerated.
  • The full packages/core suite (author: 39,351 passed, one bun-pty.test.ts failure also on main). The PR's own CI at this head is green for Test (ubuntu), Lint & Static, integration and web-shell smoke; macOS/Windows lanes were skipped on this run.
  • -i.bak, backup suffixes, symlinks, non-UTF-8 encodings beyond the BOM case were not exercised.

Methodology

Local maintainer round: three arm sources exported with git show at 35fd1f44, f51590bb and 28b5d7a3, each bundled on its own (parser and shell separately, @qwen-code/qwen-code-core external and shimmed) and run inside a working copy holding only that arm's originals plus a hardlinked node_modules. Parser cells compare applySedSubstitution with real sed -i output; endpoint cells drive ShellTool.getConfirmationDetails → execute with the configuration surface lifted verbatim from the author's shell.test.ts; the disk probe uses the real StandardFileSystemService. Unit gates run the head tree's own vitest config with only the fresh-clone dist guard relaxed. Everything is scripted under probe/ (ab-witness.mjs, parser-matrix.mjs, shell-harness.mjs, disk-crlf.mjs, r31-shell.mjs, r32.mjs, r33.mjs, run-mutations.mjs, ledger.sh); results in assertion-ledger.json, assertions.json, mutations.json (mutations-raw.json holds the runner's own tally), parser-*.json, shell-*.json.


Verification round 3 by @wenshao's local /verify-pr rig, 2026-10-29, at head 28b5d7a3. Advisory evidence for human reviewers; this round does not approve, merge, or close anything.

The write-back prediction decoded the prepared bytes as UTF-8 and compared
them with the read content, so a file carrying a BOM or a non-UTF-8 encoding
never matched and the false `sed edit applied` this PR removes survived.
Compare content strings instead, and only predict the CRLF normalization when
the writer applies it: a client-side write stores the content verbatim, so
`s/\r$//` is a real edit there and must not be suppressed.
`\0` is NUL to a `u`-flagged regex but a literal `0` to sed, so the simulator
found no match and answered `sed edit made no changes` instead of falling back
to the real binary.
@2SinceJuly

Copy link
Copy Markdown
Contributor Author

Round 3 is addressed in two commits on top of 28b5d7a3 (no rebase): b951ab0c3 and 9fc09d69b.

R3-1 (Critical — BOM / non-UTF-8 made the comparison false). The guard no longer decodes the prepared bytes. It compares content strings against the content the read returned, and applies the CRLF normalization only when the active writer applies it, so a BOM or a non-UTF-8 encoding can no longer make an unchanged file look changed.

R3-2 (a client-side write never restores CRLF). Added restoresLineEndingsOnWrite to FileSystemService: StandardFileSystemService reports true, and AcpFileSystemService reports !capabilities.writeTextFile, i.e. only the local-fallback path normalizes. When the write goes to the client the predicted content is left unnormalized, so sed -i 's/\r$//' is applied instead of being suppressed — s/.$// on a CRLF file is still reported as no change under the local writer.

R3-3 (\0). 0 is now in hasSedJavascriptDivergentEscape, so s/\0/X/ falls back to the real binary instead of simulating a miss. Verified here that BSD sed turns a0b into aXb while a u-flagged regex only matches NUL.

R3-4 (only the skip branch was pinned). shell.test.ts now also covers a CRLF edit that does change bytes (foo\r\n → bar\r\n) and the verbatim-writer case; the BOM skip case is pinned next to the existing one.

packages/core shell and sedEditParser suites: 431 passed. packages/cli ACP filesystem suite: 34 passed. tsc --noEmit clean for core and cli.

Resolving the threads. @qwen-code /review

@wenshao

wenshao commented Oct 29, 2026

Copy link
Copy Markdown
Collaborator

Verdict: findings — 36/39 scripted assertions passed at head 9fc09d69b5441fa8db0ac6808484db3503ea9d4d (base 35fd1f44ff58b29d4e9eaf1b6f1e5472b6d3de20). R3-1, R3-2 and R3-4 are fixed and pinned; R3-3 is fixed but unpinned. One blocking regression stands: the new restoresLineEndingsOnWrite declaration is false for the repository's own qwen serve client, whose write path re-applies CRLF, so sed -i 's/\r$//' on a CRLF file reports sed edit applied while the bytes on disk are unchanged (R4-1). The same runtime also keeps R3-1's BOM-shaped false success.

中文摘要
  • 结论:findings(36/39 脚本断言通过;head 9fc09d69b,base 35fd1f44)。
  • 上一轮四条:R3-1(BOM 假成功)、R3-2(逐字写入被跳过)、R3-4(CRLF 正向用例)在本地写入路径上均已修复并被测试钉住(M20/M21/M22 全部杀死)。R3-3(\0)行为已修复,但没有任何测试(M24 存活;修正后的对照变异 M25b 能杀死套件)。
  • 本轮新问题(R4-1,Critical,已复现):AcpFileSystemService.restoresLineEndingsOnWrite 取 !capabilities.writeTextFile,而 qwen serve 广告的是 writeTextFile: true,于是子进程按“逐字写入”预测;但真正的写入端——daemon 的 writeTextOverwrite——会用现有文件的 meta 重新做 CRLF 还原。用仓库真实的 ACP 服务 + 真实 daemon 写入类端到端驱动:s/\r$// 与 BOM 文件上的 s/.$// 都报 applied,磁盘字节不变。相对上一轮 head(r3 报 “no changes”)这是新引入的假成功;本地运行时不受影响。
  • A/B:真 GNU sed 4.10 为 oracle,base/r3/head 三臂;. 跨 \r、星芒单字符、\p/\u{41}/\0 回退均与 sed 一致;B5 零宽全局仍沿用 BSD 语义(既有 S1,未变)。
  • 未覆盖:Windows/BSD sed(只有 GNU 4.10)、真实 ACP 传输(进程内直调)、真正拉起的 serve 会话、-i.bak/符号链接/非 UTF-8(BOM 之外)、core 全量套件、head 的 ubuntu/lint/integration lane 当时仍在排队。

A/B: base, round-3 head and head against GNU sed 4.10

Previous-finding status

Round 3 (2026-10-29, head 28b5d7a3) reported findings. This head adds two production commits on top of that tree (b951ab0c, 9fc09d69). Every row was re-run in this round — nothing is carried forward by assumption.

# finding severity status at 9fc09d69
R1-1 -E \p{…} / \u{…} simulated as JS escapes Critical fixed — both decline; the guard's own tests and my F1/F3 cells confirm
R2-1 CRLF edit whose only effect is \r reports applied with unchanged bytes Critical fixed locally; re-opened in qwen serve — see R4-1
R1-2 gate's u-only rejection class unpinned Suggestion fixed — M25b kills the suite when p is dropped
R3-1 BOM / non-UTF-8 file: false "applied" Critical fixed locally (string comparison); stands in qwen serve (same root cause as R4-1)
R3-2 a writer that stores content verbatim gets a real edit skipped Suggestion fixed — the verbatim-writer cell applies and changes bytes
R3-3 \0 is NUL to JS, a literal 0 to sed Suggestion fixed behaviourally, unpinned — M24 survives; the commit ships no test
R3-4 no positive-write CRLF fixture: the writing direction is unpinned Suggestion fixed — M22 is killed by two tests, including the new positive-write one
S1 (r1) zero-width global over an astral character follows BSD Suggestion stands — B5 mismatch at all three arms
N-1 CRLF→LF never reaches disk under the local writer note unchanged — s/\r$// still leaves abc\r\n; the report is truthful

Central claim + A/B

Claim unchanged: the simulated sed -i fast path must treat ., bracket expressions and astral characters the way sed does. Oracle: the real binary (GNU sed 4.10), never hand-written expectations. Three arm worktrees at 35fd1f44 / 28b5d7a3 / 9fc09d69, each bundled from its own sources; parser cells call applySedSubstitution, endpoint cells drive the real ShellTool against real files.

cell base r3 head claim
A1 s/=.*$/=new/ on CRLF wrong matches sed matches sed . spans \r (issue #15675)
B1 s/^.// on 😀1 wrong (lone surrogate) matches matches one astral character is one sed character
F1 -E s/\p{L}/X/g wrong declines declines sed rejects it; fall back to the binary
ESC7 s/\0/X/g wrong (NUL) wrong (NUL) declines → aXb R3-3
L1 s/.$// on abc\r\n wrong matches matches dotAll + per-line processing
B5 zero-width global on 😀 mismatch mismatch mismatch S1, unchanged (BSD vs GNU)
E-R31 BOM CRLF s/.$// "no changes" applied, unchanged "no changes" R3-1 fixed locally
E-R32 verbatim s/\r$// applied, changed "no changes", skipped applied, changed R3-2 fixed
E-SERVE s/\r$// applied, unchanged "no changes" applied, unchanged R4-1
E-SERVE-BOM s/.$// "no changes" applied, unchanged applied, unchanged R3-1 stands in serve

The endpoint cells print in 01-ab-three-arms.png; the serve chain in 02-serve-chain.png

qwen serve chain: real ACP service + real daemon writer

Mutation matrix of the two new production commits.

Findings

R4-1 — Critical, blocking, reproduced. AcpFileSystemService declares restoresLineEndingsOnWrite = !capabilities.writeTextFile. qwen serve advertises { readTextFile: false, writeTextFile: true } (packages/acp-bridge/src/channel-handshake.ts), so in every served session the child predicts a verbatim write. The delegated write does not store content verbatim: the daemon's WorkspaceFileSystem.writeTextOverwrite re-reads the existing file's meta and encodes through encodeTextFileContentAsync, which re-applies CRLF (packages/cli/src/serve/fs/workspace-file-system.ts, mergeWriteMeta + atomicWriteTextResolvedFile). Measured on the real chain — real AcpFileSystemService, real writeTextOverwrite, only the ACP transport in-process: sed -i 's/\r$//' on abc\r\n → message sed edit applied, bytes 6162630d0a before and after. The BOM variant (EF BB BF abc\r\n, s/.$//) is the same: applied, bytes unchanged. A real edit (s/foo/bar/) still lands, so this is a false success, not a broken write. At the previous head the same two cells answered no changes (truthful); base answered applied for the non-BOM cell, so this is a re-introduced defect of R2-1's class rather than a pre-existing behaviour. The tool also records a file-history backup, commit attribution and a read-cache write for the edit, so the false success is not merely cosmetic.

Suggested fix: a capability flag says the client can write, not how it writes; the ACP service cannot distinguish a verbatim IDE from the in-repo daemon. Keep the conservative prediction (normalize) unless the writer proves verbatim — the r3 column of 02-serve-chain.png is that configuration, measured: both serve cells become truthful again, and the only cost is the hypothetical verbatim client's s/\r$// being skipped. A qwen serve fixture that drives the delegated write through writeTextOverwrite (as zz-pr15701-serve.test.ts does) should ship with the change; the author's unit tests use a mock whose property is set by hand, which is why M20/M21 stay green while the production client is mis-modelled.

R4-2 — Suggestion, measured. 9fc09d69 ("decline \0") changes only sedEditParser.ts; no test was added. Dropping 0 from the class (M24) leaves all 431 tests green, while the corrected control (dropping p, M25b) reds rejects sed escapes that diverge in JavaScript regexes in 5 ms — the suite is live and simply does not pin \0. Behaviour at head is correct (ESC7 declines; real sed writes aXb), so this is a coverage gap, not dead code. Add expect(parseSedEditCommand("sed -i 's/\\0/X/g' f.txt")).toBeNull() to the existing divergence test.

R4-3 — Suggestion, measured. Dropping needsCrlfLineEndings(filePath) from the predicate (M23) also leaves 431/431 green: the Windows .cmd axis has no fixture. This round cannot close it (macOS os.platform() makes the branch unreachable), so it is listed under Not covered as well.

Corrections

My first control mutation (M25) removed W from the class instead of p, so it survived and briefly looked like a dead suite. The corrected control (M25b) reds exactly the \p test; mutations.json carries only the corrected row. Also, round 3's oracle was GNU sed 4.5; this round's is 4.10, so \p{L} is rejected by the oracle with Invalid content of \{\} rather than being a no-op — the head's decline is correct under both.

Not covered

  • Windows and BSD sed — this rig has GNU sed 4.10 only (round 3 used 4.5); every "matches sed" verdict is GNU-specific. M23's Windows branch is unexercised.
  • A real ACP transport. The serve cells call the real AcpFileSystemService, real WorkspaceFileSystem and real encoder in-process; the AgentSideConnection object is a one-method stand-in. No daemon process was spawned.
  • Non-UTF-8 encodings beyond BOM, -i.bak, backup suffixes, symlinks.
  • The full packages/core suite (the author's two files: 431/431 at head; packages/cli ACP filesystem: 34/34). Head's Test (ubuntu), Lint & Static, integration and web-shell E2E lanes were still queued when this round captured (windows/macos lanes skipped), so no CI evidence is claimed beyond the plumbing lanes that had finished.

Methodology

Local maintainer round, three detached worktrees at base / round-3 head / head; each arm runs the same probe file (zz-pr15701-probe.test.ts) from its own packages/core/src/tools/, with parser cells comparing applySedSubstitution against /opt/homebrew/bin/gsed and endpoint cells driving the real ShellTool on real temp files (real StandardFileSystemService, plus declared stand-ins for the verbatim and daemon writers). A second probe (zz-pr15701-serve.test.ts) runs in the head arm's packages/cli with the real ACP service and the real daemon write class; core imports are aliased to packages/core/src through the package export map. Mutation rows mutate the head arm's production files, run the author's two core suites, and restore with git checkout HEAD --. Everything is scripted under probe/; results in assertions.json, mutations.json, probe/results-*.json, probe/mutations.json.


Verification round 4 by @wenshao's local /verify-pr rig, 2026-10-29, at head 9fc09d69b5. Advisory evidence for human reviewers; this round does not approve, merge, or close anything.

@qwen-code-review-bot qwen-code-review-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Partially reviewed — gaps disclosed. Suggestions are inline.

Not reviewed: build-and-test — Test (windows-latest, Node 22.x) was skipped in CI and the suite did not run locally, so the Windows-only needsCrlfLineEndings (.bat/.cmd) disjunct of the no-op guard is unexecuted.

Test Plan (not a blocker): tests: 53 passed — this review observed 21509, 2093, 41039, 2366, 355, 14832 passed.

Convergence: round 4 posted 1 inline comment(s), 1 of them reported for the first time; the previous round posted 4 (4 new). Findings keep coming back to the same files: packages/core/src/utils/sedEditParser.ts (findings in round 3; 1 more now). A cluster that keeps producing siblings usually means the fixes are treating instances of a shared root cause — triaging that cause before the next round, or splitting an independent cluster into its own pull request, tends to end the loop faster than fixing them one at a time. No Critical finding is open on this round, so merging and moving the remaining Suggestion threads to a follow-up issue is available as an ending — a merged pull request cannot diverge further. (Observation only — nothing was withheld from this review because of this observation.)

中文说明

仅完成部分审查,审查缺口已披露。 建议见行内评论。

未审查(原文为英文):build-and-test — Test (windows-latest, Node 22.x) was skipped in CI and the suite did not run locally, so the Windows-only needsCrlfLineEndings (.bat/.cmd) disjunct of the no-op guard is unexecuted.

Test Plan(非阻断):tests: 53 passed — this review observed 21509, 2093, 41039, 2366, 355, 14832 passed。

收敛情况:第 4 轮发布了 1 条行内评论,其中 1 条是首次提出;上一轮发布了 4 条(其中 4 条首次提出)。发现反复回到同一批文件:packages/core/src/utils/sedEditParser.ts(第 3 轮已出过发现,本轮又有 1 条)。一个不断再生兄弟发现的簇,通常意味着逐条修复只在处理同一根因的实例——先定位并处理该根因,或把独立的簇拆成单独的 PR,通常比逐条修复更快结束循环。本轮没有未决的 Critical,因此"合入后把剩余 Suggestion 线程转到后续 issue"是一个可选的结束方式——已合入的 PR 不会继续发散。(仅为观察——本轮评审未因此扣留任何内容。)

— deepseek-v4.1-flash-1017 via Qwen Code /review (v0.26.8)

// equivalence class. Matching only the self-contained `[[:alpha:]]` form let
// negated and mixed expressions such as `[^[:space:]]` or `[[:alpha:]abc]`
// through, and the simulation then disagreed with sed on them.
return /\[[:.=]/u.test(pattern);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Suggestion] R4-1: The widened POSIX-class guard also swallows plain bracket classes such as [.], [=] and [:], so those commands leave the simulated-edit path even though the translator handles them exactly like sed.

hasPosixBracketExpression now tests the raw pattern against /\[[:.=]/u, which matches any [ followed by :, . or =. A POSIX character class, collating symbol or equivalence class only opens inside a bracket expression ([[:alpha:]], [[.ch.]], [[=a=]]); a plain class whose first member is one of those characters is not one. sed -i 's/[.]/_/g' file.txt is a common literal-dot idiom and now parses to null, so the command runs as a real shell command: no diff preview, no prior-read gate, no undo snapshot — and on a host without a sed binary it fails outright where it previously worked.

Witness:

PR (63dc1eb9): [.] DECLINED  [=] DECLINED  [:] DECLINED  [.]$ DECLINED  [abc] ACCEPTED
BASE          : all four accepted; s/[.]/X/g on "a.b" -> "aXb"
GNU sed       : printf 'a.b\n' | sed 's/[.]/X/g'  -> aXb
still declined on both: [^[:space:]]  [[:alpha:]abc]  [[=a=]]  [a[:digit:]]
Suggested change
return /\[[:.=]/u.test(pattern);
function hasPosixBracketExpression(pattern: string): boolean {
let inBracketExpression = false;
for (let i = 0; i < pattern.length; i++) {
const char = pattern[i]!;
if (char === '\\') {
i++;
continue;
}
if (char === '[') {
if (inBracketExpression && /[:.=]/u.test(pattern[i + 1] ?? '')) {
return true;
}
inBracketExpression = true;
continue;
}
if (char === ']') {
inBracketExpression = false;
}
}
return false;
}

The guard must keep declining [^[:space:]], [[:alpha:]abc], [[=a=]] and the nested [a[:digit:]] form (pinned at sedEditParser.test.ts:240-247), which a naive two-opening-bracket regex would miss — hence the bracket-state walk. Add expect(parseSedEditCommand("sed -i 's/[.]/X/g' f.txt")).not.toBeNull() plus an apply-level row such as expect(applySed("sed -i 's/[.]$//' f.txt", 'ab.\n')).toBe('ab\n'); both are red today, and removing the bracket-state guard must red them again.

中文说明

放宽后的 POSIX 字符类守卫把普通的方括号表达式也一起拒绝了,例如 [.]、[=]、[:]——翻译器其实能正确处理它们,与 sed 行为一致。

hasPosixBracketExpression 现在直接用 /[\[:.=]/u 匹配原始 pattern,只要出现 [ 后紧跟 :、. 或 = 就判为 POSIX 结构。但字符类、排序符号、等价类只在方括号表达式内部才会开启([[:alpha:]]、[[.ch.]]、[[=a=]]);首个成员恰好是这些字符的普通字符类并不是 POSIX 结构。sed -i 's/[.]/_/g' file.txt 是很常见的「字面点」写法,现在会解析为 null,改走真实 shell 命令:失去 diff 预览、失去先读校验、失去撤销快照;在没有 sed 的主机上,原本可用的命令现在直接失败。

建议按方括号状态判断(模块里 hasBracketCharacterEscape 已经用同样的走法):进入 [ 后,只有「已在方括号表达式内部」时遇到 [ 紧跟 :/./= 才拒绝。必须继续保持拒绝 [^[:space:]]、[[:alpha:]abc]、[[=a=]] 以及嵌套的 [a[:digit:]](sedEditParser.test.ts:240-247 已钉住),简单的双开括号正则会漏掉嵌套形式,所以要走括号状态。请补上 s/[.]/X/g 的 not.toBeNull() 断言和一条 apply 用例,它们在当前代码上是红的;去掉括号状态守卫后必须重新变红。

— deepseek-v4.1-flash-1017 via Qwen Code /review (v0.26.8)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 3bb1a0d (rebased onto main as 104c467). hasPosixBracketExpression now walks the pattern and only reports a POSIX construct when [ is followed by :, . or = inside an already-open bracket expression, so [.], [=] and [:] keep the simulated path while [^[:space:]], [[:alpha:]abc], [[=a=]] and [a[:digit:]] stay declined.

Added s/[.]/X/g / s/[=]/X/g / s/[:]/X/g parse assertions and an apply-level s/[.]$// row. Verified: with the new guard removed the added case fails (1 failed / 53 passed); with it, src/utils/sedEditParser.test.ts is 54 passed.

@2SinceJuly 2SinceJuly left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both notes were addressed in b8ccdda4, which landed after you wrote this.

  1. hasPosixBracketExpression now tests /\[[:.=]/u over the raw pattern, so [^[:space:]], [[:alpha:]abc] and [[=a=]] all decline and fall through to the real sed. Your three witness cells are the cases that commit exists for. The blunt anywhere-match is deliberate and documented: it also declines [a[]], which would in fact translate correctly, because a false decline costs only the fast path.

  2. sedEditParser.test.ts covers the multibyte backreference next to the x* case, with the GNU/BSD comment naming both: s/\(.\)\1/X/g on 😀😀x asserts Xx, and the note says BSD compares bytes and leaves the file alone.

Nothing else changed on top of what you reviewed. Not run locally here: the full packages/core suite.

qqqys and others added 8 commits November 6, 2026 12:41
…6385)

With experimental.unifiedAgentLoop on, the stream processor hands a
shared round's tool batch to the round's submitQuery, which awaits it,
its completion step and the round that step sends, once its own stream
and activity ended. While another stream is active, which may be the
stream that sent the round (a response of provider replays alone), the
batch completes on its own as before.

- The Ink executor takes a scheduler hold: the scheduler's completion
  callback returns once the completion step stopped the turn or is
  about to send the next round, as the current path holds it, up to the
  round whose calls need the scheduler.
- A batch whose executed calls history all answered no longer sends its
  replay answers alone.

The design doc describes PR 3b-2b.
* feat(serve): add atomic user Skill activation

* test(serve): cover conditional user Skill activation capability

* docs(serve): index user Skill activation endpoint

* fix(serve): preserve User Skill refresh recovery
…#16422)

* feat(skills): add file-backed archive installation kernel

* feat(skills): expose native archive routes and receipts
* chore(release): v0.26.8

* chore(release): sdk-typescript v0.1.28

---------

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Shaojin Wen <shaojin.wensj@alibaba-inc.com>
Matching `[:`/`[.`/`[=` anywhere in the pattern rejected ordinary
classes such as `[.]` -- a common literal-dot idiom that sed and the
translator both handle. Only open a POSIX construct when the opening
bracket sits inside a bracket expression, so `[^[:space:]]` and
`[[=a=]]` stay declined while `s/[.]/X/g` keeps the simulated path.
@2SinceJuly

Copy link
Copy Markdown
Contributor Author

Merged origin/main at 1a20792c into the branch as 336c18af (plain merge, no rebase, no force-push). No change to the PR's own files — this also gives the red lane a fresh run.

The only failing check was Linux unit tests (shard 3/8), and it is not this PR:

@wenshao

wenshao commented Nov 6, 2026

Copy link
Copy Markdown
Collaborator

Verdict: merge-ready — 58/58 scripted assertions passed at head 7092daf6f4d844da4214f0156b725c55bb2b7ac2 (base 1b1f4455d9). The PR's own delta since round 5 is two commits, and it is load-bearing in both directions: against real GNU sed 4.10 and real BSD sed, the new scanner (i) declines every real POSIX construct (base simulates four of the named forms wrongly) and (ii) restores 180 plain bracket-class cells that the intermediate attempt over-declined — 360/360 simulated cells match both binaries. One new Suggestion (an escaped-[ POSIX-lookalike is now simulated where GNU sed rejects the pattern) and the carried round-5 Suggestions stand; no blocking finding.

中文摘要
  • 结论:merge-ready(58/58 脚本断言通过;head 7092daf6,base 1b1f4455d9)。本轮是 round-5(df0288326,215/215 merge-ready)之后的跟进轮,PR 自身新增的差异只有两个提交:b8ccdda4(POSIX 字符类检测放宽到取反/混合形态)与 3bb1a0d1(用扫描器替代正则,保留 [.] 这类普通字符类的模拟路径)。
  • 中心 A/B:见下方 Bracket-expression sweep 表。base 臂把 218 格模拟错;中间臂(b8)过度拒绝、丢掉 180 格本可安全模拟的路径;head 两件事都对——0 格错误,且把 b8 拒绝的 180 格全部恢复,180/180 与两个真实 sed 逐字节一致。
  • 端点行为未回归:见 Endpoint input closure 表。prepareSedEdit 在 r5 与 head 间逐字节相同,7 个端点格的替换输出也与 r5 完全一致(对照格证明比较器能看出差异),因此 r5 在真实 qwen serve 写链上得到的结论继续成立。
  • 变异矩阵:12 行,8 杀 4 存活;存活项各自有归因(见下表)。新增断言有精确见证:把扫描器整体退回 round-1 正则(D5)会被 [^[:space:]] 这条新增断言杀死;把扫描器退回中间臂正则(D2)会被新增的 [.] 断言杀死。
  • findings:① 新 Suggestion——s/\[[:alpha:]]/X/g 这类「被转义的 [ + POSIX 外形」现在走模拟(BSD 语义),而 GNU sed 会报错;在 [:] 输入上模拟会真的写盘(F1)。② 沿用 round-5 的三条 Suggestion(Windows .cmd 谓词仍无钉子、逐字写入客户端丢失行尾编辑、u 标志让 15/25 种字面转义回退真 sed),本轮逐条复测。
  • 未覆盖:Windows 与 BSD 之外的 sed 版本(本机 GNU 4.10 + macOS BSD sed)、真实 ACP 传输/daemon 进程、非 UTF-8 编码、packages/core 全量套件;本 head 的 CI 实质车道被 workflow 整体取消,无测试结论。

Previous-finding status

Round 5 (2026-10-31, head df0288326) reported merge-ready with 215/215 and three Suggestions. Since it, the branch adds b8ccdda4bf and 3bb1a0d1a1 (both production + tests) and merges of main. Every row below was re-measured at this head — none is carried by assumption.

# finding severity status at 7092daf6
F1 (r5) the u flag removes literal-escape spellings from the simulation Suggestion stands — re-measured with a 25-spelling list: 15 lost (\-, \&, \%, \~, \{, \}, \#, \@, \!, \,, \:, \;, \=, \_, \ ), 9 kept. Round 5 reported 16/25 from a different list; the code is unchanged, the lists differ
F2 (r5) the Windows .cmd predicate is unpinned Suggestion stands — E1 (delete needsCrlfLineEndings(filePath)) survives 431/431
F3 (r5) a verbatim ACP writer loses line-ending-only edits Suggestion, accepted stands, unchanged — prepareSedEdit and the endpoint cells are byte-identical to round 5 (see Endpoint input closure), so the accepted cost and the qwen serve fix both hold
S1 (r1) zero-width global over an astral character follows BSD, not GNU Suggestion stands, now measured on real BSD sed — head matches BSD 4.x (-😀-); GNU 4.10 interleaves at every byte. Documented, not new
S3 (r1) the gate's s flag is unobservable Suggestion stands — G1 (gate su → u) survives 431/431
R4-3 (r4) Windows .cmd axis Suggestion superseded by F2 above (same axis, same survivor)

Central claim + A/B

Claim unchanged in shape, narrower in scope this round: the simulated sed -i fast path must treat ., bracket expressions and astral characters the way sed does, and must decline every real POSIX bracket construct while keeping plain bracket classes simulated. The oracle is the real binary — both binaries: GNU sed 4.10 (/opt/homebrew/bin/gsed, LC_ALL=C.UTF-8) and BSD sed (/usr/bin/sed, macOS 26.6, en_US.UTF-8). Never hand-written expectations.

Three arms, each bundled from its own sedEditParser.ts (base = 1b1f4455d9, b8 = the intermediate regex guard, head = 7092daf6); the arm sources have distinct sha256 prefixes 3e8d6850…, 5647b025…, 608e9fe3…, and the base bundle contains zero gsu occurrences while b8/head contain exactly one (positive control that the arms are not the same build).

Sweep: 1,110 generated bracket-expression cells; 66 were rejected by the real binary and skipped; 1,044 were adjudicated against both seds.

arm simulated correct on both seds wrong when both seds agree declined
base 1b1f4455 780 562 218 264
b8 b8ccdda4 180 180 0 864
head 7092daf6 360 360 0 684

Bracket-expression sweep: base, the intermediate regex guard and head against real GNU sed 4.10 and BSD sed

01-bracket-sweep.png. The two load-bearing numbers: head restores 180 cells that b8 declined, and 180/180 of them match both seds byte-for-byte; all 218 base-wrong cells are fixed at head.

Named cells: the six rows of issue #15675 plus the bracket family and the leading-] siblings are asserted individually — 32 cells, 33 scripted assertions, 33/33 pass. The issue's x* row is the one documented GNU/BSD split: b8 and head are BSD-shaped (-😀-), base is wrong against both binaries. The Reviewer Test Plan's [abc], [^abc], [\[\]] all take the simulation path and match both binaries (C4/C5/C6).

The round-6 delta

b8ccdda4 widened the POSIX guard to any [:/[./[= and thereby also declined plain classes such as [.]; 3bb1a0d1 replaced it with a bracket-state scanner so only a construct inside a bracket expression declines. Both directions are measured:

cell base b8 (regex guard) head (scanner) oracle
s/[.]/X/g sim, match decline (over-declined) sim, match both seds
s/[.]$// sim, match decline sim, match both seds
s/[=]/X/g sim, match decline sim, match both seds
s/[:]/X/g sim, match decline sim, match both seds
s/[^[:space:]]/X/g sim, WRONG decline decline both seds
s/[[:alpha:]abc]/X/g sim, WRONG decline decline both seds
s/[[=a=]]/X/g sim, WRONG decline decline both seds
s/[[.a.]]/X/g sim, WRONG decline decline both seds

Named cells across base, the intermediate guard and head

02-delta-cells.png (full table). The sweep above generalises this to every generated bracket shape: no cell exists where head simulates bytes that both real seds disagree with.

Endpoint input closure

prepareSedEdit is byte-identical between round 5's head and this head (diff of the extracted function is empty), and packages/core/src/services/fileSystemService.ts, packages/cli/src/acp-integration/service/filesystem.ts and packages/acp-bridge/src/bridgeFileSystem.ts are unchanged (git diff --stat df0288326 7092daf6 over all three files is empty). The one changed input is applySedSubstitution, so the closure is proven at the function boundary: every endpoint cell's substitution output is byte-identical to round 5, and a bracket-pattern control proves the comparator can see a difference.

cell substitution output, r5 vs head
LOCAL-NOOP s/.$// on abc\r\n identical "abc\n"
LOCAL-BOM-NOOP identical "abc\n"
SERVE-NOOP s/\r$// identical "abc\n"
SERVE-BOM-NOOP identical "abc\n"
LOCAL-POSITIVE / SERVE-POSITIVE identical "bar\r\n"
VERBATIM-CRLF identical "abc\n"
control s/[^[:space:]]/X/g r5 simulates, head declines — comparator reports a difference

The three CRLF no-op tests in shell.test.ts also ran green at head in the mutation baseline (431/431), so the prediction itself is exercised, not only its input. The qwen serve chain result from round 5 therefore carries on a proven closure rather than on a promise that nothing changed.

Mutation matrix

Runner: the author's two core suites (sedEditParser.test.ts 54 + shell.test.ts 377 = 431) through the head worktree, baseline green before any row, every anchor asserted to match exactly once, each mutant written and read back, and every restore confirmed by git checkout HEAD -- plus git status --porcelain returning empty. G4 is the expected-kill positive control.

ID mutation expected outcome witness
D1 POSIX detection never fires kill killed rejects POSIX bracket expressions… ([[:space:]]*$, the old expect)
D2 scanner replaced by the b8 blanket regex kill killed keeps plain bracket classes whose first member is ':', '.' or '='
D3 scanner stops skipping escaped characters survive survived — (coverage gap; see F1)
D4 scanner stops closing a bracket at ] survive survived — (no test distinguishes it; the leading-] shapes it could affect are declined by hasLeadingBracketLiteral, inferred from code)
D5 whole scanner reverted to the round-1 regex kill killed rejects POSIX bracket expressions… at the new [^[:space:]] expect (line 244)
G1 gate loses dotAll (su → u) survive (S3) survived —
G2 gate loses the u flag (su → s) kill killed declines u-only invalid escapes…
G3 apply loses dotAll+unicode (gsu → g) kill killed matches one character for . and bracket expressions
G4 divergent-escape class loses p/P/u (control) kill killed rejects sed escapes that diverge in JavaScript regexes
S1 translator stops escaping a literal ] kill killed keeps backslashes literal inside bracket expressions under -E
E1 restoration predicate drops the Windows branch survive (F2) survived —
E2 no-op suppression removed entirely kill killed the two CRLF ShellTool tests

Mutation matrix at head

03-mutation-matrix.png. 12 rows: 8 killed, 4 survived, 0 unexpected. The two delta commits each have a row that isolates their contribution: D5 reverts only b8ccdda4's functional contribution and dies on the assertion that commit added; D2 reverts only 3bb1a0d1's scoping decision and dies on the test that commit added. A per-commit coverage claim over the whole range is not made: this clone is shallow (git rev-parse --is-shallow-repository = true), so git log --not <target> returns ~50 main commits alongside the PR's two, and any enumeration from it would be contaminated (see Not covered).

Findings

F1 — Suggestion, new in this delta: an escaped [ followed by a POSIX-lookalike is now simulated, where GNU sed rejects the pattern. hasPosixBracketExpression skips escaped characters, so s/\[[:alpha:]]/X/g no longer matches a POSIX construct and takes the simulation path. At round 5 the same pattern declined and the real binary ran. Measured consequences against both oracles:

input head simulation GNU sed 4.10 BSD sed
a[b\n / [a\n / x[:alpha:]y\n unchanged syntax error unchanged
[:]\n X\n (differs from the input, so the endpoint writes — inferred from prepareSedEdit) syntax error X\n

Reproduce: node escaped-bracket.mjs "$PWD/tmp/pr15701-verify-20261107-004308" (7 inputs, all agree with at least one sed). The simulation is consistently BSD-shaped, which is the module's documented choice for GNU/BSD splits, and no case was found where it disagrees with both binaries. The residual is that on a GNU-sed host a pattern the binary would reject is now applied instead of erroring, and no test pins the escape-skip: mutation D3 survives all 431 tests. Not blocking — the divergence is narrow and matches a real sed; declining it would need the scanner to treat an escaped [ differently, which is exactly the branch D3 removes. A one-line fixture on s/\[[:alpha:]]/X/g would pin it, and it should state which side (decline vs BSD-simulation) the project wants.

F2 — Suggestion, carried: the Windows .cmd predicate is still unpinned. E1 deletes needsCrlfLineEndings(filePath) from the restoration predicate and the two suites stay 431/431. The branch is unreachable on this rig (os.platform()), so only a platform-mocked test can close it. Agrees with the author's explicit deferral under the five-round rule.

F3 — Suggestion, carried and accepted: a verbatim ACP writer loses line-ending-only edits. Unchanged from round 5, and the closure above shows the behaviour is byte-identical. The skip is reported as "made no changes" — truthful about bytes, silent about the dropped request.

Not covered

  • A per-commit coverage claim. The clone is shallow, so the --not <target> enumeration mixes ~50 main commits into the range; the matrix is per-guard and the two delta commits are named individually. Commits between the previous round and this one that the PR did not author were not mutated.
  • Windows sed and Linux BSD variants. Two real binaries only: GNU 4.10 and macOS BSD sed. E1's Windows branch and the Windows "no sed binary to fall back to" path are unexercised.
  • A real ACP transport / daemon process. The endpoint closure is at the function boundary plus the author's in-process ShellTool tests; no qwen serve process was spawned this round (round 5 measured that chain, and the closure above carries it).
  • Non-UTF-8 encodings beyond BOM, -i.bak, backup suffixes, symlinks.
  • The full packages/core suite. Only the two affected files (431 tests) were run. packages/cli was not re-run.
  • CI at this head. The run (42211976083) is conclusion: cancelled; the red check names carry 0-second durations and no logs (gh run view --job … --log → log not found). Lint & Static passed in the same run; the substantive test lanes were cancelled by the workflow, not failed. No test conclusion is available from CI.

Methodology

Local maintainer round on macOS (Node 24.18.1). Four parser arms — base 1b1f4455d9, intermediate b8ccdda4bf, round-5 df0288326 (for the closure), head 7092daf6 — each esbuild-bundled from that commit's own sedEditParser.ts + errors.ts; oracles are /opt/homebrew/bin/gsed 4.10 and /usr/bin/sed, each writing a real temp file with -i and reading the bytes back. Named cells: 33 scripted comparisons. Sweep: 1,110 generated bracket shapes over six inputs, 1,044 adjudicated. Mutation runner: a detached head worktree at trees/head with hard-linked node_modules (readlink -f node_modules/@qwen-code/qwen-code-core resolves inside that worktree) and a packages/core build; every anchor/restore checked. Raw logs, bundles, runners and the JSON ledgers are in the artifact directory (logs/ab-parser.json, logs/fuzz-brackets.json, logs/closure.json, logs/mut-*.txt, mutations.json, assertions.json). Every number in this report is generated from those files.


Verification round 6 by @wenshao's local /verify-pr rig, 2026-11-07, at head 7092daf6. Advisory evidence for human reviewers; this round does not approve, merge, or close anything.

`s/\\[[:alpha:]]/X/g` is a literal `[` and an ordinary class to BSD sed, so
the simulation rewrote the file. GNU sed rejects that pattern outright as
a malformed bracket expression, so a simulated edit writes a change the
user's own sed would have refused.

The bracket scanner skipped the escaped `[` and so never saw the nested
class. Decline the POSIX-shaped lookalike outside a bracket expression and
leave a plain escaped `[`, and an escaped `[` inside a class, on the fast
path they already had.
@2SinceJuly

Copy link
Copy Markdown
Contributor Author

F1 fixed in 352f7376. The other round-5 Suggestions stand as you measured them.

F1 — an escaped [ followed by a POSIX-looking class was simulated where GNU rejects it. hasPosixBracketExpression skipped the character after a backslash, so it never inspected the [: (or [., [=) that follows a literal \[, and s/\[[:alpha:]]/X/g took the fast path. On BSD sed that is a literal [ plus an ordinary class and the simulation matches the binary; GNU sed rejects the pattern, so the simulated run writes a change the user's own sed would have refused.

The scanner now declines \[[ followed by a class opener when it is outside a bracket expression. Scope is deliberately one shape:

  • s/\[[:alpha:]]/X/g and s/\[[.a.]]/X/g now decline (the finding).
  • s/\[a\]/X/g and s/a\[b/X/g keep their fast path — a plain escaped [ is a literal in every sed, and the existing tests still simulates escapes outside brackets and escaped backslashes and still simulates ordinary bracket expressions still pass unchanged.
  • s/\[\\]/-style members inside a class still simulate.

Pinned by rejects an escaped '[' followed by a POSIX-looking opener in packages/core/src/utils/sedEditParser.test.ts; reverting the guard makes it fail and leaves the other 54 cases green.

Verified at 352f7376:

  • vitest src/utils/sedEditParser.test.ts — 55/55.
  • eslint and prettier --check clean on both files, tsc --noEmit for packages/core clean.

Caveat you should weigh: this machine has no GNU sed (/opt/homebrew/bin/gsed is absent here), so I confirmed the BSD side directly — printf 'a[:]b\n' | /usr/bin/sed 's/\[[:alpha:]]/X/g' gives aXb — and took the GNU-rejects half from your report rather than re-measuring it. If you would rather this stay simulated on BSD hosts and only decline where GNU is the sed in use, that is a different rule and I have not built it.

Not run: the bracket sweep harness, the endpoint cells, and CI — this head's lanes were cancelled, as you noted.

@yiliang114 yiliang114 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The changes correctly fix Unicode and CRLF handling in sed simulation by using su RegExp flags.

  • sedEditParser.ts: new RegExp(pattern, 'su') — the s flag makes . match \r (dotAll), and u treats characters as Unicode code points, resolving both astral-character corruption and CRLF line-matching failures.
  • POSIX bracket detection rewritten as a state machine that correctly handles negated, mixed, and equivalence-class forms while keeping plain [.]-style classes on the fast path.
  • Divergent escape detection extended for \p, \P, \u, and \0 — all of which diverge between sed and JavaScript u-mode regexes.
  • toJavascriptPattern: ] escaping scoped to outside-bracket positions only (required by u-flag strict mode).
  • shell.ts: CRLF restoration prediction prevents false "edit applied" reports when the substitution's only effect is consumed by the write path.
  • fileSystemService.ts: Export of needsCrlfLineEndings for the prediction logic.

Tests comprehensively cover astral characters, CRLF substitution, POSIX bracket edge cases, and the GNU/BSD backreference split. All patterns that cannot be safely simulated fall back to real sed.

No P0/P1 findings.

@yiliang114

Copy link
Copy Markdown
Collaborator

Closeout check on head d6166c42 — approvals and threads are fine: two approvals on this head, Risk & Scope present, and the single unresolved thread is an outdated [Suggestion] (R4-1). The gap is closeout (b).

The newest verification report is wenshao's 58/58 run at 2026-11-06T17:14Z, and it binds head 7092daf6. The head has moved to d6166c42 since. This changes what the sed tool reports for . and bracket expressions, which is user-visible output, so I'm not treating the older-head run as evidence here.

A verification pass bound to d6166c42 covering the matching behaviour in 7092daf6..d6166c42 would clear the gate. Required checks on this head are still queued.

@yiliang114

Copy link
Copy Markdown
Collaborator

Closeout (b) isn't met for head d6166c42, so I'm holding the merge even though approvals are in place.

This PR changes packages/core/src/utils/sedEditParser.ts, which changes what sed commands actually match — observable CLI behaviour. The thread has scripted assertions, but no pr-ui-verify report bound to d6166c42 with a PASS per visible claim. Once checks finish, a report bound to this head is what unblocks the merge.

Also still open on the same file: the round-5 [Suggestion] about bracket classes ([.], [=], [:]). Not blocking, but it lands on the exact guard this PR widens — worth a follow-up reference when it's addressed.

@2SinceJuly

Copy link
Copy Markdown
Contributor Author

The head has not moved since your comment — d6166c42 is still the tip, and d6166c42 is also what the previous verification bound. Re-checking your gate:

  • The pr-ui-verify marker has no producing tool in this repo: grep -rn "pr-ui-verify" .github/ scripts/ returns nothing, so I cannot manufacture a report that would satisfy (b) honestly.
  • What I can do is say what the change is observable as, from the code: sedEditParser.ts decides whether a sed run is simulated and what it matches. This head only narrows the accepted grammar — an escaped [ followed by a POSIX-looking class now declines instead of being simulated — so the observable difference is a sed invocation that used to be reported as applied now being left alone. 7092daf6..d6166c42 is that one commit, not a behaviour across the range.

I am not going to post a pr-ui-verify report I cannot back with a real run; a fabricated marker is worse than an open gate. If the gate's marker is something a contributor is expected to produce by hand, tell me the schema and the head I should bind to and I will produce it from an actual CLI run — the parser is exercisable outside an interactive shell, so a scripted before/after on the reported sed output is feasible.

On the round-5 [.] / [=] / [:] suggestion: not addressed in this head, and I am not folding it in here — it changes the guard this PR widens, and mixing it in would make the verification harder, not easier. It belongs in its own change.

@wenshao
wenshao merged commit 0a56277 into QwenLM:main Nov 7, 2026
80 of 84 checks passed
wenshao added a commit that referenced this pull request Nov 7, 2026
…n-hook

main advanced 11 commits (stacked-skill overflow #16404, workflow agent
approvals #16432, daemon UI tool blocks #16306, sibling model settings
#15829, late snapshot publication claim #16489, inline FQN imports in
runtime modules #16504, cancelled shell as interrupted #15836, CSI statement
helper fold #16503, Maven suite cache write-back #15274, sed char match
#15701, Ink TUI tool-batch round #16428). One conflict, the workflow size
ratchet:

- .size-baseline: sdk-java.yml auto-merged to 167124 bytes (HEAD recorded
  161927 / main 165241, both below the combined actual), so record the actual
  merged size 167124 per the ratchet's "record the real size in the same PR"
  rule. The absolute gate (470000) and the 4096-byte baseline allowance both
  pass.

No source conflict: main's commits did not touch this branch's A3 source
files, so RuntimeBrokerService / RuntimeExecutionRecovery (R18-1/R19-1
observe-only mount-handback guard) and the cli serve fence are unchanged.

Verification: check-workflow-size.sh green (bash5); workflow meta tests
(workflow-size + sdk-java-workflow + hosted-process-ci) 428 green; workspace
build and ClassSizeRatchetTest re-run after the merge.
@qwen-code-review-bot

Copy link
Copy Markdown
Collaborator

Released in v0.27.0.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

sed edit simulation: . does not match \r or a whole emoji, so results differ from real sed

7 participants