[Amaury Levé (Evangelink)]

Pull request

Summary

Draft investigation branch for the MSBuild team. Adds a portable Windows benchmark harness, reproduction instructions, and all 60 timing observations comparing worker parallelism, node reuse, and experimental -mt on TestFx.

No product source, build defaults, or test behavior is changed. No related issue has been filed.

At eight workers, ordinary process-mode reuse reduced median time by 26.8% for unchanged builds / 15.0% after a source timestamp touch. Adding -mt to reuse instead made the medians 33.6% / 9.3% slower on the measured host. This is evidence for investigation, not a proposal to change defaults.

Design and decision record

The investigation guide records the controls, actual approaches, limitations, reproduction commands, and task-host/incrementality leads. A separate feature design record is not applicable: this packages a measured investigation without choosing a new production design.

The branch intentionally remains based on measured revision bacb2826dbf20a0192a6753a61a98743faf5d4f8, rather than rebasing and silently changing the workload. Current head: 98fb5fad49b186b8b49242567b5290900a3f60bc.

Demonstration

Windows 11 Enterprise, 32 logical processors, 127.95 GiB RAM; pinned .NET SDK 12.0.100-alpha.1.26509.101 / MSBuild 18.13.0.51001. TestFx.slnx, Debug, restore excluded, classic UWP excluded, modern Windows/WinUI included. MSBUILDUSESERVER=1 is constant.

Median seconds, six timed observations per cell:

Workers	-mt	-nr	Unchanged	Source timestamp touch
1	false	false	42.079	63.593
8	false	false	25.476	40.950
8	false	true	18.648	34.815
8	true	false	34.599	41.221
8	true	true	24.920	38.045

The path-free CSV preserves every observation, including outliers; the guide includes ranges and both block orders. This shared-host experiment is not a CI, tail-latency, desktop MSBuild, or restore-inclusive guarantee.

Unchanged is not a pure no-op: a generated WinUI app.manifest invalidates one test project's compilation. The timestamp-touch scenario triggers 24 Csc calls. These workloads were consistent across configurations and were not repaired or excluded to improve timings.

Text evidence is more useful than screenshots for this investigation; no product UI/layout was changed. Raw binlogs and environment/process logs remain local because they may contain sensitive data. The harness generates fresh diagnostic captures for the team to inspect.

Validation and regression proof

Original experiment at the measured base: 80 successful builds (60 timed, ten warmups, ten separate diagnostic captures). All ten native binlog replays reported zero warnings/errors and equal compiler counts: one Csc call unchanged, 24 after the source touch.

Repository-local harness validation performed on 2026-11-09, using the same pinned SDK, Debug solution/TFM scope and build-only contract; these existing matching records were rechecked for this PR, not presented as a new performance baseline:

Exact command	Result
.\eng\measure-msbuild-incremental.ps1 -Cases processes-reuse,threads-reuse -Blocks 1 -Repetitions 1	Passed: ten builds, four timed samples, four separate diagnostic captures; all exit 0. Native replay matched one/24 Csc calls.
.\eng\measure-msbuild-incremental.ps1 -Cases processes-no-reuse,threads-no-reuse -Workers 2 -Blocks 1 -Repetitions 1 -SkipBinaryLogs	Passed: six builds, four timed samples; all exit 0. Disabled reuse left no tracked MSBuild workers.
.\eng\measure-msbuild-incremental.ps1 -Cases threads-reuse -Blocks 1 -Repetitions 1 -SkipBinaryLogs	Passed: three builds, two timed samples; all exit 0. Revalidated the final PID/creation-time retention refinement.

Verified the source bytes and original timestamp were restored, no owned processes remained, all 60 committed CSV observations match the original records, and invalid worker/empty/duplicate case selections fail before building. The current head's harness matches the validated code; its Windows working-copy and canonical Git hashes are recorded in the guide.

Local reproduction artifacts include runs.csv, summary.csv, metadata.json, processes.json, console logs, and separate diagnostic-*.binlog captures under artifacts\msbuild-incremental. They are deliberately not uploaded. The guide describes the executed evidence and replay command.

Not run: unit/acceptance test execution, packaging, CI, a complete new performance matrix with the revised portable harness, or cancellation/fault-injection validation. This PR does not claim a new product fix or red/green defect proof.

Test-value evidence

Contract	Comparison / counterfactual	Evidence
Preserve the measured build workload while making the harness portable	Original session harness versus repository-local harness	Executed preservation: original 80 builds and 19 reproduction-validation builds passed; four new diagnostic captures matched compiler work. Product source remains at the measured base.
Timestamp restoration and owned-process cleanup	No faulty cleanup/cancellation was injected	Executed: final sequence restored source hash/timestamp, retained the same worker identities, and left no owned processes. Paired fault-injection proof remains unverified.

No production/test behavior changes or new regression tests are part of this draft. Historical timings and current-harness correctness evidence are intentionally distinguished.

[Copilot]

🟡 Changes recommended

Cleanup has a critical exit/PID-reuse race that can abort cleanup or terminate an unrelated process, while inherited toolchain overrides and relative-path handling can produce misleading metadata or misplaced benchmark evidence.

3 open issues

• Terminate workers through retained handles to avoid PID reuse · New
• Resolve relative output paths against PowerShell's current location · New
• Clear inherited build-tool overrides before benchmarking · New

What changed in this PR

Adds a Windows reproduction package for investigating TestFx’s incremental-build performance without changing product source, build defaults, or test behavior.

Changes:

• Adds a configurable benchmark harness comparing worker counts, node reuse, and experimental -mt.
• Documents reproduction steps, workload controls, results, and validation limitations.
• Preserves all 60 historical timing observations in a path-free CSV.

Test-value evidence: For head 98fb5fad49b186b8b49242567b5290900a3f60bc, the PR reports 19 harness-validation builds separately from the original 80-build experiment. Execution artifacts remain local and were not independently verified; cancellation and fault-injection validation remain unproven.

File	Description
eng/​measure-msbuild-incremental.ps1	Runs the benchmark matrix and captures timings, diagnostics, and process metadata.
docs/​investigations/​msbuild-incremental-performance.md	Explains reproduction, results, investigation leads, and evidence limitations.
docs/​investigations/​msbuild-incremental-performance.csv	Records all 60 historical timing samples, including outliers.

_{Review effort: Balanced}

[github-actions]

Final test-triage resolution — Azure Pipelines build 1670870

Resolution: inconclusive, likely environmental one-off. No issue created. The collected evidence is incomplete: evidenceIncomplete=true and one artifact download failed. This is not a clean bill of health.

Affected leg: Windows Debug, task "Test (unit tests only)", which exited with code 2.

Failure signature

• Test: DirectInvokerPreservesFrameworkFailuresAndReportsMalformedReflectionFailures (MSTestAdapter.PlatformServices.UnitTests), .NET Framework 4.8 only.
• Error: Expected object to refer to System.Threading.Tasks.Task {Status=Running} because Timed out waiting for the owned assertion child., but found Task+DelayPromise {Status=Canceled}.
• Source: DebugAssertionChildProbe.cs:90.

Assessment (confidence: medium)

• The agent reported "Free memory is lower than 5%" (about 95% used) on this leg. A child process timing out under memory pressure fits that.
• In the 12-build history window I checked, this test passed on every other leg and TFM in this build and in the history I examined. It takes 13–20s on net48/net462 versus about 1.5s on modern TFMs, so net48 timing is tight.
• Nothing suggests a link to this PR. It adds an MSBuild incremental-build investigation and does not touch the adapter test.
• No crash or hang diagnostics were found. No retry or flaky evidence exists, and the slowness rule (at least 10 samples and a current duration at least 3× the historical p95) was not met.
• History does not show a prior failure of this test, so the one-off cannot be tied to a recurring defect.

Next step: re-run the Windows Debug leg. If the test fails again on net48, treat it as a likely timing-sensitive test, since the 13–20s runtime on net48 points to a tight timeout. File a Bug on DebugAssertionChildProbe.cs if it recurs on other commits.

The final test-triage check showed the PR head and merge SHAs matched this build.

🤖 Automated content by GitHub Copilot. Generated by the Pipeline Test Triage workflow. · copilot · auto · ~43 AIC · ⌖ ~1 AIC · ⊞ 10K