Skip to content

Unify Helix test result publishing on C# - #17679

Draft
premun wants to merge 14 commits into
dotnet:mainfrom
premun:prvysoky/remove-python-publishing
Draft

premun wants to merge 14 commits into
dotnet:mainfrom
premun:prvysoky/remove-python-publishing

Conversation

@premun

@premun premun commented Oct 8, 2026

Copy link
Copy Markdown
Member

Summary

  • Extract shared C# result downloading, parsing, publishing, transport, and metrics for the Helix SDK and Job Monitor.
  • Publish inline Helix results from the build after work-item completion; keep Job Monitor as the exclusive publisher when enabled.
  • Collect raw xUnit/JUnit XML and TRX on workers using PowerShell/shell payloads; remove the Python Azure Pipelines reporter and its JSON/pickle protocol.
  • Preserve run naming, queue mapping, attachment policies, and work-item execution results; add explicit handling for incomplete downloads/uploads and malformed inline results.
  • Add orchestration, target-ordering, collector, failure, cancellation, and concurrency tests; update reporting and migration documentation.

Compatibility

  • EnableAzurePipelinesReporter remains the inline reporting switch. Local/disabled reporting is unchanged; Job Monitor mode never also publishes inline.
  • Custom JSON/pickle report producers must emit supported raw XML/TRX instead.
  • Inline malformed results now fail reporting rather than silently omitting tests. Job Monitor retains its existing parsing policy.

Validation

  • .\build.cmd -configuration Release -test -projects src\Microsoft.DotNet.Helix\Sdk.Tests\Microsoft.DotNet.Helix.Sdk.Tests\Microsoft.DotNet.Helix.Sdk.Tests.csproj — passed: 366 tests.
  • .\build.cmd -configuration Release -test -pack — failed in two unchanged tests; compilation/package generation and Helix SDK tests passed:
    • CentralPackageManagementTests.ImplicitPackageReferences_ShouldNotConflictWithPackageVersionEntries: existing implicit Microsoft.VisualStudioEng.MicroBuild.Plugins.SwixBuild / central package-version conflict.
    • MsiTests.ItCanBuildATemplatePackMsi: Visual Studio MSBuild\Current\Bin\MSBuild.exe was not found on this machine.
  • Isolated dotnet/arcade-validation checkout, pinned to the locally built 12.0.0-dev packages and using a separate NuGet package directory:
    • & D:\repos\arcade\.dotnet\dotnet.exe build .\src\Validation\src\Validation.csproj --configuration Release — passed, no warnings/errors.
    • & D:\repos\arcade\.dotnet\dotnet.exe msbuild .\HelixReporting.proj /t:VerifyPackagedReporting /p:HelixTargetQueues=package-smoke /v:minimal — passed. Session-only smoke project checked reporting imports, both collector payloads, the shared result-processing assembly, and absence of the Python reporter; synthetic token/queue, no service requests.
  • PowerShell collector tests and independent Git Bash collector checks — passed (recursive/quoted paths, case-insensitive filenames, deduplication, missing-root errors).
  • SDK package contents and git diff --check — passed.
  • Live inline and Job Monitor Helix/AzDO pipeline smoke tests — not run.

To double check:

premun and others added 5 commits October 8, 2026 17:00
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Update the test-results ignore exception for the shared library location.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Collect raw XML results on workers, download and publish them before closing test runs, and preserve monitor ownership and existing failure policies. Add task, target ordering, collection, and incomplete-upload tests.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Remove obsolete reporter scripts and JSON/pickle contracts, retain unrelated worker tooling, and document C# build-side reporting and migration.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Place the collector under tools/test-results and suppress only NuGet's installation-script warning for this intentional runtime payload.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings October 8, 2026 15:35

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Inline MSTest identities can regress and invalid nested Azure DevOps result IDs remain accepted.

1 open finding
What changed in this PR

Unifies Helix test-result collection and Azure DevOps publishing in shared C# infrastructure used by the SDK and Job Monitor.

Changes:

  • Adds shared downloading, parsing, aggregation, publishing, retry, and metrics components.
  • Replaces worker-side Python reporting with raw XML/TRX collectors and build-side publishing.
  • Adds orchestration, failure, concurrency, collector, and compatibility tests plus documentation.
File Description
.gitignore Tracks the relocated test-results project.
Arcade.slnx Adds the shared project.
Documentation/​AzureDevOps/​SendingJobsToHelix.md Documents the publishing flow.
src/​Microsoft.DotNet.Helix/​TestResults/​WorkItemTestResults.cs Moves the result model namespace.
src/​Microsoft.DotNet.Helix/​TestResults/​TransientFailureDetector.cs Moves transient-failure detection.
src/​Microsoft.DotNet.Helix/​TestResults/​TestResultProcessor.cs Adds strict parsing support.
src/​Microsoft.DotNet.Helix/​TestResults/​TestReportingMetrics.cs Generalizes reporting metrics.
src/​Microsoft.DotNet.Helix/​TestResults/​TestNameFormatter.cs Formats fully qualified test titles.
src/​Microsoft.DotNet.Helix/​TestResults/​ResultAggregator.cs Adds shared result aggregation.
src/​Microsoft.DotNet.Helix/​TestResults/​PreparedTestResults.cs Adds prepared-result model.
src/​Microsoft.DotNet.Helix/​TestResults/​Model/​TestResultAttachmentMode.cs Adds attachment policy model.
src/​Microsoft.DotNet.Helix/​TestResults/​Model/​TestResultAttachment.cs Adds attachment model.
src/​Microsoft.DotNet.Helix/​TestResults/​Model/​TestResult.cs Adds parsed test model.
src/​Microsoft.DotNet.Helix/​TestResults/​Model/​TerminalError.cs Adds terminal transport error.
src/​Microsoft.DotNet.Helix/​TestResults/​Model/​AzureDevOpsReportingError.cs Adds reporting error type.
src/​Microsoft.DotNet.Helix/​TestResults/​Microsoft.DotNet.Helix.TestResults.csproj Defines the shared library.
src/​Microsoft.DotNet.Helix/​TestResults/​LocalTestResultsReader.cs Supports strict malformed-result handling.
src/​Microsoft.DotNet.Helix/​TestResults/​ITestResultProcessor.cs Relocates the processor contract.
src/​Microsoft.DotNet.Helix/​TestResults/​IBlobClientFactory.cs Relocates blob factory abstraction.
src/​Microsoft.DotNet.Helix/​TestResults/​IBlobClient.cs Relocates blob abstraction.
src/​Microsoft.DotNet.Helix/​TestResults/​IAzureDevOpsResultTransport.cs Adds publishing transport contract.
src/​Microsoft.DotNet.Helix/​TestResults/​IAzureDevOpsResultPublisher.cs Relocates publisher contract.
src/​Microsoft.DotNet.Helix/​TestResults/​HelixTestResultsDownloader.cs Adds shared artifact downloading.
src/​Microsoft.DotNet.Helix/​TestResults/​AzureDevOpsResultTransport.cs Adds shared Azure DevOps transport.
src/​Microsoft.DotNet.Helix/​TestResults/​AzureDevOpsResultPublisher.cs Strengthens result publication validation.
src/​Microsoft.DotNet.Helix/​TestResults/​AzureDevOpsRateLimitGate.cs Relocates throttling coordination.
src/​Microsoft.DotNet.Helix/​TestResults/​AzureBlobClientFactory.cs Relocates blob client creation.
src/​Microsoft.DotNet.Helix/​TestResults/​AzureBlobClient.cs Relocates blob downloading.
src/​Microsoft.DotNet.Helix/​Sdk/​UploadHelixTestResults.cs Adds inline C# result publishing.
src/​Microsoft.DotNet.Helix/​Sdk/​tools/​xharness-runner/​Readme.md Documents worker result staging.
src/​Microsoft.DotNet.Helix/​Sdk/​tools/​test-results/​collect-test-results.sh Adds POSIX result collection.
src/​Microsoft.DotNet.Helix/​Sdk/​tools/​test-results/​collect-test-results.ps1 Adds PowerShell result collection.
src/​Microsoft.DotNet.Helix/​Sdk/​tools/​Microsoft.DotNet.Helix.Sdk.props Registers the upload task.
src/​Microsoft.DotNet.Helix/​Sdk/​tools/​Microsoft.DotNet.Helix.Sdk.MonoQueue.targets Stages raw worker results.
src/​Microsoft.DotNet.Helix/​Sdk/​tools/​azure-pipelines/​reporter/​test_results_reader/​__init__.py Removes Python result discovery.
src/​Microsoft.DotNet.Helix/​Sdk/​tools/​azure-pipelines/​reporter/​run.py Removes Python reporter entry point.
src/​Microsoft.DotNet.Helix/​Sdk/​tools/​azure-pipelines/​reporter/​RESULTS_FORMAT.md Removes obsolete JSON protocol docs.
src/​Microsoft.DotNet.Helix/​Sdk/​tools/​azure-pipelines/​reporter/​helpers.py Removes Python helpers.
src/​Microsoft.DotNet.Helix/​Sdk/​tools/​azure-pipelines/​reporter/​formats/​xunit.py Removes Python xUnit parser.
src/​Microsoft.DotNet.Helix/​Sdk/​tools/​azure-pipelines/​reporter/​formats/​trx.py Removes Python TRX parser.
src/​Microsoft.DotNet.Helix/​Sdk/​tools/​azure-pipelines/​reporter/​formats/​result_format.py Removes Python parser abstraction.
src/​Microsoft.DotNet.Helix/​Sdk/​tools/​azure-pipelines/​reporter/​formats/​junit.py Removes Python JUnit parser.
src/​Microsoft.DotNet.Helix/​Sdk/​tools/​azure-pipelines/​reporter/​formats/​__init__.py Removes Python format registration.
src/​Microsoft.DotNet.Helix/​Sdk/​tools/​azure-pipelines/​reporter/​_helix_compat.py Removes JSON compatibility protocol.
src/​Microsoft.DotNet.Helix/​Sdk/​tools/​azure-pipelines/​AzurePipelines.props Prevents duplicate monitor publishing.
src/​Microsoft.DotNet.Helix/​Sdk/​tools/​azure-pipelines/​AzurePipelines.MultiQueue.targets Orders upload before run closure.
src/​Microsoft.DotNet.Helix/​Sdk/​tools/​azure-pipelines/​AzurePipelines.MonoQueue.targets Removes worker-side Python reporting.
src/​Microsoft.DotNet.Helix/​Sdk/​Readme.md Documents C# publishing and migration.
src/​Microsoft.DotNet.Helix/​Sdk/​Microsoft.DotNet.Helix.Sdk.csproj References and packages shared components.
src/​Microsoft.DotNet.Helix/​Sdk.Tests/​Microsoft.DotNet.Helix.Sdk.Tests/​UploadHelixTestResultsTests.cs Tests inline upload orchestration.
src/​Microsoft.DotNet.Helix/​Sdk.Tests/​Microsoft.DotNet.Helix.Sdk.Tests/​TransientFailureDetectorTests.cs Updates shared namespace usage.
src/​Microsoft.DotNet.Helix/​Sdk.Tests/​Microsoft.DotNet.Helix.Sdk.Tests/​ReportingTargetsTests.cs Tests publisher selection and ordering.
src/​Microsoft.DotNet.Helix/​Sdk.Tests/​Microsoft.DotNet.Helix.Sdk.Tests/​Microsoft.DotNet.Helix.Sdk.Tests.csproj Adds MSBuild and payload test inputs.
src/​Microsoft.DotNet.Helix/​Sdk.Tests/​Microsoft.DotNet.Helix.Sdk.Tests/​LocalTestResultsReaderTests.cs Tests strict parsing behavior.
src/​Microsoft.DotNet.Helix/​Sdk.Tests/​Microsoft.DotNet.Helix.Sdk.Tests/​HelixServiceTests.cs Tests complete-download failure handling.
src/​Microsoft.DotNet.Helix/​Sdk.Tests/​Microsoft.DotNet.Helix.Sdk.Tests/​Fakes/​InlineReportingTasks.cs Adds target-ordering test doubles.
src/​Microsoft.DotNet.Helix/​Sdk.Tests/​Microsoft.DotNet.Helix.Sdk.Tests/​Fakes/​FakeHelixService.cs Updates shared result types.
src/​Microsoft.DotNet.Helix/​Sdk.Tests/​Microsoft.DotNet.Helix.Sdk.Tests/​CollectTestResultsTests.cs Tests raw result collectors.
src/​Microsoft.DotNet.Helix/​Sdk.Tests/​Microsoft.DotNet.Helix.Sdk.Tests/​AzureDevOpsServiceTests.cs Updates transport and metrics tests.
src/​Microsoft.DotNet.Helix/​Sdk.Tests/​Microsoft.DotNet.Helix.Sdk.Tests/​AzureDevOpsResultPublisherTests.cs Tests incomplete publishing responses.
src/​Microsoft.DotNet.Helix/​JobMonitor/​TestResultUploadPipeline.cs Uses shared reporting metrics.
src/​Microsoft.DotNet.Helix/​JobMonitor/​TestResults/​Model/​PackedTestReport.cs Removes obsolete packed report model.
src/​Microsoft.DotNet.Helix/​JobMonitor/​TestResults/​Model/​AzureDevOpsReportingParameters.cs Removes obsolete reporter parameters.
src/​Microsoft.DotNet.Helix/​JobMonitor/​StatusReporter.cs Uses generalized metrics snapshots.
src/​Microsoft.DotNet.Helix/​JobMonitor/​Services/​HelixService.cs Delegates result downloading.
src/​Microsoft.DotNet.Helix/​JobMonitor/​Services/​AzureDevOpsService.cs Delegates result transport.
src/​Microsoft.DotNet.Helix/​JobMonitor/​Microsoft.DotNet.Helix.JobMonitor.csproj References the shared library.
src/​Microsoft.DotNet.Helix/​JobMonitor/​JobMonitorRunner.cs Constructs shared reporting services.
src/​Microsoft.DotNet.Helix/​JobMonitor/​Interfaces/​IHelixService.cs Imports shared result models.
src/​Microsoft.DotNet.Helix/​JobMonitor/​Design/​Components/​TestResults.md Updates result-processing architecture.
src/​Microsoft.DotNet.Helix/​JobMonitor/​Design/​Architecture.md Updates shared metrics documentation.

🧠 Review effort: Balanced


Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.

@premun
premun marked this pull request as draft October 8, 2026 17:10
Qualify TRX display names only for inline publishing, keeping Job Monitor naming and xUnit custom-display identities unchanged. Cover class collisions, existing prefixes, and data rows through parsing, aggregation, and publication.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings October 12, 2026 11:47

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The inline migration currently loses legacy xUnit, JUnit, and TRX diagnostic information during parsing.

0 open findings

1 resolved since last review
Previously missed (3)

In code that hasn't changed since last review

Medium severity Decode escaped xUnit failure messages

src/​Microsoft.DotNet.Helix/​TestResults/​TestResultProcessor.cs:46

This migration also bypasses the removed inline xUnit parser's _unescape_xunit_message step. xUnit encodes control characters in failure messages as sequences such as \n and \xNNNN; the shared reader currently publishes failure/message verbatim, so inline failures will display the escape sequences instead of the original text. Preserve that decoding in the shared reader and add a representative compatibility test.

Medium severity Preserve skipped TRX test reasons from StdOut

src/​Microsoft.DotNet.Helix/​Sdk/​UploadHelixTestResults.cs:45

Switching inline reporting to LocalTestResultsReader drops the reason for skipped TRX tests. The removed TRX parser read Output/StdOut for NotExecuted, but the shared reader sets SkipReason from ErrorInfo/Message; skipped TRX entries normally have no ErrorInfo, and Failed attachment mode also discards their stdout. As a result, Azure Pipelines will show these tests as skipped with no reason. Preserve the former StdOut mapping (and add a compatibility assertion) before routing inline reporting through this processor.

Medium severity Preserve JUnit failure messages and stack traces

src/​Microsoft.DotNet.Helix/​Sdk/​UploadHelixTestResults.cs:45

The new inline path also changes JUnit failure data: the removed parser mapped the <failure message> attribute to the error message and the element text to the stack trace, while LocalTestResultsReader.ReadJUnitResult ignores the attribute and always sets StackTrace to null. Inline JUnit failures will therefore lose their structured message/stack trace in Azure Pipelines. Port those mappings to the shared reader and cover them with an inline compatibility test.

🧠 Review effort: Balanced

Keep Microsoft.Build for reporting target tests while accepting the upstream removal of coverlet.collector.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: e810b258-0322-45b2-a0d3-b12937362f90
Copilot AI balanced review requested due to automatic review settings October 21, 2026 13:10

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The inline parser migration regresses JUnit failure diagnostics and xUnit failure-message decoding.

1 open issue

Review effort: Balanced

Disable Job Monitor jobs and override EnableHelixJobMonitor for unit and XHarness submissions. Revert this commit before merging the PR.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: e810b258-0322-45b2-a0d3-b12937362f90
Copilot AI balanced review requested due to automatic review settings October 21, 2026 13:25

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved result-processing and collection defects can misreport test outcomes.

3 open issues
1 previously missed issue

In code that hasn't changed since last review

Medium severity Avoid recopied upload artifacts during collection

src/​Microsoft.DotNet.Helix/​Sdk/​tools/​test-results/​collect-test-results.sh:12

find -path interprets its argument as a glob even when quoted. With an upload directory under run[1]/upload, the collector does not prune it and copies upload/already.trx to upload/upload/already.trx. Both artifacts can then be parsed as separate attempts. Compare directory paths literally to avoid copying files already in the upload directory.

Review effort: Balanced

Comment thread azure-pipelines-pr.yml Resolved
Restore JUnit failure messages, exception types, and stack traces; preserve TRX skip reasons from stdout; and decode xUnit failure-message escapes. Cover parsing and serialized inline publication without changing result identities or attachment policies.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: e810b258-0322-45b2-a0d3-b12937362f90
Copilot AI balanced review requested due to automatic review settings October 21, 2026 13:44

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved result-loss, run-finalization, and collector-duplication issues affect reporting correctness.

4 open issues
1 previously missed issue

In code that hasn't changed since last review

Medium severity Glob matching mishandles bracketed paths during collection

src/​Microsoft.DotNet.Helix/​Sdk/​tools/​test-results/​collect-test-results.sh:12

find -path treats its argument as a glob even when shell-quoted. With a working directory named work[1] and an upload directory inside it, the collector fails to prune that directory: an existing upload/already.trx is copied to upload/upload/already.trx. The duplicate can then be downloaded and aggregated as another attempt. Compare directory paths literally, and add a bracket-containing root to the deduplication test.

1 resolved since last review

Review effort: Balanced

Read nested test results in postorder with output scoped to each result. Publish work-item execution results and close test runs after reporting failures without retrying uploads or hiding errors. Fix both collectors for bracketed working directories and cover the failure and publication paths.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: e810b258-0322-45b2-a0d3-b12937362f90
Copilot AI balanced review requested due to automatic review settings October 21, 2026 14:14
@premun

premun commented Oct 21, 2026

Copy link
Copy Markdown
Member Author

Addressed the new review batch in 9634210:

  • Nested TRX rows are parsed/published, with each result retaining only its own diagnostics.
  • Failed inline result uploads now still publish completed work-item execution results and close runs, preserving the reporting error and without retrying the failed uploader.
  • Also fixed the collector issue from the review summary: the shell collector prunes upload directories using literal path equality rather than find glob matching. The same bracketed-directory regression exposed Windows PowerShell resetting its provider location at process startup; the PowerShell collector now uses the inherited process working directory. Both collectors keep existing uploads intact and avoid duplicates.
  • The xUnit decoding finding was already fixed in aebb18f. Temporary pipeline overrides remain intentionally in place for the requested inline-flow trial and must be reverted before merge.

Validation:

  • .\build.cmd -configuration Release -test -projects src\Microsoft.DotNet.Helix\Sdk.Tests\Microsoft.DotNet.Helix.Sdk.Tests\Microsoft.DotNet.Helix.Sdk.Tests.csproj — passed, 400 tests (including nested publication, failure-finalization ordering, and bracketed collector paths).
  • .\build.cmd -configuration Release -test -pack — compilation/package generation and all 400 Helix tests passed; overall failed only on the same two known tests documented earlier: implicit SwixBuild central-package conflict and missing Visual Studio MSBuild for the MSI test.
  • Independent Git Bash collector scenarios — passed for normal/bracketed roots, upload roots inside/outside the working directory, quoted nested paths, and two collection passes without duplication.
  • Rebuilt SDK package contents — shared result-processing DLL, reporting targets, and both collector payloads match the updated build/source files.
  • git diff --check — passed.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Failure isolation remains unresolved, and the cross-component migration still needs live inline and Job Monitor validation.

2 open issues
3 resolved since last review

Review effort: Balanced

Let healthy in-flight and queued uploads finish before reporting collected download, parse, or publication failures. Preserve the original exception for single failures and caller cancellation for the batch. Add mixed-stage regression coverage beyond the parallelism limit.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: e810b258-0322-45b2-a0d3-b12937362f90
Copilot AI balanced review requested due to automatic review settings October 21, 2026 14:38

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Compatibility and download-isolation issues remain unresolved, and the cross-component migration still needs human review and live pipeline validation.

1 open issue
1 previously missed issue

In code that hasn't changed since last review

Medium severity Concurrent uploads can collide on sanitized download directories

src/​Microsoft.DotNet.Helix/​TestResults/​HelixTestResultsDownloader.cs:50

The inline uploader processes up to 16 work items concurrently, but sanitizing their names does not give each download a distinct directory. On Windows, work:item and work?item both become work-item; submission only replaces / (Sdk/Helpers.cs:17–20). If both contain testResults.xml, their downloads can overwrite one another before parsing or fail with sharing violations. Give each download an isolated directory and add a regression with colliding names. The suggested change adds a unique parent directory while preserving the final directory name used by the reader's JUnit fallback.

2 resolved since last review

Review effort: Balanced

<HelixPostCommands>
$(HelixPostCommands);
find . -maxdepth 5 '(' -iname '*.trx' -o -iname 'testResults.xml' -o -iname 'test-results.xml' -o -iname 'test_results.xml' -o -iname 'junit-results.xml' -o -iname 'junitresults.xml' ')' -print0 | xargs -0 -I {} sh -c 'mkdir -p "$HELIX_WORKITEM_UPLOAD_ROOT/$$(dirname "{}")" &amp;&amp; cp -f "{}" "$HELIX_WORKITEM_UPLOAD_ROOT/{}"' || true
sh "$HELIX_CORRELATION_PAYLOAD/test-results/collect-test-results.sh" || exit $?
Show high-importance download, parsing, and publication stages with per-work-item file/result counts and a final summary. Explain the all-jobs completion gate in build output and document logging behavior.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: e810b258-0322-45b2-a0d3-b12937362f90
Copilot AI balanced review requested due to automatic review settings October 21, 2026 15:44
@premun

premun commented Oct 21, 2026

Copy link
Copy Markdown
Member Author

Added high-importance inline reporting progress logs in 4c80639, visible in normal pipeline output:

  • The build explicitly says it is waiting for all submitted Helix jobs before result processing, then announces work-item discovery.
  • Each work item logs download start, downloaded file count (or no supported result files), parsing/aggregation completion, publication start, and completed-work-item progress.
  • A final summary reports successful/failed work-item counts and the total published results. Existing per-work-item failure isolation and cancellation semantics are unchanged.

The supplied excerpt still has the Windows job outstanding; the existing all-jobs completion gate explains why result processing has not started yet. This change makes that gate visible without changing execution order.

Validation:

  • .\build.cmd -configuration Release -test -projects src\Microsoft.DotNet.Helix\Sdk.Tests\Microsoft.DotNet.Helix.Sdk.Tests\Microsoft.DotNet.Helix.Sdk.Tests.csproj — passed, 406 tests. Added stage-order/visibility/count assertions for populated and empty results, plus mixed-success/failure summary assertions.
  • .\build.cmd -configuration Release -test -pack — compilation/package generation and all 406 Helix tests passed; overall failed in unchanged SDK/workload tests: the known implicit SwixBuild central-package conflict and missing Visual Studio MSBuild for the MSI test, plus CreateVisualStudioWorkloadTests.ItCanCreateWorkloadsThatSupportArm64InVisualStudio (Assert.True failure) in this run.
  • Rebuilt SDK package contains the matching task assembly and MultiQueue targets; git diff --check passed.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Unresolved result-correctness issues and missing live validation of both publishing modes require fixes and human verification.

1 open issue
1 previously missed issue

In code that hasn't changed since last review

Medium severity Result collector ignores symlinked regular files

src/​Microsoft.DotNet.Helix/​Sdk/​tools/​test-results/​collect-test-results.sh:12

The -type f predicate excludes symbolic links to result files, unlike the removed Python reader and previous monitor collector. With testResults.xml symlinked to a runner's output outside the working directory, the collector exits successfully but stages nothing; reporting then treats the work item as having no tests. Include symlinks whose targets are regular files without following directory symlinks, and add a collector regression for this case.

Review effort: Balanced

Keep batch boundaries and one outcome summary per work item visible by default. Move processing stages to detailed verbosity and replace repeated generic results links with test run IDs.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: e810b258-0322-45b2-a0d3-b12937362f90
Copilot AI balanced review requested due to automatic review settings October 22, 2026 12:58
@premun

premun commented Oct 22, 2026

Copy link
Copy Markdown
Member Author

Reduced inline result-processing log noise in 8715cb8:

  • Default output retains batch start/end and one summary per work item identifying its Helix job, published result count, top-level outcome breakdown, and completed/total work-item progress.
  • Download, parsing/aggregation, publication-stage messages and shared informational diagnostics now require detailed/diagnostic MSBuild verbosity. Warnings and errors stay visible.
  • Test-run closing messages show the run ID instead of repeating the same build-wide Results tab URL.

Validation: .\build.cmd -configuration Release -test -projects src\Microsoft.DotNet.Helix\Sdk.Tests\Microsoft.DotNet.Helix.Sdk.Tests\Microsoft.DotNet.Helix.Sdk.Tests.csproj passed (406 tests), including assertions for concise default messages and detailed-only stages. .\build.cmd -configuration Release -test -pack was started but interrupted before a final outcome; its Helix SDK tests passed and it reported the previously observed unrelated CentralPackageManagementTests implicit SwixBuild PackageVersion conflict. This is not a passing full-suite validation.

The paused retry/quarantine design and other outstanding review fixes are unchanged.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The cross-component migration has unresolved outcome-handling defects and still needs live inline and Job Monitor validation plus final human review.

1 open issue

Review effort: Balanced

Retain UploadHelixTestResults without the reverted Architecture workaround and update upstream blob client tests for the shared publisher namespace.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: ca6021bb-42a2-4bc8-9962-8d30c3e875e7
Copilot AI balanced review requested due to automatic review settings November 3, 2026 13:21

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Critical issues remain: sanitized work-item names can collide and corrupt concurrently downloaded results, while standalone mono-queue submissions collect results without publishing them.

3 open issues

Review effort: Balanced

Comment on lines +33 to +34
<ItemGroup Condition="'$(EnableHelixJobMonitor)' == 'true' or '$(ImportAzurePipelinesTargets)' == 'true'">
<HelixCorrelationPayload Include="$(MSBuildThisFileDirectory)test-results" IncludeDirectoryName="true" />
return new WorkItemTestResults(jobName, workItemName, []);
}

string workItemDirectory = _fileSystem.PathCombine(outputDirectory, SanitizeDirName(workItemName));

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants