Skip to content

[Microsoft.Android.Run] Report partial results after interruption - #13313

Open
simonrozsival wants to merge 3 commits into
mainfrom
simonrozsival-ci-crash-result-observability
Open

simonrozsival wants to merge 3 commits into
mainfrom
simonrozsival-ci-crash-result-observability

Conversation

@simonrozsival

@simonrozsival simonrozsival commented Oct 29, 2026 •

Copy link
Copy Markdown
Member

Context: #13278

Android instrumentation already streams individual test results through Microsoft Testing Platform, but an interrupted run may never produce the final results bundle or a usable TRX report. Losing that report must not erase the outcomes already observed, and an absent report must not look like a complete pass.

Retain completed outcomes and unfinished test names in the existing in-memory streaming state before asynchronous result publication. On abnormal termination, print explicitly partial counts, completed outcomes and unfinished context independently of TRX generation. The last observed active test is context, not a proven crash cause; missing bundles and adb failures are not described as proof of a native crash. Preserve normal TRX generation, the legacy device-TRX fallback, publication, retries and gating without adding a wrapper, journal or collector.

The pinned MTP 2.2.1 TRX provider also reported a misleading missing-ITrxReportCapability error for an interrupted active test because its error node omitted TRX class metadata. Carry the streamed class name into that node and use one non-causal unfinished-context message for both the test state and the TRX exception. Completed results retain their outcomes; the interrupted active test is serialized as Failed with an infrastructure-error explanation rather than left in progress. Unfinished streamed tests take this error path even if adb exits successfully or supplies a final bundle, instead of falling into the legacy TRX fallback.

Add thirteen regressions to the existing Xamarin.Android.Tools.AndroidSdk-Tests project, alongside its process and adb tooling tests. Coverage includes real-provider checks of the saved TRX outcomes and exact error text, normal and interrupted runs with unwritable reports, missing bundles, adb failures, process-startup errors, start-only runs and synchronized cancellation after streamed progress. The existing public and official/1ES test jobs already select this project; no separate test project, solution registration or pipeline change remains.

Validation

All commands use the stable .NET host. The existing SDK-discovery suite requires an isolated HOME on this machine: its pre-existing Constructor_SetValuesFromPath test otherwise finds the local android-toolchain/jdk-21 before its fake JDK. The isolated-home baseline passed with 509 tests and 7 skips before moving the unchanged thirteen test cases.

  • dotnet build tests/Xamarin.Android.Tools.AndroidSdk-Tests/Xamarin.Android.Tools.AndroidSdk-Tests.csproj -c Debug -p:AndroidToolsDisableMultiTargeting=false -p:DotNetTargetFrameworkVersion=10.0 -v minimal — passed with 0 errors.
  • env HOME=<isolated-test-home> NUGET_PACKAGES=<existing-package-cache> dotnet test tests/Xamarin.Android.Tools.AndroidSdk-Tests/Xamarin.Android.Tools.AndroidSdk-Tests.csproj --no-build --no-restore -c Debug -p:AndroidToolsDisableMultiTargeting=false -p:DotNetTargetFrameworkVersion=10.0 -v minimal --logger 'trx;LogFileName=pr13313-existing-suite-after.trx' --results-directory <validation-artifacts> — passed: 522 tests, 7 existing environment/platform skips, 0 failures. The saved TRX confirms all thirteen moved streaming cases ran and passed, with no new skips.
  • dotnet build tests/Xamarin.Android.Tools.AndroidSdk-Tests/Xamarin.Android.Tools.AndroidSdk-Tests.csproj -c Release -p:AndroidToolsDisableMultiTargeting=false -p:DotNetTargetFrameworkVersion=10.0 -v minimal — passed with 0 errors.
  • dotnet test tests/Xamarin.Android.Tools.AndroidSdk-Tests/Xamarin.Android.Tools.AndroidSdk-Tests.csproj --no-build --no-restore -c Release -p:AndroidToolsDisableMultiTargeting=false -p:DotNetTargetFrameworkVersion=10.0 -v minimal --filter 'FullyQualifiedName~StreamingTests' — passed: 13 tests, 0 skipped.
  • make all CONFIGURATION=Release — passed after the move with 0 errors; existing SourceLink warnings remain non-fatal.
  • The strengthened real-provider regression failed before the message fix: the saved TRX contained only The test process crashed: Process crashed. instead of the unfinished/non-causal explanation. It passes with the fix and also checks the preceding Passed and interrupted Failed results.
  • Both start-only cases failed before their production correction because the TRX fallback threw. They now save a Failed active-test result with the exact non-causal error text, with and without a final bundle. The cancellation test waits for the preceding pass and next active-test update, cancels the still-running invocation, and verifies cancellation propagates while completed outcomes and unfinished context remain in the console summary.
  • Local source/XML/YAML checks passed: all test bodies are unchanged by the move, both public and official/1ES callers retain the existing test project selection, and the shared pipeline template matches its pre-PR baseline byte-for-byte. git diff --check passed.

Earlier validation with a session-only two-test CoreCLR fixture on an owned Android 35 arm64 emulator observed the first passing result while the target was still alive. Normal execution produced two passing TRX results; a controlled native abort preserved the preceding pass and produced a failed active-test record; deliberately preventing TRX writing still preserved the partial console output. These device checks precede the final error-message and start-only corrections; the final source was rebuilt and the corrections were verified by the real-provider host regressions.

Azure server-side template expansion and Test Results ingestion have not been independently exercised. If TRX writing itself fails, the fallback is task-log evidence, not structured Azure Tests-tab results. Terminating the host itself can also prevent the final partial summary; this change does not introduce a durable crash-forensics protocol.


Pull Request
title and
description
should follow the
commit-messages.md workflow documentation, and in particular should include:

  • Useful description of why the change is necessary.
  • Links to issues fixed
  • Unit tests

Context: #13278

Instrumentation already streams individual test results, but an interrupted run can lose its final bundle or TRX report.  Retain observed outcomes and unfinished tests in the existing streaming state, and print explicitly partial diagnostics on abnormal termination without depending on TRX generation.  The last active test is context, not a proven crash cause.

The pinned MTP TRX provider also reported a misleading missing-ITrxReportCapability error because the interrupted test node omitted its class metadata.  Preserve that metadata and use the same non-causal error message for the test state and the saved TRX exception.  Keep normal TRX publication, retries and gating unchanged.

Add ten standalone regressions using the real TRX provider, including saved error text and outcomes, and select them in the existing macOS test step shared by public and 1ES pipelines.  The final Release SDK build and all ten tests pass in both Release and CI-configured Debug.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: b06f18f2-44ec-4686-be15-78b67a14dcca
Copilot AI balanced review requested due to automatic review settings October 29, 2026 13:28

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

A start-only adb run exiting with code 0 can throw from the legacy TRX fallback instead of saving a failed infrastructure result for the active test, and cancellation after streamed progress lacks regression coverage.

2 open issues
What changed in this PR

This PR improves interrupted on-device test reporting in Microsoft.Android.Run, preserving observed outcomes even when no usable TRX report is produced.

Changes:

  • Retains completed outcomes and unfinished test names for explicitly partial console summaries.
  • Adds TRX class metadata and non-causal error messages for interrupted active tests.
  • Adds ten standalone regression cases, macOS CI selection, and reporting documentation.
File Description
tests/​Microsoft.Android.Run-Tests/​StreamingTests.cs Tests streaming, interruption, TRX output, and reporting failures.
tests/​Microsoft.Android.Run-Tests/​Microsoft.Android.Run-Tests.csproj Defines the standalone adapter test project.
src/​Microsoft.Android.Run/​AndroidTestAdapter.cs Preserves partial observations and improves interrupted-test reporting.
Microsoft.Android.slnx Adds the test project to the solution.
Documentation/​building/​windows/​instructions.md Documents streamed results and partial summaries.
Documentation/​building/​unix/​instructions.md Documents streamed results and partial summaries.
build-tools/​automation/​yaml-templates/​stage-xamarin-android-tools-tests.yaml Selects the suite in the shared macOS standalone-test step.
.github/​skills/​tests/​references/​test-catalog.md Lists the standalone suite and its command.

Review effort: Balanced

Context: #13313

A start-only instrumentation run can exit adb successfully without finishing its active test.  With no completed results, the adapter entered the legacy TRX fallback and threw instead of saving an infrastructure-error result.  Route unfinished streamed tests through the existing abnormal-termination path, even if a final results bundle is present, without describing the interruption as a proven process crash.

Add real-provider regressions for both start-only cases; both fail before the fix and save a Failed result with non-causal context afterward.  Add synchronized cancellation coverage that waits for a preceding pass and the next active test before cancelling the still-running invocation, then verifies cancellation propagates and the partial summary survives.

All 13 tests pass in Release and CI-configured Debug, and make all CONFIGURATION=Release succeeds.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: b06f18f2-44ec-4686-be15-78b67a14dcca
Context: #13313 (comment)

Put the thirteen adapter streaming and TRX regressions in the existing Xamarin.Android.Tools.AndroidSdk-Tests project, alongside the process and adb tooling tests already selected by public and official CI.  Keep the test bodies and source-linked production behavior unchanged; add explicit imports and retain the existing nullable context so the linked files do not depend on project-wide implicit usings.

Remove the separate test project, solution entry and additional CI project selection.  The shared pipeline template is now identical to its pre-PR version.

The full CI-configured Debug suite passes with 522 passed and seven existing skips, including all thirteen moved cases.  Baseline comparison uses an isolated HOME because the normal local toolchain overrides the existing fake-JDK discovery test.  The thirteen Release regressions and final Release SDK build also pass.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: b06f18f2-44ec-4686-be15-78b67a14dcca

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants