Repository navigation
Clarify spec intent: common xtra field names, labels vs xtra.traits, stdout/stderr line-splitting #53
Description
Context
We are adding a CTRF reporter to Microsoft.Testing.Platform (the engine behind MSTest, xunit.v3, NUnit, and various other .NET test frameworks). Brad Wilson (xunit) recently published a side-by-side comparison (microsoft/testfx#8903) of his xunit.v3 CTRF emitter vs ours, and several differences boiled down to "the spec doesn't say". Before each ecosystem settles on its own conventions, we would love your guidance on the following so that CTRF documents produced by different .NET reporters are interoperable.
1. Canonical key names for common extra fields
The spec marks environment and per-test as additionalProperties: false for everything except extra, which is the freeform escape hatch. That works, but means two producers reporting the same concept (machine name, exception type, etc.) can pick different keys inside extra, hurting tooling that wants to consume both.
Two concrete examples we hit:
- Machine / hostname: we emit
environment.extra.machine; xunit.v3 emitsenvironment.extra.computer. - Exception type name: we emit
tests[].extra.exceptionType(e.g.System.InvalidOperationException); xunit.v3 emitstests[].extra.exception.
Are there "recommended" key names for these (and other widely useful fields like build agent, runtime version, framework name)? It would be great either to (a) bless one of the two existing names per concept, or (b) publish a non-normative "common extra keys" appendix so future producers can converge.
2. What is labels for?
The spec says labels is "Structured key-value metadata for the test case (e.g. priority, severity, external identifiers)" typed as an object whose values are string | number | boolean.
We read the e.g. as non-exhaustive and use labels for the test framework's user-defined key/value traits (e.g. Category=Smoke, Owner=alice). xunit.v3 instead puts traits under tests[].extra.traits because Brad reads labels as restricted to the three listed examples (priority/severity/external IDs).
Could you clarify the intent?
- Option A -
labelsis the right place for any structured key/value test metadata (our reading). - Option B -
labelsis reserved for a narrow administrative set; arbitrary trait-style metadata should go underextra. - Option C - something else (e.g.
parametersfor data-driven inputs,labelsonly for the three listed examples,extra.traitsfor everything else).
We are happy to follow whichever interpretation matches your intent - just want to avoid two .NET CTRF producers diverging on something this central.
3. Are stdout / stderr meant to be one entry per physical line?
The schema describes both as "Standard output lines from test execution", typed as an array of strings. We initially emitted a single multi-line string (one array element with embedded \n); we have since switched to one element per LF-terminated line (with CRLF normalization and the trailing newline not producing an extra empty entry).
Two follow-ups:
- Is that one-element-per-physical-line interpretation what the spec author intended?
- For tests that produce no captured output, should the array be present-and-empty (
[]), or omitted entirely? We currently omit it; the spec doesn't say.
Thanks for the work on CTRF - it has made a real difference for us standardizing across .NET test frameworks.
Activity
bradwilson commentedon Jun 8, 2026
I will add my comment on issue 2, which is that I don't see
labelsanywhere in the spec, so Amaury's description of our differences isn't quite correct.The one thing I did interpret on my own is that
tagsis described asUser-defined tags or labels for categorization (e.g., ["smoke", "regression"]), so the mapping I chose was traits namedCategory. A user who adds[Trait("Category", "Smoke")]not only gets that trait listed intests[].extra.traits(along with all other traits, regardless of name) but also gettingstests[].tagsset to["Smoke"].Ma11hewThomas commentedon Jun 8, 2026
It's great to hear you are adding a CTRF reporter to MTP, and thanks for the feedback.
For point 1, it’s preferred to promote common properties to first class CTRF fields rather than establish recommendations or non-normative conventions for extra, which could become difficult to maintain over time.
Further specification guidance is being drafted for v1 about avoiding collisions in extra, and optional namespacing. While that should help producers avoid conflicts, it would not address the issue raised here unfortunately.
Regarding point 2, option A is correct.
Labels were added recently to help plug the gap between keyless tags and key/value metadata. As CTRF grew, it became clear we’d need to support frameworks that expose simple tag-style classifications, as well as those that emit key/value metadata like traits. @bradwilson was an early adopter of CTRF, and labels weren’t part of the spec at the time.
I'll add an item to clarify this in the schema and specification prior to v1.
Regarding
stdoutandstderrYes, the intended representation for stdout and stderr is one array entry per output line.
For tests with no captured output, both omission and an empty array are valid, this is currently left to producer to decide.
Thanks again for the feedback!
bradwilson commentedon Jun 8, 2026
Can you point to this in the spec?
Also, you should allow arrays for label values, since our users can specify multiple values for a single name.
Ma11hewThomas commentedon Jun 9, 2026
https://github.com/ctrf-io/ctrf/blob/main/spec/ctrf.md#915-labels
I'm investigating an inconsistency with the website documentation at the moment, so apologies for the confusion.
Regarding allowing arrays for label values, this is a reasonable use case, and I can confirm I will add support for it. This means the following will be CTRF compliant:
Arrays will accept the same primitive types as individual label values: string, number, and boolean.
Thanks for raising this and again for the feedback, much appreciated.
bradwilson commentedon Jun 9, 2026
Makes sense, I had been relying on the website spec, not the GitHub spec. Thanks for the clarity!
bradwilson commentedon Jun 9, 2026
I guess I should ask: will
labelsreplacetagsin the 1.0 spec, then? It feels liketagsbecomes a degenerate case for a specific "kind" of label (without prescribing what that should be, where I choseCategoryas the most likely mapping based on wording of the spec).Restore CTRF labels per spec 9.15 and fix build
Ma11hewThomas commentedon Jun 14, 2026
My preference is to keep
tagsandlabelsas complementary fields rather than make labels replace tags.tagsare for simple, keyless classification, e.g.["smoke", "regression", "api"]labelsare for structured key/value metadata, e.g.{ "priority": "high", "owner": "qa" }While tags could technically be represented as a specific label such as
tags, that would prescribe a mapping that CTRF should not impose. Keeping both allows simple categorisation to stay simple, while labels provide a more structured option where needed.Prior to 1.0, I intend to provide clearer guidance and documentation around the intended use of tags and labels.