Skip to content

feat(http_telemetry)!: add route, header and body fields to server telemetry - #987

Open
Evgeny Fedorov (evgenyfedorov2) wants to merge 16 commits into
microsoft:mainfrom
evgenyfedorov2:u/efedorov/http-telemetry-pr2
Open

Evgeny Fedorov (evgenyfedorov2) wants to merge 16 commits into
microsoft:mainfrom
evgenyfedorov2:u/efedorov/http-telemetry-pr2

Conversation

@evgenyfedorov2

@evgenyfedorov2 Evgeny Fedorov (evgenyfedorov2) commented Nov 14, 2026 •

Copy link
Copy Markdown
Member

Summary

Adds route, header and body fields to the server telemetry of http_telemetry.

const REQUEST_ID: HeaderName = HeaderName::from_static("x-request-id");

#[routerama::resolver]
enum BookRoute<'p> {
    #[route(GET, "/books/{book}")]
    GetBook { book: &'p str },

    #[route(GET, "/health")]
    Health,
}

let layer = Telemetry::server(sink)
    .clock(clock)
    .excluded_paths([BasePath::from_static("/health/")])
+   .route_resolver(BookRoute::resolver())
+   .route_parameter_data_classes([("book", PUBLIC)])
+   .request_header_data_classes([(REQUEST_ID, PUBLIC)])
+   .request_body_media_types(["application/json"])?
+   .response_body_media_types(["application/json"])?;
let handler = Execute::new(move |_request: HttpRequest| {
    let response = HttpResponse::new(body_builder.empty());
    async move { Ok::<_, HttpError>(response) }
});
let service = (layer, handler).into_service();
  • Route: with route_resolver, the middleware logs the template of the route that matches the request, such as /books/{book}, as http.route. http.route is also a dimension of the http.server.request.duration metric. The middleware logs the route parameters that you pass to route_parameter_data_classes as http.route.parameter.<name>, and redacts each value with the data class of its parameter.
  • Headers: the middleware logs the headers that you pass to request_header_data_classes and response_header_data_classes as http.request.header.<name> and http.response.header.<name>. It redacts each value with the data class of its header.
  • Bodies: with request_body_media_types and response_body_media_types, the middleware logs the start of request and response bodies as http.request.body.content and http.response.body.content. It logs a body whose Content-Type is one of these media types, any text type, or an application type that ends with +json or +xml. It does not log a body whose Content-Encoding names a coding other than identity. It logs the body as text and does not redact it. body_size_limit sets how many bytes of each body the middleware logs: 32 KiB by default, 1.5 MiB at most.
  • The middleware never reads a request body itself. It copies a streamed body while the handler reads it.

A follow-up PR adds client telemetry for fetch.

Changes in other crates

  • routerama: adds the RouteTable trait, which matches a request to a route of a resolver and gives the template of the route and the values of its parameters. Runtime resolvers and the resolvers that #[resolver] generates implement it.
  • observed (breaking): an #[event] field marked #[flatten] adds fields whose names are known only at run time, and EventDescription::contains_metrics also counts the metrics of those fields. visit_fields moves from Event to the new VisitFields trait, which Event extends, so a hand-written Event must also implement VisitFields, including its CONTAINS_METRICS constant.

Add the `RouteTable` trait and its `RouteHit` match result. A resolver lists
the template and the parameter names of each route, and matches a request to
the number of its route and the raw values of its parameters. `RawResolver`
implements it, so a `DynBuilder` resolver does too.
Every resolver that `#[resolver]` generates implements `RouteTable`. The
templates and the parameter names of its static routes are tables built into
the program. `match_route` uses a second copy of the matching code of
`resolve`, so both pick the same route. A snapshot test shows that the output
of `Generator::generate()` does not change.
An `#[event]` struct can get fields whose names are known only at run
time from a field marked `#[flatten]`. The type of that field implements
the new `VisitFields` trait.

`visit_fields` moves from `Event` and `DynEvent` to `VisitFields`, which
both traits now extend. A hand-written `Event` or `DynEvent` must also
implement `VisitFields`.
With a route resolver, the complete, error and abandoned events of the
server have `http.route`, the template of the matched route, and
`http.route.parameter.<name>` for each route parameter that has a data
class. The layer matches the request again with the resolver, only when
a processor wants the event.

The events also have the configured request and response headers as
`http.request.header.<name>` and `http.response.header.<name>`. Each
value is redacted according to the data class of its header.
With `response_body_media_types`, the complete, error and abandoned
events of the server have `http.response.body`, the start of the
response body as text, when the `Content-Type` of the response matches
one of the configured media types. A media type also covers its
structured suffixes, so `application/json` covers
`application/problem+json`. An entry that is not a media type returns
`MediaTypeError`.

`body_size_limit` sets how many bytes are logged, 32 KiB by default.
Bytes that are not valid UTF-8 become the replacement character, and
the body is logged unredacted.
@codecov

codecov Bot commented Nov 14, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.0%. Comparing base (6542ec1) to head (edde471).
⚠️ Report is 3 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff            @@
##            main     #987      +/-   ##
=========================================
  Coverage   99.9%   100.0%              
=========================================
  Files        851      349     -502     
  Lines      74592    24071   -50521     
=========================================
- Hits       74583    24071   -50512     
+ Misses         9        0       -9     
Flag Coverage Δ
linux 100.0% <100.0%> (ø)
linux-arm 100.0% <100.0%> (ø)
scheduled ?
windows 100.0% <100.0%> (+<0.1%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

With `request_body_media_types`, the complete, error and abandoned
events of the server have `http.request.body`, the start of the
request body as text. The middleware does not read a request body by
itself. It copies the bytes of a streamed body while the handler reads
them, as the HTTP logging middleware of ASP.NET Core does, so a body
that the handler does not read logs an empty text. A body that is
already in memory is copied when the request arrives.

Both lists of media types now cover a `Content-Type` by the rule of
`Microsoft.Extensions.Http.Diagnostics`: a type equal to one of the
entries, any `text` type, and an `application` type whose subtype ends
with `+json` or `+xml`. An empty list logs no body, and an entry whose
type or subtype is `*` returns `MediaTypeError`.

The workspace now needs `http-body-util` 0.1.4, which added
`inspect_frame`.
@evgenyfedorov2 Evgeny Fedorov (evgenyfedorov2) changed the title feat(http_telemetry): add route, header and body fields to server telemetry feat(http_telemetry)!: add route, header and body fields to server telemetry Nov 18, 2026
The middleware no longer logs a body whose `Content-Encoding` names a
coding other than `identity`, because the text of compressed bytes
cannot be read. A layer placed before the server layer of
`http_compression`, as the crate docs advise, logged a gzip response
body as broken text. To log such bodies, put the layer after
`http_compression`.

Two new tests make a processor stop the visit of the event fields at a
route parameter and at a header, and check that no later field is
visited.
@evgenyfedorov2
Evgeny Fedorov (evgenyfedorov2) marked this pull request as ready for review November 18, 2026 14:08
Copilot AI balanced review requested due to automatic review settings November 18, 2026 14:08

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Events whose measurements come only from flattened fields are incorrectly marked as containing no metrics, causing metrics-only processors to silently discard those measurements.

1 open issue
What changed in this PR

This PR expands http_telemetry server events with route, header, and body fields. It adds supporting route lookup APIs in routerama and runtime-defined event fields in observed.

Changes:

  • Adds route dimensions, classified route parameters and headers, and size-limited body logging.
  • Introduces RouteTable for runtime and macro-generated resolvers.
  • Adds #[flatten] and moves field visitation into VisitFields, a breaking change for handwritten event implementations.
File Description
crates/​routerama/​tests/​snapshots/​route_table_macro__static_resolver_lists_each_route_attribute.snap Records generated route listings.
crates/​routerama/​tests/​snapshots/​route_table__route_listing_prints_templates_and_parameters.snap Records runtime route listings.
crates/​routerama/​tests/​route_table.rs Tests runtime route tables and captures.
crates/​routerama/​tests/​route_table_macro.rs Tests generated route tables and matching parity.
crates/​routerama/​tests/​bolero_routerama_resolver.rs Adds differential route-table tests.
crates/​routerama/​src/​walk.rs Separates path and resolver lifetimes.
crates/​routerama/​src/​route_table.rs Defines RouteTable and RouteHit.
crates/​routerama/​src/​raw_resolver.rs Stores route metadata and implements route-table matching.
crates/​routerama/​src/​lib.rs Exports and documents route-table APIs.
crates/​routerama/​src/​dyn_builder.rs Passes route definitions into resolver construction.
crates/​routerama/​src/​codegen_helpers/​mod.rs Exports route-table generation helpers.
crates/​routerama/​README.md Updates generated resolver documentation.
crates/​routerama/​Cargo.toml Adds snapshot-testing dependency.
crates/​routerama_build/​tests/​snapshots/​macro_impl__resolver_without_routes_has_empty_route_table.snap Records empty route-table expansion.
crates/​routerama_build/​tests/​snapshots/​macro_impl__dynamic_resolver_route_table_is_runtime_table.snap Records dynamic route-table expansion.
crates/​routerama_build/​tests/​macro_impl.rs Adds route-table expansion tests.
crates/​routerama_build/​tests/​generator.rs Adds a generator-output snapshot test.
crates/​routerama_build/​src/​macro_impl/​resolver.rs Generates route-table implementations.
crates/​routerama_build/​src/​generator.rs Adds internal route-table generation entry point.
crates/​routerama_build/​Cargo.toml Adds snapshot-testing dependency.
crates/​observed/​src/​sink/​core.rs Migrates test events to VisitFields.
crates/​observed/​src/​processing/​event_view.rs Migrates dynamic test event visitation.
crates/​observed/​src/​processing/​event_state.rs Separates field visitation from event metadata.
crates/​observed/​src/​lib.rs Exports VisitFields and documents flattening.
crates/​observed/​src/​interop.rs Makes DynEvent extend VisitFields.
crates/​observed/​src/​event.rs Introduces the shared field-visitation trait.
crates/​observed_testing/​tests/​event_sampling.rs Migrates sampling fixture to VisitFields.
crates/​observed_testing/​tests/​derive_event.rs Tests flattened runtime fields.
crates/​observed_testing/​tests/​coverage_dyn_sink.rs Migrates dynamic sink fixtures.
crates/​observed_macros/​src/​lib.rs Documents both generated trait implementations.
crates/​observed_macros/​README.md Updates generated macro documentation.
crates/​observed_macros_impl/​tests/​snapshots/​expansion__the_event_attribute_accepts_the_disabled_flag_and_a_trailing_comma.snap Updates disabled-event expansion snapshots.
crates/​observed_macros_impl/​tests/​snapshots/​expansion__flatten_calls_visit_fields.snap Records flattened visitor generation.
crates/​observed_macros_impl/​tests/​snapshots/​expansion__flatten_attribute_is_validated.snap Records flattening validation errors.
crates/​observed_macros_impl/​tests/​snapshots/​expansion__every_severity_attribute_maps_to_its_own_variant.snap Updates severity expansion snapshots.
crates/​observed_macros_impl/​tests/​snapshots/​expansion__every_instrument_kind_records_the_field_it_names.snap Updates instrument expansion snapshot.
crates/​observed_macros_impl/​tests/​snapshots/​expansion__dimension_routes_the_log_and_metric_signals_independently.snap Updates dimension-routing expansion.
crates/​observed_macros_impl/​tests/​snapshots/​expansion__an_optional_field_follows_its_if_none_setting.snap Updates optional-field expansion.
crates/​observed_macros_impl/​tests/​snapshots/​expansion__an_instrument_without_a_name_is_named_after_the_event.snap Updates unnamed-instrument expansion.
crates/​observed_macros_impl/​tests/​snapshots/​expansion__an_instrument_looks_through_parentheses_around_its_value_type.snap Updates wrapped-type instrument expansion.
crates/​observed_macros_impl/​tests/​snapshots/​expansion__an_event_with_no_signal_at_all_visits_nothing.snap Updates empty visitor expansion.
crates/​observed_macros_impl/​tests/​snapshots/​expansion__an_event_re_emits_its_struct_without_the_attributes_it_consumed.snap Updates attribute-stripping expansion.
crates/​observed_macros_impl/​tests/​snapshots/​expansion__an_event_field_may_hold_a_shared_reference.snap Updates shared-reference expansion.
crates/​observed_macros_impl/​tests/​snapshots/​expansion__an_attribute_the_macro_does_not_own_is_left_untouched.snap Updates preserved-attribute expansion.
crates/​observed_macros_impl/​tests/​snapshots/​expansion__a_unit_struct_event_expands_to_an_empty_visitor.snap Updates unit-event expansion.
crates/​observed_macros_impl/​tests/​snapshots/​expansion__a_raw_identifier_field_is_reported_under_its_domain_name.snap Updates raw-identifier expansion.
crates/​observed_macros_impl/​tests/​snapshots/​expansion__a_macro_substituted_type_is_seen_through_its_invisible_group.snap Updates substituted-type expansion.
crates/​observed_macros_impl/​tests/​snapshots/​expansion__a_log_message_may_only_reference_an_attribute_that_exists.snap Updates message-validation snapshots.
crates/​observed_macros_impl/​tests/​snapshots/​expansion__a_generic_field_routed_to_no_signal_is_bound_only_by_send_and_sync.snap Records bounds for unvisited generic fields.
crates/​observed_macros_impl/​tests/​snapshots/​expansion__a_generic_event_spells_out_the_bounds_its_body_relies_on.snap Updates generic trait-bound expansion.
crates/​observed_macros_impl/​tests/​snapshots/​expansion__a_fieldless_counter_becomes_the_event_level_metric.snap Updates fieldless-counter expansion.
crates/​observed_macros_impl/​tests/​snapshots/​expansion__a_field_type_that_only_resembles_an_option_is_treated_as_a_plain_field.snap Updates field-type handling snapshots.
crates/​observed_macros_impl/​tests/​snapshots/​expansion__a_disabled_event_records_that_in_its_description.snap Updates disabled-event metadata expansion.
crates/​observed_macros_impl/​tests/​snapshots/​expansion__a_borrowed_string_is_copied_but_a_static_one_is_not.snap Updates string-field expansion.
crates/​observed_macros_impl/​tests/​expansion.rs Adds flattening and generic-bound tests.
crates/​observed_macros_impl/​src/​lib.rs Updates macro implementation documentation.
crates/​observed_macros_impl/​src/​event.rs Parses flattening and generates separate traits.
crates/​observed_macros_impl/​README.md Updates generated implementation documentation.
crates/​http_telemetry/​tests/​server_routes.rs Tests route fields and parameter redaction.
crates/​http_telemetry/​tests/​server_response_body_logging.rs Tests body filtering, limits, and stream preservation.
crates/​http_telemetry/​tests/​server_headers.rs Tests header capture and redaction.
crates/​http_telemetry/​tests/​server_events.rs Makes field comparisons order-independent.
crates/​http_telemetry/​tests/​common/​mod.rs Adds redaction and visitor-control test helpers.
crates/​http_telemetry/​src/​telemetry.rs Adds configuration and request-body capture.
crates/​http_telemetry/​src/​server/​route_fields.rs Prepares route templates and parameter descriptors.
crates/​http_telemetry/​src/​server/​response_body.rs Captures response bytes during polling.
crates/​http_telemetry/​src/​server/​mod.rs Registers and exports new telemetry helpers.
crates/​http_telemetry/​src/​server/​media_types.rs Validates media types and filters body logging.
crates/​http_telemetry/​src/​server/​interner.rs Reuses configured field-name storage.
crates/​http_telemetry/​src/​server/​header_fields.rs Creates classified header descriptors.
crates/​http_telemetry/​src/​server/​events.rs Adds route, header, and body event fields.
crates/​http_telemetry/​src/​server/​drop_guard.rs Carries captured values into terminal events.
crates/​http_telemetry/​src/​server/​body_buffer.rs Buffers bounded body prefixes as text.
crates/​http_telemetry/​src/​lib.rs Documents fields and body-logging behavior.
crates/​http_telemetry/​README.md Updates generated telemetry documentation.
crates/​http_telemetry/​examples/​support/​print.rs Exposes the example processor’s redaction engine.
crates/​http_telemetry/​Cargo.toml Adds dependencies and public-type allowances.
crates/​fetch_winhttp_impl/​src/​telemetry.rs Imports VisitFields for telemetry tests.
Cargo.toml Updates the http-body-util dependency requirement.
Cargo.lock Records added telemetry and test dependencies.

Review effort: Balanced

`RouteTable` now has only `match_route`. A `RouteHit` carries the
template and the parameter names of its route, and
`RouteHit::parameter` returns the value of a parameter by name. Route
numbers, the listing of routes and the generated table of templates
are gone.

`http_telemetry` builds the list of configured route parameters once
and asks each hit for their values, instead of preparing fields for
every route. A route of a `#[resolver]` type lends its template for
`'static`, so `http.route` still needs no copy.

Also fixes findings of a review against the repository rules:
`route_resolver` takes the resolver without an `Arc`, the interner
uses `parking_lot::Mutex` like the rest of the crate, and tests compare
values and panic payloads.
Copilot AI balanced review requested due to automatic review settings November 19, 2026 10:02

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Flattened metrics are omitted from event metadata, causing metric-filtering processors to discard events and lose measurements.

1 open issue
1 previously missed issue

In code that hasn't changed since last review

Medium severity Parse comma-separated Content-Encoding values

crates/​http_telemetry/​src/​server/​media_types.rs:59

This compares each whole Content-Encoding field line to one coding. A value such as identity, identity contains no non-identity coding, but this check rejects it and omits the body field even when its media type matches. Parse the comma-separated members and trim whitespace before checking each coding, as crates/http_compression/src/compression.rs:999-1007 does. Continue to reject unreadable values and any non-identity coding.

1 resolved since last review

Review effort: Balanced

Comment thread crates/routerama/src/route_table.rs Resolved
`VisitFields` has a required `CONTAINS_METRICS` constant. For an event
with a `#[flatten]` field, `EventDescription::contains_metrics` now also
counts the constant of the field type. Before, an event whose only
measurements came from a flattened value reported no metrics, so a
processor that selects metric events never received them.

`DynEvent` declares `visit_fields` again, as it does on `main`, because
a trait with a constant cannot be used as `dyn DynEvent`. A foreign
event adaptor therefore needs no `VisitFields` impl.
Copilot AI balanced review requested due to automatic review settings November 19, 2026 12:05

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved issues make flattened counter-only events advertise metrics without measurements and incorrectly suppress body logging for identity-only Content-Encoding values.

1 open issue
1 previously missed issue

In code that hasn't changed since last review

Medium severity Normalize Content-Encoding tokens before checking for identity

crates/​http_telemetry/​src/​server/​media_types.rs:59

Comparing the entire Content-Encoding value with one token disables body logging for identity, identity or identity\t, even though neither names a non-identity coding. Split comma-separated values, trim whitespace, and ignore empty members, as the decoder does in crates/http_compression/src/compression.rs:994–1007. Continue rejecting any non-identity coding so encoded request and response bodies remain excluded.

2 resolved since last review

Review effort: Balanced

The function keeps its name from `main`. Renaming it only added noise to the change.
Copilot AI balanced review requested due to automatic review settings November 19, 2026 12:20

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Flattening a fieldless counter incorrectly marks an otherwise non-metric parent as containing metrics, causing metric-filtered processors to receive an event with no measurements.

No open issues 🎉

1 resolved since last review

Review effort: Balanced

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Flattened counter-only events can reach metric processors without measurements, while valid identity-only Content-Encoding lists and whitespace variants incorrectly disable body logging.

1 open issue

Review effort: Balanced

Its last snapshot tests were removed in fc5843d, so cargo udeps reports it as unused.
Copilot AI balanced review requested due to automatic review settings November 19, 2026 12:50

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human assessment is needed to judge the impact of flattened fieldless counters reporting metrics despite emitting no metric data points.

1 open issue

Review effort: Balanced

The body fields are `http.request.body.content` and
`http.response.body.content`, the names that OpenTelemetry defines.
When the copy of a body reaches the size limit, its text leaves out an
incomplete character at its end instead of turning it into `�`,
because OpenTelemetry says to cut a body on a character boundary.

`http.route` of a route added at run time copies its template once
instead of twice.

The docs of the data class options say "Adds", because each call adds
to the pairs of the earlier calls. The docs of `observed::VisitFields`
say that a field name built at run time must stay in memory until the
process ends.
Copilot AI balanced review requested due to automatic review settings November 19, 2026 14:10

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Flattened counter-only events incorrectly report metrics despite exposing no measurements, and Content-Encoding parsing suppresses body logging for valid identity-only lists or whitespace.

1 open issue

Review effort: Balanced

A flattened value adds only its fields to the outer event. The counter
of a flattened event itself is not one of them, yet it made
`CONTAINS_METRICS` true, so a processor that selects metric events got
an event with nothing to record. The constant now comes from the field
metrics of the type.

The docs of `VisitFields::visit_fields` say that it returns `Break` as
soon as the visitor does, and the docs of `#[flatten]` say that the
fields of a flattened `#[event]` reach the log only if it has a
severity attribute.
Each `Content-Encoding` line is split at commas and trimmed, and empty
items are ignored, as `http_compression` reads this header. Before,
`identity, identity` turned body logging off.

New tests check the route, a route parameter and the request and
response headers on the error, panic and abandoned events. The docs say
that the logs can contain personal data from the bodies, and that route
parameters are logged only if the layer has a resolver.
Copilot AI balanced review requested due to automatic review settings November 19, 2026 15:08

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues remain for the telemetry enhancements and supporting API changes.

No open issues 🎉

1 resolved since last review

Review effort: Balanced

Copilot AI balanced review requested due to automatic review settings November 19, 2026 15:13

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The review reports no active issues requiring changes.

No open issues 🎉

Review effort: Balanced

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues are blocking approval.

No open issues 🎉

Review effort: Balanced

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants