Kafka Streams instrumentation no longer enabled since 3.7.4 (kafka.enabled → kafka-clients.enabled) #4877

Expected behavior

With instrumentation.kafka.enabled (default true), Kafka Streams applications get the OpenTelemetry kafka-streams instrumentation, as in 3.6.x and 3.7.0–3.7.3. Producer spans created inside a topology (e.g. .to("topic")) are children of the process span.

Actual behavior

Since 3.7.4 the kafka-streams instrumentation module is not enabled. The process request is now only created by the kafka-clients consumer instrumentation when records are polled, so it is not current while the topology runs. Every producer span from the topology becomes a root span:

the trace is broken between the consumed record and the produced record
the producer dependency no longer inherits the parent's sampling decision, so a sampling override on the process request (e.g. 1 %) no longer applies. In one of our apps this took stored publish dependencies from ~360k/day to ~33M/day.
Measured across our Kafka Streams apps in production: share of Kafka publish dependencies without a parent went from 0 % on 3.6.2 to ~100 % on 3.7.9.

Cause (as far as we can tell)

AiConfigCustomizer used to set otel.instrumentation.kafka.enabled=true. kafka is a shared module name for kafka-clients, kafka-streams (and metrics). In #4177 (OTel update) this was changed to otel.instrumentation.kafka-clients.enabled=true plus otel.instrumentation.kafka-clients-metrics.enabled=false, presumably to keep Kafka metrics off after open-telemetry/opentelemetry-java-instrumentation#13803 removed otel.instrumentation.kafka.metric-reporter.enabled. Since common.default-enabled=false, kafka-streams is no longer enabled by anything. Still the case in 3.7.10 and on main.

To Reproduce

Kafka Streams app (Kafka clients/streams 4.x, Spring Boot 4, Java 25 – also expected on older versions), agent via applicationinsights-runtime-attach 3.7.9.
Topology: builder.stream("in").mapValues(...).to("out").
Produce a message to in.
In Application Insights: out publish is a root dependency; it is not a child of in process.
Set OTEL_INSTRUMENTATION_KAFKA_STREAMS_ENABLED=true (or the system property) and repeat: the out publish dependency is a child of in process again, same as on 3.6.2.
Suggested fix

Also set otel.instrumentation.kafka-streams.enabled=true when instrumentation.kafka.enabled is true.

System information

SDK Version: applicationinsights-runtime-attach 3.7.9 (also checked 3.7.10 source)
OS type and version: Alpine Linux (container), AKS
Application Server type and version: Spring Boot 4.0.8 / 4.1.1 (embedded Tomcat)
Using spring-boot? Yes
Additional relevant libraries: kafka-streams 4.x, spring-kafka 4.x
Workaround

OTEL_INSTRUMENTATION_KAFKA_STREAMS_ENABLED=true (env) or -Dotel.instrumentation.kafka-streams.enabled=true before attach.