Skip to content

test(otel): cover the Helm chart's default-on OTLP endpoint on EKS and AKS - #802

Draft
miconeilaws wants to merge 1 commit into
aws:mainfrom
miconeilaws:feat/otlp-endpoint-e2e
Draft

miconeilaws wants to merge 1 commit into
aws:mainfrom
miconeilaws:feat/otlp-endpoint-e2e

Conversation

@miconeilaws

@miconeilaws miconeilaws commented Nov 9, 2026 •

Copy link
Copy Markdown
Collaborator

Description of the issue

aws-observability/helm-charts#425 makes the amazon-cloudwatch-observability chart serve an OTLP endpoint on the node agent by default (4317 gRPC, 4318 HTTP). Workloads send to the cloudwatch-agent Service. The chart builds the metrics/, traces/, logs/ and profiles/cw_k8s_otlp pipelines as YAML in otelConfig: they enrich the telemetry like Container Insights enriches the same pod, set the service identity with the chart-wide chain in templates/_service-identity.tpl, apply otelFilters, compute span metrics, and route logs to /aws/cwagent/<cluster>/otlp. otlp.enabled alone turns the endpoint on or off; agents[].config default:otel is ignored and builds the same config as default.

aws-observability/helm-charts#429 (stacked on #425, same chart major) moves OTEL Container Insights metrics, application logs and kube-state-metrics onto the same chain, so a pod's Container Insights telemetry gets the same service.name as its OTLP telemetry.

No suite covers this today. The AKS and GKE default:otel suites send over hostNetwork to 127.0.0.1, so they never go through the Service or exercise pod association from a normal pod, and nothing checks labels, otelFilters or the identity chain on OTLP telemetry.

Description of changes

New EKS suite test/otel/otlp + terraform/eks/daemon/otel-otlp (registered in the generator). Chart defaults plus otelContainerInsights.enabled=true, two nodes. Ordinary pods (no hostNetwork, no pod attributes) send through the Service, one workload per identity case:

Case Sender Expected service.name
app Deployment with app.kubernetes.io/name and /version labels, one pod per node the Deployment name (otelServiceIdentity.useAppLabels is false by default, so the labels are not used and there is no service.version)
annotation resource.opentelemetry.io/service.name and /service.version annotations plus app labels the annotation
explicit sends service.name, has an app label the sent value
unknown sends unknown_service:sh the Deployment name
daemonset DaemonSet, no identity labels the DaemonSet name
cronjob CronJob, no identity labels the CronJob name
grpc telemetrygen on gRPC 4317, sends unknown_service:telemetrygen the Deployment name

All senders but grpc post OTLP/HTTP JSON to cloudwatch-agent.amazon-cloudwatch:4318.

The tests check:

  • the Service and the agent's host network have 4317 and 4318, and the senders' posts are accepted;
  • the resource attributes of the app sender's metrics, logs and spans from each pod, against the pod and its node from the Kubernetes API (pod name and UID, node, instance ID, instance type, zone, cloud.resource_id, k8s.workload.*, service.name, service.namespace, service.instance.id as <namespace>.<pod>.<container>, deployment.environment.name, pod and node labels); that labels on the built-in exclusion list and service.version are absent; and that no internal attribute (_tmp.*, which covers _tmp.service_identity.*, or aws.log.*) is exported. Attributes outside both lists are allowed, so a detector attribute added in a later agent release does not fail the suite;
  • scope tags cloudwatch.source=cloudwatch-agent, cloudwatch.solution=otel-otlp;
  • the identity cases above, with no internal attribute;
  • span metrics with the span's HTTP dimensions and the sender's resource attributes, labels included;
  • the exact set of metric names with the OTLP scope for the cluster;
  • logs in /aws/cwagent/<cluster>/otlp, stream <namespace>/<service.namespace>/<service.name>, including the stream of the unknown_service case;
  • traces in Transaction Search (aws/spans);
  • OTEL Container Insights for the app sender pods (container_cpu_usage_seconds_total, k8s.pod.cpu.usage, kube_pod_info): its own scope, the same service.name, service.namespace and deployment.environment.name as the OTLP telemetry (the Add EKS integration test for FluentBit log emission with Instrumentation source #429 chain), no service.version and no internal attribute; and its application log group gets the senders' stdout but no OTLP log record.

No suite checks k8s.pod.ip: #425 associates pods by k8s.pod.uid, then name and namespace, then the connection.

New EKS suite test/otel/otlp_disabled + terraform/eks/daemon/otel-otlp-disabled (registered in the generator), following the one-terraform-directory-per-chart-configuration precedent. Same install with otlp.enabled=false: no OTLP port on the Service (the operator creates no Service when the agent's configuration opens no port, which also counts) or the host network, every post through the Service fails, and no metric with the OTLP scope arrives once Container Insights metrics are seen.

test/otel/filters gains an OTLP sender container in every workload, so OTLP telemetry from each namespace carries the same pod labels. New tests check that the namespace filter, the pod and node label filters and the metric name filter apply to OTLP metrics, logs and spans. metricNames.include now also names otlp_e2e_requests; the unlisted otlp_e2e_unlisted and the span metrics must be dropped.

test/azure/aks (containerinsights mode), the mode that installs the chart with the values scripts/azure/setup.sh uses: a sender through the Service, and checks of its metrics, span metrics, the OTLP metric name set, logs and traces with the AKS resource attributes, each polled until a shared deadline. The install values are unchanged; agents[0].config=default:otel stays valid because the chart ignores it. The raw-agent default:otel mode (test/azure/aks) and test/gcp/gke do not install the chart, so they are unchanged.

Shared pieces: terraform/otlp_service_sender.sh (the HTTP sender; it logs the HTTP status of every post, which the tests read) and test/otel/otlpendpoint (Kubernetes and log helpers and the EKS suite setup, with unit tests). The otel-otlp, otel-otlp-disabled and otel-filters EKS terraform take helm_chart_repo (precedent: otel-llm-serving) and the AKS terraform takes helm_charts_repo, so a chart branch on a fork can be tested; the EKS ones stop with a clear error when the chart has no otlp values block. OTLP log groups are deleted when a test ends, also on failure.

Note for reviewers: the first records a new pod sends can reach the agent before its pod cache has the pod. Those records are exported without pod attributes, so the identity and attribute checks select telemetry the agent matched to a pod, and the CronJob sender sends three rounds per run.

Coverage gaps and follow-ups

  • GKE: no chart e2e coverage. terraform/gcp/gke runs a raw agent DaemonSet without Helm, so it cannot test the chart's OTLP endpoint. A GKE chart mode is a follow-up.
  • otelServiceIdentity.useAppLabels=true: not covered here, because it is cluster-wide and would need another cluster. The chart's render and execution tests cover the opt-in order.
  • Profiles: no profiles e2e suite. The profiles/cw_k8s_otlp pipeline needs a profiles-capable agent image; a suite is a follow-up after the chart's agent image bump.

Related PRs and merge order

License

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Tests

On the current commit, locally (Go 1.26 in a container):

  • go vet -tags integration clean for every changed package (Linux, and Windows for the new packages); make simple-lint (license and import order) passes; go test -run=NO_MATCH ./... (make compile) compiles every package; go test -tags integration -c builds the four changed suites; go test ./test/otel/otlpendpoint/ passes; gofmt clean; terraform fmt -check -recursive and terraform validate pass for the four changed terraform directories.
  • The otel-otlp sender manifests pass kubectl apply --dry-run=client. The grpc sender's telemetrygen arguments were run against a local collector on 4317: otlp_e2e_requests arrived with otlp_e2e_case=grpc and service.name=unknown_service:telemetrygen.
  • Expected values were checked against kueue metrics integration tests #425 at 80a5183 and Add EKS integration test for FluentBit log emission with Instrumentation source #429 at 11a316a (_service-identity.tpl, pod association, and the rendered otelConfig with otlp.enabled=false).

Pre-change evidence (on an earlier revision of this PR, against an earlier #425 head, before the otelServiceIdentity chain): a minikube run tapped with a debug exporter resolved the senders through the Service with the then-expected identities, applied the otel-filters values to metrics, logs and spans, and with otlp.enabled=false removed 4317/4318 from the Service. It has not been repeated on the current commit.

Not run yet: the EKS and AKS suites against real accounts. They need a chart with #425 and #429 (a fork branch through helm_chart_repo/helm_charts_repo, or the released chart major) and an AWS account (plus an Azure subscription for AKS). The default agent image in the chart crash-loops on the profiles pipeline until the agent image bump, so a run uses a profiles-capable agent image under test. A run will be linked here.

PR Checklist

…d AKS

Add e2e suites for the chart's default-on OTLP endpoint (4317 gRPC,
4318 HTTP) and its service identity chain.

- terraform/eks/daemon/otel-otlp + test/otel/otlp: chart defaults plus
  OTel Container Insights. Ordinary pods send through the
  cloudwatch-agent Service without pod attributes. Identity cases:
  app.kubernetes.io labels (unused by default, so service.name is the
  Deployment name and there is no service.version), annotations, a
  service.name the sender sets, unknown_service, DaemonSet, CronJob, and
  a telemetrygen sender on gRPC 4317. The tests check the resource
  attributes against the sender pod and its node, that no
  _tmp.service_identity.* or other internal attribute is exported, scope
  tags, span metrics, the exact OTLP metric name set, logs in
  /aws/cwagent/<cluster>/otlp, traces in Transaction Search, and that
  OTel Container Insights metrics of the same pods keep their scope and
  get the same service identity.
- terraform/eks/daemon/otel-otlp-disabled + test/otel/otlp_disabled:
  otlp.enabled=false. No OTLP Service or host ports, a send through the
  Service fails, and no metric with the OTLP scope arrives.
- test/otel/filters: an OTLP sender container in every workload. The
  namespace, pod and node label and metric name filters apply to OTLP
  metrics, logs and spans; a non-empty metricNames.include also drops an
  unlisted OTLP metric and the span metrics.
- test/azure/aks containerinsights mode: a sender through the Service
  and checks of its metrics, span metrics, metric names, logs and traces,
  each polled until a deadline. The install values are unchanged; the
  chart ignores default:otel.
- terraform/otlp_service_sender.sh: the shared OTLP/HTTP sender. It logs
  the HTTP status of every post, which the tests read.
- test/otel/otlpendpoint: helpers and the EKS suite setup shared by the
  suites, with unit tests.
- The EKS otel-otlp, otel-otlp-disabled and otel-filters terraform take
  helm_chart_repo and the AKS terraform helm_charts_repo, so a chart
  branch on a fork can be tested. The EKS ones stop with a clear error
  when the chart has no otlp values block.
- OTLP log groups are deleted when a test ends, also on failure.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant