Repository navigation
test(otel): cover the Helm chart's default-on OTLP endpoint on EKS and AKS - #802
Draft
miconeilaws wants to merge 1 commit into
Draft
miconeilaws wants to merge 1 commit into
miconeilaws wants to merge 1 commit into
Conversation
Open
7 of 10 tasks
…d AKS Add e2e suites for the chart's default-on OTLP endpoint (4317 gRPC, 4318 HTTP) and its service identity chain. - terraform/eks/daemon/otel-otlp + test/otel/otlp: chart defaults plus OTel Container Insights. Ordinary pods send through the cloudwatch-agent Service without pod attributes. Identity cases: app.kubernetes.io labels (unused by default, so service.name is the Deployment name and there is no service.version), annotations, a service.name the sender sets, unknown_service, DaemonSet, CronJob, and a telemetrygen sender on gRPC 4317. The tests check the resource attributes against the sender pod and its node, that no _tmp.service_identity.* or other internal attribute is exported, scope tags, span metrics, the exact OTLP metric name set, logs in /aws/cwagent/<cluster>/otlp, traces in Transaction Search, and that OTel Container Insights metrics of the same pods keep their scope and get the same service identity. - terraform/eks/daemon/otel-otlp-disabled + test/otel/otlp_disabled: otlp.enabled=false. No OTLP Service or host ports, a send through the Service fails, and no metric with the OTLP scope arrives. - test/otel/filters: an OTLP sender container in every workload. The namespace, pod and node label and metric name filters apply to OTLP metrics, logs and spans; a non-empty metricNames.include also drops an unlisted OTLP metric and the span metrics. - test/azure/aks containerinsights mode: a sender through the Service and checks of its metrics, span metrics, metric names, logs and traces, each polled until a deadline. The install values are unchanged; the chart ignores default:otel. - terraform/otlp_service_sender.sh: the shared OTLP/HTTP sender. It logs the HTTP status of every post, which the tests read. - test/otel/otlpendpoint: helpers and the EKS suite setup shared by the suites, with unit tests. - The EKS otel-otlp, otel-otlp-disabled and otel-filters terraform take helm_chart_repo and the AKS terraform helm_charts_repo, so a chart branch on a fork can be tested. The EKS ones stop with a clear error when the chart has no otlp values block. - OTLP log groups are deleted when a test ends, also on failure.
miconeilaws
force-pushed
the
feat/otlp-endpoint-e2e
branch
from
November 11, 2026 18:52
e7d4320 to
815a036
Compare
This was referenced Nov 12, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of the issue
aws-observability/helm-charts#425 makes the
amazon-cloudwatch-observabilitychart serve an OTLP endpoint on the node agent by default (4317 gRPC, 4318 HTTP). Workloads send to thecloudwatch-agentService. The chart builds themetrics/,traces/,logs/andprofiles/cw_k8s_otlppipelines as YAML inotelConfig: they enrich the telemetry like Container Insights enriches the same pod, set the service identity with the chart-wide chain intemplates/_service-identity.tpl, applyotelFilters, compute span metrics, and route logs to/aws/cwagent/<cluster>/otlp.otlp.enabledalone turns the endpoint on or off;agents[].configdefault:otelis ignored and builds the same config asdefault.aws-observability/helm-charts#429 (stacked on #425, same chart major) moves OTEL Container Insights metrics, application logs and kube-state-metrics onto the same chain, so a pod's Container Insights telemetry gets the same
service.nameas its OTLP telemetry.No suite covers this today. The AKS and GKE
default:otelsuites send over hostNetwork to127.0.0.1, so they never go through the Service or exercise pod association from a normal pod, and nothing checks labels,otelFiltersor the identity chain on OTLP telemetry.Description of changes
New EKS suite
test/otel/otlp+terraform/eks/daemon/otel-otlp(registered in the generator). Chart defaults plusotelContainerInsights.enabled=true, two nodes. Ordinary pods (no hostNetwork, no pod attributes) send through the Service, one workload per identity case:service.nameapp.kubernetes.io/nameand/versionlabels, one pod per nodeotelServiceIdentity.useAppLabelsis false by default, so the labels are not used and there is noservice.version)resource.opentelemetry.io/service.nameand/service.versionannotations plus app labelsservice.name, has an app labelunknown_service:shtelemetrygenon gRPC 4317, sendsunknown_service:telemetrygenAll senders but
grpcpost OTLP/HTTP JSON tocloudwatch-agent.amazon-cloudwatch:4318.The tests check:
cloud.resource_id,k8s.workload.*,service.name,service.namespace,service.instance.idas<namespace>.<pod>.<container>,deployment.environment.name, pod and node labels); that labels on the built-in exclusion list andservice.versionare absent; and that no internal attribute (_tmp.*, which covers_tmp.service_identity.*, oraws.log.*) is exported. Attributes outside both lists are allowed, so a detector attribute added in a later agent release does not fail the suite;cloudwatch.source=cloudwatch-agent,cloudwatch.solution=otel-otlp;/aws/cwagent/<cluster>/otlp, stream<namespace>/<service.namespace>/<service.name>, including the stream of theunknown_servicecase;aws/spans);container_cpu_usage_seconds_total,k8s.pod.cpu.usage,kube_pod_info): its own scope, the sameservice.name,service.namespaceanddeployment.environment.nameas the OTLP telemetry (the Add EKS integration test for FluentBit log emission with Instrumentation source #429 chain), noservice.versionand no internal attribute; and its application log group gets the senders' stdout but no OTLP log record.No suite checks k8s.pod.ip: #425 associates pods by
k8s.pod.uid, then name and namespace, then the connection.New EKS suite
test/otel/otlp_disabled+terraform/eks/daemon/otel-otlp-disabled(registered in the generator), following the one-terraform-directory-per-chart-configuration precedent. Same install withotlp.enabled=false: no OTLP port on the Service (the operator creates no Service when the agent's configuration opens no port, which also counts) or the host network, every post through the Service fails, and no metric with the OTLP scope arrives once Container Insights metrics are seen.test/otel/filtersgains an OTLP sender container in every workload, so OTLP telemetry from each namespace carries the same pod labels. New tests check that the namespace filter, the pod and node label filters and the metric name filter apply to OTLP metrics, logs and spans.metricNames.includenow also namesotlp_e2e_requests; the unlistedotlp_e2e_unlistedand the span metrics must be dropped.test/azure/aks(containerinsights mode), the mode that installs the chart with the valuesscripts/azure/setup.shuses: a sender through the Service, and checks of its metrics, span metrics, the OTLP metric name set, logs and traces with the AKS resource attributes, each polled until a shared deadline. The install values are unchanged;agents[0].config=default:otelstays valid because the chart ignores it. The raw-agentdefault:otelmode (test/azure/aks) andtest/gcp/gkedo not install the chart, so they are unchanged.Shared pieces:
terraform/otlp_service_sender.sh(the HTTP sender; it logs the HTTP status of every post, which the tests read) andtest/otel/otlpendpoint(Kubernetes and log helpers and the EKS suite setup, with unit tests). Theotel-otlp,otel-otlp-disabledandotel-filtersEKS terraform takehelm_chart_repo(precedent:otel-llm-serving) and the AKS terraform takeshelm_charts_repo, so a chart branch on a fork can be tested; the EKS ones stop with a clear error when the chart has nootlpvalues block. OTLP log groups are deleted when a test ends, also on failure.Note for reviewers: the first records a new pod sends can reach the agent before its pod cache has the pod. Those records are exported without pod attributes, so the identity and attribute checks select telemetry the agent matched to a pod, and the CronJob sender sends three rounds per run.
Coverage gaps and follow-ups
terraform/gcp/gkeruns a raw agent DaemonSet without Helm, so it cannot test the chart's OTLP endpoint. A GKE chart mode is a follow-up.otelServiceIdentity.useAppLabels=true: not covered here, because it is cluster-wide and would need another cluster. The chart's render and execution tests cover the opt-in order.profiles/cw_k8s_otlppipeline needs a profiles-capable agent image; a suite is a follow-up after the chart's agent image bump.Related PRs and merge order
TestContainerInsightsintest/otel/otlpexpects.release-*chart branch (and fail fast if it has nootlpblock), and the AKS containerinsights suite installshelm_charts_branch,mainby default. Against a chart with kueue metrics integration tests #425 but not Add EKS integration test for FluentBit log emission with Instrumentation source #429,TestContainerInsightsfails by design.default:otelfrom the setup scripts. No order between the two: the AKS install values here keepdefault:otel, which is valid with and without #2402.License
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.
Tests
On the current commit, locally (Go 1.26 in a container):
go vet -tags integrationclean for every changed package (Linux, and Windows for the new packages);make simple-lint(license and import order) passes;go test -run=NO_MATCH ./...(make compile) compiles every package;go test -tags integration -cbuilds the four changed suites;go test ./test/otel/otlpendpoint/passes;gofmtclean;terraform fmt -check -recursiveandterraform validatepass for the four changed terraform directories.otel-otlpsender manifests passkubectl apply --dry-run=client. Thegrpcsender'stelemetrygenarguments were run against a local collector on 4317:otlp_e2e_requestsarrived withotlp_e2e_case=grpcandservice.name=unknown_service:telemetrygen.80a5183and Add EKS integration test for FluentBit log emission with Instrumentation source #429 at11a316a(_service-identity.tpl, pod association, and the renderedotelConfigwithotlp.enabled=false).Pre-change evidence (on an earlier revision of this PR, against an earlier #425 head, before the
otelServiceIdentitychain): a minikube run tapped with a debug exporter resolved the senders through the Service with the then-expected identities, applied theotel-filtersvalues to metrics, logs and spans, and withotlp.enabled=falseremoved 4317/4318 from the Service. It has not been repeated on the current commit.Not run yet: the EKS and AKS suites against real accounts. They need a chart with #425 and #429 (a fork branch through
helm_chart_repo/helm_charts_repo, or the released chart major) and an AWS account (plus an Azure subscription for AKS). The default agent image in the chart crash-loops on the profiles pipeline until the agent image bump, so a run uses a profiles-capable agent image under test. A run will be linked here.PR Checklist
makepasses locally (build, unit tests, lint):make simple-lint,make compile's test compile, unit teststest/otel/otlpendpoint