Harden Agentless Feature Flags EVP delivery by leoromanovsky · Pull Request #12477 · DataDog/dd-trace-java

leoromanovsky · 2026-09-12T03:52:08Z

Motivation

Java already delivers agentless exposures (#12195) and flag evaluations (#12204) directly to EVP. This follow-up closes reliability gaps found while checking the cross-SDK contract: unsafe replay, inconsistent writer routes, lost Agent URL prefixes and permanent disablement when no route exists at startup. Tracks FFLSDK-187.

Changes

Share Agentless route state between writers, preserve URL prefixes, and require both identity-forwarding capabilities before selecting local EVP.
Keep direct routing sticky; recover only from unavailable state with bounded probes.
Disable underlying retries and redirects; replay only after 404/405 or a proven pre-connect failure. Keep credentials route-specific and Remote Config fixed to Agent EVP v2.
Bound exposure shutdown flushing, including trace-disabled operation, with lifecycle and concurrency coverage.

Decisions

SDK origin/version emission is isolated in stacked identity PR Send Java SDK identity with Feature Flagging EVP events #12575. This bottom PR alone is not completion of the identity contract.
Bounded shutdown remains explicit additional lifecycle scope here; it is independent of identity attribution.
No new product surface or system-test changes. Existing delivery is hardened rather than reimplemented from scratch.

Validation

Existing default-branch base: 3110972ccf744f304a7cf59973e240f9ccd1a071.
Exact reduced candidate: 1b5766b56c67bb7151a084abb752e62e5bb4b332.
./gradlew :communication:spotlessApply :products:feature-flagging:feature-flagging-lib:spotlessApply :communication:test :products:feature-flagging:feature-flagging-lib:test --console=plain --max-workers=4: PASS; communication 234/234, feature-flagging-lib 279/279.
Capability regression proves missing or partial forwarding support is rejected even without identity-header configuration. Credential and no-replay tests remain below the identity split.
git diff --check: PASS. New scope-reduction commit is GitHub verified: verified=true, reason=valid.
No refreshed whole-agent lifecycle, CI, system-tests, dogfooding or backend-intake evidence for this candidate. Earlier results belong to e20a9d77c333ddb0c0444835f23fafba70952373 and must not be carried forward as current proof.
Fresh CI remains separate from local validation. This PR remains a draft.

Add capability-gated local discovery, safe sticky direct fallback, shared route state, send-once semantics, and bounded lifecycle handling for Java Feature Flags telemetry. Environment: Datadog workspace

linear-code · 2026-09-12T03:52:11Z

FFLSDK-187

datadog-official · 2026-09-12T04:05:41Z

🎯 Code Coverage (details)
• Patch Coverage: 80.51%
• Overall Coverage: 59.40% (+0.05%)

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 450f8b3 | Docs | Give us feedback!}

dd-octo-sts · 2026-09-12T04:15:52Z

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite	Status
Startup	🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results

Scenario	Candidate	master	Δ (95% CI of mean)
startup:insecure-bank:iast:Agent	14.72 s	14.56 s	[+0.4%; +1.8%] (maybe worse)
startup:insecure-bank:tracing:Agent	13.56 s	13.49 s	[+0.0%; +1.1%] (maybe worse)
startup:petclinic:appsec:Agent	17.12 s	17.02 s	[-0.4%; +1.6%] (no difference)
startup:petclinic:iast:Agent	16.90 s	16.85 s	[-0.5%; +1.0%] (no difference)
startup:petclinic:profiling:Agent	16.10 s	16.79 s	[-8.1%; -0.0%] (maybe better)
startup:petclinic:sca:Agent	17.00 s	16.86 s	[+0.2%; +1.6%] (maybe worse)
startup:petclinic:tracing:Agent	16.16 s	16.10 s	[-0.6%; +1.2%] (no difference)

Commit: 450f8b38 · CI Pipeline · Benchmarking Platform UI

Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

Move identity attribution to a follow-up branch while keeping forwarding-capability requirements explicit and independent of emitted headers. Preserve credential isolation, no-replay coverage, and existing lifecycle hardening. Validation: communication tests 234/234 and feature-flagging-lib tests 279/279 pass; scoped Spotless formatting passes. Environment: Datadog workspace

chatgpt-codex-connector · 2026-09-21T14:42:54Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-21T14:47:50.191853Z	`1b5766b`	Draft marked ready
🔒 Security Review	✅ Completed 2026-09-21T14:51:15.140290Z	`1b5766b`	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

datadog-official

⚠️ Autotest was unable to complete this review. View session

Please try again by commenting @autotest review.

dougqh

🤖 Generated with Claude Code

Environment: Datadog workspace

Gate shutdown instrumentation on tracing or resolved feature flag enablement, and share Agent path joining. Pin recovery generations, discovery concurrency, replay, idle recovery, and disabled-product behavior with regression tests. Environment: Datadog workspace

vjfridge

LGTM! My earlier feedback is addressed, and local tests passed for flag evaluations and both exposure and evaluation telemetry.

I commented one non-blocking P2 follow-up for direct intake timeout initialization, backed by a reproduction and tested fix.

Initialize shared HTTP settings before the direct client is cached when Feature Flagging starts first. Preserve Remote Config and credential-free construction, and cover both event types with real-client regressions. Environment: Datadog workspace

leoromanovsky · 2026-10-16T15:58:14Z

/merge

gh-worker-devflow-routing-ef8351 · 2026-10-16T15:58:19Z

View all feedbacks in Devflow UI.

2026-10-16 15:58:19 UTC ℹ️ Start processing command /merge

2026-10-16 15:58:25 UTC ℹ️ MergeQueue: Pull request is not mergeable yet

It will be processed automatically as soon as GitHub reports it as mergeable. View in MergeQueue UI.

Run /code blockers to see what is blocking it.
Run /remove to cancel it.

2026-10-16 18:11:21 UTC ℹ️ MergeQueue: merge request added to the queue

The expected merge time in master is approximately 1h (p90).

2026-10-16 19:24:45 UTC ℹ️ MergeQueue: This merge request was merged

Keep the Feature Flags code of this branch: the master copies of the SDK come from the stacked PRs, and master #12884 and #12477 change modules this branch removes. They are ported in the following commits.

Port master #12477 to the SDK and the agent event proxy: - Send each batch once: direct intake no longer retries, the agent proxy uses the send-once client factory and retries only connection refusal, and only with the Remote Configuration source. - Send Remote Configuration events to the fixed Agent EVP v2 endpoint. - Switch to direct intake for good on the first proxy failure, replaying the batch only after a connection refusal or a 404/405 response. - Keep the proxy route when it is the only one, and discover its endpoint again at most once a minute until it recovers.

Port the agent part of master #12477 to the event proxy: use the send-once client factory, send Remote Configuration events to the fixed Agent EVP v2 endpoint with a connection refusal only retry, treat only connection refusals and 404/405 responses as definitive, and keep the proxy route when it is the only one, discovering its endpoint again at most once a minute until it recovers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

feat(ffe): Port the per-subject evaluation telemetry opt-out to the SDK Port master #12884 to the dd-openfeature SDK: the dd_evaluation_telemetry_opt_out context attribute suppresses evaluation events, evaluation metrics and span enrichment for the subject when enabled by Provider.Options.enableEvaluationTelemetryOptOutParameter or DD_FEATURE_FLAGS_EVALUATION_TELEMETRY_PER_SUBJECT_OPT_OUT_ENABLED. Exposures are unchanged. Provider.Options now defaults to a 30 seconds initialization timeout, as on master. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> feat(ffe): Send each Feature Flags event batch once Port the SDK part of master #12477: direct intake no longer retries, and the event proxy fallback switches to direct intake for good on the first proxy failure, replaying the batch only when the proxy definitively did not receive it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> feat(ffe): Flush pending Feature Flags events at application shutdown The runtime now stops at application shutdown: through the new optional ShutdownSource bridge component when the Datadog Java agent provides it, so the agent flushes the SDK before the JVM shutdown hooks run, or from a JVM shutdown hook otherwise. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> refactor(ffe): Make the bridge shutdown component abstract like the others No released agent implements the bridge yet, so the shutdown component needs no default method for compatibility. The no-op bridge returns null, like the other optional components. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> fix(ffe): Share one shutdown budget between the Feature Flags event pipelines The runtime closed the flag evaluation pipeline, waiting up to 5 seconds, before even signaling the exposure pipeline, then waited up to another 800 ms: a stalled final POST could hold the JVM exit for almost 6 seconds and starve the exposures flush. Port the two-phase shutdown of the agent writers: signal both pipelines, then await both against one 800 ms deadline, without interrupting a current or final HTTP request. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> fix(ffe): Fall back to a JVM shutdown hook when the agent shutdown subscription fails A failing agent shutdown subscription escaped the shared runtime acquisition after the runtime started, leaking it and failing the provider initialization. Fall back to the JVM shutdown hook instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> fix(ffe): Prevent startup failures with older Java agents by returning caller defaults when the RC connector is unavailable Co-authored-by: sarahchen6 <sarah.chen@datadoghq.com> Co-authored-by: bruce.bujon <bruce.bujon@datadoghq.com>

Port the agent part of master #12477 to the event proxy: use the send-once client factory, send Remote Configuration events to the fixed Agent EVP v2 endpoint with a connection refusal only retry, treat only connection refusals and 404/405 responses as definitive, and keep the proxy route when it is the only one, discovering its endpoint again at most once a minute until it recovers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Port the agent-side behaviors of the removed writers: send each event batch once with proxy discovery and recovery, and the Java SDK identity headers (#12477, #12575); flush the SDK pending events at agent shutdown, with or without tracing; declare the SDK settings the agent exposes (#12884, #12774). Move the Feature Flags shutdown tests from the agent tests to the OpenFeature smoke tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Harden agentless Feature Flags EVP delivery

612ef52

Add capability-gated local discovery, safe sticky direct fallback, shared route state, send-once semantics, and bounded lifecycle handling for Java Feature Flags telemetry. Environment: Datadog workspace

leoromanovsky added type: feature Enhancements and improvements tag: ai generated Largely based on code generated by an AI or LLM comp: openfeature OpenFeature labels Sep 12, 2026

leoromanovsky mentioned this pull request Sep 12, 2026

Harden Agentless Feature Flags EVP delivery #12200

Closed

test(openfeature): cover EVP fallback edge states

e20a9d7

leoromanovsky mentioned this pull request Sep 12, 2026

test(java): enable agentless EVP egress DataDog/system-tests#7702

Draft

leoromanovsky mentioned this pull request Sep 19, 2026

Send Java SDK identity with Feature Flagging EVP events #12575

Merged

leoromanovsky marked this pull request as ready for review September 21, 2026 14:42

leoromanovsky requested review from a team as code owners September 21, 2026 14:42

leoromanovsky requested review from bric3, btthomas, pavlokhrebto and vandonr and removed request for a team September 21, 2026 14:42

datadog-official Bot reviewed Sep 21, 2026

View reviewed changes

dougqh reviewed Sep 22, 2026

View reviewed changes

leoromanovsky added 2 commits October 7, 2026 01:55

Merge master into Java EVP fallback hardening

909f9ab

Environment: Datadog workspace

leoromanovsky requested a review from dougqh October 7, 2026 02:58

leoromanovsky requested review from aarsilv and vjfridge October 13, 2026 02:45

vjfridge approved these changes Oct 16, 2026

View reviewed changes

Comment thread ...feature-flagging-lib/src/main/java/com/datadog/featureflag/FeatureFlagBackendApiFactory.java

gh-worker-dd-mergequeue-cf854d Bot merged commit ae26c2b into master Oct 16, 2026
822 of 824 checks passed

gh-worker-dd-mergequeue-cf854d Bot deleted the leoromanovsky/fflsdk-187-java-evp-fallback branch October 16, 2026 19:24

github-actions Bot added this to the 1.68.0 milestone Oct 16, 2026

This was referenced Oct 20, 2026

Port master Feature Flags changes to the dd-openfeature SDK #12957

Merged

Connect the dd-openfeature SDK to the agent #12844

Open

PerfectSlayer mentioned this pull request Oct 21, 2026

Restore the agent Feature Flags behaviors through the bridge #12978

Draft

ethan-romero-dg mentioned this pull request Oct 22, 2026

Add debug logging for feature flag onboarding #12897

Open

Navigation Menu

Repository navigation

Harden Agentless Feature Flags EVP delivery - #12477

Harden Agentless Feature Flags EVP delivery#12477
gh-worker-dd-mergequeue-cf854d[bot] merged 11 commits into
masterfrom
leoromanovsky/fflsdk-187-java-evp-fallback

leoromanovsky commented Sep 12, 2026 •

edited

Loading

Uh oh!

linear-code Bot commented Sep 12, 2026

Uh oh!

datadog-official Bot commented Sep 12, 2026 •

edited by datadog-prod-us1-4 Bot

Loading

Uh oh!

dd-octo-sts Bot commented Sep 12, 2026 •

edited

Loading

Uh oh!

chatgpt-codex-connector Bot commented Sep 21, 2026 •

edited

Loading

Uh oh!

datadog-official Bot left a comment

Uh oh!

dougqh left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vjfridge left a comment

Uh oh!

Uh oh!

leoromanovsky commented Oct 16, 2026

Uh oh!

gh-worker-devflow-routing-ef8351 Bot commented Oct 16, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

leoromanovsky commented Sep 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Changes

Decisions

Validation

Uh oh!

linear-code Bot commented Sep 12, 2026

Uh oh!

datadog-official Bot commented Sep 12, 2026 • edited by datadog-prod-us1-4 Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dd-octo-sts Bot commented Sep 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🟢 Java Benchmark SLOs — All performance SLOs passed

Uh oh!

chatgpt-codex-connector Bot commented Sep 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

datadog-official Bot left a comment

Choose a reason for hiding this comment

Uh oh!

dougqh left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vjfridge left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

leoromanovsky commented Oct 16, 2026

Uh oh!

gh-worker-devflow-routing-ef8351 Bot commented Oct 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

leoromanovsky commented Sep 12, 2026 •

edited

Loading

datadog-official Bot commented Sep 12, 2026 •

edited by datadog-prod-us1-4 Bot

Loading

dd-octo-sts Bot commented Sep 12, 2026 •

edited

Loading

chatgpt-codex-connector Bot commented Sep 21, 2026 •

edited

Loading

gh-worker-devflow-routing-ef8351 Bot commented Oct 16, 2026 •

edited

Loading