Skip to content

Add per-subject evaluation telemetry opt-out - #12884

Merged
gh-worker-dd-mergequeue-cf854d[bot] merged 3 commits into
masterfrom
leoromanovsky/evaluation-telemetry-opt-out
Oct 20, 2026
Merged

gh-worker-dd-mergequeue-cf854d[bot] merged 3 commits into
masterfrom
leoromanovsky/evaluation-telemetry-opt-out

Conversation

@leoromanovsky

@leoromanovsky leoromanovsky commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

Applications need to evaluate flags while excluding selected subjects from evaluation telemetry. This adds the same explicit, per-subject control as the client SDKs. Applications keep ownership of their consent rules.

Changes and Decisions

  • Add Provider.Options.enableEvaluationTelemetryOptOutParameter(boolean), disabled by default.
  • Also support DD_FEATURE_FLAGS_EVALUATION_TELEMETRY_PER_SUBJECT_OPT_OUT_ENABLED=true. An explicit provider option overrides deployment configuration, including false.
  • Honor only boolean true at dd_evaluation_telemetry_opt_out. Missing, false, null, and other types preserve telemetry.
  • Suppress new evaluation EVP events/counts, OTel evaluation metrics, and automatic flag span enrichment.
  • Preserve flag results, defaults, exposures, experimentation, application hooks, unrelated tracing, and queued data.
  • Keep each decision within its evaluation. Concurrent and nested evaluations do not share subject state.
flowchart TD
    C[Per-evaluation subject context] --> P[Datadog provider]
    P --> E[Evaluate flag]
    E --> R[Return flag result]
    E --> X[Existing exposure handling: unchanged]
    E --> G{Evaluation telemetry opted out?}
    G -->|Yes| S[Skip new evaluation telemetry]
    G -->|No| T[Evaluation EVP, OTel metrics, flag span enrichment]
Loading
// Configure once.
OpenFeatureAPI api = OpenFeatureAPI.getInstance();
api.setProviderAndWait(new Provider(
    new Provider.Options().enableEvaluationTelemetryOptOutParameter(true)));
Client client = api.getClient();

// Supply each request's subject decision. Do not replace the global context.
boolean enabled = client.getBooleanValue("my-feature", false,
    new ImmutableContext(user.getId(), Collections.singletonMap(
        "dd_evaluation_telemetry_opt_out",
        new Value(user.hasDeclinedEvaluationTelemetry()))));

The parameter name is identical across languages. This control does not opt a subject out of experiments or exposure logging.

The module's existing coverage gate still fails on the unchanged base and this branch. This PR does not change its thresholds.

@leoromanovsky leoromanovsky added type: feature Enhancements and improvements tag: ai generated Largely based on code generated by an AI or LLM comp: openfeature OpenFeature labels Oct 15, 2026
@dd-octo-sts

dd-octo-sts Bot commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 13.94 s 13.98 s [-1.1%; +0.5%] (no difference)
startup:insecure-bank:tracing:Agent 12.85 s 12.91 s [-1.1%; +0.3%] (no difference)
startup:petclinic:appsec:Agent 17.20 s 17.08 s [-0.3%; +1.7%] (no difference)
startup:petclinic:iast:Agent 17.04 s 17.04 s [-1.0%; +0.9%] (no difference)
startup:petclinic:profiling:Agent 16.87 s 16.96 s [-1.8%; +0.8%] (no difference)
startup:petclinic:sca:Agent 17.07 s 16.76 s [+1.0%; +2.7%] (significantly worse)
startup:petclinic:tracing:Agent 16.18 s 16.09 s [-0.4%; +1.5%] (no difference)

Commit: f6d47500 · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

@leoromanovsky
leoromanovsky marked this pull request as ready for review October 15, 2026 12:11
@leoromanovsky
leoromanovsky requested a review from a team as a code owner October 15, 2026 12:11
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 15, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-15T12:15:13.953043Z 39ace88 Draft marked ready
🔒 Security Review ✅ Completed 2026-10-15T12:16:35.864516Z 39ace88 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@datadog-datadog-prod-us1-2 datadog-datadog-prod-us1-2 Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: PASS

More details

The opt-out gates evaluation telemetry independently of exposure logging and flag resolution. The completed static review identified no reportable defects.

Was this helpful? React 👍 or 👎

Open Bits AI session

🤖 Bits Code Review · Commit 39ace88

@leoromanovsky
leoromanovsky requested a review from a team as a code owner October 15, 2026 21:37
@leoromanovsky
leoromanovsky requested review from AlexeyKuznetsov-DD and removed request for a team October 15, 2026 21:37
@datadog-datadog-prod-us1-2

This comment has been minimized.

@datadog-datadog-prod-us1-2 datadog-datadog-prod-us1-2 Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: PASS

More details

The evaluation telemetry gate leaves exposure handling and flag results intact. The completed static review identified no actionable defects.

Was this helpful? React 👍 or 👎

Open Bits AI session

🤖 Bits Code Review · Commit 3787306

@leoromanovsky
leoromanovsky added this pull request to the merge queue Oct 20, 2026
@dd-octo-sts

dd-octo-sts Bot commented Oct 20, 2026

Copy link
Copy Markdown
Contributor

/merge

@gh-worker-devflow-routing-ef8351

gh-worker-devflow-routing-ef8351 Bot commented Oct 20, 2026 •

Copy link
Copy Markdown

View all feedbacks in Devflow UI.

2026-10-20 04:34:03 UTC ℹ️ Start processing command /merge


2026-10-20 04:34:07 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in master is approximately 1h (p90).


2026-10-20 05:32:08 UTC ℹ️ MergeQueue: This merge request was merged

@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 20, 2026
@gh-worker-dd-mergequeue-cf854d
gh-worker-dd-mergequeue-cf854d Bot merged commit 8d0d779 into master Oct 20, 2026
609 checks passed
@gh-worker-dd-mergequeue-cf854d
gh-worker-dd-mergequeue-cf854d Bot deleted the leoromanovsky/evaluation-telemetry-opt-out branch October 20, 2026 05:32
@github-actions github-actions Bot added this to the 1.68.0 milestone Oct 20, 2026
PerfectSlayer added a commit that referenced this pull request Oct 20, 2026
Port master #12884 to the dd-openfeature SDK: the
dd_evaluation_telemetry_opt_out context attribute suppresses evaluation
events, evaluation metrics and span enrichment for the subject when
enabled by Provider.Options.enableEvaluationTelemetryOptOutParameter or
DD_FEATURE_FLAGS_EVALUATION_TELEMETRY_PER_SUBJECT_OPT_OUT_ENABLED.
Exposures are unchanged. Provider.Options now defaults to a 30 seconds
initialization timeout, as on master.
PerfectSlayer added a commit that referenced this pull request Oct 20, 2026
Keep the Feature Flags code of this branch: the master copies of the SDK
come from the stacked PRs, and master #12884 and #12477 change modules
this branch removes. They are ported in the following commits.
PerfectSlayer added a commit that referenced this pull request Oct 20, 2026
…gent

Declare DD_FEATURE_FLAGS_EVALUATION_TELEMETRY_PER_SUBJECT_OPT_OUT_ENABLED
in the agent Feature Flags configuration and document the opt-out in
the SDK README, completing the port of master #12884.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PerfectSlayer added a commit that referenced this pull request Oct 20, 2026
…gent

Declare DD_FEATURE_FLAGS_EVALUATION_TELEMETRY_PER_SUBJECT_OPT_OUT_ENABLED
in the agent Feature Flags configuration and document the opt-out in
the SDK README, completing the port of master #12884.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PerfectSlayer added a commit that referenced this pull request Oct 20, 2026
…gent

Declare DD_FEATURE_FLAGS_EVALUATION_TELEMETRY_PER_SUBJECT_OPT_OUT_ENABLED
in the agent Feature Flags configuration and document the opt-out in
the SDK README, completing the port of master #12884.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
gh-worker-dd-mergequeue-cf854d Bot pushed a commit that referenced this pull request Oct 21, 2026
feat(ffe): Port the per-subject evaluation telemetry opt-out to the SDK

Port master #12884 to the dd-openfeature SDK: the
dd_evaluation_telemetry_opt_out context attribute suppresses evaluation
events, evaluation metrics and span enrichment for the subject when
enabled by Provider.Options.enableEvaluationTelemetryOptOutParameter or
DD_FEATURE_FLAGS_EVALUATION_TELEMETRY_PER_SUBJECT_OPT_OUT_ENABLED.
Exposures are unchanged. Provider.Options now defaults to a 30 seconds
initialization timeout, as on master.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

feat(ffe): Send each Feature Flags event batch once

Port the SDK part of master #12477: direct intake no longer retries, and
the event proxy fallback switches to direct intake for good on the first
proxy failure, replaying the batch only when the proxy definitively did
not receive it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

feat(ffe): Flush pending Feature Flags events at application shutdown

The runtime now stops at application shutdown: through the new optional
ShutdownSource bridge component when the Datadog Java agent provides
it, so the agent flushes the SDK before the JVM shutdown hooks run, or
from a JVM shutdown hook otherwise.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

refactor(ffe): Make the bridge shutdown component abstract like the others

No released agent implements the bridge yet, so the shutdown component
needs no default method for compatibility. The no-op bridge returns
null, like the other optional components.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

fix(ffe): Share one shutdown budget between the Feature Flags event pipelines

The runtime closed the flag evaluation pipeline, waiting up to 5 seconds,
before even signaling the exposure pipeline, then waited up to another
800 ms: a stalled final POST could hold the JVM exit for almost 6 seconds
and starve the exposures flush. Port the two-phase shutdown of the agent
writers: signal both pipelines, then await both against one 800 ms
deadline, without interrupting a current or final HTTP request.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

fix(ffe): Fall back to a JVM shutdown hook when the agent shutdown subscription fails

A failing agent shutdown subscription escaped the shared runtime
acquisition after the runtime started, leaking it and failing the
provider initialization. Fall back to the JVM shutdown hook instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

fix(ffe): Prevent startup failures with older Java agents by returning caller defaults when the RC connector is unavailable

Co-authored-by: sarahchen6 <sarah.chen@datadoghq.com>
Co-authored-by: bruce.bujon <bruce.bujon@datadoghq.com>
PerfectSlayer added a commit that referenced this pull request Oct 21, 2026
…gent

Declare DD_FEATURE_FLAGS_EVALUATION_TELEMETRY_PER_SUBJECT_OPT_OUT_ENABLED
in the agent Feature Flags configuration and document the opt-out in
the SDK README, completing the port of master #12884.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PerfectSlayer added a commit that referenced this pull request Oct 21, 2026
Port the agent-side behaviors of the removed writers: send each event
batch once with proxy discovery and recovery, and the Java SDK identity
headers (#12477, #12575); flush the SDK pending events at agent
shutdown, with or without tracing; declare the SDK settings the agent
exposes (#12884, #12774). Move the Feature Flags shutdown tests from the
agent tests to the OpenFeature smoke tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PerfectSlayer added a commit that referenced this pull request Oct 21, 2026
Port the agent-side behaviors of the removed writers: send each event
batch once with proxy discovery and recovery, and the Java SDK identity
headers (#12477, #12575); flush the SDK pending events at agent
shutdown, with or without tracing; declare the SDK settings the agent
exposes (#12884, #12774). Move the Feature Flags shutdown tests from the
agent tests to the OpenFeature smoke tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PerfectSlayer added a commit that referenced this pull request Oct 21, 2026
Port the agent-side behaviors of the removed writers: send each event
batch once with proxy discovery and recovery, and the Java SDK identity
headers (#12477, #12575); flush the SDK pending events at agent
shutdown, with or without tracing; declare the SDK settings the agent
exposes (#12884, #12774). Move the Feature Flags shutdown tests from the
agent tests to the OpenFeature smoke tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PerfectSlayer added a commit that referenced this pull request Oct 22, 2026
Port the agent-side behaviors of the removed writers: send each event
batch once with proxy discovery and recovery, and the Java SDK identity
headers (#12477, #12575); flush the SDK pending events at agent
shutdown, with or without tracing; declare the SDK settings the agent
exposes (#12884, #12774). Move the Feature Flags shutdown tests from the
agent tests to the OpenFeature smoke tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PerfectSlayer added a commit that referenced this pull request Oct 22, 2026
Port the agent-side behaviors of the removed writers: send each event
batch once with proxy discovery and recovery, and the Java SDK identity
headers (#12477, #12575); flush the SDK pending events at agent
shutdown, with or without tracing; declare the SDK settings the agent
exposes (#12884, #12774). Move the Feature Flags shutdown tests from the
agent tests to the OpenFeature smoke tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PerfectSlayer added a commit that referenced this pull request Oct 22, 2026
Port the agent-side behaviors of the removed writers: send each event
batch once with proxy discovery and recovery, and the Java SDK identity
headers (#12477, #12575); flush the SDK pending events at agent
shutdown, with or without tracing; declare the SDK settings the agent
exposes (#12884, #12774). Move the Feature Flags shutdown tests from the
agent tests to the OpenFeature smoke tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PerfectSlayer added a commit that referenced this pull request Oct 22, 2026
Port the agent-side behaviors of the removed writers: send each event
batch once with proxy discovery and recovery, and the Java SDK identity
headers (#12477, #12575); flush the SDK pending events at agent
shutdown, with or without tracing; declare the SDK settings the agent
exposes (#12884, #12774). Move the Feature Flags shutdown tests from the
agent tests to the OpenFeature smoke tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PerfectSlayer added a commit that referenced this pull request Oct 22, 2026
Port the agent-side behaviors of the removed writers: send each event
batch once with proxy discovery and recovery, and the Java SDK identity
headers (#12477, #12575); flush the SDK pending events at agent
shutdown, with or without tracing; declare the SDK settings the agent
exposes (#12884, #12774). Move the Feature Flags shutdown tests from the
agent tests to the OpenFeature smoke tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PerfectSlayer added a commit that referenced this pull request Oct 22, 2026
Port the agent-side behaviors of the removed writers: send each event
batch once with proxy discovery and recovery, and the Java SDK identity
headers (#12477, #12575); flush the SDK pending events at agent
shutdown, with or without tracing; declare the SDK settings the agent
exposes (#12884, #12774). Move the Feature Flags shutdown tests from the
agent tests to the OpenFeature smoke tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp: openfeature OpenFeature tag: ai generated Largely based on code generated by an AI or LLM type: feature Enhancements and improvements

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants