Repository navigation
Send feature flag exposures after evaluation, and let applications record them with ExposureHook - #12774
Conversation
The provider now deduplicates exposures itself and sends them from a finally-stage hook instead of at resolution. Resolution only checks the new provider cache and stamps __dd_exposure_cache_hit in the flag metadata. The hook records the exposure and sends it when the allocation logs exposures, the subject is not already exposed, and the evaluation has no error, so an evaluation that an application hook fails no longer sends an exposure or marks the subject as exposed. The agent's writer keeps its own cache for now, because an older provider still sends every evaluation.
Test each condition under which the exposure hook sends nothing, and list the exposure cache's key and value classes as POJOs excluded from coverage verification, as the writer's cache classes are.
🟢 Java Benchmark SLOs — All performance SLOs passed
PR vs. master results
Commit: Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion. |
SpotBugs reports IA_AMBIGUOUS_INVOCATION_OF_INHERITED_OR_OUTER_METHOD because the anonymous LinkedHashMap's size() call could resolve to the cache's own size(). A static nested map class has no outer instance, so the call is unambiguous.
This comment has been minimized.
This comment has been minimized.
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 268677e167
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
Logging allocations can fail on older agents because the new cache lookup reads an unsupported serial-ID field before checking compatibility. The provider cache can also suppress exposures after concurrent dispatch reordering or queue rejection.
🤖 Bits Code Review · Commit 268677e
The exposure cache check read split.serialId before exposureSerialId could apply SPLIT_SERIAL_ID_SUPPORTED. With an older dd-java-agent whose Split has no serialId field, every evaluation of an allocation that logs exposures then threw NoSuchFieldError. The read now sits behind the same guard as the other serial id reads.
Exposures are now sent from a provider hook. OpenFeature SDK 1.20.1 to 1.22.1 does not run the hooks of providers wrapped in a MultiProvider, so a Datadog provider inside one sent no exposures. SDK 1.23.0 runs them. The README states 1.23.0 as the minimum for MultiProvider, and a test covers the MultiProvider path; it fails on 1.20.1.
The ImmutableMetadata getters build a debug message for every absent key, whether or not debug logging is on. __dd_do_log is absent from most evaluations by default, so the hook built that string on most of them. It now reads the keys from asUnmodifiableMap().
The flag now names the capability it detects, like SPLIT_SERIAL_ID_SUPPORTED next to it, instead of the case to avoid. Its value is inverted, and every reader is inverted with it.
They have the same size but refresh entries at different times, so once full they can evict different subjects. The comment said they evict the same ones.
leoromanovsky
left a comment
There was a problem hiding this comment.
Catching up; solid iteration and noticing the OF bump needed.
Read feature.flags.exposures.datadog.logging.enabled, enabled by default, as introduced by #12774 for the previous provider.
Port the event side of #12774: exposures carry their exposure features, and flag evaluations carry their evaluation features, which split the aggregation buckets.
Port #12774 to the dd-openfeature provider. Exposures are sent from a finally-stage hook once the evaluation succeeded, deduplicated by a provider-side cache that replaces the exposure pipeline cache, and applications can record them with ExposureHook. Split features are added to the flag metadata. Require OpenFeature SDK 1.23.0 so provider hooks run inside a MultiProvider.
Port the event side of #12774: exposures carry their exposure features, and flag evaluations carry their evaluation features, which split the aggregation buckets. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Port #12774 to the dd-openfeature provider. Exposures are sent from a finally-stage hook once the evaluation succeeded, deduplicated by a provider-side cache that replaces the exposure pipeline cache, and applications can record them with ExposureHook. Split features are added to the flag metadata. Require OpenFeature SDK 1.23.0 so provider hooks run inside a MultiProvider. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Port the event side of #12774: exposures carry their exposure features, and flag evaluations carry their evaluation features, which split the aggregation buckets. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Port #12774 to the dd-openfeature provider. Exposures are sent from a finally-stage hook once the evaluation succeeded, deduplicated by a provider-side cache that replaces the exposure pipeline cache, and applications can record them with ExposureHook. Split features are added to the flag metadata. Require OpenFeature SDK 1.23.0 so provider hooks run inside a MultiProvider. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Port the event side of #12774: exposures carry their exposure features, and flag evaluations carry their evaluation features, which split the aggregation buckets. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Port #12774 to the dd-openfeature provider. Exposures are sent from a finally-stage hook once the evaluation succeeded, deduplicated by a provider-side cache that replaces the exposure pipeline cache, and applications can record them with ExposureHook. Split features are added to the flag metadata. Require OpenFeature SDK 1.23.0 so provider hooks run inside a MultiProvider. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
feat(ffe): Add dd-openfeature SDK settings and configuration delivery Add the SDK settings, the connector SPI the agent can implement, the agentless configuration delivery from the CDN and the direct intake event transport. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> refactor(ffe): Parse the configuration source like the agent Parse the Feature Flags configuration source like the agent enum config parsing: case insensitive, with no trimming of surrounding spaces, so the SDK and the agent keep resolving the same configuration. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> fix(ffe): Validate Feature Flags endpoint URLs Restore the endpoint validation the agent URL builders provided: - reject custom configuration URLs with a scheme other than http(s), - keep the custom URL query when adding the default configuration path, - reject a site that does not make a plain host name, so the API key cannot be sent to another host. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> fix(ffe): Parse boolean and integer settings like the agent Read boolean and integer settings from their raw value instead of a trimmed one: the agent reads a padded boolean as false and ignores a padded integer. Also skip blank values and integers that do not parse in favor of the next source, like the agent, so the SDK resolves the same settings with or without the agent attached. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> fix(ffe): Report only parsing failures as malformed Remote Configuration Keep the configuration listener call out of the parsing error handling, so a listener failure is not logged as a malformed configuration. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> fix(ffe): Fix HTTP timeouts, cancellations, and retries fix(ffe): Reject null events before reserving queue capacity feat(ffe): Add a setting to stop sending exposures to Datadog Read feature.flags.exposures.datadog.logging.enabled, enabled by default, as introduced by #12774 for the previous provider. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> refactor(ffe): Rename the SDK connector to agent bridge Rename the internal.connector SPI package to internal.bridge, Connector to AgentBridge, NoConnector to NoopAgentBridge and Connectors to AgentBridges. The detect() entry point stays out of AgentBridge: the agent injects the bridge implementations when instrumenting it, and they cannot be defined while their own interface is being defined. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> fix(ffe): Log ignored Feature Flags settings and configuration responses Warn when a positive integer setting is not positive, as the agent did, and when the configuration endpoint returns an unexpected HTTP status, so a wrong custom endpoint is diagnosable. Clarify that configuration listeners receive null when the configuration is removed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> fix(ffe): Fix HTTP call deadlock on cancellation HttpCall cancelled the HttpClient exchange while holding its own monitor, and the HttpClient delivered the response body subscription while holding its internal lock and then took that monitor, deadlocking on a timeout or interruption racing with the response body subscription. Track the cancellation and subscription with volatile fields instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: sarahchen6 <sarah.chen@datadoghq.com>
feat(ffe): Add dd-openfeature SDK exposure and flag evaluation pipelines Add the exposure and flag evaluation event pipelines: deduplication, aggregation, payload building and batching over the event transport. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> feat(ffe): Send split features with exposures and flag evaluations Port the event side of #12774: exposures carry their exposure features, and flag evaluations carry their evaluation features, which split the aggregation buckets. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> refactor(ffe): Use the agent bridge in the event pipelines Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> fix(ffe): Omit null map entries from event payloads Co-authored-by: sarahchen6 <sarah.chen@datadoghq.com>
feat(ffe): Add dd-openfeature SDK OpenFeature provider Add the com.datadog.openfeature.Provider OpenFeature provider, its evaluator and hooks (metrics, logging, span enrichment), and the runtime wiring the configuration delivery and event pipelines together. The SDK is still not published, the agent-coupled provider remains the shipped one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> refactor(ffe): Pass evaluator metadata through to OpenFeature The evaluator now builds OpenFeature ImmutableMetadata directly, so the provider no longer copies it on every evaluation. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> refactor(ffe): Delegate evaluation to the OpenFeature-typed evaluator The evaluator now takes the OpenFeature context and returns the ProviderEvaluation, so the provider no longer adapts the context and converts the result on every evaluation. Exposures are dispatched through the evaluator ExposureListener. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> fix(ffe): Isolate Feature Flags configuration listeners Keep notifying the other configuration listeners when one of them fails. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> feat(ffe): Send exposures after evaluation and add ExposureHook Port #12774 to the dd-openfeature provider. Exposures are sent from a finally-stage hook once the evaluation succeeded, deduplicated by a provider-side cache that replaces the exposure pipeline cache, and applications can record them with ExposureHook. Split features are added to the flag metadata. Require OpenFeature SDK 1.23.0 so provider hooks run inside a MultiProvider. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> refactor(ffe): Use the agent bridge in the provider Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> fix(ffe): Allow shutdown while initialization waits for configuration fix(ffe): Flush pending exposures when releasing the runtime chore(ffe): Remove unused supported resolution types from DDEvaluator The type mapping moved to FlagEvaluator, which owns its own copy. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> chore(ffe): Keep the OpenTelemetry API out of the SDK public API No OpenTelemetry type is exposed by the SDK public API, so declare it as an implementation dependency rather than leaking it onto the consumers' compile classpath. It remains a runtime dependency. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: sarahchen6 <sarah.chen@datadoghq.com> Co-authored-by: bruce.bujon <bruce.bujon@datadoghq.com>
Port the agent-side behaviors of the removed writers: send each event batch once with proxy discovery and recovery, and the Java SDK identity headers (#12477, #12575); flush the SDK pending events at agent shutdown, with or without tracing; declare the SDK settings the agent exposes (#12884, #12774). Move the Feature Flags shutdown tests from the agent tests to the OpenFeature smoke tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Port the agent-side behaviors of the removed writers: send each event batch once with proxy discovery and recovery, and the Java SDK identity headers (#12477, #12575); flush the SDK pending events at agent shutdown, with or without tracing; declare the SDK settings the agent exposes (#12884, #12774). Move the Feature Flags shutdown tests from the agent tests to the OpenFeature smoke tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Port the agent-side behaviors of the removed writers: send each event batch once with proxy discovery and recovery, and the Java SDK identity headers (#12477, #12575); flush the SDK pending events at agent shutdown, with or without tracing; declare the SDK settings the agent exposes (#12884, #12774). Move the Feature Flags shutdown tests from the agent tests to the OpenFeature smoke tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Port the agent-side behaviors of the removed writers: send each event batch once with proxy discovery and recovery, and the Java SDK identity headers (#12477, #12575); flush the SDK pending events at agent shutdown, with or without tracing; declare the SDK settings the agent exposes (#12884, #12774). Move the Feature Flags shutdown tests from the agent tests to the OpenFeature smoke tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Port the agent-side behaviors of the removed writers: send each event batch once with proxy discovery and recovery, and the Java SDK identity headers (#12477, #12575); flush the SDK pending events at agent shutdown, with or without tracing; declare the SDK settings the agent exposes (#12884, #12774). Move the Feature Flags shutdown tests from the agent tests to the OpenFeature smoke tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Port the agent-side behaviors of the removed writers: send each event batch once with proxy discovery and recovery, and the Java SDK identity headers (#12477, #12575); flush the SDK pending events at agent shutdown, with or without tracing; declare the SDK settings the agent exposes (#12884, #12774). Move the Feature Flags shutdown tests from the agent tests to the OpenFeature smoke tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Port the agent-side behaviors of the removed writers: send each event batch once with proxy discovery and recovery, and the Java SDK identity headers (#12477, #12575); flush the SDK pending events at agent shutdown, with or without tracing; declare the SDK settings the agent exposes (#12884, #12774). Move the Feature Flags shutdown tests from the agent tests to the OpenFeature smoke tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Port the agent-side behaviors of the removed writers: send each event batch once with proxy discovery and recovery, and the Java SDK identity headers (#12477, #12575); flush the SDK pending events at agent shutdown, with or without tracing; declare the SDK settings the agent exposes (#12884, #12774). Move the Feature Flags shutdown tests from the agent tests to the OpenFeature smoke tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Port the agent-side behaviors of the removed writers: send each event batch once with proxy discovery and recovery, and the Java SDK identity headers (#12477, #12575); flush the SDK pending events at agent shutdown, with or without tracing; declare the SDK settings the agent exposes (#12884, #12774). Move the Feature Flags shutdown tests from the agent tests to the OpenFeature smoke tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Port the agent-side behaviors of the removed writers: send each event batch once with proxy discovery and recovery, and the Java SDK identity headers (#12477, #12575); flush the SDK pending events at agent shutdown, with or without tracing; declare the SDK settings the agent exposes (#12884, #12774). Move the Feature Flags shutdown tests from the agent tests to the OpenFeature smoke tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
### What does this PR do? Customers need exposures and holdout metadata for their own pipelines without copying Datadog eligibility and deduplication logic. This adds customer exposure hooks and the `split.features` support from Java ([#12772](DataDog/dd-trace-java#12772), [#12775](DataDog/dd-trace-java#12775), [#12797](DataDog/dd-trace-java#12797), [#12774](DataDog/dd-trace-java#12774)). - Customer hooks read one provider-owned exposure-cache decision through `IsExposure()`, `IsCacheHit()`, and `ShouldSend()`. - `Features()` returns scalar split metadata for the `HOOK` destination, including holdout fields. Event writers filter `EXPOSURE` and `EVALUATION` fields for their respective events. Raw `Details.FlagMetadata` can contain internal fields for other destinations. - Evaluation aggregation separates different feature values, including its degraded tier. Routing and canonical scalar snapshots are compiled once per split; each writer shares immutable feature maps within a flush window. Public flag metadata remains scalar. Parsing retains the first valid entry for each key. - Go retains at most 256 unique features per split. Keys and string values must fit within 256 UTF-8 bytes. Invalid entries are skipped with debug diagnostics, not truncated. - `DD_FEATURE_FLAGS_EXPOSURES_DATADOG_LOGGING_ENABLED=false` disables Datadog exposure delivery, not customer hooks, deduplication, or evaluation telemetry. - Flag assignments and experimentation opt-out remain unchanged. Customers own queueing and retries. After registering the provider, add the hook to the OpenFeature client. `ddopenfeature` aliases `github.com/DataDog/dd-trace-go/v2/openfeature`. ```go client.AddHooks(ddopenfeature.NewExposureHook(func(e ddopenfeature.ExposureEvaluation) { if !e.ShouldSend() { return } // Application code: enqueue copied fields without blocking. allocation, _ := e.AllocationKey() serialID, hasSerialID := e.SplitSerialID() enqueueRow(e.Details.FlagKey, e.Context.TargetingKey(), e.Details.Variant, allocation, serialID, hasSerialID, e.Features()) })) ``` Create each hook once and reuse it, including with `of.WithHooks`. The callback must support concurrent calls. Its first panic warns with the flag key; subsequent panics log at debug level. Use `AllocationKey()` and `SplitSerialID()` to copy allocation fields, preserving absent versus zero serial IDs. **Upgrade behavior change / release note:** Applications using the Datadog provider inside OpenFeature MultiProvider will stop sending Datadog exposures after this upgrade. Register the Datadog provider directly to retain exposure delivery. This guard also disables customer exposure callbacks and exposure-cache updates under MultiProvider; flag evaluation continues. OpenFeature `v1.18.0` supplies zero-value `Finally` details for some early errors and retains success-looking fields after `After` returns an error. [go-sdk#623](open-feature/go-sdk#623) addresses that separately. The direct-provider bridge prevents reported errors from consuming the exposure cache; remove it only when `go.mod` requires the fixed SDK release. Application `After` panics propagate while deferred `Finally` hooks still run without an `Error` call, so those panics can consume the cache. Complete MultiProvider support remains deferred. ### Motivation Customers need exposures and holdout metadata for their own pipelines without copying Datadog's eligibility and deduplication logic. ### Test coverage Tests cover destination isolation, malformed entries, metadata limits, first-valid-key handling, shared cache decisions, shutdown, privacy modes, and feature-aware aggregation. Regression tests cover both overflow paths, concurrent warning deduplication, and cache-hit features. Additional regressions cover cancellation before versus after resolution, customer hooks at API/client/invocation levels with rejection at each application-hook level, independent panic warnings, disabled-delivery startup, and immutable feature snapshots across configuration changes and flush windows. Benchmarks cover the client path, feature routing, aggregation, and hook construction. Companion system tests: [#8094](DataDog/system-tests#8094). Cases cover client/invocation rejection, sequential MultiProvider calls, direct-provider retries, subject identity, serial transitions, and destination routing. Evaluation-event aggregation cases require test-agent intake support. Upstream MultiProvider concurrency is not validated. ### Validation These observations apply to the explicitly listed commits. The feedback changes at `885b16d2b348924508a687e59404c883b077c4e8` have not yet been rerun through system tests or staging. | Tested commit | Validation | Result | | --- | --- | --- | | `DataDog/dd-trace-go@7d8b15e` → `885b16d2b348924508a687e59404c883b077c4e8` | Same client benchmark, warm exposure cache and synchronous evaluation-queue drain; Go 1.26.0, Linux/amd64, three repetitions | Two features: 50 → 37 allocations/evaluation (5208 → 4048 B); with customer hook: 54 → 41 (5888 → 4728 B). No-feature path unchanged at 35 allocations / 3720 B. | | `DataDog/dd-trace-go@7d8b15e`; `DataDog/system-tests@c3ecef5595f094a476cf89e13441fbad6bbc7d51` | System tests: shared cache, identity, serial transitions, split destinations, rejection/retry, delivery controls, MultiProvider guard; Linux/arm64, OpenFeature `v1.18.0`, stock agent `v1.67.0` | 8 passed / 0 failed / 2 deselected: unsupported evaluation-event capture. | | `DataDog/dd-trace-go@7d8b15e` | Feature-flag dogfooding: live staging Remote Configuration, paired hooks, cache hits, holdout metadata, exposure delivery on/off; Linux/arm64, local EVP capture | 19 checks passed / 0 failed / 0 skipped. Exposures: 18 enabled, 0 disabled. Evaluation count: 37 in each app. Downstream storage not checked. | ### Reviewer's Checklist <!-- Authors can use this list as a reference. Reviewers sign off. --> - [ ] Changed code has unit tests for its functionality at or near 100% coverage. - [ ] [System-Tests](https://github.com/DataDog/system-tests/) covering this feature have been added and enabled with the va.b.c-dev version tag. - [ ] There is a benchmark for any new code, or changes to existing code. - [ ] If this interacts with the agent in a new way, a system test has been added. - [ ] New code is free of linting errors. You can check this by running `make lint` locally. - [ ] New code doesn't break existing tests. You can check this by running `make test` locally. - [ ] Add an appropriate team label so this PR gets put in the right place for the release notes. - [ ] All generated files are up to date. You can check this by running `make generate` locally. - [ ] Non-trivial go.mod changes, e.g. adding new modules, are reviewed by @DataDog/dd-trace-go-guild. Make sure all nested modules are up to date by running `make fix-modules` locally. Unsure? Have a question? Request a review! Co-authored-by: leo.romanovsky <leo.romanovsky@datadoghq.com>
Port the agent-side behaviors of the removed writers: send each event batch once with proxy discovery and recovery, and the Java SDK identity headers (#12477, #12575); flush the SDK pending events at agent shutdown, with or without tracing; declare the SDK settings the agent exposes (#12884, #12774). Move the Feature Flags shutdown tests from the agent tests to the OpenFeature smoke tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
What Does This Do
This PR combines #12775 and #12797, which were reviewed and approved separately and squashed into this branch.
Exposures after a successful evaluation
ExposureLoggingHook, at the OpenFeature finally stage. A failed evaluation sends nothing.ExposureDeduplicationCache. Resolution checks it and stamps__dd_exposure_cache_hit; the hook records the exposure and sends it.MultiProvider.ExposureHook (#12775)
ExposureHook, which hands each completed evaluation to a callback with the provider's exposure decision (isExposure(),isCacheHit(),shouldSend(),getFeatures()).HOOKfeatures into the flag metadata as__dd_feature.<key>.DD_FEATURE_FLAGS_EXPOSURES_DATADOG_LOGGING_ENABLED(defaulttrue). Whenfalse, the provider sends no exposures to Datadog, and the exposure cache keeps running.EXPOSURE and EVALUATION features (#12797)
EXPOSUREfeatures on exposure events and itsEVALUATIONfeatures on flag evaluation events. Flag evaluation buckets include the features.dd-java-agent, the provider checks for each capability and sends events without features.Motivation
Some customers use Datadog feature flags but keep exposures in their own data store. They need each exposure, with the holdout key and assignment group, in their own code, and they need to stop sending the same exposures to Datadog. Datadog also routes other features, such as a bandit policy id, to exposures and flag evaluations.
Additional Notes
dd-openfeature) and the agent (dd-java-agent) ship separately.Providerevaluation methods directly runs no hooks.EXPOSUREorEVALUATIONuntil ddoghq/dd-source#124673 is deployed in every datacenter.Jira ticket: EX-3800 EX-3815 EX-3801 EX-3828