Repository navigation
Conversation
Replace the tracker ladder with the current one: convenience functions, config() with handler packages, custom handlers, and manual track() for feedback. Rewrite each provider reference around handlers, document what the SDK records and what it doesn't (time to first token, feedback), add a legacy-to-current mapping, and fix the metrics API timestamps, which are milliseconds.
…lers
Custom handlers now fill {{name}} placeholders with parse_template and drop the tools key from model.parameters before calling the provider. Note that the Python OpenAI handler doesn't forward parameters, that inspect_config is async, and that the to_lang_graph runner is called with invoke().
…icModel anthropic 2.x removed both arguments from messages.create, so forwarding them from the variation raises TypeError.
Skill eval results
Only suites whose source actually changed since their last recorded score were re-run. Soft-failing while we stabilise the baseline. |
jeffdupont
left a comment
jeffdupont
left a comment
There was a problem hiding this comment.
Every Python pip install line leaves out launchdarkly-server-sdk, so the first invoke() raises. Feedback attribution has no path for stream() users.
skills/agentcontrol/built-in-metrics/references/openai-tracking.md:9 — Python also needs launchdarkly-server-sdk. launchdarkly-ai-server loads it as an optional peer, and without it the first invoke() raises RuntimeError: LaunchDarkly server SDK not installed (python-ai-sdk lifecycle.py:271-280). The same line needs adding in anthropic-tracking.md:7, bedrock-tracking.md:10, gemini-tracking.md:10 and langchain-tracking.md:51. #220 already has the right form: pip install launchdarkly-server-sdk launchdarkly-ai-server ....
skills/agentcontrol/built-in-metrics/SKILL.md:78 — RuntimeError also comes from a run that already started: the handler packages raise Tool loop exceeded the maximum number of steps (10) and claude-agents raises Claude agent run ended with .... A fallback here would re-run the request. Suggest: "If you fall back on RuntimeError, first check (await inspect_config(key, context))["enabled"] and re-raise when it's True. The config was served, so the error came from the run."
skills/agentcontrol/built-in-metrics/SKILL.md:81 — On launchdarkly-ai-server 0.4.0, a list works too: handler_key() normalizes list and tuple tags (python-ai-sdk #152). Only 0.3.0 needed a tuple. Every handler package on PyPI requires launchdarkly-ai-server>=0.3.1, so anyone who installs a handler today gets 0.4.0. Suggest dropping this sentence, keeping the tuple in the example, and bumping the compatibility line to 0.4.0+.
skills/agentcontrol/built-in-metrics/references/openai-tracking.md:79 — Correct for openai-messages 0.2.5. But python-ai-sdk #107 (merged 10-23, unreleased) makes it forward the spec allowlist, and js-ai-sdk #118 changes the Node list to match. Suggest pinning the sentence to versions ("≤ 0.2.5") and saying the next release forwards the spec list in both SDKs, so the note doesn't silently become wrong.
skills/agentcontrol/built-in-metrics/references/anthropic-tracking.md (Tier 2) — Nice to have. Heads-up worth one line: python-ai-sdk #201 / js-ai-sdk #135 (merged, unreleased) make create_claude_agents_handler() run with no Claude Code built-ins unless the variation attaches a NativeTool, and load no host settings (setting_sources=[]). Apps that reach Bedrock/Vertex/a gateway through ~/.claude/settings.json will need setting_sources=["user"] or the env vars.
| getClient().track(event, context, result.trackData, 1); | ||
| ``` | ||
|
|
||
| `track_data` identifies the config, variation, version, and model, so the feedback lands on the variation that produced the response. In Python, convert the context dict with `Context.from_dict()` before calling `track()`. |
There was a problem hiding this comment.
Should fix. Tier 4 relies on result.track_data, but stream() never exposes it. Python's done event has response, usage and judge_results (client.py:272-277). JS's has response, usage and judgeResults (published @launchdarkly/ai-server@0.4.0, where the internal event has trackData but config().stream() drops it). streaming-tracking.md:5 notes this for TTFT. Feedback has the same gap, and this tier is where a streaming chat UI will look. Suggest one line: "For stream(), the current SDKs don't return track_data, so feedback can't be attributed to the variation yet." I'll raise exposing it on the done event as a cross-SDK change.
There was a problem hiding this comment.
Added your sentence to Tier 4 in 5005e1e. Thanks for raising the cross-SDK change.
stream() doesn't return track_data, so streamed feedback can't be attributed yet. Point agents at get-agentcontrol-config-health for a quick check and keep the REST metrics API for totals.
anthropic 2.x rejects top_k along with temperature and top_p.
jeffdupont
left a comment
jeffdupont
left a comment
There was a problem hiding this comment.
Follow-up to my 10-26 review. The stream() feedback line landed in 5005e1e, thanks. The other five points are still open. Since then the SDKs shipped 1.0.0 (both, 10-30), and the current LaunchDarkly MCP server dropped the health tool the new Verify step names. Together these break some of the skill's code as written.
Earlier points
launchdarkly-server-sdkmissing from the Pythonpip installlines: not fixed. Inline onopenai-tracking.md:9.RuntimeErrorfallback can re-run a request that already started: not fixed. Inline onSKILL.md:78.- Tuple sentence and the compatibility line: not fixed. 1.0 now makes the version bump required, not optional. Inline on
SKILL.md:5. - OpenAI parameter note: not fixed, and now wrong. openai-messages 0.3.0 (10-27) forwards parameters. Inline on
openai-tracking.md:79. - claude-agents defaults (nice to have): not done. This shipped in claude-agents 0.3.0, so it now affects anyone who installs today. Inline on
anthropic-tracking.md:35. stream()andtrack_data: fixed atSKILL.md:134. That is still accurate on 1.0: the Pythondoneevent has notrack_data(client.py:284-289).
New since the last review
7. The 1.0 factory renames break four samples. Inline on openai-tracking.md:48, openai-tracking.md:67 and langchain-tracking.md:79.
8. get-agentcontrol-config-health has been removed from the current MCP server. Inline on SKILL.md:90.
The top_k change in strands-tracking.md looks fine. CI is green on cc40c64.
| One provider, one mode, no tools. Duration, tokens, and success/error are recorded with no extra code. | ||
|
|
||
| **Python** — `ManagedModel` via `ai_client.create_model()`: | ||
| **Python** (`pip install launchdarkly-ai-server launchdarkly-ai-openai-messages`): |
There was a problem hiding this comment.
Must fix (still open from the last review). Without launchdarkly-server-sdk, the first invoke() raises RuntimeError: LaunchDarkly server SDK not installed (python-ai-sdk lifecycle.py:349-360 on 1.0). The 1.0 README's form is pip install "launchdarkly-ai-server[ld]" (packages/client/README.md:14), so: pip install "launchdarkly-ai-server[ld]" launchdarkly-ai-openai-messages. The same change is needed in anthropic-tracking.md:7, bedrock-tracking.md:10, gemini-tracking.md:10 and langchain-tracking.md:51. Node is fine, because @launchdarkly/ai-node bundles @launchdarkly/node-server-sdk.
There was a problem hiding this comment.
Fixed in 4d3cdd4: updated the Python install commands in openai-tracking.md, anthropic-tracking.md, bedrock-tracking.md, gemini-tracking.md, and langchain-tracking.md to pip install "launchdarkly-ai-server[ld]" ....
| 2. **Wrap the existing call, don't rewrite it.** Tier 2 and Tier 3 are designed to slot around an unmodified provider call. If you find yourself rewriting the call to fit the tracker, you're at the wrong tier — drop down one. | ||
| 3. **Errors are handled inside `trackMetricsOf`.** The wrapper catches exceptions, records `trackError()` internally, and re-raises — do **not** add `except: tracker.trackError()` on top, it's a noop that also trips the at-most-once guard. Tier 1 handles both paths automatically. At Tier 4 (manual, streaming, `track_duration_of`) the caller does own the error-tracking call. | ||
| 4. **Always flush before close.** Call `ldClient.flush()` (Python: `ldclient.get().flush()`; Node: `await ldClient.flush()`) before closing the client. Trailing events are at risk of being lost otherwise — in short-lived scripts and long-running services alike. In Node, `ldClient.close()` returns a Promise; await it. | ||
| 1. **Handle disabled configs.** `invoke()` raises a `RuntimeError` when the served variation is disabled or the config can't be reached. Catch it and fall back to whatever the app does when the feature is off, or call `await inspect_config(key, context)` / `await inspectConfig(key, context)` first, which returns the `enabled` state without raising. |
There was a problem hiding this comment.
Should fix (still open). On 1.0, handlers still raise RuntimeError after a run has started. claude-messages and langchain-messages raise Tool loop exceeded the maximum number of steps, and claude-agents raises Claude agent run ended with .... A blanket fallback here sends the request a second time. Suggest: "If you fall back on RuntimeError, first check (await inspect_config(key, context))["enabled"] and re-raise when it's True. The config was served, so the error came from the run."
There was a problem hiding this comment.
Fixed in 4d3cdd4: added the inspect_config / inspectConfig check before falling back on RuntimeError.
| description: "Instrument an existing codebase so LaunchDarkly records config metrics (duration, tokens, success/error, tool calls). Walks the ladder (convenience function → config() with a handler package → custom handler → manual track()) and picks the lowest-ceremony option that still captures what the Monitoring tab needs." | ||
| license: Apache-2.0 | ||
| compatibility: Requires the LaunchDarkly server-side AI SDK (`launchdarkly-server-sdk-ai>=0.20.0` for Python or `@launchdarkly/server-sdk-ai>=0.20.0` for Node) and an existing config. | ||
| compatibility: Requires `launchdarkly-ai-server` 0.3.0+ (Python) or `@launchdarkly/ai-node` 0.3.0+ (Node.js), a handler package for your provider, and an existing config. |
There was a problem hiding this comment.
Must fix. Both SDKs are now on 1.0.0 (10-30), and 1.0 renamed the factories this skill imports (see the comments on openai-tracking.md and langchain-tracking.md). Once those samples use the 1.0 names, they won't run on 0.3.x. Suggest launchdarkly-ai-server 1.0.0+ / @launchdarkly/ai-node 1.0.0+ here and in the PR description. While you're here, drop the last sentence of line 81 ("must be a tuple"): handler_key() takes a list as well as a tuple on every release since 0.4.0 (utils.py:219-233). Keep the tuple in the example.
There was a problem hiding this comment.
Fixed in 4d3cdd4: bumped compatibility to 1.0.0+ and dropped the tuple-only sentence.
| **Errors.** If the OpenAI call fails, the SDK records the duration and a generation error, then re-raises. Let it propagate to your own error handling; don't add tracking calls. | ||
|
|
||
| ## Tier 3 — Custom extractor (fallback) | ||
| **Parameters in Python.** `launchdarkly-ai-openai-messages` 0.2.x sends no variation parameters to OpenAI, so `temperature`, `max_output_tokens`, and the rest are ignored. The Node.js handler forwards them, mapping `max_tokens` to `max_output_tokens`. If a Python variation depends on its parameters, use a custom handler (Tier 3) and check the CHANGELOG for changes. |
There was a problem hiding this comment.
Must fix (was a version-pin suggestion, now wrong). launchdarkly-ai-openai-messages 0.3.0 (10-27) forwards the variation's parameters, and so does 1.0.0. It also renames max_tokens/max_completion_tokens to max_output_tokens (handler.py:61-82, 155-167), the same as Node. As written, this sends readers to write a Tier 3 custom handler they don't need. Suggest replacing the paragraph with: "Both SDKs forward the variation's parameters to OpenAI and map max_tokens to max_output_tokens. Keys the Responses API doesn't accept are dropped."
There was a problem hiding this comment.
Fixed in 4d3cdd4: updated the parameter note to match 1.0 forwarding and max_tokens → max_output_tokens.
|
|
||
| result = await config( | ||
| key="customer-support-chat", | ||
| handler=[create_claude_messages_handler(), create_claude_agents_handler()], |
There was a problem hiding this comment.
Should fix (was nice-to-have; now released). In claude-agents 0.3.0+, create_claude_agents_handler() runs without Claude Code's built-in tools unless the variation attaches a NativeTool. It also loads no host settings (launchdarkly/python-ai-sdk#201, launchdarkly/js-ai-sdk#135). An app that reaches Bedrock, Vertex or a gateway through ~/.claude/settings.json now fails until it passes setting_sources=["user"] or sets the env vars. One line under this example would cover it.
There was a problem hiding this comment.
Fixed in 4d3cdd4: added a note on create_claude_agents_handler() / createClaudeAgentsHandler() requiring NativeTool for built-ins and setting_sources=["user"] for host settings.
|
|
||
| support = config( | ||
| key="customer-support-chat", | ||
| handler=[create_openai_messages_handler(), create_openai_agent_handler()], |
There was a problem hiding this comment.
Must fix (new). 1.0 renamed this to create_openai_agents_handler (launchdarkly/python-ai-sdk#225), and the old name is gone. On launchdarkly-ai-openai-agents 1.0.0, line 44 raises ImportError. Fix lines 44 and 48.
There was a problem hiding this comment.
Fixed in 4d3cdd4: renamed to create_openai_agents_handler.
| **Error handling.** `trackMetricsOf` catches exceptions internally, records `trackError()` on the tracker, and re-throws — so you do **not** need a try/catch block that calls `trackError()` yourself. Call the wrapper directly; if the caller wants to log or handle the exception, do that in addition to (not instead of) letting it propagate: | ||
| const support = config({ | ||
| key: 'customer-support-chat', | ||
| handler: [createOpenAIHandler(), createOpenAIAgentHandler()], |
There was a problem hiding this comment.
Must fix (new). Neither of these names exists on 1.0. They are now createOpenAIMessagesHandler (launchdarkly/js-ai-sdk#173) and createOpenAIAgentsHandler (launchdarkly/js-ai-sdk#174). Fix lines 62, 63 and 67. streaming-tracking.md:27 and :29 use createOpenAIHandler too.
There was a problem hiding this comment.
Fixed in 4d3cdd4: updated to createOpenAIMessagesHandler and createOpenAIAgentsHandler here and in streaming-tracking.md.
| ### Why aggregate per message | ||
|
|
||
| `get_ai_metrics_from_response` / `getAIMetricsFromResponse` is defined on a single LangChain `AIMessage`. A LangGraph run produces N messages (model turn, tool result, model turn, tool result, final). If you pass the whole `result` to the extractor, you miss most of the token usage. Iterating and summing is deliberate — it's the same pattern the LaunchDarkly LangGraph guide uses. | ||
| The Node.js completion-mode factory is `createLangChainHandler`, not `createLangChainMessagesHandler`. |
There was a problem hiding this comment.
Must fix (new). This is now backwards. launchdarkly/js-ai-sdk#173 (released in @launchdarkly/ai-langchain-messages 1.0.0) renamed createLangChainHandler to createLangChainMessagesHandler, and the old name is no longer exported. Fix lines 69 and 74, and drop or flip this sentence.
There was a problem hiding this comment.
Fixed in 4d3cdd4: updated to createLangChainMessagesHandler and removed the outdated note.
| | `track_tool_call(name)` / `trackToolCall(name)` | any | Record a single tool invocation by name. Available on both SDKs. | | ||
| | `track_tool_calls([names])` / `trackToolCalls([names])` | any | Batch variant — record a list of tool invocations in one call. | | ||
| | `track_judge_result(result)` / `trackJudgeResult(result)` | any | Record a programmatic judge evaluation. `result.sampled` indicates whether evaluation ran. | | ||
| - [ ] Or check from the agent: the `get-agentcontrol-config-health` MCP tool reports recent generation successes and errors per environment. It doesn't return token or duration totals; for those, use the [metrics API](references/metrics-api.md). |
There was a problem hiding this comment.
Must fix (new in 5005e1e). The current LaunchDarkly MCP server has removed get-agentcontrol-config-health, and a call to it returns "this tool has been removed from the server". Its replacement, get-agent-configs with section: "health", reports config completeness only, with no generation successes or errors, so this step can't confirm metrics. Suggest dropping the bullet and pointing to the metrics API, and removing the tool name from metrics-api.md:3 too.
There was a problem hiding this comment.
Fixed in 4d3cdd4: removed get-agentcontrol-config-health from SKILL.md and metrics-api.md and pointed to the metrics API.
…ry renames, inspect_config fallback, and MCP health removal) Co-authored-by: Cursor <cursoragent@cursor.com>
Summary
config()+ handler package → custom handler → manualtrack()for feedback.{{name}}withparse_templateand droptoolsfrommodel.parameters; Strands dropstemperature/top_pfor anthropic 2.x.metrics-api.mdto use milliseconds.Jira: DEVREL-1645
Testing
Ran each reference's code in the cookbook notebooks (OpenAI, Anthropic, Bedrock, Gemini, LangChain, LangGraph, Strands, streaming, feedback, metrics API). Verified feedback events through the metrics API.
Notes
launchdarkly-ai-server0.3.0 (Python) and@launchdarkly/ai-node0.3.0 (Node.js).skills.json, so later merges may needpython3 scripts/generate_catalog.pyre-run.Note
Overview
Bumps the
built-in-metricsskill to2.0.0-experimentaland rewrites it forlaunchdarkly-ai-server/@launchdarkly/ai-nodewith handler packages, instead of the legacyldai/@launchdarkly/server-sdk-aitracker APIs.The instrumentation model changes: metrics come from routing calls through
config().invoke()orstream()(or convenience helpers)—not fromcreate_tracker(),trackMetricsOf, or manual duration/token events. The ladder is now convenience function →config()+ handler package → custom handler → manualtrack()only for feedback and business outcomes.SKILL.mdadds what the SDK auto-records (including tool calls), guardrails (no double-counting,shutdown(), disabled-config handling), a legacy-to-current replacement table, and Tier 4 user-feedback examples usingresult.track_data.skills.jsonupdates description, version, and compatibility strings.Per-provider references (OpenAI, Anthropic, Bedrock, Gemini, LangChain/LangGraph, Strands, streaming) are replaced with handler-package patterns, custom-handler shapes (
parse_template, droptoolsfrom parameters), andstream()usage. Streaming docs drop manual TTFT (not emitted by current SDKs).metrics-api.mdfixes query windows to millisecond Unix timestamps and notes TTFT stays zero on the new SDKs.Reviewed by Cursor Bugbot for commit 4d3cdd4. Bugbot is set up for automated code reviews on this repo. Configure here.