[misbahsy]

Summary

Adds openai.LLM.with_litellm(...), an OpenAI-compatible helper for the LiteLLM proxy, in the same style as with_sambanova, with_openrouter and with_ovhcloud.

The LiteLLM proxy exposes one OpenAI-compatible endpoint in front of many providers (Anthropic, Bedrock, Vertex, Azure, Groq, self-hosted models and others), with virtual keys, budgets, fallbacks and spend tracking. Teams that already run the proxy can now point an agent at it in one line:

from livekit.plugins import openai

llm = openai.LLM.with_litellm(
    model="anthropic/claude-sonnet-4-5",   # any model name from the proxy config
    base_url="https://litellm.internal.example.com",
)

Behavior

• api_key falls back to LITELLM_PROXY_API_KEY. A ValueError is raised if neither is set.
• base_url falls back to LITELLM_PROXY_API_BASE, then http://localhost:4000 (the proxy's default port). The URL is validated the same way as with_letta.
• _strict_tool_schema=False, because the proxy often fronts non-OpenAI models that reject strict schemas.
• Passes through extra_body, extra_headers and timeout, so callers can send LiteLLM request metadata such as tags.
• No new dependency: it uses the existing OpenAI client.

Relation to earlier PRs

#5734 and #6527 proposed a separate livekit-plugins-litellm package that embeds the LiteLLM SDK. This PR takes the smaller route: proxy mode only, inside the existing OpenAI plugin, with no new package or dependency. If you would rather have the embedded plugin, I am happy to close this.

Tests

• New tests/test_plugin_openai_litellm.py (unit marker): defaults, env fallback, argument precedence, missing key, invalid URLs, and a mocked request that checks the URL, auth header, model and extra_body.
• make check passes locally (ruff format, ruff lint, mypy).

Disclosure: I work on LiteLLM.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[devin-ai-integration]

Devin Review found 1 potential issue.

[devin-ai-integration]

🟡 LiteLLM request timeout remains ten seconds

When a proxy responds after ten seconds, timeout=... cannot extend the request deadline. LLMStream._run overrides the configured client timeout with conn_options.timeout, which defaults to ten seconds.

Learn more

The helper passes timeout into the LLM constructor, which uses it when creating the HTTP client. During each chat request, LLMStream._run passes a separate request timeout from APIConnectOptions, overriding the client's setting. The default request timeout is ten seconds, even when the helper configured a longer HTTP timeout.

Example: An agent sets timeout=httpx.Timeout(30) for a LiteLLM proxy that takes 15 seconds to start streaming. The request expires after ten seconds instead of receiving the response.

Recommended fix: Route the configured HTTP timeout into the per-request timeout policy, or document and remove the ineffective timeout argument in favor of configuring conn_options.timeout at chat time. Verify with a delayed mocked proxy response.

---

Was this helpful? React with 👍 or 👎 to provide feedback.