[tendant]

Summary

Adds zoom_ai.realtime.RealtimeModel to livekit-plugins-zoom-ai. It's a speech-to-speech model on the Zoom Voice Agent API: one WebSocket (voice-agent.zoom.v2) carries the caller's audio in and the agent's speech, transcripts and function calls out.

session = AgentSession(llm=zoom_ai.realtime.RealtimeModel(voice="jenna"))

It works as the model of a whole session or of one agent in a multi-agent handoff.

How it maps onto the framework

• Configuration: Zoom requires the first client event to be a complete session.update (listen / think / speak). The plugin connects only once the agent has supplied its instructions and tools, then sends one update instead of the framework's incremental three.
• Audio and turns: audio goes in and out as 16 kHz PCM16. Turn detection and barge-in are Zoom's and always on (turn_detection=True, can_disable_turn_detection=False).
• Tools: Zoom asks for a call with response.function_call_arguments.done, the result goes back as a function_call_output item, and response.create asks for the follow-up reply (auto_tool_reply_generation=False).
• One response at a time: Zoom allows a single active response, so generate_reply() requests are queued until response.done. The "active response" error is handled, including the race where a server-started response answers the request.
• Interruptions: interrupt() sends response.cancel with the response ID, and truncate() sends conversation.item.truncate.
• History: Zoom can't take message items mid-conversation (mutable_chat_context=False). History that exists before the session starts, from a handoff or a reconnect after the 60-minute session limit, is added to the instructions.
• Errors: a configuration rejected on the first session.update (for example an unknown voice) ends the session with an unrecoverable error instead of leaving the agent silent. Handshake errors don't include the request headers, so the API key never appears in them.
• Bring your own model: your own OpenAI or Anthropic key goes in provider_headers.

Instructions and tools can change mid-session through session.update. Per-reply instructions aren't supported by the API; they're ignored with a one-time warning.

Changes

• livekit-plugins/livekit-plugins-zoom-ai/livekit/plugins/zoom_ai/realtime/: new realtime model
• tests/test_plugin_zoom_ai_realtime.py: 22 hermetic unit tests against a fake Voice Agent server, including one AgentSession tool turn end to end
• examples/voice_agents/zoom_voice_agent.py: a Zoom-only agent with a function tool
• examples/voice_agents/zoom_multi_agent.py: three agents handing off (pipeline, OpenAI Realtime, Zoom)
• Plugin README: usage, options, limitations, debugging. Package description and keywords updated. No version changes.

Testing

• uv run pytest tests/test_plugin_zoom_ai_realtime.py tests/test_plugin_zoom_ai_stt.py --unit: 110 passed
• uv run pytest --unit --audio_eot: no new failures. The failures seen locally also happen on main or come from the local environment: the Azure Voice Live SDK isn't installed, a Google credentials test fails on main too, and one worker-status test is flaky under parallel load.
• make check style: ruff format/lint clean, and strict mypy clean for livekit.plugins.zoom_ai
• Manually tested against the live Zoom Voice Agent API in console mode: greeting, conversation, function call, barge-in.

The Voice Agent docs don't list the fields of conversation.item.input_audio_transcription.completed or of response.done usage. The plugin reads item_id/transcript and usage.input_tokens/output_tokens, with fallbacks.

🤖 Generated with Claude Code

[CLAassistant]

All committers have signed the CLA.