Repository navigation
Port initial realtime session configuration from agents-js - #8474
rosetta-livekit-bot[bot] wants to merge 1 commit into
Conversation
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| self._initial_chat_ctx_task: asyncio.Task[None] | None = None | ||
| if is_given(chat_ctx) and chat_ctx.items: | ||
| self._initial_chat_ctx_task = asyncio.create_task(self.update_chat_ctx(chat_ctx)) |
There was a problem hiding this comment.
🟡 First response omits initial history
Calling generate_reply() immediately after session(chat_ctx=...) queues the reply before _initial_chat_ctx_task runs. The first response therefore omits the supplied history across OpenAI-compatible sessions.
Learn more
The constructor queues instructions and tools synchronously, but schedules the initial chat context as a separate task. generate_reply() synchronously queues response.create, so a caller can enqueue it before the event loop runs the context task. The send loop preserves that order and the provider receives the reply request first. This affects the core OpenAI implementation and subclasses such as xAI, ThunderPhone, and LiveKit Inference.
Example: A caller creates model.session(chat_ctx=history) and immediately calls session.generate_reply(). The outgoing queue contains session.update, then response.create; conversation items from history arrive later. The model answers without that history.
Recommended fix: Ensure initial conversation-item events are queued before the constructor returns, or gate generate_reply() and other turn-starting operations until initial context synchronization completes. Preserve acknowledgement and reconnect bookkeeping when moving the initial synchronization.
Was this helpful? React with 👍 or 👎 to provide feedback.
| instructions: NotGivenOr[str] = NOT_GIVEN, | ||
| chat_ctx: NotGivenOr[llm.ChatContext] = NOT_GIVEN, | ||
| tools: NotGivenOr[list[llm.Tool]] = NOT_GIVEN, | ||
| ) -> RealtimeSession: | ||
| """Return a new RealtimeSession bound to this model instance.""" | ||
| # disabling server-side turn detection is unsupported (can_disable_turn_detection=False) | ||
| sess = RealtimeSession(self) |
There was a problem hiding this comment.
🟡 Initial configuration discarded by providers
Passing initial configuration to session() on AWS leaves the new session empty. Duplex, Azure, NVIDIA, Rime, and Ultravox do the same, breaking direct API callers.
Learn more
The expanded public RealtimeModel.session() contract accepts initial instructions, chat context, and tools. This implementation accepts all three values but constructs RealtimeSession(self), so none reaches the session. The same incomplete transformation appears in DuplexRealtimeAdapter.session, Azure RealtimeModel.session, NVIDIA RealtimeModel.session, Rime RealtimeModel.session, and Ultravox RealtimeModel.session. AgentActivity masks this by subsequently calling the private _update_session(), but direct users of the new public arguments receive an unconfigured session.
Example: aws_model.session(instructions="Answer briefly", tools=[lookup]) creates a Nova session with its default prompt and no lookup tool. No later public operation applies those arguments automatically.
Recommended fix: Pass the values into each concrete session and seed its state before connection, or explicitly apply them through a provider-supported initialization path before returning. For providers that cannot support an initial field, reject it rather than silently discarding it.
Was this helpful? React with 👍 or 👎 to provide feedback.
Ports livekit/agents-js#2325.
Realtime sessions now accept their initial instructions, chat context, and tools at creation time. AgentActivity supplies that configuration before providers connect, avoiding Gemini's redundant startup connection while keeping OpenAI-compatible, LiveKit Inference, xAI, and Phonic sessions configured from their first request.
Google and Phonic session state is isolated per session, and Google model option changes are forwarded only to open sessions.
Source diff coverage
.changeset/google-realtime-single-connect.md: not applicable. The Python repository does not use Changesets; versions are changed only by release PRs.agents/etc/agents.api.md: not applicable. Python does not commit generated API Extractor snapshots.agents/src/inference/realtime.ts: adapted tolivekit-agents/livekit/agents/inference/realtime/openai.py; Python keyword arguments are forwarded to the inference session.agents/src/llm/index.ts: adapted through the publicRealtimeModel.session()signature inlivekit-agents/livekit/agents/llm/realtime.py; Python needs no separate type export.agents/src/llm/openai_realtime/realtime_model.test.ts: adapted totests/test_realtime/test_openai_realtime_model.pywith the same three initial-option scenarios.agents/src/llm/openai_realtime/realtime_model.ts: adapted tolivekit-agents/livekit/agents/llm/_realtime/openai.py; initial instructions/tools are queued before connection and initial chat context uses the standard update path.agents/src/llm/realtime.ts: adapted tolivekit-agents/livekit/agents/llm/realtime.pyusing idiomatic keyword-only arguments andNOT_GIVENsentinels.agents/src/voice/agent_activity.ts: adapted tolivekit-agents/livekit/agents/voice/agent_activity.py; instruction messages are removed before session creation and the initial configuration is passed atomically.agents/src/voice/realtime_tool_output_commit.test.ts: adapted totests/fake_realtime.pyandtests/test_realtime_auto_disable_turn_detection.py; the fake records creation arguments and the voice test asserts the same configuration.plugins/google/etc/agents-plugin-google.api.md: not applicable. Python has no generated API snapshot.plugins/google/src/realtime/realtime_api.ts: adapted tolivekit-plugins/livekit-plugins-google/livekit/plugins/google/realtime/realtime_api.py; sessions copy model options, seed configuration before connecting, receive model updates, and unregister when closed or their connection loop ends.plugins/google/src/realtime/realtime_session_options.test.ts: adapted totests/test_plugin_google_realtime.pywith all five source scenarios.plugins/openai/etc/agents-plugin-openai.api.md: not applicable. Python has no generated API snapshot; the plugin inherits the core OpenAI implementation.plugins/phonic/etc/agents-plugin-phonic.api.md: not applicable. Python has no generated API snapshot.plugins/phonic/src/realtime/realtime_model.test.ts: adapted tolivekit-plugins/livekit-plugins-phonic/tests/test_realtime_session_options.pywith all five source scenarios.plugins/phonic/src/realtime/realtime_model.ts: adapted tolivekit-plugins/livekit-plugins-phonic/livekit/plugins/phonic/realtime/realtime_model.py; initial configuration is seeded before connect and instructions remain session-local.plugins/xai/etc/agents-plugin-xai.api.md: not applicable. Python has no generated API snapshot.plugins/xai/src/realtime/realtime_model.ts: adapted tolivekit-plugins/livekit-plugins-xai/livekit/plugins/xai/realtime/realtime_model.py; initial configuration is forwarded to the OpenAI-compatible base session.The expanded signature is also carried through target-only realtime providers and adapters so AgentActivity can use the API uniformly. Providers that do not require handshake-time configuration retain their existing
_update_session()behavior.Validation:
make checkRealtimeSessionvia__new__without required existing fieldslivekit-agents,livekit-plugins-google,livekit-plugins-phonic,livekit-plugins-xai,livekit-plugins-aws,livekit-plugins-azure,livekit-plugins-nvidia,livekit-plugins-rime,livekit-plugins-thunderphone, andlivekit-plugins-ultravoxcue-cliruntime validation could not be run because the binary is unavailable in the environment and nocue-clipackage is published foruvx; the hermetic voice-pipeline propagation test passes.Ported from livekit/agents-js#2325
Original PR description
Google Realtime requires instructions/tools to be passed in during initial connection. if they are not passed in, setting them for the first time would cause WS to be re-established, wasting time in the beginning.
updating realtime session establishment to include these parameters when available.
fixes: #2324