[longcw]

GPT-Live decides on its own which caller turns to delegate to its backend model. When the backend asks the caller a question and the answer is short (okay, mm-hmm, yep) or said over the question, GPT-Live often transcribes it and never delegates it, so the call goes quiet or the model promises an action that never starts. In 20 calls built on that pattern, GPT-Live dropped 15 answers; a complete answer given in turn is dropped about 1 time in 200.

A prompt does not fix it: a strict and a relaxed delegation policy drop the same share, and GPT-Live transcribes every answer it drops.

GPTLiveModel(delegation_recovery=True) adds a check in AgentActivity. Once both sides stop speaking for a grace period, a DelegationDetector judges each caller turn the model left undelegated, and RealtimeSession.delegate() starts the delegation when the turn needs the backend. With it, the same 20 calls dropped 1 answer.

Why the check runs in AgentActivity and reads the agent's chat context:

• It judges the conversation as the caller heard it. The agent's chat context holds the played text and the playback start and stop of every message. The detector marks a turn that started while an agent message was playing, because the transcript alone shows a talk-over yep as an ordinary reply.
• The realtime model only reports and starts delegations. RealtimeSession gains delegate() and a delegation_created event. GPT-Live implements both and the duplex adapter passes them through, so another realtime model can join by implementing the same two.
• A talk-over answer goes to the backend with a note. The note says the caller answered while the agent was still speaking. The backend then confirms before an action, which keeps a backchannel from becoming consent.

GPTLiveModel(delegation_recovery=True)  # DelegationDetector() on google/gemma-4-31b-it, LiveKit Inference
GPTLiveModel(delegation_recovery={"detector": llm.DelegationDetector(inference.DecisionModel("openai/gpt-6-luna-decisions"))})

The option also takes grace_period (0.5 s by default). A detector is any async callable from DelegationCheck to bool | str; a string delegates the turn and goes to the backend as a note. DelegationDetector accepts an LLM, a DecisionModel or a LiveKit Inference model name.

TODO: realtime models other than GPT-Live. A first request that GPT-Live answers by asking for details itself is left alone by design, since the model then delegates the caller's next answer.

Closes #7551.

Context for reviewing and coding agents

Where the numbers come from

Live gpt-live-1 calls on LiveKit Inference with Responses delegation and fixed backend tools, driven through a real AgentSession with in-process audio. A scripted caller answers the backend's confirmation question with Okay., Mm-hmm. or Yep., after the question or over the readback; each of 20 calls ran once without and once with the option. The 1-in-200 figure comes from 206 answers given in turn by LLM callers in 200 calls. Across those calls, every model listed below caught all dropped answers; on all 561 recorded undelegated turns, Gemma scored 95.5% accuracy (7.3% missed, 3.9% false alarms, 1.1 s) and the decision model 95.9% (12.5% missed, 2.4% false alarms, 0.4 s).

What changes for existing users

Nothing runs unless delegation_recovery is set. GPTLiveDelegation now subclasses DelegationCreatedEvent, gains source and mode, and is emitted for responses-mode delegations too, so a delegation_created listener now also sees backend delegations, with mode="responses". InputTranscriptionCompleted gains delegated, set by GPT-Live on a final transcript.

Why not in the duplex adapter

An earlier version ran the loop there. It only served duplex models, and the adapter's chat context has no playback times; syncing the activity's context to the realtime session after every reply would change provider traffic for every realtime model.

[IngLP]

Two observations from a live Azure GPT-Live session (gpt-live-1, backend gpt-6.1-terra), relevant to GPTLiveSession.delegate():

1. Azure does not echo the response.create event id on session.delegation.created.
After response.item.create (developer message) + response.create with event_id="fin_create", the service answered:

{"type": "session.delegation.created", "offset_ms": 3600,
 "delegation": {"id": "item_…", "type": "delegation", "target": "responses", "response_id": "resp_…"},
 "event_id": "event_…"}

There is no client_event_id, so _handle_delegation_created never matches _delegate_event_id: the delegation is reported with source="model", and _delegate_event_id stays set (so _backend_busy stays true until a reconnect). The delegation was created 0.36 s after the response.create, before its response.created. Is the echo specific to the OpenAI endpoint?

2. A use case for a per-delegation tool_choice.
We finalize every call with one forced tool call on the backend at the end of the conversation (finalize_call). Asking the voice model with generate_reply does not work: the ask reaches it on the commentary channel, and in our tests it never delegated from there. delegate() does: the backend called the tool 1.2 s after response.create.

The tool has to be forced. Since response.create takes no body, the only way is a session.update with delegation.responses.tool_choice before the request. That tool_choice must then be put back before the continuation: without the restore, the continuation after the tool output called the same tool again (a loop). With the restore sent when the delegation's first response.completed arrives, which is before the continuation's response.create, the continuation ran under auto.

A minimal extension of the API in this PR:

def delegate(self, instructions: str | None = None, *, tool_choice: NotGivenOr[llm.ToolChoice] = NOT_GIVEN) -> None

It sets tool_choice on the session before the request, and restores the session's own value when the delegation's first response ends (completed, failed or incomplete), or on an error for the response.create. The limit: tool_choice is session-wide, so any backend response that starts in between runs under it as well.

We run this as a fork patch and can open a PR on top of this one once it lands.

[devin-ai-integration]

Note

Newer findings are available below. Devin Review posted a newer report on this PR, in addition to the findings presented here.

Devin Review found 3 new potential issues.

1 flag not posted on this PR by your GitHub settings — view it in Devin Review. (Configure)

[longcw]

@IngLP

Thanks, both points are useful.

1. Fixed. On Azure, a delegation without client_event_id now ends the wait for a delegate() request, so _backend_busy no longer stays true. OpenAI and LiveKit Inference echo the id, so they still match by id.

agents/livekit-agents/livekit/agents/llm/_realtime/gpt_live.py

Lines 962 to 965 in e0bebcf

	)
	# Azure never echoes client_event_id, so there any delegation without one ends the wait
	if manual or (self._opts.is_azure and event.client_event_id is None):
	self._delegate_event_id = None

There is one limitation on Azure: a delegation that delegate() starts reports source="model", because nothing on the wire tells it apart from the model's own.

2. The finalize_call use case makes sense, and a follow-up PR on top of this one is welcome. Because tool_choice applies to the whole session, it would be better if the service accepted it on response.create itself. If it does not, the restore must cover every way the response can end, as you describe.

[jl982]

Thanks for working on this! I played with this code, and after some testing, I’m comfortable trying this with the default 0.5-second grace period.

My initial concern was that a late native delegation could arrive after recovery sends response.create, resulting in duplicate actions or repeated answers. But we investigated, and here’s what we found:

1. Our production timing supports the default. In our filtered September/October cohorts, p90 native-delegation delay was approximately 0.40s / 0.46s, measured from both sides becoming quiet, accounting for turns already covered by backend work. Detector inference adds further time before recovery sends a delegation, and native delegation cancels a pending check. These are estimates from traces, but they make the timing look reasonable.

2. Forced extra delegations did not repeat tools in our tests. Using real GPT-Live/backend sessions with synthetic booking, cancellation and knowledge tools, we ran 45 sessions: 44 extra bare delegations were accepted and server-confirmed, with zero repeated tool invocations. Five randomized booking-reference controls also preserved the previous result without calling the booking tool again. This supports the claim that the backend can recognize completed work.

3. Repeated speech is still possible. Some extra delegations produced another spoken confirmation despite no additional tool execution. I consider occasional repetition an acceptable tradeoff for recovering otherwise dropped turns.

Overall, I support merging this and starting with the default delay. Thank you again for implementing it!