fix(voice): keep the session busy while a reply is still owed by longcw · Pull Request #8459 · livekit/agents

longcw · 2026-11-09T09:16:02Z

Problem: A speech ends on listening while another speech is already owed, such as a finished tool's deferred reply or a realtime tool called inside a preamble, so clients close the turn early and the user-away timer arms. GPT-Live also marks the user away while its backend model still works, because the voice layer cannot see that work.

Fix: A speech now ends on thinking when another speech is already committed to follow it, and running tools hold the user-away timer through counted holds. Realtime and duplex sessions expose a read-only background_work_running, which GPT-Live returns from its backend chain state and the user-away timer reads before it starts.

Fixes #8362. Fixes #8317. Supersedes #8379 and #8345.

Context for reviewing and coding agents

How to see it

The script in #8362 runs a text-only session with a scripted LLM and prints speaking -> listening -> thinking on main when the tool finishes during a second reply. The log in #8317 shows the user going away at 15 s while the GPT-Live backend reasons with no tool. Driving a backend tool round trip through GPTLiveSession._handle_event reads background_work_running as False, then True with the call open, after the output and response.create, and during the continuation, then False after its response.completed.

Which paths change

_speech_follows replaces the bare _background_speeches check at every speech end: agent_activity.py:3216, :3470, :4047, :4710 and :5016. A realtime speech whose preamble called a tool now joins _background_speeches before the state is set, and the finally in _realtime_generation_task removes it. _set_user_away_timer no longer scans _RunningTasks; the tool executor takes and releases a hold per running task (tool_executor.py:460, :470) and moves it when set_owning_activity rescopes an AsyncToolset. A realtime model that leaves background_work_running at its default keeps today's behaviour.

Why a property read when the timer starts, and not an event

An edge event loses its state whenever no listener is attached: a handoff that reuses the realtime session detaches the old activity's listener and awaits cleanup before the new one subscribes. Reading the property when _set_user_away_timer runs is correct across reuse, reconnect and a fallback swap without any handling. Nothing restarts the timer when the work ends, because the answer is spoken next and the listening after it starts the timer; a backend that ends without speech leaves the user unmarked until the next state change.

Why the tool hold is counted per owner

The owner keeps #6883: a paused agent's tool cannot speak until that agent resumes, so its hold does not count while another agent is current. A per-owner count follows the existing _hold_interruptions and _idle_holds pattern and needs no keys or tokens.

Why not delegation tracking or generation matching

#8379 matched each generation_created to one tool turn, which breaks when GPT-Live answers several tool results in one generation. A delegation start and end pair needs a rule for when a delegation ends and assumes one delegation at a time. GPT-Live already tracks whether its backend works: running responses, open calls, and a pending or sent response.create. _backend_response_requested covers the gap between sending response.create and its response.created, where the other three are all clear.

What it does not cover

Client delegation runs in the application, so GPT-Live reports nothing for it. A backend output-token limit ends in a top-level error with no response.completed, so the chain stays stuck as it does on main and the timer stays off until a reconnect.

Conflicts

#8404 also adds _backend_busy to gpt_live.py, with _delegate_event_id as one more term. Whichever lands second keeps both terms, and #8404 can also use _backend_response_requested to mark a turn spoken in that gap as covered.

A speech ended on "listening" when a finished tool's deferred reply, or a realtime tool called inside a preamble, still had to speak, so clients closed the turn early and the user-away timer armed. GPT-Live also let the user go away while its backend model was still working. A speech now ends on "thinking" when another speech is already committed to follow it. The session holds the user-away timer through counted holds that running tools and a realtime model's background work take, and realtime models report that work with a new background_work_changed event, which GPT-Live emits from its backend chain state.

A reused realtime session reports background work only when it changes, so an agent that took over a busy session never held the user-away timer. The detaching activity now records its state in the reusable resources, and the activity that reuses the session takes the hold.

…sion reuse" This reverts commit fa4016b.

background_work_changed fires only on change, so an activity that attaches to a busy session, or one that finishes while no listener is attached during a handoff, held a wrong state. Realtime and duplex sessions now expose background_work_running, which the adapters pass through and GPT-Live implements, and the activity reads it after it subscribes and on reconnect.

devin-ai-integration

Note

Newer findings are available below. Devin Review posted a newer report on this PR, in addition to the findings presented here.

Devin Review found 1 new potential issue.

1 flag not posted on this PR by your GitHub settings — view it in Devin Review. (Configure)

devin-ai-integration · 2026-11-09T09:41:31Z

AgentActivity._detach_reusable_resources

+    @property
+    def background_work_running(self) -> bool:
+        """Whether background work is running, as the last ``background_work_changed`` said."""
+        return False


🟡 Busy custom sessions lose away hold

On handoff, background_work_running returns false for custom sessions that emit busy events but do not override the property. The reused session emits no repeat event, so the user-away timer can fire while work remains active.

Learn more

A realtime session reports model work through background_work_changed. The base property's constant false disagrees with a subclass that emits this event but does not override the property. During handoff, AgentActivity._detach_reusable_resources releases the old hold. AgentActivity._start_session_impl reads the reused session's property, so it cannot restore that hold. The same default exists on DuplexSession, whose events pass through the duplex adapter.

Example: A custom session emits background_work_changed(running=True) when a backend reply starts. The agent hands off to another agent using the same model while that reply runs. The new activity reads false, starts the user-away timer, and receives no new event until the reply ends.

Recommended fix: Cache background_work_changed in both session base classes when emitted, or explicitly require event-emitting implementations to expose their current state and preserve the activity's event-derived state across reuse for implementations without that property.

Was this helpful? React with 👍 or 👎 to provide feedback.

An edge event for background work lost its state whenever no listener was attached, such as during a handoff that reuses the realtime session. The session now exposes background_work_running only, GPT-Live returns its backend state from it, and the user-away timer reads it when it would start. The answer that follows the work restarts the timer.

devin-ai-integration

Devin Review found 1 new potential issue.

devin-ai-integration · 2026-11-09T10:55:05Z

_end_speech

+        if (
+            self._activity is not None
+            and (rt_session := self._activity.realtime_llm_session) is not None
+            and rt_session.background_work_running
        ):
-            # a tool that can speak when it lands holds the window; a paused agent's can't (#6883)
+            # the answer speaks when the work lands, and that speech restarts the window
            return


🔴 User marked away during backend reply

If response.created arrives after the user stops speaking, background_work_running turns true after the away timer starts. No state change cancels that timer, so a slow GPT-Live reply marks the waiting user away.

Learn more

The away timer starts when both user and agent are listening. GPT-Live ends the user's turn through _end_speech, which can start that timer before the backend sends its first response event. Later, _handle_response_event marks the backend busy without changing either voice state. This property check only runs when the timer is armed, so it cannot stop an existing countdown.

Example: With a 15-second timeout, the user stops speaking at second 0 and response.created arrives at second 1. If backend reasoning lasts 20 seconds, the user becomes away at second 15 while the reply is still pending.

Recommended fix: Cancel an armed away timer when backend work starts, using a state-change notification or equivalent callback. Keep the property read for timer starts and session reuse, and handle the case where work begins after the timer was armed.

Was this helpful? React with 👍 or 👎 to provide feedback.

@longcw I think this is valid.

related to this, we start the timer here before activity.start() installs the reused realtime session. The guard here will see None for rt_session. a busy session could survive the handoff and then setting the user as away

davidzhao · 2026-11-09T15:59:16Z

        self.send_event(types.ResponseCreateEvent(event_id=utils.shortuuid("response_create_")))

+    @property
+    def _backend_busy(self) -> bool:


do we need a separate property here? or just move the implementation to background_work_running?

davidzhao · 2026-11-09T16:22:24Z

+        if (
+            self._activity is not None
+            and (rt_session := self._activity.realtime_llm_session) is not None
+            and rt_session.background_work_running
        ):
-            # a tool that can speak when it lands holds the window; a paused agent's can't (#6883)
+            # the answer speaks when the work lands, and that speech restarts the window
            return


@longcw I think this is valid.

davidzhao · 2026-11-09T16:25:03Z

            self._chat_ctx._upsert_item(item)

-    def _tool_execution_updated(self, ev: ToolExecutionUpdatedEvent) -> None:
+    def _hold_user_away(self, owner: AgentActivity | None) -> None:


Suggested change

def _hold_user_away(self, owner: AgentActivity | None) -> None:

def _create_away_timer_hold(self, owner: AgentActivity | None) -> None:

creating a hold on the timer.

davidzhao · 2026-11-09T16:28:22Z

+        if owner in (None, self._activity):
+            self._cancel_user_away_timer()
+
+    def _release_user_away(self, owner: AgentActivity | None) -> None:


Suggested change

def _release_user_away(self, owner: AgentActivity | None) -> None:

def _release_away_timer_hold(self, owner: AgentActivity | None) -> None:

davidzhao · 2026-11-09T16:28:57Z

        return not self._speech_q and (not self._current_speech or self._current_speech.done())

+    @property
+    def _speech_follows(self) -> bool:


Suggested change

def _speech_follows(self) -> bool:

def _has_followup_work(self) -> bool:

davidzhao · 2026-11-09T16:32:06Z

+        if (
+            self._activity is not None
+            and (rt_session := self._activity.realtime_llm_session) is not None
+            and rt_session.background_work_running
        ):
-            # a tool that can speak when it lands holds the window; a paused agent's can't (#6883)
+            # the answer speaks when the work lands, and that speech restarts the window
            return


related to this, we start the timer here before activity.start() installs the reused realtime session. The guard here will see None for rt_session. a busy session could survive the handoff and then setting the user as away

longcw requested a review from a team as a code owner November 9, 2026 09:16

This comment was marked as resolved.

Sign in to view

longcw added 2 commits November 9, 2026 17:37

Revert "fix(voice): keep the background-work hold across realtime ses…

e4efa59

…sion reuse" This reverts commit fa4016b.

devin-ai-integration Bot reviewed Nov 9, 2026

View reviewed changes

davidzhao reviewed Nov 9, 2026

View reviewed changes

	def _hold_user_away(self, owner: AgentActivity \| None) -> None:
	def _create_away_timer_hold(self, owner: AgentActivity \| None) -> None:

	def _release_user_away(self, owner: AgentActivity \| None) -> None:
	def _release_away_timer_hold(self, owner: AgentActivity \| None) -> None:

	def _speech_follows(self) -> bool:
	def _has_followup_work(self) -> bool:

Conversation

longcw commented Nov 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

devin-ai-integration Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Nov 9, 2026

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot left a comment

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Nov 9, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

longcw commented Nov 9, 2026 •

edited

Loading

devin-ai-integration Bot left a comment •

edited

Loading