Repository navigation
Conversation
|
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). View this failed invocation of the CLA check for more information. For the most up to date status, view the checks section at the bottom of the pull request. |
Signed-off-by: Harbor404 <2657212322@qq.com>
# Conflicts: # agent/workflowagent/workflow.go
|
Synced with the latest upstream |
karolpiotrowicz
left a comment
There was a problem hiding this comment.
Three behaviors that work on main today break with this change, so it needs fixing before merge. The single-pause confirmation fix itself works: the new tests fail on the merge-base and pass on this branch.
Must change before merge:
-
A re-entry node that finished on an earlier turn runs again on every later resume.
persistence.goL336-341 setsansweredThisTurnfor any interrupt answered once anywhere in the invocation, andresume.goL148 now gates only on that flag. Main gated on whether the current turn's responses contain one of the node's interrupts (resume.goL120-127 at the merge-base).- With two
RerunOnResumeapproval steps in a chain, answering the second re-runs the first and runs the second twice: 3 runs each instead of 2. - Two parallel askers answered on separate turns fail with
multiple terminal nodes produced output. - Because this PR makes LlmAgent nodes re-entry by default, the PR's own shape hits it too. With two confirmation-gated LlmAgent nodes in a chain, approving the second call re-invokes the first agent's model and calls the second agent's model twice.
The property to keep is that a resume acts only on the interrupts the current message answers, and a node that completed earlier in the invocation does not run again. None of the new tests has two pauses in one invocation, which is why the suite stays green.
- With two
-
A reply whose resume failed can no longer be retried with the same payload. The runner persists the user's reply before the resume runs. If the resume errors, re-sending the same reply puts two identical payloads in history,
persistence.goL165-167 marks it replayed, and the retry getsErrNothingToResume. Every later identical retry gets the same error, so the paused node never completes. On main the retry re-runs the node and the workflow finishes. A duplicate of a consumed reply and a retry of a reply that was never consumed need to be told apart. -
A confirmation inside a non-LlmAgent
AgentNodeis approved, but the tool never runs. Matching by ID (workflow.goL159) now routes the reply to anAgentNodethat wraps, for example, a sequential agent containing the gated LlmAgent. That node stays in handoff mode (agent_node.goL71-73), so the approval map{"confirmed": true}becomes the node's output and flows to the successor. The gated tool never executes and no error is returned. On main the tool runs, by way of the graph restart this PR removes. An approval should either reach the tool or be reported, never be forwarded as the agent's output.
The reproduction below passes on current main (6576191) and all three tests fail on this branch (c6ea0e9).
runner/resume_regression_test.go — run with go test ./runner -run TestResumeRegression -v
package runner_test
import (
"iter"
"sync/atomic"
"testing"
"google.golang.org/genai"
"google.golang.org/adk/v2/agent"
"google.golang.org/adk/v2/agent/llmagent"
"google.golang.org/adk/v2/agent/workflowagent"
"google.golang.org/adk/v2/agent/workflowagents/sequentialagent"
"google.golang.org/adk/v2/runner"
"google.golang.org/adk/v2/session"
"google.golang.org/adk/v2/tool"
"google.golang.org/adk/v2/tool/functiontool"
"google.golang.org/adk/v2/workflow"
)
// countingAsker pauses on RequestInput and, when re-entered with the
// reply, emits it as output. failFirstResume makes its first re-entry
// return an error.
type countingAsker struct {
workflow.BaseNode
id string
runs atomic.Int32
failFirstResume bool
failed atomic.Bool
}
func newCountingAsker(name, id string) *countingAsker {
yes := true
return &countingAsker{BaseNode: workflow.NewBaseNode(name, "", workflow.NodeConfig{RerunOnResume: &yes}), id: id}
}
func (n *countingAsker) Run(ctx agent.Context, _ any) iter.Seq2[*session.Event, error] {
return func(yield func(*session.Event, error) bool) {
n.runs.Add(1)
if resp, ok := ctx.ResumedInput(n.id); ok {
if n.failFirstResume && !n.failed.Swap(true) {
yield(nil, errTransient)
return
}
ev := session.NewEvent(ctx, ctx.InvocationID())
ev.Output = resp
yield(ev, nil)
return
}
yield(workflow.NewRequestInputEvent(ctx, session.RequestInput{InterruptID: n.id, Message: n.id}), nil)
}
}
type transientErr struct{}
func (transientErr) Error() string { return "transient failure" }
var errTransient = transientErr{}
func runReturningErr(t *testing.T, r *runner.Runner, msg *genai.Content) ([]*session.Event, error) {
t.Helper()
var evs []*session.Event
var first error
for ev, err := range r.Run(t.Context(), nodeTestUser, nodeTestSession, msg, agent.RunConfig{}) {
if err != nil {
if first == nil {
first = err
}
continue
}
evs = append(evs, ev)
}
return evs, first
}
// Two approval steps in sequence. Answering the second must not re-run the first.
func TestResumeRegression_ChainedReentryAskers(t *testing.T) {
a, b := newCountingAsker("approveA", "a"), newCountingAsker("approveB", "b")
r := newWorkflowRunner(t, workflow.Chain(workflow.Start, a, b))
t1, _ := runReturningErr(t, r, userText("start"))
idA, nameA := findLongRunningInterrupt(t1)
t2, _ := runReturningErr(t, r, resumeContent(idA, nameA, "ok-a"))
idB, nameB := findLongRunningInterrupt(t2)
if _, err := runReturningErr(t, r, resumeContent(idB, nameB, "ok-b")); err != nil {
t.Fatalf("answering b: %v", err)
}
if a.runs.Load() != 2 || b.runs.Load() != 2 {
t.Errorf("runs: approveA=%d approveB=%d, want 2 and 2", a.runs.Load(), b.runs.Load())
}
}
// A resume that fails must stay retryable with the same reply.
func TestResumeRegression_RetryAfterFailedResume(t *testing.T) {
a := newCountingAsker("asker", "q")
a.failFirstResume = true
var sinkRuns atomic.Int32
sink := workflow.NewFunctionNode("sink", func(_ agent.Context, in any) (any, error) {
sinkRuns.Add(1)
return in, nil
}, workflow.NodeConfig{})
r := newWorkflowRunner(t, workflow.Chain(workflow.Start, a, sink))
t1, _ := runReturningErr(t, r, userText("start"))
id, name := findLongRunningInterrupt(t1)
if _, err := runReturningErr(t, r, resumeContent(id, name, "ok")); err == nil {
t.Fatal("first resume: want the injected failure")
}
if _, err := runReturningErr(t, r, resumeContent(id, name, "ok")); err != nil {
t.Fatalf("retry with the same reply: %v", err)
}
if sinkRuns.Load() != 1 {
t.Errorf("sink runs = %d, want 1", sinkRuns.Load())
}
}
// An AgentNode wrapping a non-LlmAgent (here a sequential agent) whose
// inner LlmAgent asks for tool confirmation. Approving must run the tool.
func TestResumeRegression_ConfirmationInsideSequentialAgentNode(t *testing.T) {
var toolRuns atomic.Int32
confirmTool, err := functiontool.New(functiontool.Config{
Name: "confirm_action", Description: "x", RequireConfirmation: true,
}, func(agent.Context, struct{}) (map[string]string, error) {
toolRuns.Add(1)
return map[string]string{"result": "executed"}, nil
})
if err != nil {
t.Fatal(err)
}
gated, err := llmagent.New(llmagent.Config{Name: "gated", Model: &scriptedModel{responses: []*genai.Content{
genai.NewContentFromFunctionCall("confirm_action", map[string]any{}, "model"),
genai.NewContentFromText("finished", "model"),
}}, Tools: []tool.Tool{confirmTool}})
if err != nil {
t.Fatal(err)
}
seq, err := sequentialagent.New(sequentialagent.Config{AgentConfig: agent.Config{Name: "seq", SubAgents: []agent.Agent{gated}}})
if err != nil {
t.Fatal(err)
}
node, err := workflow.NewAgentNode(seq, workflow.NodeConfig{})
if err != nil {
t.Fatal(err)
}
wf, err := workflowagent.New(workflowagent.Config{Name: workflowAgentName, Edges: workflow.Chain(workflow.Start, node)})
if err != nil {
t.Fatal(err)
}
svc := session.InMemoryService()
newNodeTestSession(t, t.Context(), svc)
r := newNodeTestRunner(t, wf, svc)
t1, _ := runReturningErr(t, r, userText("start"))
id, name := findLongRunningInterrupt(t1)
approve := &genai.Content{Role: genai.RoleUser, Parts: []*genai.Part{{FunctionResponse: &genai.FunctionResponse{
ID: id, Name: name, Response: map[string]any{"confirmed": true},
}}}}
if _, err := runReturningErr(t, r, approve); err != nil {
t.Fatalf("approve: %v", err)
}
if toolRuns.Load() != 1 {
t.Errorf("confirmed tool runs = %d, want 1", toolRuns.Load())
}
}Output on this branch:
resume_regression_test.go:89: runs: approveA=3 approveB=3, want 2 and 2
--- FAIL: TestResumeRegression_ChainedReentryAskers
resume_regression_test.go:110: retry with the same reply: workflow: no waiting node matched the supplied responses
--- FAIL: TestResumeRegression_RetryAfterFailedResume
resume_regression_test.go:162: confirmed tool runs = 0, want 1
--- FAIL: TestResumeRegression_ConfirmationInsideSequentialAgentNode
Also changed from main, non-blocking:
- The
Workflow.Resumedoc (resume.goL61-72) still says a duplicate resume of a re-entry node "re-runs it a second time" and "is never a no-op in the re-entry case". With this change, an exact duplicate to a single-interrupt re-entry node returnsErrNothingToResume, while a node with two interrupts still re-runs. Callers that setRerunOnResumeexplicitly get the new behavior, so the doc should describe it. The comment at L107-113, which promises "only for interrupts answered in THIS turn", no longer matches the code either. - The
NodeConfig.RerunOnResumedoc (config.goL62-66) and theNewAgentNodegodoc don't mention that LlmAgent nodes now default to re-entry.
Track whether a resume response was consumed instead of comparing payloads, keep root LlmAgent resume routing unchanged, retain resume history for prompt assembly, and default AgentNodes to re-entry so confirmations inside wrapped composite agents execute.
|
@karolpiotrowicz addressed the review-body blockers in 4fe90df:
I also updated the Workflow.Resume, NodeConfig.RerunOnResume, NodeState, and NewAgentNode documentation to describe re-entry defaults and consumed-response behavior. Verification on 4fe90df (latest main included): |
Superseded by the round-2 review of 4fe90df.
karolpiotrowicz
left a comment
There was a problem hiding this comment.
Four behaviors still need fixing before merge, and three of them work on main today. The three cases from the last round are fixed for the shapes those tests cover: all three pass on this branch. Tracking whether a reply was consumed is the right direction. The remaining problems come from when a reply counts as consumed, plus one prompt-assembly change.
Must change before merge:
-
A resume that fails after the node has emitted anything can never be retried.
persistence.goL184-187 marks a re-entry node's reply consumed on its first event, and the bit is never cleared. A node that emits a progress event and then errors getsErrNothingToResumeon every retry, so it never completes. Onmainthe retry re-runs the node and the workflow finishes. This change's own path hits it: an LlmAgent node emits the confirmed tool's response before its next model call. So a transient model error after approval leaves the invocation stuck for good, with the tool already executed. -
In parallel branches, one successor finishing consumes the other branch's reply.
persistence.goL188-196 marks every handoff reply in the invocation consumed whenever any handoff-mode node emits. If two askers are answered in one message, A's successor finishes and B's successor fails, the retry of B's reply getsErrNothingToResumeand B's successor never runs. Onmainthe retry runs it. The updatedResumedoc says a handoff reply is consumed when "a successor runs", but the code accepts any node. -
A handoff asker whose successor is a confirmation-gated
AgentNodeis never marked consumed, so an unrelated reply restarts that agent. Events from re-entry nodes only take the L184-187 branch, andagent_node.goL72-75 now makes everyAgentNodere-entry. With a parallel asker still open, answering that asker hands the first reply off again. The waiting agent then starts from scratch, calls its model again and issues a second confirmation while the first is still open. Onmainthe agent is left alone. A re-entry node that emits nothing when it resumes has the same gap and re-runs on every later resume. -
On resume, a single-turn LlmAgent node gets the whole session instead of its own exchange.
contents_processor.goL75-80 falls back tobuildContentsDefaultwhenever the user content carries a function response. In awriter → redact → gatedchain, the gated node's first call sees only the redacted text. After approval, its request contains the writer's unredacted message and the workflow's original user message. It no longer contains the redacted input that the function call was made from.TestResumeRegression_ResumePromptIncludesHistorycannot catch this because its gated node followsStartdirectly, so the user prompt and the node input are the same text.
The properties to keep are the ones from the last round. A reply counts as consumed only once the node it resumed has finished with it, so a failed resume stays retryable even after the node emitted something. A reply belongs to its own asker and successors, so finishing one branch does not consume another's. On resume, the model sees the same context its first call saw plus the call and its response.
The reproduction below passes on current main (f2f9ea3), and all four tests fail on this branch (4fe90df). I ran it 10 times on each with identical results. It uses only helpers that exist on main.
runner/resume_round2_test.go — run with go test ./runner -run TestResumeRound2 -v
package runner_test
import (
"context"
"errors"
"iter"
"slices"
"sync/atomic"
"testing"
"time"
"google.golang.org/genai"
"google.golang.org/adk/v2/agent"
"google.golang.org/adk/v2/agent/llmagent"
"google.golang.org/adk/v2/agent/workflowagent"
"google.golang.org/adk/v2/model"
"google.golang.org/adk/v2/runner"
"google.golang.org/adk/v2/session"
"google.golang.org/adk/v2/tool"
"google.golang.org/adk/v2/tool/functiontool"
"google.golang.org/adk/v2/workflow"
)
func run2(t *testing.T, r *runner.Runner, msg *genai.Content) ([]*session.Event, error) {
t.Helper()
var evs []*session.Event
var first error
for ev, err := range r.Run(t.Context(), nodeTestUser, nodeTestSession, msg, agent.RunConfig{}) {
if err != nil {
if first == nil {
first = err
}
continue
}
evs = append(evs, ev)
}
return evs, first
}
func interrupts2(evs []*session.Event) map[string]string {
out := map[string]string{}
for _, ev := range evs {
if ev == nil || ev.Content == nil {
continue
}
for _, p := range ev.Content.Parts {
if fc := p.FunctionCall; fc != nil && slices.Contains(ev.LongRunningToolIDs, fc.ID) {
out[fc.ID] = fc.Name
}
}
}
return out
}
// recModel records every request and answers a confirm_action response with
// text, anything else with a confirm_action call.
type recModel struct {
calls [][]*genai.Content
text string
}
func (m *recModel) Name() string { return "rec" }
func (m *recModel) GenerateContent(_ context.Context, req *model.LLMRequest, _ bool) iter.Seq2[*model.LLMResponse, error] {
m.calls = append(m.calls, append([]*genai.Content(nil), req.Contents...))
return func(yield func(*model.LLMResponse, error) bool) {
if m.text != "" {
yield(&model.LLMResponse{Content: genai.NewContentFromText(m.text, "model")}, nil)
return
}
if n := len(req.Contents); n > 0 {
for _, p := range req.Contents[n-1].Parts {
if p.FunctionResponse != nil && p.FunctionResponse.Name == "confirm_action" {
yield(&model.LLMResponse{Content: genai.NewContentFromText("done", "model")}, nil)
return
}
}
}
yield(&model.LLMResponse{Content: genai.NewContentFromFunctionCall("confirm_action", map[string]any{}, "model")}, nil)
}
}
func gatedAgentNode(t *testing.T, name string, m model.LLM) *workflow.AgentNode {
t.Helper()
ct, err := functiontool.New(functiontool.Config{Name: "confirm_action", Description: "x", RequireConfirmation: true},
func(agent.Context, struct{}) (map[string]string, error) {
return map[string]string{"result": "executed"}, nil
})
if err != nil {
t.Fatal(err)
}
a, err := llmagent.New(llmagent.Config{Name: name, Model: m, Tools: []tool.Tool{ct}})
if err != nil {
t.Fatal(err)
}
n, err := workflow.NewAgentNode(a, workflow.NodeConfig{})
if err != nil {
t.Fatal(err)
}
return n
}
func approve2(id string) *genai.Content {
return &genai.Content{Role: genai.RoleUser, Parts: []*genai.Part{{FunctionResponse: &genai.FunctionResponse{
ID: id, Name: "adk_request_confirmation", Response: map[string]any{"confirmed": true},
}}}}
}
// progressThenFail is a re-entry node whose first resume emits a progress
// event and then fails.
type progressThenFail struct {
workflow.BaseNode
failed atomic.Bool
}
func (n *progressThenFail) Run(ctx agent.Context, _ any) iter.Seq2[*session.Event, error] {
return func(yield func(*session.Event, error) bool) {
if resp, ok := ctx.ResumedInput("q"); ok {
if !n.failed.Swap(true) {
ev := session.NewEvent(ctx, ctx.InvocationID())
ev.Content = genai.NewContentFromText("working", "model")
if yield(ev, nil) {
yield(nil, errors.New("transient failure"))
}
return
}
ev := session.NewEvent(ctx, ctx.InvocationID())
ev.Output = resp
yield(ev, nil)
return
}
yield(workflow.NewRequestInputEvent(ctx, session.RequestInput{InterruptID: "q", Message: "q"}), nil)
}
}
// A resume that fails after the node emitted an event must stay retryable.
func TestResumeRound2_RetryAfterProgressThenFailure(t *testing.T) {
yes := true
a := &progressThenFail{BaseNode: workflow.NewBaseNode("asker", "", workflow.NodeConfig{RerunOnResume: &yes})}
var sinkRuns atomic.Int32
sink := workflow.NewFunctionNode("sink", func(_ agent.Context, in any) (any, error) {
sinkRuns.Add(1)
return in, nil
}, workflow.NodeConfig{})
r := newWorkflowRunner(t, workflow.Chain(workflow.Start, a, sink))
t1, _ := run2(t, r, userText("start"))
id, name := findLongRunningInterrupt(t1)
if _, err := run2(t, r, resumeContent(id, name, "ok")); err == nil {
t.Fatal("first resume: want the injected failure")
}
if _, err := run2(t, r, resumeContent(id, name, "ok")); err != nil {
t.Fatalf("retry with the same reply: %v", err)
}
if sinkRuns.Load() != 1 {
t.Errorf("sink runs = %d, want 1", sinkRuns.Load())
}
}
// successor emits an event. When failFirst is set it first waits for done
// (closed by the other branch's successor) and then fails once.
type successor struct {
workflow.BaseNode
failFirst bool
wait chan struct{}
done chan struct{}
runs atomic.Int32
failed atomic.Bool
}
func (n *successor) Run(ctx agent.Context, _ any) iter.Seq2[*session.Event, error] {
return func(yield func(*session.Event, error) bool) {
n.runs.Add(1)
if n.failFirst && !n.failed.Swap(true) {
select {
case <-n.wait:
case <-time.After(2 * time.Second):
}
yield(nil, errors.New("transient failure"))
return
}
ev := session.NewEvent(ctx, ctx.InvocationID())
ev.Content = genai.NewContentFromText(n.Name()+" ran", "model")
yield(ev, nil)
if n.done != nil {
close(n.done)
n.done = nil
}
}
}
// Two parallel handoff askers answered in one message. A's successor
// finishes; B's successor fails. Retrying B's reply must run B's successor.
func TestResumeRound2_ParallelHandoffRetry(t *testing.T) {
saDone := make(chan struct{})
a, b := newHitlAsker("askA", "qa", false), newHitlAsker("askB", "qb", false)
sa := &successor{BaseNode: workflow.NewBaseNode("sa", "", workflow.NodeConfig{}), done: saDone}
sb := &successor{BaseNode: workflow.NewBaseNode("sb", "", workflow.NodeConfig{}), failFirst: true, wait: saDone}
r := newWorkflowRunner(t, append(workflow.Chain(workflow.Start, a, sa), workflow.Chain(workflow.Start, b, sb)...))
t1, _ := run2(t, r, userText("start"))
ints := interrupts2(t1)
both := &genai.Content{Role: genai.RoleUser}
for id, name := range ints {
both.Parts = append(both.Parts, &genai.Part{FunctionResponse: &genai.FunctionResponse{
ID: id, Name: name, Response: map[string]any{"payload": "ans-" + id},
}})
}
if _, err := run2(t, r, both); err == nil {
t.Fatal("first resume: want sb's injected failure")
}
if _, err := run2(t, r, resumeContent("qb", ints["qb"], "ans-qb")); err != nil {
t.Fatalf("retry of B: %v", err)
}
if sb.runs.Load() != 2 {
t.Errorf("sb runs = %d, want 2", sb.runs.Load())
}
}
// A handoff asker feeding a confirmation-gated AgentNode, next to an
// unrelated parallel asker. Answering the unrelated asker must not restart
// the gated agent while its confirmation is open.
func TestResumeRound2_UnrelatedReplyDoesNotRestartGatedNode(t *testing.T) {
m := &recModel{}
exec := gatedAgentNode(t, "executor", m)
approve, other := newHitlAsker("approve", "ia", false), newHitlAsker("other", "io", false)
osink := workflow.NewFunctionNode("osink", func(_ agent.Context, in any) (any, error) { return in, nil }, workflow.NodeConfig{})
r := newWorkflowRunner(t, append(workflow.Chain(workflow.Start, approve, exec), workflow.Chain(workflow.Start, other, osink)...))
t1, _ := run2(t, r, userText("start"))
ints := interrupts2(t1)
if _, err := run2(t, r, resumeContent("ia", ints["ia"], "go")); err != nil {
t.Fatalf("answering approve: %v", err)
}
before := len(m.calls)
t3, err := run2(t, r, resumeContent("io", ints["io"], "other"))
if err != nil {
t.Fatalf("answering other: %v", err)
}
if len(m.calls) != before {
t.Errorf("executor model calls %d -> %d after an unrelated reply", before, len(m.calls))
}
if n := len(interrupts2(t3)); n != 0 {
t.Errorf("unrelated reply raised %d new confirmation(s)", n)
}
}
// writer -> redact -> gated. The gated node's first call sees only the
// redacted text. After approval it must not see the writer's text, and it
// must still see the input it made the call from.
func TestResumeRound2_SingleTurnNodeKeepsItsContextOnResume(t *testing.T) {
writer, err := llmagent.New(llmagent.Config{Name: "writer", Model: &recModel{text: "WRITER-PRIVATE"}})
if err != nil {
t.Fatal(err)
}
wn, err := workflow.NewAgentNode(writer, workflow.NodeConfig{})
if err != nil {
t.Fatal(err)
}
redact := workflow.NewFunctionNode("redact", func(agent.Context, any) (any, error) { return "REDACTED", nil }, workflow.NodeConfig{})
m := &recModel{}
gn := gatedAgentNode(t, "gated", m)
wf, err := workflowagent.New(workflowagent.Config{Name: workflowAgentName, Edges: workflow.Chain(workflow.Start, wn, redact, gn)})
if err != nil {
t.Fatal(err)
}
svc := session.InMemoryService()
newNodeTestSession(t, t.Context(), svc)
r := newNodeTestRunner(t, wf, svc)
t1, _ := run2(t, r, userText("start"))
id, _ := findLongRunningInterrupt(t1)
if _, err := run2(t, r, approve2(id)); err != nil {
t.Fatalf("approve: %v", err)
}
if len(m.calls) < 2 {
t.Fatalf("gated model calls = %d, want 2", len(m.calls))
}
var sawPrivate, sawInput bool
for _, c := range m.calls[1] {
for _, p := range c.Parts {
sawPrivate = sawPrivate || p.Text == "WRITER-PRIVATE" || p.Text == "[writer] said: WRITER-PRIVATE"
sawInput = sawInput || p.Text == "REDACTED"
}
}
if sawPrivate || !sawInput {
t.Errorf("resume request: contains writer text = %v, contains own input = %v; want false, true", sawPrivate, sawInput)
}
}Output on this branch:
resume_round2_test.go:154: retry with the same reply: workflow: no waiting node matched the supplied responses
--- FAIL: TestResumeRound2_RetryAfterProgressThenFailure
resume_round2_test.go:214: retry of B: workflow: no waiting node matched the supplied responses
--- FAIL: TestResumeRound2_ParallelHandoffRetry
resume_round2_test.go:242: executor model calls 1 -> 2 after an unrelated reply
resume_round2_test.go:245: unrelated reply raised 1 new confirmation(s)
--- FAIL: TestResumeRound2_UnrelatedReplyDoesNotRestartGatedNode
resume_round2_test.go:288: resume request: contains writer text = true, contains own input = false; want false, true
--- FAIL: TestResumeRound2_SingleTurnNodeKeepsItsContextOnResume
Maintainers need to decide whether every AgentNode should now default to re-entry. That question and a doc wording point are on the lines.
|
@karolpiotrowicz Addressed the four review-body items in
The Focused verification passed:
The new regression set also passed 5 consecutive runs. |
karolpiotrowicz
left a comment
There was a problem hiding this comment.
Three behaviors still need fixing before merge. Two of the four items from my last review are fixed outright: a re-entry node that emits progress and then fails is now retryable (including an LlmAgent whose model call fails after approval), and finishing one parallel branch no longer consumes the other branch's reply (20 of 20 runs). The other two are fixed for the graphs their tests use but not for the neighboring cases below.
Must change before merge
-
A re-entry node that finishes without Output, a route or a new interrupt never consumes its reply.
persistence.goL181-183 settles a re-entry reply only on an event thatisConsumptionEventaccepts. A node whose resume emits only content, or nothing at all, stays unconsumed. That covers aFunctionNodeor dynamic node returning nil and a custom node that only reports progress. Answering an unrelated question later in the same invocation then runs that node a second time. Onmainit is left alone. This is the second half of the third item from my last review, which is still open. A duplicate of the node's own reply also re-runs it and its successor, despite theResumedoc saying duplicates yieldErrNothingToResume. The property to keep: a re-entry reply is consumed once the node finishes its resume successfully, whatever its last event carries. -
A resumed single-turn node still loses its input in two graph shapes. The property from my last review still applies: on resume the model sees the context its first call saw, plus the call and its response, and nothing else.
- In a loop.
seedInsertAtputs the seed before the first event in the invocation whose path matches the node. Every activation of a node in a loop carries the same path (gated@1on both passes), so on the second pass the seed lands before the first pass and gets cut off. The request after the second approval holds only the call and its response, withoutINPUT-2. - Behind a handoff asker, which is the graph in
TestResumeRegression_UnrelatedReplyDoesNotRestartGatedNode. The gated node's first call sees the replygo-PAYLOADas its input. After approval its request no longer containsgo-PAYLOAD. It containsFor context: [test_workflow] called tool adk_request_input ... "payload":"start"instead, which is the asker's question and the workflow's original input. The node is rebuilt with a nil input becausepredecessorInputonly finds outputs recorded as events, and a handoff asker's reply is not one. With a nil input no seed is created.
- In a loop.
-
A handoff asker with no successor never consumes its reply.
persistence.goL187-199 settles a handoff reply only when a successor emits, so an asker at the end of a graph is never settled. WithChain(Start, ask), sending the same reply again and sending a different reply both return nil on this branch. Onmainboth returnErrNothingToResume, which is what the newResumedoc promises.
Reproduction
The reproduction below fails all four tests on this branch (00ad29a), 10 runs out of 10. On current main (f2f9ea3) the first and last tests pass. The loop and handoff tests fail earlier there, because main never resumes the confirmation at all. It uses only helpers that exist on main.
runner/resume_round3_test.go — run with go test ./runner -run TestResumeRound3 -v
package runner_test
import (
"context"
"fmt"
"iter"
"strings"
"sync/atomic"
"testing"
"google.golang.org/genai"
"google.golang.org/adk/v2/agent"
"google.golang.org/adk/v2/agent/llmagent"
"google.golang.org/adk/v2/agent/workflowagent"
"google.golang.org/adk/v2/model"
"google.golang.org/adk/v2/runner"
"google.golang.org/adk/v2/session"
"google.golang.org/adk/v2/tool"
"google.golang.org/adk/v2/tool/functiontool"
"google.golang.org/adk/v2/workflow"
)
func run3(t *testing.T, r *runner.Runner, msg *genai.Content) ([]*session.Event, error) {
t.Helper()
var evs []*session.Event
var first error
for ev, err := range r.Run(t.Context(), nodeTestUser, nodeTestSession, msg, agent.RunConfig{}) {
if err != nil {
if first == nil {
first = err
}
continue
}
evs = append(evs, ev)
}
return evs, first
}
// model3 records every request. It answers a confirm_action response with
// text and anything else with a confirm_action call.
type model3 struct{ calls [][]*genai.Content }
func (m *model3) Name() string { return "model3" }
func (m *model3) GenerateContent(_ context.Context, req *model.LLMRequest, _ bool) iter.Seq2[*model.LLMResponse, error] {
m.calls = append(m.calls, append([]*genai.Content(nil), req.Contents...))
return func(yield func(*model.LLMResponse, error) bool) {
if n := len(req.Contents); n > 0 {
for _, p := range req.Contents[n-1].Parts {
if p.FunctionResponse != nil && p.FunctionResponse.Name == "confirm_action" {
yield(&model.LLMResponse{Content: genai.NewContentFromText("done", "model")}, nil)
return
}
}
}
yield(&model.LLMResponse{Content: genai.NewContentFromFunctionCall("confirm_action", map[string]any{}, "model")}, nil)
}
}
func (m *model3) callSees(i int, text string) bool {
for _, c := range m.calls[i] {
for _, p := range c.Parts {
if strings.Contains(p.Text, text) {
return true
}
}
}
return false
}
func gated3(t *testing.T, name string, m model.LLM) *workflow.AgentNode {
t.Helper()
ct, err := functiontool.New(functiontool.Config{Name: "confirm_action", Description: "x", RequireConfirmation: true},
func(agent.Context, struct{}) (map[string]string, error) {
return map[string]string{"result": "executed"}, nil
})
if err != nil {
t.Fatal(err)
}
a, err := llmagent.New(llmagent.Config{Name: name, Model: m, Tools: []tool.Tool{ct}})
if err != nil {
t.Fatal(err)
}
n, err := workflow.NewAgentNode(a, workflow.NodeConfig{})
if err != nil {
t.Fatal(err)
}
return n
}
func approve3(id string) *genai.Content {
return &genai.Content{Role: genai.RoleUser, Parts: []*genai.Part{{FunctionResponse: &genai.FunctionResponse{
ID: id, Name: "adk_request_confirmation", Response: map[string]any{"confirmed": true},
}}}}
}
func graphRunner3(t *testing.T, edges []workflow.Edge) *runner.Runner {
t.Helper()
wf, err := workflowagent.New(workflowagent.Config{Name: workflowAgentName, Edges: edges})
if err != nil {
t.Fatal(err)
}
svc := session.InMemoryService()
newNodeTestSession(t, t.Context(), svc)
return newNodeTestRunner(t, wf, svc)
}
// contentOnly is a re-entry node whose successful resume emits only content.
type contentOnly struct {
workflow.BaseNode
resumes atomic.Int32
}
func (n *contentOnly) Run(ctx agent.Context, _ any) iter.Seq2[*session.Event, error] {
return func(yield func(*session.Event, error) bool) {
if resp, ok := ctx.ResumedInput("q"); ok {
n.resumes.Add(1)
ev := session.NewEvent(ctx, ctx.InvocationID())
ev.Content = genai.NewContentFromText(fmt.Sprintf("handled %v", resp), "model")
yield(ev, nil)
return
}
yield(workflow.NewRequestInputEvent(ctx, session.RequestInput{InterruptID: "q", Message: "q"}), nil)
}
}
// A re-entry node that finished its resume without Output has still consumed
// the reply. Answering an unrelated question afterwards must not run it again.
func TestResumeRound3_ReentryNodeWithoutOutputConsumesReply(t *testing.T) {
yes := true
a := &contentOnly{BaseNode: workflow.NewBaseNode("asker", "", workflow.NodeConfig{RerunOnResume: &yes})}
other := newHitlAsker("other", "io", false)
osink := workflow.NewFunctionNode("osink", func(_ agent.Context, in any) (any, error) { return in, nil }, workflow.NodeConfig{})
r := newWorkflowRunner(t, append(workflow.Chain(workflow.Start, a), workflow.Chain(workflow.Start, other, osink)...))
t1, _ := run3(t, r, userText("start"))
names := map[string]string{}
for _, ev := range t1 {
if ev.Content == nil {
continue
}
for _, p := range ev.Content.Parts {
if p.FunctionCall != nil {
names[p.FunctionCall.ID] = p.FunctionCall.Name
}
}
}
if _, err := run3(t, r, resumeContent("q", names["q"], "ok")); err != nil {
t.Fatalf("answering asker: %v", err)
}
if _, err := run3(t, r, resumeContent("io", names["io"], "other")); err != nil {
t.Fatalf("answering other: %v", err)
}
if got := a.resumes.Load(); got != 1 {
t.Errorf("asker resumed %d times, want 1 (an unrelated reply re-ran it)", got)
}
}
type writer3 struct {
workflow.BaseNode
runs atomic.Int32
}
func (n *writer3) Run(ctx agent.Context, _ any) iter.Seq2[*session.Event, error] {
return func(yield func(*session.Event, error) bool) {
ev := session.NewEvent(ctx, ctx.InvocationID())
ev.Output = fmt.Sprintf("INPUT-%d", n.runs.Add(1))
yield(ev, nil)
}
}
type router3 struct {
workflow.BaseNode
runs atomic.Int32
}
func (n *router3) Run(ctx agent.Context, in any) iter.Seq2[*session.Event, error] {
return func(yield func(*session.Event, error) bool) {
ev := session.NewEvent(ctx, ctx.InvocationID())
ev.Output = in
ev.Routes = []string{"finish"}
if n.runs.Add(1) == 1 {
ev.Routes = []string{"loop"}
}
yield(ev, nil)
}
}
// writer -> gated -> router, looping once. After the second approval the
// gated node must still see its second input.
func TestResumeRound3_SingleTurnResumeInLoopKeepsInput(t *testing.T) {
m := &model3{}
w := &writer3{BaseNode: workflow.NewBaseNode("writer", "", workflow.NodeConfig{})}
g := gated3(t, "gated", m)
rt := &router3{BaseNode: workflow.NewBaseNode("router", "", workflow.NodeConfig{})}
fin := workflow.NewFunctionNode("fin", func(_ agent.Context, in any) (any, error) { return in, nil }, workflow.NodeConfig{})
r := graphRunner3(t, []workflow.Edge{
{From: workflow.Start, To: w},
{From: w, To: g},
{From: g, To: rt},
{From: rt, To: w, Route: workflow.StringRoute("loop")},
{From: rt, To: fin, Route: workflow.StringRoute("finish")},
})
t1, _ := run3(t, r, userText("start"))
id1, _ := findLongRunningInterrupt(t1)
t2, err := run3(t, r, approve3(id1))
if err != nil {
t.Fatalf("first approval: %v", err)
}
id2, _ := findLongRunningInterrupt(t2)
if _, err := run3(t, r, approve3(id2)); err != nil {
t.Fatalf("second approval: %v", err)
}
if len(m.calls) != 4 {
t.Fatalf("gated model calls = %d, want 4", len(m.calls))
}
if !m.callSees(3, "INPUT-2") {
t.Errorf("request after the second approval does not contain INPUT-2")
}
}
// approve (handoff) -> gated. The gated node's first call sees the reply as
// its input. After the confirmation it must still see that input.
func TestResumeRound3_HandoffIntoGatedNodeKeepsInput(t *testing.T) {
m := &model3{}
g := gated3(t, "executor", m)
r := newWorkflowRunner(t, workflow.Chain(workflow.Start, newHitlAsker("approve", "ia", false), g))
t1, _ := run3(t, r, userText("start"))
_, name := findLongRunningInterrupt(t1)
t2, err := run3(t, r, resumeContent("ia", name, "go-PAYLOAD"))
if err != nil {
t.Fatalf("answering approve: %v", err)
}
id, _ := findLongRunningInterrupt(t2)
if _, err := run3(t, r, approve3(id)); err != nil {
t.Fatalf("approval: %v", err)
}
if len(m.calls) != 2 {
t.Fatalf("executor model calls = %d, want 2", len(m.calls))
}
if !m.callSees(0, "go-PAYLOAD") || !m.callSees(1, "go-PAYLOAD") {
t.Errorf("go-PAYLOAD in first call = %v, in resume call = %v; want true, true", m.callSees(0, "go-PAYLOAD"), m.callSees(1, "go-PAYLOAD"))
}
if m.callSees(1, "adk_request_input") {
t.Errorf("resume call contains the approve node's adk_request_input call")
}
}
// A handoff asker with no successor. Once answered, a duplicate or a
// different reply must be rejected.
func TestResumeRound3_TerminalHandoffReplyRejected(t *testing.T) {
r := newWorkflowRunner(t, workflow.Chain(workflow.Start, newHitlAsker("ask", "q", false)))
t1, _ := run3(t, r, userText("start"))
id, name := findLongRunningInterrupt(t1)
if _, err := run3(t, r, resumeContent(id, name, "ok")); err != nil {
t.Fatalf("resume: %v", err)
}
_, dup := run3(t, r, resumeContent(id, name, "ok"))
_, changed := run3(t, r, resumeContent(id, name, "changed"))
if dup == nil || changed == nil {
t.Errorf("duplicate reply err = %v, changed reply err = %v; want ErrNothingToResume for both", dup, changed)
}
}Output on this branch:
resume_round3_test.go:155: asker resumed 2 times, want 1 (an unrelated reply re-ran it)
--- FAIL: TestResumeRound3_ReentryNodeWithoutOutputConsumesReply
resume_round3_test.go:218: request after the second approval does not contain INPUT-2
--- FAIL: TestResumeRound3_SingleTurnResumeInLoopKeepsInput
resume_round3_test.go:242: go-PAYLOAD in first call = true, in resume call = false; want true, true
resume_round3_test.go:245: resume call contains the approve node's adk_request_input call
--- FAIL: TestResumeRound3_HandoffIntoGatedNodeKeepsInput
resume_round3_test.go:261: duplicate reply err = <nil>, changed reply err = <nil>; want ErrNothingToResume for both
--- FAIL: TestResumeRound3_TerminalHandoffReplyRejected
The AgentNode default
The narrowed AgentNode default matches adk-python for LlmAgent. Maintainers still need to decide the remote A2A case, which is on the line.
# Conflicts: # internal/llminternal/contents_processor.go
|
Addressed the three review-body items in
The Verification:
|
|
@wolo-lab this needs a design decision from you before I review it again.
The question is whether a persisted field on @Harbor404, nothing is needed from you until this is decided. I have reproductions for all four failures and will post them with the outcome, since which of them still apply depends on the direction. |
|
Gentle check-in on this one. I know the resume-design question went to @wolo-lab on 2026-10-13, and I don't want to push code in a direction that isn't settled — so the branch is parked on purpose. If a decision lands, I'll re-sync with No action needed if it's simply still queued; happy to wait. |
Linked issue
Closes: #1658
Problem
A graph
AgentNodepaused on a non-adk_request_inputlong-running tool call such asadk_request_confirmationcould not be resumed by its matchingFunctionResponse. Resume detection filtered on the function name, so the follow-up turn started a freshWorkflow.Run, replayed completed upstream nodes, and never re-entered the paused agent/tool.Solution
adk_request_inputstale-reply diagnostic.AgentNodewrapping anLlmAgenttoRerunOnResume=trueunless the caller explicitly configured it.internal/utils.UnwrapResponse, preserving multi-key confirmation maps and recognizing{"result": v}.Behavior change
Tool confirmation and consent replies now resume the paused graph node in place. Already-completed upstream nodes do not run again, approved tools execute, denied tools do not, and exact duplicate replies do not replay side effects.
Testing Plan
Unit Tests:
Commands run:
GOTOOLCHAIN=go1.26.6 go test -race -mod=readonly -count=1 ./agent/workflowagent ./workflow ./runner ./tool/functiontool ./internal/utils ./agent/llmagent ./internal/llminternal ./internal/configurablego test -race -mod=readonly -count=1 -shuffle=on workgo build -mod=readonly workGOTOOLCHAIN=go1.26.6 golangci-lint runin the root andplugin/agentanalytics: 0 issuesgo mod tidy -diffin both modules: cleanWith your source change reverted and your tests kept, which test fails?
TestRunner_WorkflowAgentNodeConfirmationRepliesResumePausedGraphfails on the baseline for both confirm and deny: upstream runs twice and the approved tool never executes.TestRunner_WorkflowAgentNodeDuplicateConfirmationReplyIsNoopfails because the duplicate replay starts a fresh graph and runs the approved tool twice.TestRunner_WorkflowAgentNodeResumeCancellationDoesNotRestartGraphfails because a cancelled resume replays the completed upstream node.TestDecodeWorkflowInputResponsefails for the multi-key confirmation map and{"result": v}shapes.The new regression tests cover confirm, deny, input reply, paused node state, duplicate reply, and cancellation.
Manual End-to-End (E2E) Tests:
No live model or cloud call is required; these are full Runner + WorkflowAgent + in-memory session tests with real
functiontoolconfirmation handling and a scripted model.Checklist
Additional context
The implementation intentionally keeps generic
AgentNodebehavior as handoff; the LlmAgent default mirrors adk-python'sbuild_nodebehavior for agent nodes.