Skip to content

[fix] Refuse agent configurations the runtime cannot run - #8341

Open
mmabrouk wants to merge 5 commits into
release/v0.128.14from
feat/agent-save-checks
Open

mmabrouk wants to merge 5 commits into
release/v0.128.14from
feat/agent-save-checks

Conversation

@mmabrouk

@mmabrouk mmabrouk commented Nov 17, 2026 •

Copy link
Copy Markdown
Member

Context

An agent could be saved with a configuration that the runtime cannot run. For example, a skill without a body saved with a 200, and after that every run of the agent failed with a 500 and an error that named no field. The save path checked only a few fields (the harness kind, the instructions and the model connection), while the runtime parses the whole agent template with AgentTemplate.from_params before every run.

Two smaller problems sat next to it. The agenta_tools entry in tools has no name, so the operations engine could not address it, and turning on one Agenta tool meant replacing the whole tools list. The default agent template also had no skills key, so the first skill had to be written differently from every later one.

These fixes come from pull request 7409. They do not depend on the cross-agent tools, so they ship on their own first.

Changes

Refuse a configuration the runtime cannot run, on every agent save. A new rule, _reject_unrunnable_agent in api/oss/src/core/workflows/service.py, runs the runtime's own parse on the configuration that would be stored. It replaces the narrow harness check and runs on every write path: the revision build that every commit goes through, and the simple create and edit paths of workflows, applications and evaluators, before their first write. The rule reads the result, not the change, so an agent stored before this rule with a broken field refuses other changes until the same change fixes that field. Parts that hold an embed are skipped, because embeds resolve only before a run.

The refusal is a 422 with the usual agent error envelope, and it names each field:

{
  "code": "invalid_agent_configuration",
  "message": "This agent's setup is not valid, so it cannot run.",
  "retryable": false,
  "next_step": "Correct the fields listed in `details.issues` and send the commit again. ...",
  "details": {"issues": ["parameters.agent.skills[0].body is required"]}
}

To name the field, the SDK parsers of skills, MCP servers and tools now raise one shared error type, ConfigEntryError in sdks/python/agenta/sdk/agents/config_errors.py. It carries the entry's position and the problems pydantic found inside it. The workflow, evaluator and template loader routes map the refusal to 422, and the agent's own commit_revision tool answers with the same envelope. The harness kind refusal (invalid_harness_kind) keeps its code and shape.

Key the agenta_tools entry by its type. In the operations engine (change_set.py), a tools entry whose type allows one entry per list is keyed by that type. The set of such types comes from the SDK (SINGLE_ENTRY_TOOL_TYPES, today only agenta_tools).

Before, the only way to turn on one Agenta tool was to set the whole tools list. After:

{"operation": "set",
 "target": ["parameters", "agent", {"list": "tools", "key": "agenta_tools"}, "tools", "create_schedule"],
 "value": "allow"}

Seed skills: [] in the default agent template (build_agent_v0_default in the SDK), so the first skill is an add_item like every later one.

Not release-ready on its own

Not release-ready on its own. This PR is one part of the builder agent. It becomes release-ready only together with the builder agent's UI and its first behavior, which other PRs build. Do not merge it alone.

The PR also adds the builder agent's OpenSpec change (openspec/changes/builder-agent/), the approved design for the whole program.

Tests

  • API: new test_commit_agent_template_validation.py covers the refusal on every write path, the field names in the issues, embeds, the bare and prompt template shapes, a broken stored agent, and the 422 mapping on each route. test_commit_harness_validation.py now runs through the new rule. test_change_set.py covers the keyed agenta_tools entry. test_config_handlers.py covers the commit_revision answer.
  • SDK: test_config_entry_errors.py covers the shared entry error. A tool entry whose type is not text is now refused as an unsupported shape instead of raising a TypeError.
  • Services: test_default_agent_template.py checks the seeded skills: [].
  • Unit suites run locally: API, SDK and services.

How to review

  1. sdks/python/agenta/sdk/agents/config_errors.py, then the small changes in the skill, MCP and tool parsers that raise it.
  2. _reject_unrunnable_agent in api/oss/src/core/workflows/service.py, and its call sites. The simple workflow and application edit paths now read the stored data before the artifact write, so a refused edit writes nothing.
  3. InvalidAgentConfigurationError in types.py and the route mappings.
  4. _tool_name in change_set.py.

Agenta Product Agent added 2 commits November 17, 2026 18:03
Every agent save now runs the runtime's own AgentTemplate.from_params on the
configuration it would store, and names each refused field. The change-set
engine keys a nameless single-entry tools item (agenta_tools) by its type, and
the default agent template seeds skills: [].

Split out of the cross-agent tools PR (#7409).
@coderabbitai

coderabbitai Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Explain Change →

No actionable comments were generated in the recent review.

Summary

Summary by CodeRabbit

  • Bug Fixes
    • Invalid agent configurations are rejected before workflows, evaluators, and applications are saved, with details identifying affected fields.
    • Editing an application with an invalid stored revision no longer proceeds with the write.
    • Errors for skills, tools, and MCP entries now provide clearer entry and field details.
    • Configuration errors returned by workflow, evaluator, application, and revision-commit actions include structured details.
  • New Features
    • Eligible tool entries can be targeted by type; duplicate entries are rejected where only one is allowed.
    • New agent defaults include an empty skills list.
  • Documentation
    • Added specifications for a proposed project builder agent and cross-agent tools.

The changes add runtime agent-configuration validation across workflow, application, and evaluator writes. They add structured SDK configuration errors and type-based addressing for single-entry tools. OpenSpec documents describe a proposed platform builder agent and cross-agent tools.

Changes: Agent validation and tool behavior

Layer / File(s) Summary
Structured SDK configuration errors
sdks/python/agenta/sdk/agents/config_errors.py, sdks/python/agenta/sdk/agents/{skills,mcp,tools}/*, sdks/python/oss/tests/pytest/unit/agents/test_config_entry_errors.py
SDK parsing now preserves structured validation issues for skill, MCP, and tool entries. Error descriptions include entry indexes and field paths.
Agent validation before writes
api/oss/src/core/workflows/*, api/oss/src/core/{applications,evaluators}/service.py, api/oss/src/apis/fastapi/{workflows,evaluators,agent_templates}/*, api/oss/src/core/tools/platform_handlers.py, api/oss/tests/pytest/unit/workflows/*, api/oss/tests/pytest/unit/tools/test_config_handlers.py
Workflow validation uses the runtime agent-template parser before relevant writes, including edits that reuse stored revision data. API handlers map configuration, harness, and instructions errors to HTTP 422 responses; the platform commit handler returns configuration failures as failed results.
Single-entry tool addressing
sdks/python/agenta/sdk/agents/tools/*, api/oss/src/core/workflows/change_set.py, api/oss/tests/pytest/unit/workflows/test_change_set.py, api/oss/tests/pytest/unit/workflows/test_commit_wrapper.py
agenta_tools entries are keyed by type in change sets. Tests cover nested operations, duplicate rejection, and selector normalization.
Default agent template shape
sdks/python/agenta/sdk/utils/types.py, services/oss/tests/pytest/unit/agent/test_default_agent_template.py
The default agent template now includes an empty skills list when no skill reference is supplied.

Builder-agent specifications

Layer / File(s) Summary
Builder-agent proposal and design
openspec/changes/builder-agent/{proposal.md,design.md,tasks.md}
The documents describe the proposed builder agent, its scope and permissions, its shared learning drive, and planned work items.
Builder-agent and cross-agent requirements
openspec/changes/builder-agent/specs/*
The specifications define builder-agent resolution and access, tool approvals, learning-drive behavior, release updates, and cross-agent tool permissions.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant WorkflowService
  participant AgentTemplate
  participant WorkflowErrorHandler
  participant APIClient
  WorkflowService->>AgentTemplate: Parse agent configuration
  AgentTemplate-->>WorkflowService: Return configuration issues
  WorkflowService->>WorkflowErrorHandler: Raise mapped validation error
  WorkflowErrorHandler-->>APIClient: Return HTTP 422 details
Loading

Possibly related PRs

  • Agenta-AI/agenta#7409: Changes the agent-save path, configuration validation, agenta_tools addressing, and the default skills list.
Merge Risk: Low · up to cd5a3

This change refuses agent configurations the runtime cannot run and lets Agenta tools be edited by type. Known follow-ups remain: simple metadata edits can overwrite a concurrent configuration commit, and the builder-agent spec leaves privacy rules for the shared learning drive undefined. These are bounded and mostly pre-existing or documentation-level. The PR is mergeable with owner awareness, though the author notes it should ship together with the builder agent UI.

Pre-merge checks: Passed 4

Check name Status Explanation
Title check Passed The title "[fix] Refuse agent configurations the runtime cannot run" matches the main change: a new save-time validation rule (_reject_unrunnable_agent) that runs the runtime's AgentTemplate parser an…
Description check Passed The description is closely related to the changeset. It explains the problem (agents saved with configurations the runtime cannot run), the new _reject_unrunnable_agent rule and its call sites, the Co…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Title check

Explanation

The title "[fix] Refuse agent configurations the runtime cannot run" matches the main change: a new save-time validation rule (_reject_unrunnable_agent) that runs the runtime's AgentTemplate parser and returns a 422 with an invalid_agent_configuration envelope before writes. The title is concise and specific, and it does not mention the secondary changes (agenta_tools keying, skills seeding, OpenSpec docs), which is acceptable.

Full details: Description check

Explanation

The description is closely related to the changeset. It explains the problem (agents saved with configurations the runtime cannot run), the new _reject_unrunnable_agent rule and its call sites, the ConfigEntryError shared type, the agenta_tools keying in change_set.py, the skills: [] seeding in the default template, the 422 envelope, and the tests added. It also states the PR is not release-ready on its own.

Finishing Touches

Generate unit tests (beta)

  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

Recent review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 4f55c0f and cd5a3e2.

Files selected for processing: 3

  • api/oss/src/core/workflows/service.py
  • api/oss/tests/pytest/unit/workflows/test_commit_agent_template_validation.py
  • sdks/python/agenta/sdk/agents/tools/compat.py

Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 3 remain after this review.

devin-ai-integration[bot]

This comment was marked as resolved.

… guard a non-text tool type

Review round 1: the commit_revision tool maps invalid_agent_configuration to
its envelope instead of a 500, the template loader maps the connection
refusal too, and a tool entry whose type is not text has no key instead of
raising.
coderabbitai[bot]

This comment was marked as resolved.

@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Railway Preview Environment

Item Value
Status stopped
Commit cd5a3e269063bdccd4f2577bf5fdddefaca38c31
Manual expiry (UTC) Not granted
CI deadline (UTC) Not granted
Workflow https://github.com/Agenta-AI/agenta/actions/runs/44145367254

Dependent tests finished. Post /preview for manual review. Environment absence verified; retained storage requires separate provider accounting.

Post /preview as a new comment to request a one-hour preview. Repeating it while active does not extend the hour.
Preview data is disposable. Expiry is enforced by the next successful cleanup sweep, not an exact timer.
Environment deletion stops its compute. Provider-retained storage and permanent services may still be billed.

Review round 2: the runtime parse raised TypeError on {"type": []}, so the
save answered 500. The tool parser and the legacy gateway check now treat a
non-text type as an unsupported shape.
coderabbitai[bot]

This comment was marked as resolved.

@mmabrouk

Copy link
Copy Markdown
Member Author

🤖 The AI agent says:

Review loop result for head 4f55c0f69b447962622d776eff04f0f6e5af3693 (base dd913fc5d062422775730fedfbe1470c61261949), 2 rounds.

Lens Final rating
Simplify Low
Slop Low

Fixed: the commit_revision tool now answers invalid_agent_configuration with its envelope instead of a 500. The comment on the parse except now says that the parse's ValueError is a configuration refusal.
Left on purpose: an embed anywhere inside a top-level template field (for example harness.extras) skips the parse of that whole field. The rule documents this, no product path writes an embed inside harness, and checking part of a field gives false refusals (an embedded agents_md leaves an empty object). The runtime still refuses such a field at run time.

@mmabrouk

Copy link
Copy Markdown
Member Author

🤖 The AI agent says:

Review loop result for head 4f55c0f69b447962622d776eff04f0f6e5af3693 (base dd913fc5d062422775730fedfbe1470c61261949), 3 rounds.

Lens Final rating
Correctness Medium
Security Low

Fixed: commit_revision maps the new refusal; the template loader maps the connection refusal; a tool entry whose type is not text is refused as an unsupported shape instead of raising (_tool_name, coerce_tool_config, _is_legacy_gateway_entry).
Left after the three-round cap: some malformed values still make the runtime parser raise a type that is not a configuration error, for example parameters.prompt.llm_config as a string (AttributeError in _parse_agent_fields). Such a save answers 500 instead of 422. Nothing is written, and the same value already fails every run today. A follow-up should harden the parser against wrong types in one place rather than one case per review round.

@mmabrouk

Copy link
Copy Markdown
Member Author

🤖 The AI agent says:

Review loop result for head 4f55c0f69b447962622d776eff04f0f6e5af3693 (base dd913fc5d062422775730fedfbe1470c61261949), 2 rounds.

Lens Final rating
Naming Low

Fixed: the template loader maps InvalidAgentConnectionError like its three siblings, and commit_revision maps InvalidAgentConfigurationError.
Left on purpose: details.issues stays a list of strings, the shape final_validation_failed already uses. A shared base for the four agent configuration errors, and dropping the underscore of _reject_unrunnable_agent (applications imported the underscore helper before this PR), are follow-ups.

…t tool type before the slug fallback

CodeRabbit: an empty @ag.embed skipped the save check, and a tool entry with
a non-text type and a gateway slug name parsed as a gateway tool.
@mmabrouk

Copy link
Copy Markdown
Member Author

🤖 The AI agent says:

Ship record for head cd5a3e2690. Tier: Program (one PR of the builder agent).

Gate Result Head Link
Spec Approved by Mahmoud on 2026-11-17 n/a openspec/changes/builder-agent/ on this branch
Design simplify review Not run in this PR; the spec was approved as written n/a n/a
Slop and simplicity loop Low 4f55c0f #8341 (comment)
Naming loop Low 4f55c0f #8341 (comment)
Correctness and security loop Security Low; correctness Medium after the three-round cap (malformed values that make the runtime parser raise a non-configuration error answer 500; listed in the record) 4f55c0f #8341 (comment)
Tests Passed in CI: API, SDK, services and runner unit and integration; locally: API, SDK and services unit suites cd5a3e2 https://github.com/Agenta-AI/agenta/pull/8341/checks
Live QA Not run: no live stack for this PR; the change is a save-time check covered by unit tests n/a n/a
Screenshots No UI change n/a n/a
CodeRabbit No open items cd5a3e2 #8341
CI Required checks green, web tests not awaited. tests / wait-for-readiness fails on the runner image ("Playwright does not support chromium on ubuntu26.04-x64"), the same on other open PRs (for example #8342), so the Railway API acceptance jobs were skipped. Entire Gates also fails on other open PRs. cd5a3e2 https://github.com/Agenta-AI/agenta/pull/8341/checks
Docs in sync No docs describe this cd5a3e2 n/a

Commits after the review-loop head: cd5a3e2 (CodeRabbit) checks a template behind an empty @ag.embed, and refuses a non-text tool type before the gateway slug fallback. Both have tests.

Deploy steps: none.
Not covered: live QA, and the Railway API acceptance jobs (skipped by the readiness failure above). Not release-ready on its own: it ships together with the builder agent's other PRs.

@mmabrouk

Copy link
Copy Markdown
Member Author

🤖 The AI agent says: Ready for a code review. This PR moves the agent-save checks out of #7409: every agent save runs the runtime's own parse and refuses an unrunnable configuration with a 422 that names the field, the agenta_tools entry is keyed by its type, and the default template seeds skills: []. One item is left after the review cap: some malformed values still make the parser raise a non-configuration error, which answers 500 (listed in the correctness record). It is not release-ready on its own; #7409 is stacked on it.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant