Skip to content

feat(tools): add confidence-aware route selector for Jev Choice results - #9437

Open
Rosman Cariño (Rosman-Carino) wants to merge 1 commit into
crewAIInc:mainfrom
Rosman-Carino:feat/jev-route-selector
Open

Rosman Cariño (Rosman-Carino) wants to merge 1 commit into
crewAIInc:mainfrom
Rosman-Carino:feat/jev-route-selector

Conversation

@Rosman-Carino

@Rosman-Carino Rosman Cariño (Rosman-Carino) commented Nov 14, 2026 •

Copy link
Copy Markdown

Related issue

Fixes #8067
Part of #8065

Summary

Adds select_route, a small pure function in
crewai_tools.tools.jev_decision_tool.route_selector that maps a Jev Choice
answer (label + TypeSafe confidence) onto a caller-provided set of Flow route labels.

from crewai_tools.tools.jev_decision_tool.route_selector import select_route

@router(classify)
def route(self, decision: dict[str, Any]) -> str:
    return select_route(
        decision.get("choice"),
        decision.get("confidence"),
        allowed_routes=("refund", "tech_support", "triage"),
        fallback="triage",
        min_confidence=0.8,
    )
  • A known label whose confidence is ≥ min_confidence selects that route.
  • Low confidence, unknown/missing/non-string label, or missing/NaN/±inf/out-of-range/
    non-numeric confidence selects the explicit fallback. A malformed payload never
    raises from inside a router.
  • Invalid configuration (a bare string as allowed_routes, fallback not in routes —
    which also covers empty routes — or threshold not a number in [0, 1]) raises
    ValueError, checked before the answer so misconfiguration can never hide behind
    a fallback.
  • Confidence is used exactly as TypeSafe reports it; nothing is recomputed.

Three new files (291 lines), nothing modified. No new dependency; stdlib only.
The example and Flow test consume dicts shaped like JevDecisionTool's result in #8076
({"success", "choice", "probabilities", "confidence"}), so the two compose directly.

Design notes

  • Config args are keyword-only so allowed_routes / fallback / min_confidence
    can't be transposed.
  • min_confidence is required, no default: the issue calls the threshold
    "configurable" and names none, and a routing cutoff should be an explicit choice.
    Happy to add a default if preferred.
  • bool is rejected as a confidence/threshold (True == 1 would read as 1.0).
  • isinstance(choice, str) both narrows the type for strict mypy and keeps an
    unhashable payload from raising against a set of routes.
  • The router takes the previous step's return value as its argument (not
    self.state); the Flow test asserts the confident path actually reaches the
    refund listener, not just that the fallback path works.
  • Package dir jev_decision_tool/ matches feat(tools): add jev decision tool #8076 so the two PRs compose. Both PRs add
    jev_decision_tool/__init__.py; ours is empty so the add/add conflict for whichever
    merges second is trivial (take feat(tools): add jev decision tool #8076's). Happy to rebase onto feat(tools): add jev decision tool #8076 if it lands first.
  • Imported from its own module; no top-level crewai_tools re-export and no alias.
    The package __init__ re-exports tool classes and their config/schema types, not
    helper functions, and this helper is internal to the Jev integration rather than a
    tool. (JevDecisionTool in feat(tools): add jev decision tool #8076 is the public surface.)

Verification

  • Tests added or updated for the changed behavior
  • Relevant tests and quality checks pass locally

28 behavioral tests, no credentials or network: below/equal/above/just-below threshold,
inclusive bounds, int values, unknown/None/non-string labels,
None/NaN/inf/out-of-range/bool/string confidence, each invalid-config path, config
validated before the answer for every config error, and two real Flow runs
through a @start → @router → @listen chain with synthetic decisions shaped like
JevDecisionTool's result (including its {"success": False, "error": ...} form).

  • uv run pytest lib/crewai-tools/tests/tools/test_jev_route_selector.py → 28 passed on Python 3.10–3.13
  • uv run mypy lib/crewai-tools/src/crewai_tools/tools/jev_decision_tool/ → clean (strict)
  • uv run ruff check / ruff format --check / pre-commit run → clean

Additional context

Docs. No docs/edge/ page added: the helper has no user-facing tool surface, the
Jev tool (#8076) has no page yet to hang a section on, and comparable helper modules
in crewai-tools are undocumented. Happy to add one (with ar/ko/pt-BR sync) if
maintainers want it.

Relationship to #8156. #8156 (open) addresses the same issue from a separate
jev_tool/ package and edits the two package __init__s to re-export the helper.
This PR is the assignee's implementation; it is written to compose with #8076's
package and modifies nothing existing. Happy to defer to maintainers on which to
take forward.

🤖 Generated with Claude Code


Note

Low Risk
New isolated helper and tests under jev_decision_tool; no changes to existing tools or runtime paths.

Overview
Adds a select_route helper under jev_decision_tool/route_selector so Flow @router steps can turn Jev Choice outputs (choice + TypeSafe confidence) into a route name with an explicit confidence floor and safe fallback.

Callers pass allowed_routes, a fallback, and required min_confidence. Known labels at or above the threshold return that route; bad or low-trust inputs (missing/wrong type, unknown label, invalid confidence including NaN/bool) return fallback without raising. Misconfiguration (allowed_routes as a bare string, fallback not in routes, invalid threshold) raises ValueError before evaluating the decision.

Adds 28 behavioral tests, including a small Flow (@start → @router → @listen) that mirrors JevDecisionTool-shaped dicts (success and error payloads).

Reviewed by Cursor Bugbot for commit 1e742c4. Bugbot is set up for automated code reviews on this repo. Configure here.

@strix-security

strix-security Bot commented Nov 14, 2026 •

Copy link
Copy Markdown

Strix Security Review

✅ No open security findings.

PR summary: Adds a new `crewai_tools.tools.jev_decision_tool.route_selector` module containing a small pure function, `select_route`, that maps a Jev Choice label plus its confidence onto one of a caller-supplied set of Flow route labels, falling back when the label or confidence cannot be trusted. It also adds an empty package `__init__.py` and a test module covering threshold, payload and configuration behaviour plus two Flow runs.

Review summary

Reviewed the three added files: route_selector.py (the select_route helper), the empty package __init__.py, and the behavioural test module. The helper is pure and sink-free — it imports only collections.abc.Collection, performs string/float comparisons, and has no filesystem, network, subprocess, SQL, template, serialization or logging surface, so there is no injection, SSRF, path, deserialization or secret-handling path to trace. Its return value is constrained to a member of the caller-supplied allowed_routes (or the fallback, which is validated to be a member), so a hostile Choice payload or label cannot produce a route the caller did not allow; malformed confidence values (None, NaN, infinities, out-of-range, bool, numeric strings) and non-string labels fall back instead of raising, and misconfiguration raises before the answer is examined. I also considered the trust assumption that the confidence value originates from an upstream classifier whose output drives branch selection, and found no exploitability introduced here: the classification decision precedes the helper, and the helper can only map it onto a route label the caller already registered. No production code in the repository imports the new module yet, and nothing in the existing tree is modified or newly exposed.

Review 1 on this PR · 1e742c4 · requested Nov 14 07:39 UTC when the PR was opened


Reviewed by Strix
Re-run review · Configure security review settings · Try a deeper review with @strix-security /deep

@Rosman-Carino

Copy link
Copy Markdown
Author

This PR was written with AI assistance (Claude Code). Per CONTRIBUTING.md it needs the llm-generated label, but I don't have permission to add labels — could a maintainer apply it? Thanks.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[FEATURE] Add a confidence-aware Python route selector for Jev Choice results

1 participant