Skip to content

feat(cost): LLM gateways (LiteLLM, Agentgateway, Langfuse) - #987

Draft
induwara-yaala wants to merge 11 commits into
feature/970-cost-management-foundationfrom
feature/971-cost-management-llm-gateways
Draft

induwara-yaala wants to merge 11 commits into
feature/970-cost-management-foundationfrom
feature/971-cost-management-llm-gateways

Conversation

@induwara-yaala

@induwara-yaala induwara-yaala commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Description

Gateways for cost management (#954). This PR adds the LLMGateway seam and three built-in gateways: LiteLLM, Agentgateway and Langfuse. It also adds the LLMGatewayContract suite, which every gateway passes.

Nothing calls the gateways yet. RunMeterFactory still returns the no-op meter (#970), so behaviour does not change with or without a cost block. Spec: docs/specs/954-pluggable-cost-management/spec.md, §4–§5, deviations 23–27.

Each gateway was checked against vendor source rather than docs alone:

  • LiteLLM: the installed proxy at 1.105.7.
  • Agentgateway: v1.7.0 on GitHub, plus sqlx's SQLite encoding.
  • Langfuse: the installed SDK at 4.15.1 and the Langfuse server source.

Several of the spec's sketches turned out wrong. Those are recorded as deviations below.

Type of Change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation update
  • Refactoring (no functional changes)
  • Performance improvement
  • Test update
  • CI/CD update
  • Other (please describe):

Related Issues

Fixes #971
Relates to #969, #954
Depends on #970 (#986)

Changes Made

The commits follow the ticket's iterations; 2.5 spans three commits.

  • 2.1 Gateway seam (cost/gateway/{base,http,factory}.py, cost/testing.py):
    • LLMGateway base class:
      • Each optional operation raises UnsupportedCapabilityError unless the gateway declares that capability.
      • gateway_user_id / raw_user_id encode and decode ids (rule 5). Ids that already start with ak~ are also encoded, so decoding always gives back the original (deviation 23).
      • _next_batch is a shared helper that cuts one batch of rows after a cursor, exactly. It raises UsageTruncatedError rather than loop forever when more than a page shares one second.
    • GatewayHTTP:
      • Every failure becomes UsageRetrievalError; 401 and 403 set auth=True.
      • Logs carry the method and path only. httpx's own HTTP Request: line includes the query string, which carries user ids, so a filter strips the query from it for gateway URLs.
      • One client per event loop; a call from another loop gets a short-lived client of its own.
    • LLMGatewayFactory builds the three built-ins, or a bring-your-own gateway named by dotted path.
    • LLMGatewayContract runs against a fake gateway in test_cost_gateway.py.
  • 2.2 LiteLLM headers and is_budget_refusal:
    • Capabilities per api mode.
    • x-litellm-* headers. Agent tags use rule 5 instead of the spec's lossy ,→_.
    • is_budget_refusal reads both body shapes: the OpenAI SDK keeps only the inner error object as body (openai/_client.py:687), so the spec's body["error"]["type"] would miss it (deviation 24).
  • 2.3 LiteLLM usage reads:
    • /spend/logs/v2 paging. A total of 10 000 or more, or total_is_capped, raises UsageTruncatedError.
    • Rows are kept only inside [start, end), applied on the client because the route filters to the second.
    • fetch_usage_since uses the shared batch helper.
  • 2.4 LiteLLM native budgets and probe:
    • Budget rows are created with /budget/new and updated through /budget/update when they already exist. Customer rows use /customer/new and /customer/update the same way.
    • Any other client error (4xx) raises AKConfigError.
    • UsageRetrievalError.detail carries the response text, so "already exists" can be told apart from other 400s.
    • probe refuses LiteLLM below 1.111 in customer_management mode.
    • customer_management budget sync is deferred (deviation 25). Its 1.111 API (prerelease only) differs from the spec's sketch and can't be verified, so the mode no longer declares native_user_budget.
  • 2.5 Agentgateway (cost/gateway/agentgateway.py) and the new cost-agentgateway extra (psycopg):
    • Reads the request-log database read-only: SQLite through the standard library, Postgres through psycopg.
    • SQLite stores started_at with a 0/3/6/9-digit fraction (sqlx's encoding), which compares correctly as text only against whole seconds. SQLite reads therefore bound by whole seconds, filter exactly in Python, and compute total_spend from rows.
    • Postgres keeps the SQL aggregate, with cost::numeric in both reads, so totals equal row sums exactly.
    • Only model calls are read.
    • probe checks the migration version (deviation 26).
  • 2.6 Langfuse (cost/gateway/langfuse.py): reads go through v2 observations, following the vendor (deviation 27):
    • Filters go in a filter JSON array.
    • Results come newest first, so the adapter sorts them.
    • Time bounds are compared to the millisecond, so they are widened and rows filtered exactly.
    • An empty page that still carries a cursor isn't the end.
    • Only type = GENERATION rows are read.
    • The metrics API has no cached-token measure, so total_spend sums observation rows.
    • Only the 1 KiB per-value header limit is checked: four values can't reach 8 KiB.

Testing

  • Unit tests pass locally: full suite, 6985 passed, 0 failed.
  • Integration tests pass locally: the Agentgateway Postgres contract against a throwaway postgres:16 container. It is gated on AK_TEST_AGW_POSTGRES_URL and skipped in CI.
  • Manual testing completed
  • New tests added for changes:
    • New: test_cost_gateway.py, test_cost_gateway_litellm.py, test_cost_gateway_agentgateway.py and test_cost_langfuse_compat.py.
    • Extended: test_packaging_extras.py.
    • The contract runs against LiteLLM (both modes), Agentgateway (SQLite and Postgres) and Langfuse.
    • Agentgateway fixtures are the v1.7.0 migrations, copied verbatim (Apache-2.0, tests/fixtures/agentgateway/).
    • Langfuse fake responses are validated with the SDK's own ObservationsV2Response, and the query parameter names are checked against the SDK's request code.
    • For each iteration I broke the code on purpose (cursor ids ignored, no time bounds, no LLM-row filter, no "already exists" handling, no version check, and so on). Each break failed at least one test.

make lint-check-all passes. mypy reports nothing new beyond the type-abstract on resolve_dotted(..., base=LLMGateway), which identity/provider/factory.py already has.

Checklist

  • My code follows the project's style guidelines
  • I have performed a self-review of my code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • Any dependent changes have been merged and published

Additional Notes

@coderabbitai

coderabbitai Bot commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@induwara-yaala induwara-yaala left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Solid gateway seam that follows the house pattern (ABC + factory + dotted-path BYO + LLMGatewayContract, constructor-only settings). One correctness bug in LiteLLM's time bounds, plus a few smaller points.

  • Findings: 1 blocker, 3 suggestions, 1 nit (inline), plus the notes below
  • Spec test-table coverage: the promises in the gateway rows are covered: the Postgres extra, SQLite migration 3, read-only open, customer_management below 1.111, and is_budget_refusal on body, response.json() and message. The deferrals (deviation 25, #973, #974 and the live LiteLLM test) are stated in the description.
  • Spec (spec.md): deviations 23–27 are recorded and line up with the code. One factual premise fails against the vendor source: §5.1 says the /spend/logs/v2 "resolution is one second". At 1.105.7 the route compares full-precision startTime against a whole-second bound (spend_management_endpoints.py:2688-2691; see the inline comment).
  • Spec test table (spec.md ~L1718): the test_cost_gateway_litellm.py row still lists "the member upsert" (deferred by deviation 25) and "floored to the second". Both should be updated with the fix.
  • [nit] Classes, not scripts: the module-level helpers _second/_json_value (litellm), _floor/_sqlite_text (agentgateway) and _ceil_ms/_millis (langfuse) each serve one class. Per ak-dev-architecture House Pattern 3, make them @staticmethods on that class.
  • Docs and skills: deferred to #974, as the description says. No finding here.
  • Done well: the Agentgateway fixtures are the real v1.7.0 migrations on real SQLite. The Langfuse fakes are validated with the SDK's own ObservationsV2Response. The SQLite AutoSi handling and the Postgres cost::numeric parity are thought through.

CI: 84 checks passed, 2 skipped.

@induwara-yaala

Copy link
Copy Markdown
Contributor Author

Thanks for the review. All points are addressed in adc196c (code and tests) and 822ba0d (spec):

  • Blocker: LiteLLM end_date is now the next whole second. The fake proxy compares full timestamps, and a regression test covers the 12:00:00.400 / .700 case.
  • Suggestions:
    • The §5.1 bound wording and the test-table row are corrected.
    • langfuse-metadata values are reversible (ak~-quoted when they hold a colon). This changes that header's format.
    • Response text is kept out of AKConfigError.
  • Nits:
    • Currency is checked before any write.
    • The six module-level helpers are now static methods on their classes (_date_param/_json_value, _floor/_sqlite_text, _ceil_ms/_millis); the Agentgateway bounds helper became a classmethod.

The full suite passes locally (6986 passed, 0 failed) and make lint-check-all is clean.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant