Repository navigation
feat(cost): LLM gateways (LiteLLM, Agentgateway, Langfuse) - #987
Draft
induwara-yaala wants to merge 11 commits into
Draft
induwara-yaala wants to merge 11 commits into
induwara-yaala wants to merge 11 commits into
Conversation
…udget synchronization methods
Contributor
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. |
… retrieval and error handling
induwara-yaala
commented
Nov 19, 2026
induwara-yaala
left a comment
Contributor
Author
There was a problem hiding this comment.
Solid gateway seam that follows the house pattern (ABC + factory + dotted-path BYO + LLMGatewayContract, constructor-only settings). One correctness bug in LiteLLM's time bounds, plus a few smaller points.
- Findings: 1 blocker, 3 suggestions, 1 nit (inline), plus the notes below
- Spec test-table coverage: the promises in the gateway rows are covered: the Postgres extra, SQLite migration 3, read-only open,
customer_managementbelow 1.111, andis_budget_refusalon body,response.json()and message. The deferrals (deviation 25, #973, #974 and the live LiteLLM test) are stated in the description. - Spec (
spec.md): deviations 23–27 are recorded and line up with the code. One factual premise fails against the vendor source: §5.1 says the/spend/logs/v2"resolution is one second". At 1.105.7 the route compares full-precisionstartTimeagainst a whole-second bound (spend_management_endpoints.py:2688-2691; see the inline comment). - Spec test table (
spec.md~L1718): thetest_cost_gateway_litellm.pyrow still lists "the member upsert" (deferred by deviation 25) and "floored to the second". Both should be updated with the fix. - [nit] Classes, not scripts: the module-level helpers
_second/_json_value(litellm),_floor/_sqlite_text(agentgateway) and_ceil_ms/_millis(langfuse) each serve one class. Perak-dev-architectureHouse Pattern 3, make them@staticmethods on that class. - Docs and skills: deferred to #974, as the description says. No finding here.
- Done well: the Agentgateway fixtures are the real v1.7.0 migrations on real SQLite. The Langfuse fakes are validated with the SDK's own
ObservationsV2Response. The SQLite AutoSi handling and the Postgrescost::numericparity are thought through.
CI: 84 checks passed, 2 skipped.
…d readability and consistency
…for improved accuracy
Contributor
Author
|
Thanks for the review. All points are addressed in adc196c (code and tests) and 822ba0d (spec):
The full suite passes locally (6986 passed, 0 failed) and |
9 of 21 tasks
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Gateways for cost management (#954). This PR adds the
LLMGatewayseam and three built-in gateways: LiteLLM, Agentgateway and Langfuse. It also adds theLLMGatewayContractsuite, which every gateway passes.Nothing calls the gateways yet.
RunMeterFactorystill returns the no-op meter (#970), so behaviour does not change with or without acostblock. Spec:docs/specs/954-pluggable-cost-management/spec.md, §4–§5, deviations 23–27.Each gateway was checked against vendor source rather than docs alone:
Several of the spec's sketches turned out wrong. Those are recorded as deviations below.
Type of Change
Related Issues
Fixes #971
Relates to #969, #954
Depends on #970 (#986)
Changes Made
The commits follow the ticket's iterations; 2.5 spans three commits.
cost/gateway/{base,http,factory}.py,cost/testing.py):LLMGatewaybase class:UnsupportedCapabilityErrorunless the gateway declares that capability.gateway_user_id/raw_user_idencode and decode ids (rule 5). Ids that already start withak~are also encoded, so decoding always gives back the original (deviation 23)._next_batchis a shared helper that cuts one batch of rows after a cursor, exactly. It raisesUsageTruncatedErrorrather than loop forever when more than a page shares one second.GatewayHTTP:UsageRetrievalError; 401 and 403 setauth=True.HTTP Request:line includes the query string, which carries user ids, so a filter strips the query from it for gateway URLs.LLMGatewayFactorybuilds the three built-ins, or a bring-your-own gateway named by dotted path.LLMGatewayContractruns against a fake gateway intest_cost_gateway.py.is_budget_refusal:apimode.x-litellm-*headers. Agent tags use rule 5 instead of the spec's lossy,→_.is_budget_refusalreads both body shapes: the OpenAI SDK keeps only the innererrorobject asbody(openai/_client.py:687), so the spec'sbody["error"]["type"]would miss it (deviation 24)./spend/logs/v2paging. Atotalof 10 000 or more, ortotal_is_capped, raisesUsageTruncatedError.[start, end), applied on the client because the route filters to the second.fetch_usage_sinceuses the shared batch helper.probe:/budget/newand updated through/budget/updatewhen they already exist. Customer rows use/customer/newand/customer/updatethe same way.AKConfigError.UsageRetrievalError.detailcarries the response text, so "already exists" can be told apart from other 400s.proberefuses LiteLLM below 1.111 incustomer_managementmode.customer_managementbudget sync is deferred (deviation 25). Its 1.111 API (prerelease only) differs from the spec's sketch and can't be verified, so the mode no longer declaresnative_user_budget.cost/gateway/agentgateway.py) and the newcost-agentgatewayextra (psycopg):psycopg.started_atwith a 0/3/6/9-digit fraction (sqlx's encoding), which compares correctly as text only against whole seconds. SQLite reads therefore bound by whole seconds, filter exactly in Python, and computetotal_spendfrom rows.cost::numericin both reads, so totals equal row sums exactly.probechecks the migration version (deviation 26).cost/gateway/langfuse.py): reads go through v2 observations, following the vendor (deviation 27):filterJSON array.type = GENERATIONrows are read.total_spendsums observation rows.Testing
postgres:16container. It is gated onAK_TEST_AGW_POSTGRES_URLand skipped in CI.test_cost_gateway.py,test_cost_gateway_litellm.py,test_cost_gateway_agentgateway.pyandtest_cost_langfuse_compat.py.test_packaging_extras.py.tests/fixtures/agentgateway/).ObservationsV2Response, and the query parameter names are checked against the SDK's request code.make lint-check-allpasses. mypy reports nothing new beyond thetype-abstractonresolve_dotted(..., base=LLMGateway), whichidentity/provider/factory.pyalready has.Checklist
Additional Notes
feature/970-cost-management-foundation, so the diff shows only [TASK] Cost management: LLM gateways (LiteLLM, Agentgateway, Langfuse) #971. Retarget tofeature/954-pluggable-cost-managementonce feat(cost): cost management foundation and core changes #986 merges.customer_managementmode now only attributes calls. TheCostManagercapability check must refuse abudgetsblock with it.UsageRetrievalError.test_cost_litellm_live.py(planned) should confirm them.usageDetailsis unconfirmed. It is the first of three candidate keys, marked with ashortcut:comment.langfuse-tagsandlangfuse-metadatainto observation fields.