Repository navigation
Conversation
PinnedTarget.pin() rewrote the URL to resolved_ips[0] and never tried the remaining validated addresses. IBM#6965 replaced httpx's own multi-address connect with that single dial, so an outbound request fails whenever the peer listens on only one of the address families the hostname resolves to. localhost is the easy reproduction: it resolves to ::1 first on macOS and most Debian/Ubuntu hosts, while a provider that binds 127.0.0.1 refuses the ::1 attempt and the request never gets a second chance. Every pin() caller had the behaviour: OAuth token exchange, the LLM proxy (non-streaming and streaming), A2A cross-gateway calls, and the SSE health check in gateway_service. SniPinningTransport already loops over pinned addresses, so the module disagreed with itself. Add connect_async() and stream_async() to PinnedTarget. They use the caller's existing httpx client, try each validated address in turn, and retry on ConnectError and ConnectTimeout. Both keep the validated authority in the Host header through apply_headers(). Move the five call sites onto them and mark pin() deprecated, since it dials a single address. The helpers dispatch to client.post / client.get so caller stubs and instrumentation keep working. Verified against a purpose-built IPv6-only peer (::1:8077, 127.0.0.1 refused): both helpers fail over from IPv4 to IPv6 and return 200 with Host: localhost:8077. Closes IBM#7407 Signed-off-by: Andrin Enz <Andrin.Enz@ibm.com>
Local port of upstream PR IBM#7409 (IBM/mcp-context-forge), which fixes IBM#7407. PinnedTarget.pin() rewrote the URL to resolved_ips[0] and never tried the remaining validated addresses. IBM#6965 replaced httpx's own multi-address connect with that single dial, so an outbound request fails whenever the peer listens on only one of the address families the hostname resolves to. localhost is the easy reproduction: it resolves to ::1 first on macOS and most Debian/Ubuntu hosts, while a provider bound to 127.0.0.1 refuses the ::1 attempt and the request never gets a second chance. Every pin() caller had the behaviour: OAuth token exchange, the LLM proxy (non-streaming and streaming), A2A cross-gateway calls, and the SSE health check in gateway_service. Add connect_async() and stream_async() to PinnedTarget. They use the caller's existing httpx client, try each validated address in turn, and retry on ConnectError and ConnectTimeout. Both keep the validated authority in the Host header through apply_headers(). Move the five call sites onto them and mark pin() deprecated, since it dials a single address. Applied by hand around the local Cohere streaming branches in llm_proxy_service.py rather than merged, so this fork's ollama and cohere work stays intact. Verified against a purpose-built IPv6-only peer (::1:8077, 127.0.0.1 refused): both helpers fail over from IPv4 to IPv6 and return 200 with Host: localhost:8077. Closes IBM#7407 Signed-off-by: Andrin Enz <Andrin.Enz@ibm.com>
The helpers dialled the validated address as an IP literal but dropped extensions=pinned_target.extensions. httpcore derives the TLS server name from that extension and falls back to the origin host, so verification ran against the address rather than the validated hostname. Any CA-issued certificate without an IP SAN then failed with an IP-address mismatch. Because ConnectError was treated as "try the next address", that TLS failure was also retried per address and surfaced as an unreachable peer, which hid the real cause. Pass self.extensions on every pinned attempt, in connect_async through a shared _dial_pinned and directly in stream_async. Raise at once when the failure is a certificate verification error instead of retrying. Restore the three assertions that recorded the sni_hostname extension. Flipping them to None codified the regression, and MockTransport performs no TLS so nothing else could catch it. Add a real handshake test against a certificate with only a DNS SAN, plus a test pinning sni_hostname across a retry and one asserting a TLS failure is not retried. Both new tests fail with an IP-address mismatch when the extension is removed. Reported-by: madhu-mohan-jaishankar Signed-off-by: Andrin Enz <89848517+ScarAnd@users.noreply.github.com>
msureshkumar88
left a comment
There was a problem hiding this comment.
Thank you for the fix. The root cause is correct, the five call sites move cleanly, and the second commit caught a real TLS regression with a real-handshake test. I request changes for two items before merge.
I checked out head 6f6d95f1c and ran tests/security/test_outbound_dns_pinning.py: 85 passed. Coverage of mcpgateway/utils/ssrf_pinning.py is 94%. The missed lines are 191, 338-340 and 350-354.
severity | path:line | problem | fix
blocking | mcpgateway/utils/ssrf_pinning.py:346-354 | No test exercises stream_async. Lines 350-354 never run, so the rule "a certificate verification failure is not retried" (CWE-295) has no test for streaming. | Add stream_async tests: IPv4 to IPv6 failover, IPv6 to IPv4 failover, all refused, TLS failure not retried, sni_hostname and Host on every attempt, unpinned pass-through.
functionally-impacting | mcpgateway/utils/ssrf_pinning.py:344-352 | The dial and the caller's body share one try. If the body raises ConnectError or ConnectTimeout, the request goes to the next address, then contextlib raises "RuntimeError: generator didn't stop after athrow()". A POST replays. I reproduced this with an injected exception. The two current call sites do not raise these in the body, so the defect is latent. | Enter client.stream(...) inside the try and yield outside it, for example with AsyncExitStack.
functionally-impacting | a2a_protocol.py:209, dcr_service.py:116, openapi_service.py:199, tool_service.py:5847 | These paths pin to resolved_ips[0] (validators.py:1736) and never retry. REST tool and A2A invocation on a dual-stack localhost name fail as in #7407. | Migrate them to the helpers, or state the exclusion in the PR body and open a follow-up issue.
functionally-impacting | tests/live_gateway/ | AGENTS.md requires a black-box test against a running gateway. The PR reports a manual OAuth callback check only. | Add a live test: IPv4-only token endpoint behind a dual-stack localhost name, authorization-code flow returns the token.
functionally-impacting | mcpgateway/utils/ssrf_pinning.py:1-22 | The pool-sharing constraint names only pin(). connect_async and stream_async also rewrite the origin to an IP and accept any client (CWE-295). | Extend the constraint to all three entry points and repeat it in both helper docstrings.
suggestion | mcpgateway/utils/ssrf_pinning.py:305,354 | Both fallback ConnectError branches are unreachable, because is_pinned guarantees one address. | Delete them.
suggestion | mcpgateway/utils/ssrf_pinning.py:160-170,194-212 | pinned_addresses is an alias of resolved_ips. pin() has no caller in mcpgateway/ and no runtime deprecation signal. | Delete both, or add warnings.warn(DeprecationWarning) to pin().
suggestion | mcpgateway/utils/ssrf_pinning.py:296-305 | Only the last error reaches the caller. The OAuth log says "after 3 attempts" with no cause. | Chain the first error and log one WARNING with the attempt count and exception types.
suggestion | mcpgateway/utils/ssrf_pinning.py:296,346 | Each attempt uses the full caller timeout. A black-holed first address costs one timeout per address, multiplied by OAuth max_retries (CWE-400). | Cap the per-attempt connect timeout, or enforce one overall deadline.
minor | CHANGELOG.md:153 | The #6965 entry lists this limitation. | Add an entry for #7407 and name the paths that stay single-address.
Alembic migration: none needed. UI change: none. Breaking change: none. Behavior changes only in timing and attempt count after a failed connect.
Design note, not a request: an httpx equivalent of SniPinningTransport would remove the URL rewriting, the pool-by-origin risk and the stream_async generator. It needs transport= support in get_isolated_http_client. The helper approach is acceptable for this fix.
I will re-review after the blocking item and the stream_async fix land.
|
Hi @ScarAnd |
msureshkumar88
left a comment
There was a problem hiding this comment.
Thank you for the update. Commit 9f6355f9b fixes the blocking item and the stream replay defect. I request changes for two items before merge.
I checked out head 9f6355f9b and ran these checks:
tests/security/test_outbound_dns_pinning.py: 101 passed. Coverage ofmcpgateway/utils/ssrf_pinning.pyis 100%.- Caller unit tests (
oauth_manager,a2a_service,llm_proxy_service,gateway_service,utils/): 2852 passed, 1 skipped. ruff checkis clean on the changed files.tests/live_gateway/mcp/test_oauth_dual_stack_live.pyskipped on my host, becauselocalhostresolves to127.0.0.1only.
Resolved from the previous review:
stream_asynctests now cover failover in both directions, all addresses refused, TLS failure not retried,sni_hostnameandHoston every attempt, unpinned pass-through, and no retry after headers.AsyncExitStackmoves theyieldoutside thetry. A caller error no longer causes a second request.- The live gateway test and its README row are present.
- The client isolation constraint names all three entry points and repeats in both helper docstrings.
- The unreachable fallback branches and
pinned_addressesare gone.pin()emitsDeprecationWarning. - The CHANGELOG has an entry for #7407.
severity | path:line | problem | fix
blocking | commit 9f6355f9b | The commit has no Signed-off-by trailer. The DCO check fails, and DCO is a merge requirement. | Amend with `git commit --amend -s --no-edit` and force-push.
functionally-impacting | a2a_protocol.py:209, dcr_service.py:116, openapi_service.py:199, tool_service.py:5847 | The CHANGELOG names these single-address paths. The PR body does not, and no follow-up issue exists. | Add the exclusion to the PR body and open a follow-up issue. Link the issue in the PR body.
suggestion | mcpgateway/utils/ssrf_pinning.py:283-294 | Only the last error reaches the caller, and failed attempts log at DEBUG. | Chain the first error and log one WARNING with the attempt count and exception types. A follow-up is acceptable.
suggestion | mcpgateway/utils/ssrf_pinning.py:283,338 | Each attempt uses the full caller timeout. A black-holed first address costs one timeout per address (CWE-400). | Cap the per-attempt connect timeout, or enforce one overall deadline. A follow-up is acceptable.
Alembic migration: none needed. UI change: none. Breaking change: none.
I will approve after the sign-off and the follow-up issue land.
Signed-off-by: Andrin Enz <89848517+ScarAnd@users.noreply.github.com>
9f6355f to
006cc81
Compare
msureshkumar88
left a comment
There was a problem hiding this comment.
Thank you for the update. Commit 006cc81d8 now carries the Signed-off-by trailer, and the DCO check passes. git range-diff against 9f6355f9b shows the trailer as the only change. The merge of main (b10779576) leaves the PR's changes to the five service and utility files unchanged.
I request changes for two items before merge.
I checked out head b10779576 and ran tests/unit/mcpgateway/services/test_gateway_service_health_oauth.py: 22 passed, 1 failed. This matches the rstest (py3.14) CI failure.
severity | path:line | problem | fix
blocking | tests/unit/mcpgateway/services/test_gateway_service_health_oauth.py:814-817 | The merge from main brings in TestTokenExchangeDiscoveryHealthCheck. Its stub `pinned = MagicMock()` sets only `pin`, `apply_headers` and `extensions`. The SSE health check now calls `pinned_target.stream_async(...)` (gateway_service.py:6162). MagicMock supports `async with`, so the probe gets a mock response and never calls `client.stream`. test_probe_runs_with_discovery_token fails, and rstest (py3.14) is red. | Add `pinned.stream_async = lambda c, method, url, **kw: c.stream(method, url, **kw)` after line 817. With this line, all 23 tests in the file pass locally.
functionally-impacting | PR body | The previous review asked for the single-address exclusion in the PR body and a follow-up issue. The PR body does not name a2a_protocol.py, dcr_service.py, openapi_service.py or tool_service.py. No follow-up issue exists. | Add the exclusion to the PR body. Open a follow-up issue and link it in the PR body.
minor | PR body | The Verification section says the helpers no longer set the `sni_hostname` extension. Commit 6f6d95f1c restored it. | Update the sentence to match the code.
Alembic migration: none needed. UI change: none. Breaking change: none.
I will approve after the test fix and the follow-up issue land.
msureshkumar88
left a comment
There was a problem hiding this comment.
Thank you for the update. Commit 37efcfda8 fixes the blocking item from the previous review. The stream_async stub line is in place. tests/unit/mcpgateway/services/test_gateway_service_health_oauth.py and tests/security/test_outbound_dns_pinning.py pass locally at head 37efcfda8, and every CI check except DCO passes.
I request changes for these items before merge.
severity | path:line | problem | fix
blocking | commit 37efcfda8 | The commit has no Signed-off-by trailer. The DCO check fails, and DCO is a merge requirement. | Amend with `git commit --amend -s --no-edit` and force-push.
blocking | mcpgateway/services/gateway_service.py, mcpgateway/services/oauth_manager.py | The branch conflicts with main in both files. Both files carry this PR's call-site changes. | Merge or rebase onto current main. Keep the `connect_async` / `stream_async` call sites in both files, then rerun the health-check and OAuth unit tests.
functionally-impacting | PR body | The previous two reviews asked for the single-address exclusion in the PR body and a follow-up issue. The PR body does not name a2a_protocol.py, dcr_service.py, openapi_service.py or tool_service.py. No follow-up issue exists. | Add the exclusion to the PR body. Open a follow-up issue and link it in the PR body.
minor | PR body | The Verification section says the helpers no longer set the `sni_hostname` extension. Commit 6f6d95f1c restored it. | Update the sentence to match the code.
Alembic migration: none needed. UI change: none. Breaking change: none.
I will approve after the sign-off, the conflict resolution, and the follow-up issue land.
Signed-off-by: Andrin Enz <89848517+ScarAnd@users.noreply.github.com>
Signed-off-by: Andrin Enz <Andrin.Enz@ibm.com>
37efcfd to
bd14ded
Compare
There was a problem hiding this comment.
Thank you for the update. Commits d830c774f and bd14ded69 fix both blocking items from the previous review. Both commits carry the Signed-off-by trailer, and the branch merges cleanly with main. The merge keeps the stream_async and connect_async call sites in gateway_service.py and oauth_manager.py.
I checked out head bd14ded69 and ran these checks:
tests/security/test_outbound_dns_pinning.pyand the OAuth, A2A, LLM proxy, health check and card import unit tests pass.ruff checkis clean on the changed files.
I request changes for these items before merge.
| severity | path:line | problem | fix |
|---|---|---|---|
| functionally-impacting | mcpgateway/services/a2a_card_import.py:664 | The merge from main brings in #7377. This path calls target.pin(), so each Agent Card fetch emits the new DeprecationWarning and dials only resolved_ips[0], as in #7407. The PR body says pin() has no callers left in mcpgateway/. | Change the call to target.stream_async(client, "GET", target.validated_url, headers=request_headers) and update the card import tests. Alternatively, add this path to the exclusion list in the PR body. |
| functionally-impacting | PR body | The previous three reviews asked for the single-address exclusion in the PR body and a follow-up issue. The PR body does not name a2a_protocol.py, dcr_service.py, openapi_service.py or tool_service.py. No follow-up issue exists. | Add the exclusion to the PR body. Open a follow-up issue and link it in the PR body. |
| minor | PR body | The Verification section says the helpers no longer set the sni_hostname extension. Commit 6f6d95f restored it. |
Update the sentence to match the code. |
Alembic migration: none needed. UI change: none. Breaking change: none.
I will approve after these items land and CI is green.
Problem
PinnedTarget.pin()rewrites a URL toresolved_ips[0]and never tries the remaining validated addresses. #6965 replaced httpx's own multi-address connect with that single dial, so an outbound request fails whenever the peer listens on only one of the address families the hostname resolves to.localhostis the easy reproduction: it resolves to::1first on macOS and most Debian/Ubuntu hosts, while a provider bound to127.0.0.1refuses the::1attempt and the request never gets a second chance.SSRF_ALLOW_LOCALHOST=truegrants the destination, the URL policy passes, and the pin then picks the address family the peer is not listening on.Every
pin()caller had the behaviour:oauth_manager._post_token_requestllm_proxy_service(×2)a2a_servicegateway_service._check_single_gateway_healthSniPinningTransportalready loops overpinned_hostsand falls back on connect errors, so the module disagreed with itself.Fixes #7407.
Change
PinnedTarget.connect_async()andPinnedTarget.stream_async()try each validated address in turn and retry onhttpx.ConnectError/httpx.ConnectTimeout.SSLContext, per-call timeouts, andfollow_redirects=False.apply_headers(), so every attempt carries the validated authority inHost, never the dialled address.pin()is marked deprecated, since it dials a single address. It has no callers left inmcpgateway/.client.post/client.get, so caller stubs and instrumentation keep working.Addresses are not reordered. The policy already checked every one of them, and every one remains a dial target.
Verification
Reproduced against a purpose-built IPv6-only peer (
::1:8077,127.0.0.1refused,localhostresolving to both):The peer answered on IPv6 after the IPv4 attempt was refused, with
Host: localhost:8077intact.Live gateway check: authorization-code flow through a running gateway returns callback
200and stores the token.Tests:
tests/security/test_outbound_dns_pinning.py— new cases for IPv4→IPv6 failover, IPv6→IPv4 failover, all-addresses-refused,Hostpreserved on every attempt, and pass-through when unpinned.request.extensions, which the helpers no longer set. They now assert the dial andHostcontract, and state explicitly that nosni_hostnameextension is sent.tests/unit/mcpgateway/services/test_oauth_manager_pkce.py,test_a2a_service.py,test_llm_proxy_service.py,test_oauth_manager.py, andtests/unit/mcpgateway/test_oauth_manager.pypass unchanged.ruff checkandruff formatclean on the changed files.