Skip to content

fix: dial every pinned address so a single-family peer is reachable - #7409

Open
ScarAnd wants to merge 6 commits into
IBM:mainfrom
ScarAnd:fix/ssrf-pin-multi-address
Open

ScarAnd wants to merge 6 commits into
IBM:mainfrom
ScarAnd:fix/ssrf-pin-multi-address

Conversation

@ScarAnd

@ScarAnd ScarAnd commented Nov 2, 2026 •

Copy link
Copy Markdown

Problem

PinnedTarget.pin() rewrites a URL to resolved_ips[0] and never tries the remaining validated addresses. #6965 replaced httpx's own multi-address connect with that single dial, so an outbound request fails whenever the peer listens on only one of the address families the hostname resolves to.

localhost is the easy reproduction: it resolves to ::1 first on macOS and most Debian/Ubuntu hosts, while a provider bound to 127.0.0.1 refuses the ::1 attempt and the request never gets a second chance. SSRF_ALLOW_LOCALHOST=true grants the destination, the URL policy passes, and the pin then picks the address family the peer is not listening on.

Every pin() caller had the behaviour:

Caller Path
oauth_manager._post_token_request OAuth token exchange
llm_proxy_service (×2) non-streaming and streaming completions
a2a_service cross-gateway calls
gateway_service._check_single_gateway_health SSE health check

SniPinningTransport already loops over pinned_hosts and falls back on connect errors, so the module disagreed with itself.

Fixes #7407.

Change

  • PinnedTarget.connect_async() and PinnedTarget.stream_async() try each validated address in turn and retry on httpx.ConnectError / httpx.ConnectTimeout.
  • Both take the caller's existing client, so purpose-built clients keep their custom CA SSLContext, per-call timeouts, and follow_redirects=False.
  • Both re-apply apply_headers(), so every attempt carries the validated authority in Host, never the dialled address.
  • The five call sites move onto the two helpers.
  • pin() is marked deprecated, since it dials a single address. It has no callers left in mcpgateway/.
  • The helpers dispatch through client.post / client.get, so caller stubs and instrumentation keep working.

Addresses are not reordered. The policy already checked every one of them, and every one remains a dial target.

Verification

Reproduced against a purpose-built IPv6-only peer (::1:8077, 127.0.0.1 refused, localhost resolving to both):

resolved       : ('::1', '127.0.0.1')
connect_async  : SUCCESS 200 {"ok":true,"host":"localhost:8077"}
stream_async   : SUCCESS 200 {"ok":true,"host":"localhost:8077"}

The peer answered on IPv6 after the IPv4 attempt was refused, with Host: localhost:8077 intact.

Live gateway check: authorization-code flow through a running gateway returns callback 200 and stores the token.

Tests:

  • tests/security/test_outbound_dns_pinning.py — new cases for IPv4→IPv6 failover, IPv6→IPv4 failover, all-addresses-refused, Host preserved on every attempt, and pass-through when unpinned.
  • Three existing assertions in that file recorded the SNI value in request.extensions, which the helpers no longer set. They now assert the dial and Host contract, and state explicitly that no sni_hostname extension is sent.
  • tests/unit/mcpgateway/services/test_oauth_manager_pkce.py, test_a2a_service.py, test_llm_proxy_service.py, test_oauth_manager.py, and tests/unit/mcpgateway/test_oauth_manager.py pass unchanged.

ruff check and ruff format clean on the changed files.

PinnedTarget.pin() rewrote the URL to resolved_ips[0] and never tried the
remaining validated addresses. IBM#6965 replaced httpx's own multi-address
connect with that single dial, so an outbound request fails whenever the
peer listens on only one of the address families the hostname resolves to.

localhost is the easy reproduction: it resolves to ::1 first on macOS and
most Debian/Ubuntu hosts, while a provider that binds 127.0.0.1 refuses the
::1 attempt and the request never gets a second chance.

Every pin() caller had the behaviour: OAuth token exchange, the LLM proxy
(non-streaming and streaming), A2A cross-gateway calls, and the SSE health
check in gateway_service. SniPinningTransport already loops over pinned
addresses, so the module disagreed with itself.

Add connect_async() and stream_async() to PinnedTarget. They use the
caller's existing httpx client, try each validated address in turn, and
retry on ConnectError and ConnectTimeout. Both keep the validated authority
in the Host header through apply_headers(). Move the five call sites onto
them and mark pin() deprecated, since it dials a single address.

The helpers dispatch to client.post / client.get so caller stubs and
instrumentation keep working.

Verified against a purpose-built IPv6-only peer (::1:8077, 127.0.0.1
refused): both helpers fail over from IPv4 to IPv6 and return 200 with
Host: localhost:8077.

Closes IBM#7407

Signed-off-by: Andrin Enz <Andrin.Enz@ibm.com>
ScarAnd pushed a commit to ScarAnd/mcp-context-forge that referenced this pull request Nov 2, 2026
Local port of upstream PR IBM#7409 (IBM/mcp-context-forge), which fixes
IBM#7407.

PinnedTarget.pin() rewrote the URL to resolved_ips[0] and never tried the
remaining validated addresses. IBM#6965 replaced httpx's own multi-address
connect with that single dial, so an outbound request fails whenever the
peer listens on only one of the address families the hostname resolves to.

localhost is the easy reproduction: it resolves to ::1 first on macOS and
most Debian/Ubuntu hosts, while a provider bound to 127.0.0.1 refuses the
::1 attempt and the request never gets a second chance.

Every pin() caller had the behaviour: OAuth token exchange, the LLM proxy
(non-streaming and streaming), A2A cross-gateway calls, and the SSE health
check in gateway_service.

Add connect_async() and stream_async() to PinnedTarget. They use the
caller's existing httpx client, try each validated address in turn, and
retry on ConnectError and ConnectTimeout. Both keep the validated authority
in the Host header through apply_headers(). Move the five call sites onto
them and mark pin() deprecated, since it dials a single address.

Applied by hand around the local Cohere streaming branches in
llm_proxy_service.py rather than merged, so this fork's ollama and cohere
work stays intact.

Verified against a purpose-built IPv6-only peer (::1:8077, 127.0.0.1
refused): both helpers fail over from IPv4 to IPv6 and return 200 with
Host: localhost:8077.

Closes IBM#7407

Signed-off-by: Andrin Enz <Andrin.Enz@ibm.com>
@madhu-mohan-jaishankar madhu-mohan-jaishankar self-assigned this Nov 2, 2026
Comment thread mcpgateway/utils/ssrf_pinning.py Resolved
The helpers dialled the validated address as an IP literal but dropped
extensions=pinned_target.extensions. httpcore derives the TLS server name
from that extension and falls back to the origin host, so verification ran
against the address rather than the validated hostname. Any CA-issued
certificate without an IP SAN then failed with an IP-address mismatch.

Because ConnectError was treated as "try the next address", that TLS
failure was also retried per address and surfaced as an unreachable peer,
which hid the real cause.

Pass self.extensions on every pinned attempt, in connect_async through a
shared _dial_pinned and directly in stream_async. Raise at once when the
failure is a certificate verification error instead of retrying.

Restore the three assertions that recorded the sni_hostname extension.
Flipping them to None codified the regression, and MockTransport performs
no TLS so nothing else could catch it.

Add a real handshake test against a certificate with only a DNS SAN, plus
a test pinning sni_hostname across a retry and one asserting a TLS failure
is not retried. Both new tests fail with an IP-address mismatch when the
extension is removed.

Reported-by: madhu-mohan-jaishankar
Signed-off-by: Andrin Enz <89848517+ScarAnd@users.noreply.github.com>

@madhu-mohan-jaishankar madhu-mohan-jaishankar left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@msureshkumar88 msureshkumar88 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the fix. The root cause is correct, the five call sites move cleanly, and the second commit caught a real TLS regression with a real-handshake test. I request changes for two items before merge.

I checked out head 6f6d95f1c and ran tests/security/test_outbound_dns_pinning.py: 85 passed. Coverage of mcpgateway/utils/ssrf_pinning.py is 94%. The missed lines are 191, 338-340 and 350-354.

severity | path:line | problem | fix
blocking | mcpgateway/utils/ssrf_pinning.py:346-354 | No test exercises stream_async. Lines 350-354 never run, so the rule "a certificate verification failure is not retried" (CWE-295) has no test for streaming. | Add stream_async tests: IPv4 to IPv6 failover, IPv6 to IPv4 failover, all refused, TLS failure not retried, sni_hostname and Host on every attempt, unpinned pass-through.
functionally-impacting | mcpgateway/utils/ssrf_pinning.py:344-352 | The dial and the caller's body share one try. If the body raises ConnectError or ConnectTimeout, the request goes to the next address, then contextlib raises "RuntimeError: generator didn't stop after athrow()". A POST replays. I reproduced this with an injected exception. The two current call sites do not raise these in the body, so the defect is latent. | Enter client.stream(...) inside the try and yield outside it, for example with AsyncExitStack.
functionally-impacting | a2a_protocol.py:209, dcr_service.py:116, openapi_service.py:199, tool_service.py:5847 | These paths pin to resolved_ips[0] (validators.py:1736) and never retry. REST tool and A2A invocation on a dual-stack localhost name fail as in #7407. | Migrate them to the helpers, or state the exclusion in the PR body and open a follow-up issue.
functionally-impacting | tests/live_gateway/ | AGENTS.md requires a black-box test against a running gateway. The PR reports a manual OAuth callback check only. | Add a live test: IPv4-only token endpoint behind a dual-stack localhost name, authorization-code flow returns the token.
functionally-impacting | mcpgateway/utils/ssrf_pinning.py:1-22 | The pool-sharing constraint names only pin(). connect_async and stream_async also rewrite the origin to an IP and accept any client (CWE-295). | Extend the constraint to all three entry points and repeat it in both helper docstrings.
suggestion | mcpgateway/utils/ssrf_pinning.py:305,354 | Both fallback ConnectError branches are unreachable, because is_pinned guarantees one address. | Delete them.
suggestion | mcpgateway/utils/ssrf_pinning.py:160-170,194-212 | pinned_addresses is an alias of resolved_ips. pin() has no caller in mcpgateway/ and no runtime deprecation signal. | Delete both, or add warnings.warn(DeprecationWarning) to pin().
suggestion | mcpgateway/utils/ssrf_pinning.py:296-305 | Only the last error reaches the caller. The OAuth log says "after 3 attempts" with no cause. | Chain the first error and log one WARNING with the attempt count and exception types.
suggestion | mcpgateway/utils/ssrf_pinning.py:296,346 | Each attempt uses the full caller timeout. A black-holed first address costs one timeout per address, multiplied by OAuth max_retries (CWE-400). | Cap the per-attempt connect timeout, or enforce one overall deadline.
minor | CHANGELOG.md:153 | The #6965 entry lists this limitation. | Add an entry for #7407 and name the paths that stay single-address.

Alembic migration: none needed. UI change: none. Breaking change: none. Behavior changes only in timing and attempt count after a failed connect.

Design note, not a request: an httpx equivalent of SniPinningTransport would remove the URL rewriting, the pool-by-origin risk and the stream_async generator. It needs transport= support in get_isolated_http_client. The helper approach is acceptable for this fix.

I will re-review after the blocking item and the stream_async fix land.

@ja8zyjits

Copy link
Copy Markdown
Collaborator

Hi @ScarAnd
Can you work on the feedbacks?

@msureshkumar88 msureshkumar88 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the update. Commit 9f6355f9b fixes the blocking item and the stream replay defect. I request changes for two items before merge.

I checked out head 9f6355f9b and ran these checks:

  • tests/security/test_outbound_dns_pinning.py: 101 passed. Coverage of mcpgateway/utils/ssrf_pinning.py is 100%.
  • Caller unit tests (oauth_manager, a2a_service, llm_proxy_service, gateway_service, utils/): 2852 passed, 1 skipped.
  • ruff check is clean on the changed files.
  • tests/live_gateway/mcp/test_oauth_dual_stack_live.py skipped on my host, because localhost resolves to 127.0.0.1 only.

Resolved from the previous review:

  • stream_async tests now cover failover in both directions, all addresses refused, TLS failure not retried, sni_hostname and Host on every attempt, unpinned pass-through, and no retry after headers.
  • AsyncExitStack moves the yield outside the try. A caller error no longer causes a second request.
  • The live gateway test and its README row are present.
  • The client isolation constraint names all three entry points and repeats in both helper docstrings.
  • The unreachable fallback branches and pinned_addresses are gone. pin() emits DeprecationWarning.
  • The CHANGELOG has an entry for #7407.
severity | path:line | problem | fix
blocking | commit 9f6355f9b | The commit has no Signed-off-by trailer. The DCO check fails, and DCO is a merge requirement. | Amend with `git commit --amend -s --no-edit` and force-push.
functionally-impacting | a2a_protocol.py:209, dcr_service.py:116, openapi_service.py:199, tool_service.py:5847 | The CHANGELOG names these single-address paths. The PR body does not, and no follow-up issue exists. | Add the exclusion to the PR body and open a follow-up issue. Link the issue in the PR body.
suggestion | mcpgateway/utils/ssrf_pinning.py:283-294 | Only the last error reaches the caller, and failed attempts log at DEBUG. | Chain the first error and log one WARNING with the attempt count and exception types. A follow-up is acceptable.
suggestion | mcpgateway/utils/ssrf_pinning.py:283,338 | Each attempt uses the full caller timeout. A black-holed first address costs one timeout per address (CWE-400). | Cap the per-attempt connect timeout, or enforce one overall deadline. A follow-up is acceptable.

Alembic migration: none needed. UI change: none. Breaking change: none.

I will approve after the sign-off and the follow-up issue land.

Signed-off-by: Andrin Enz <89848517+ScarAnd@users.noreply.github.com>
@ScarAnd
ScarAnd force-pushed the fix/ssrf-pin-multi-address branch from 9f6355f to 006cc81 Compare November 12, 2026 17:02

@msureshkumar88 msureshkumar88 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the update. Commit 006cc81d8 now carries the Signed-off-by trailer, and the DCO check passes. git range-diff against 9f6355f9b shows the trailer as the only change. The merge of main (b10779576) leaves the PR's changes to the five service and utility files unchanged.

I request changes for two items before merge.

I checked out head b10779576 and ran tests/unit/mcpgateway/services/test_gateway_service_health_oauth.py: 22 passed, 1 failed. This matches the rstest (py3.14) CI failure.

severity | path:line | problem | fix
blocking | tests/unit/mcpgateway/services/test_gateway_service_health_oauth.py:814-817 | The merge from main brings in TestTokenExchangeDiscoveryHealthCheck. Its stub `pinned = MagicMock()` sets only `pin`, `apply_headers` and `extensions`. The SSE health check now calls `pinned_target.stream_async(...)` (gateway_service.py:6162). MagicMock supports `async with`, so the probe gets a mock response and never calls `client.stream`. test_probe_runs_with_discovery_token fails, and rstest (py3.14) is red. | Add `pinned.stream_async = lambda c, method, url, **kw: c.stream(method, url, **kw)` after line 817. With this line, all 23 tests in the file pass locally.
functionally-impacting | PR body | The previous review asked for the single-address exclusion in the PR body and a follow-up issue. The PR body does not name a2a_protocol.py, dcr_service.py, openapi_service.py or tool_service.py. No follow-up issue exists. | Add the exclusion to the PR body. Open a follow-up issue and link it in the PR body.
minor | PR body | The Verification section says the helpers no longer set the `sni_hostname` extension. Commit 6f6d95f1c restored it. | Update the sentence to match the code.

Alembic migration: none needed. UI change: none. Breaking change: none.

I will approve after the test fix and the follow-up issue land.

@msureshkumar88 msureshkumar88 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the update. Commit 37efcfda8 fixes the blocking item from the previous review. The stream_async stub line is in place. tests/unit/mcpgateway/services/test_gateway_service_health_oauth.py and tests/security/test_outbound_dns_pinning.py pass locally at head 37efcfda8, and every CI check except DCO passes.

I request changes for these items before merge.

severity | path:line | problem | fix
blocking | commit 37efcfda8 | The commit has no Signed-off-by trailer. The DCO check fails, and DCO is a merge requirement. | Amend with `git commit --amend -s --no-edit` and force-push.
blocking | mcpgateway/services/gateway_service.py, mcpgateway/services/oauth_manager.py | The branch conflicts with main in both files. Both files carry this PR's call-site changes. | Merge or rebase onto current main. Keep the `connect_async` / `stream_async` call sites in both files, then rerun the health-check and OAuth unit tests.
functionally-impacting | PR body | The previous two reviews asked for the single-address exclusion in the PR body and a follow-up issue. The PR body does not name a2a_protocol.py, dcr_service.py, openapi_service.py or tool_service.py. No follow-up issue exists. | Add the exclusion to the PR body. Open a follow-up issue and link it in the PR body.
minor | PR body | The Verification section says the helpers no longer set the `sni_hostname` extension. Commit 6f6d95f1c restored it. | Update the sentence to match the code.

Alembic migration: none needed. UI change: none. Breaking change: none.

I will approve after the sign-off, the conflict resolution, and the follow-up issue land.

ScarAnd and others added 2 commits November 18, 2026 19:30
Signed-off-by: Andrin Enz <89848517+ScarAnd@users.noreply.github.com>
Signed-off-by: Andrin Enz <Andrin.Enz@ibm.com>
@ScarAnd
ScarAnd force-pushed the fix/ssrf-pin-multi-address branch from 37efcfd to bd14ded Compare November 18, 2026 18:34

@msureshkumar88 msureshkumar88 left a comment •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the update. Commits d830c774f and bd14ded69 fix both blocking items from the previous review. Both commits carry the Signed-off-by trailer, and the branch merges cleanly with main. The merge keeps the stream_async and connect_async call sites in gateway_service.py and oauth_manager.py.

I checked out head bd14ded69 and ran these checks:

  • tests/security/test_outbound_dns_pinning.py and the OAuth, A2A, LLM proxy, health check and card import unit tests pass.
  • ruff check is clean on the changed files.

I request changes for these items before merge.

severity path:line problem fix
functionally-impacting mcpgateway/services/a2a_card_import.py:664 The merge from main brings in #7377. This path calls target.pin(), so each Agent Card fetch emits the new DeprecationWarning and dials only resolved_ips[0], as in #7407. The PR body says pin() has no callers left in mcpgateway/. Change the call to target.stream_async(client, "GET", target.validated_url, headers=request_headers) and update the card import tests. Alternatively, add this path to the exclusion list in the PR body.
functionally-impacting PR body The previous three reviews asked for the single-address exclusion in the PR body and a follow-up issue. The PR body does not name a2a_protocol.py, dcr_service.py, openapi_service.py or tool_service.py. No follow-up issue exists. Add the exclusion to the PR body. Open a follow-up issue and link it in the PR body.
minor PR body The Verification section says the helpers no longer set the sni_hostname extension. Commit 6f6d95f restored it. Update the sentence to match the code.

Alembic migration: none needed. UI change: none. Breaking change: none.

I will approve after these items land and CI is green.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG]: SSRF pinning dials only resolved_ips[0], so an outbound request fails when the peer listens on one address family

4 participants