Skip to content

bug(network): policy DNS asks one nameserver once with a 2 s timeout, so lookups fail intermittently under load #5770

Open

Description

@jbadeau

What happens

Under load, a workload in a sandbox intermittently fails to resolve a host its policy allows: failed to lookup address information: No address associated with hostname (EAI_NODATA). Run alone, the same lookup always succeeds.

Seen with v0.1.8 and the Podman driver; the resolver code below is unchanged on main (6c3d773).

Why

Policy DNS asks the upstream once: crates/openshell-supervisor-network/src/policy_dns/resolver.rs sends one UDP query to the first nameserver of the supervisor's /etc/resolv.conf, waits DEFAULT_EXCHANGE_TIMEOUT (2 s, line 21) and does not retry, nor try another nameserver. A single lost reply becomes a failed lookup for the workload; since AAAA queries are always answered empty, glibc reports it as "No address associated with hostname".

Container-runtime resolvers drop queries under concurrency. Asking Docker Desktop/OrbStack's resolver exactly as the supervisor does (one UDP A query, 2 s, no retry, 2000 queries):

at once answered lost SERVFAIL
20 2000 0 0
100 1995 5 0
400 236 0 1764

With many sandboxes, each with its own supervisor asking the same host resolver, this is reached in practice.

Related: #5197 (the first nameserver is fixed at supervisor start).

Workaround

A local caching resolver (Unbound) as the supervisors' nameserver: 4000 of 4000 answered at 800 at once.

Suggested fix

Retry a lost exchange (with backoff, within the workload's own timeout) and fall back to the other nameservers in /etc/resolv.conf; or document that the supervisor's nameserver should be a local caching resolver.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    state:triage-neededOpened without agent diagnostics and needs triage

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions