Skip to content

Forget the host neighbor entry of a reused guest address - #553

Draft
yummybomb wants to merge 1 commit into
mainfrom
hypeship/forget-reused-guest-neighbor
Draft

yummybomb wants to merge 1 commit into
mainfrom
hypeship/forget-reused-guest-neighbor

Conversation

@yummybomb

Copy link
Copy Markdown
Contributor

Summary

The first connection from the host to a new guest could stall for about ten seconds when the guest was given an address that an earlier guest had used. CreateAllocation now deletes the host's neighbor entry for the address it hands out.

Why it happened

Guest addresses are picked at random from the subnet and reused once released. The host's neighbor (ARP) entry for a released address stays behind and still points at the previous guest's MAC. Linux keeps using a stale entry while it re-verifies it, so the first packets to the new guest go to a MAC nobody owns. The entry is only dropped once the delay and probe timers have run out, about 8 s later, and a TCP connect then gets through on its next SYN retransmit.

A guest forked from a snapshot is the worst case, because it sends nothing until the host contacts it. A cold-booted guest usually repairs the entry itself when it first resolves the gateway. On a host with fewer than gc_thresh1 (128) neighbor entries the kernel never garbage-collects them, so a stale entry can sit there for hours.

What changes

CreateAllocation removes the bridge's neighbor entry for the address it has just reserved, before it creates the TAP. A missing entry is the normal case and is not an error. A failure is logged and does not fail the allocation. Restore from standby is unchanged: the guest keeps its address and MAC.

Testing

  • Two new tests run against a real bridge and the kernel's neighbor table. They need root and are skipped without it. One covers the helper. The other plants stale entries on every address of a /29 and checks that CreateAllocation clears the one it hands out and leaves the rest. Each fails when the call or the delete is removed.
  • I reproduced the stall on a host by replacing a fresh fork's neighbor entry with a stale one before first contact: the first request took 11.7 s in 3 of 3 runs, against 0.5 s for forks left alone. This matches a fork I had seen take 11.7 s on its own, whose address had last been used 72 minutes earlier.
  • A build with this change then served about 150 forks and cold boots on that host, and logged no neighbor warnings.

The address a guest gets is random, so the host run could not force a reuse. The tests above are what cover the fixed path.

Guest addresses are picked at random from the subnet and reused. The host
kept the previous owner's neighbor entry, so the first connection to a new
guest on that address was sent to a MAC nobody owned until the entry timed
out, about ten seconds later. Delete the entry when the address is
allocated.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant