[yummybomb]

Summary

A fork that was still being brought up when Hypeman was killed stayed behind as an ordinary instance, holding its VMM, its memory and its vGPU. A fork is now marked pending until its request succeeds, and Hypeman deletes the forks that are still marked when it starts.

Why it happened

A fork is made in two steps: its metadata is saved, then it is brought to the state the caller asked for. If the process dies in between, the fork stays wherever it got to. Nothing on disk says it is unfinished, so after the restart it looks like any other standby or paused instance. The caller was disconnected without a response and has no ID to delete.

For a QEMU guest with a vGPU this is expensive: a restore that was cut short keeps its VMM, the guest memory it had pinned and its virtual function until someone deletes the instance by hand.

What changes

• StoredMetadata.ForkPending is set when a fork's metadata is first saved, for instance forks and snapshot forks, and cleared once the fork has reached its target state. If clearing it fails, the fork request fails and the fork is cleaned up like any other failed fork, so a fork whose caller got a success is never still marked.
• RecoverInterruptedForks deletes the forks that are still marked. It runs once at startup, before the API listens and before the vGPU reconciler starts. Like the network and device reconciliation next to it, it assumes that no other API process is using the data directory.
• Snapshot metadata never carries the mark, so nothing restored or forked from a snapshot of a pending fork inherits it.

Forks that completed are not touched, and a graceful restart still lets in-flight forks finish.

Testing

• TestForkIsPendingOnlyUntilItsRequestSucceeds covers the mark through an instance fork, a snapshot fork, a fork left half-made and a snapshot of that fork. TestRecoverInterruptedForksDeletesOnlyPendingForks covers the startup pass. Five mutants (mark never cleared for either kind of fork, mark kept in snapshots, recovery deleting everything, recovery deleting nothing) each fail one of them.
• On a host with two GPUs I killed Hypeman with SIGKILL while four vGPU snapshot forks were restoring, on a build of Add opt-in vGPU snapshots and forks on a managed QEMU runtime #552 plus this change:

Killed after	Fork requests answered	Deleted at startup	Left after restart
0.3 s	0 of 4	4	nothing
0.7 s	0 of 4	4	nothing
1.0 s	0 of 4	4	nothing
1.5 s	0 of 4	4	nothing
2.2 s	1 of 4	3	the answered fork, running and usable

No VMM, virtual function, TAP or guest directory was left in any case, and the next fork passed. Without the change the same kill at 1.0 s left two forks Paused and two in Standby, with two VMMs still holding two virtual functions (two runs).

• A graceful restart 1.0 s into four forks: all four requests succeeded, nothing was deleted at startup, and all four guests were running and usable.

Not covered

• A VMM started in the instant before the kill, before its control socket exists and before its PID is saved, cannot be found by the delete. That window is the same as for any start and is not changed here.
• A fork from a running instance puts its source in standby first. If Hypeman dies during that, the source stays in standby. This change only deals with the fork.