[casaroli]

Summary

In the kernel build a vfork() child is in the address environment of its parent. nxtask_setup_fork() allocates the stack of the child (up_create_stack()), its task information (group_allocate()) and the copy of its environment (env_dup()) in the user heap of the current address environment: the heap of the parent. When the child ends (_exit(), or exec() and then _exit(0) in execve()), its group ends, and the kernel-build exit code assumes that the user heap of the group goes away with its address environment. For a vfork() child that is not true: the heap is the heap of the parent, and it stays. So each vfork() lost the task information and the environment of the child (1288 bytes on qemu-armv8a:knsh) in the heap of the parent. A shell that runs each command with vfork() grows with each command.

A new group flag, GROUP_FLAG_VFORK (bit 7 of tg_flags, which was free), is set by nxtask_setup_fork() when a vfork() child joins the address environment of its parent. task_uninit_info() then frees the task information and the environment of the child. group_leave() clears the flag when the caller is not in the address environment of the child (code in another process ends the child): a free from there would go into the heap of the caller, so the blocks stay in the heap of the parent, as before.

Impact

• Kernel builds: vfork() no longer loses memory in the heap of the parent.
• A vfork() child that code in another process ends (task_delete() from the kernel, for example) still leaves its blocks in the heap of the parent, as before. A child that ends itself (also by a fatal signal) frees them.
• Flat and protected builds: no change. Only CONFIG_ARCH_ADDRENV sets the flag, and only CONFIG_BUILD_KERNEL code uses it. No architecture code.
• No ABI change (tg_flags is kernel-internal). Not a breaking change.

Testing

Test program forkstress.c: N rounds of a case, then the page pool and the kernel heap (/proc/meminfo), the processes in /proc, the lowest free file descriptor and the heap of the test (mallinfo()) are compared with the values after the first round. Cases: fork() + _exit(); fork() + exec(); two children joined by a pipe; the same in a forked child; posix_spawn(); vfork() + _exit(); vfork() + exec().

qemu-armv8a:knsh (QEMU, HVF) with CONFIG_PIPES, CONFIG_SCHED_WAITPID, CONFIG_SCHED_CHILD_STATUS, on master 6b9b216:

Case	master	this PR
fork 1000, exec 300, pipe 200, nest 200, spawn 300	PASS (no growth)	PASS
vfork 200	FAIL: heap in use +257616 bytes (1288 per child), page pool +245760	PASS (+0)
vexec 200	FAIL: heap in use +257600 bytes, page pool +253952	PASS (+0)

rv-virt:knsh64 (QEMU TCG), the same options: master vexec 200 +177624 bytes, vfork 100 +88800 (888 per child); this PR +0 in both. ostest exits with status 0 on master and with this PR.

ostest exits with status 0 on qemu-armv8a:knsh on master and with this PR, and on qemu-armv8a:nsh with this PR.

Builds with EXTRAFLAGS="-Wno-cpp -Werror", each OK and with group_leave.c compiled: qemu-armv8a:knsh, qemu-armv8a:nsh (arm64 kernel, flat), rv-virt:knsh64, rv-virt:pnsh64 (rv64 kernel, protected), mps2-an500:knsh (Cortex-M protected), sim:nsh.

tools/checkpatch.sh -c -u -m -g passes.

[github-actions]

MemBrowse Memory Report · Explore what changed →

No memory changes detected for:

• arduino-mega2560
• esp32-devkitc
• hifive1-revb
• mirtoo
• qemu-armv8a
• qemu-intel64
• rx65n-rsk2mb
• s698pm-dkit
• stm32-nucleo-f103rb

---

_{Catch memory regressions in CI with MemBrowse. Free for open source.}

[casaroli]

This PR is split to make the review easier. Review order:

1. This PR: free the task information of a vfork() child in the parent's heap.
2. sched/group: Do not free the stacks of an ending process into its heap. #21465: do not free the stacks of an ending process into its heap.