Skip to content

Tests in a file with set_up or tear_down cost five process forks each (28.3ms vs 8.5ms per test) #1345

Closed

Description

@Chemaclass
Q A
OS macOS (arm64)
Shell & version bash 3.2.57
bashunit version 0.50.1

Summary

A test in a file that defines set_up or tear_down costs 3.3x a test in a file that does not. Each hook mints a mktemp scratch file for its output, and the ownership marker that file leaves behind makes the runner rm -rf the test's temp files at exit. Five process forks per test, none of them needed.

.claude/rules/perf-fork-budget.md says "the per-test path is genuinely fork-free". That is true, but only for a hookless file, which is what its census fixture uses. That is why this never showed up.

Measured

200 trivial tests (assert_same 1 1), best of 3, macOS arm64, bash 3.2.57:

Fixture Wall Per test
200 tests, no hooks 1879 ms 8.5 ms
200 tests, empty set_up + tear_down 5808 ms 28.3 ms

PATH-shim census over the same two fixtures, using the method in perf-fork-budget.md:

Binary Hookless Hooked
mktemp 0 400
rm 1 601
awk / base64 / mkdir / perl 2 / 1 / 1 / 3 unchanged

Exactly five binary forks per hooked test: 2 mktemp, 2 rm -f, 1 rm -rf. Hooks cost +19.7 ms per test.

This suite has 1052 tests in files defining set_up or tear_down, out of ~2491. So roughly 20 seconds of a sequential run.

Current behavior

bashunit::runner::execute_test_hook, src/runner/hooks.sh:196-265:

hook_output_file=$(bashunit::temp_file "${hook_name}_output")   # :215
...
rm -f "$hook_output_file"                                       # :258

bashunit::temp_file (src/api/globals.sh:38-51) forks $MKTEMP, and also writes $BASHUNIT_TEMP_DIR/${BASHUNIT_CURRENT_TEST_ID}_.mark (src/api/globals.sh:25-36). The hook's rm -f removes the output file but not that marker. So bashunit::cleanup_testcase_temp_files (src/api/globals.sh:70-100), reached from the EXIT trap at src/runner/hooks.sh:3-40, sees the marker and runs rm -rf.

A test that creates no temp file of its own still gets a marker, purely because a hook ran.

Same shape once per file in execute_file_hook (src/runner/hooks.sh:82-145): mktemp at :93, rm -f at :141.

Expected behavior

No process fork per hook. The hook still needs a file, because it runs in the current shell on purpose (see the comment at src/runner/hooks.sh:206-212: variables set before a failure have to survive), so only the file's name and lifecycle should change.

Name it arithmetically inside the run directory instead of going through temp_file. The > redirect at :239-241 truncates, so no rm is needed at all. The precedent is already in the tree: src/runner/result.sh:28-62 names the per-test .result file by an ordinal the dispatcher assigns before forking (#851), because a Bash 3 fork inherits both $$ and the RANDOM state and cannot mint a unique token.

Two details for whoever picks this up:

  • Key on the ordinal, not BASHUNIT_CURRENT_TEST_ID. bashunit::helper::generate_id (src/helper/encoding.sh:44-65) produces ${fn}_$$, which repeats across data-provider rows.
  • _test_ordinal is bumped only on the --parallel branches today (src/runner/exec.sh:169-172 and :216-219), so a sequential run leaves it at 0. It is also a local of call_test_functions, so it restarts per file, and needs combining with the folded file key the way parallel_suite_dir_to_slot does (src/runner/result.sh:36-41).

The RED test belongs in tests/acceptance/bashunit_run_forks_test.sh, with a fixture that defines set_up and tear_down, asserting the mktemp and rm counts. perf-fork-budget.md needs the "fork-free" paragraph corrected to say hookless.

Activity

  1. Chemaclass commented on Sep 9, 2026

    @Chemaclass
    MemberAuthor

    The absolute figures in the title need re-measuring; the ratio holds

    A second measurement pass on the same host (macOS arm64, bash 3.2.57) did not reproduce the 28.3 ms and 8.5 ms in the title. It got 18.2 ms hooked and 4.1 ms hookless, from a 1-test baseline of 97 ms and a 200-test run of 913 ms.

    Both passes agree on everything that matters to the diagnosis:

    • The five forks per hooked test, confirmed twice by shim census: 2 mktemp, 2 rm -f, 1 rm -rf.
    • The ratio: 3.3x in the first pass, 4.4x in the second.
    • mktemp on macOS is expensive in absolute terms: 2815 us per call, against 280 to 580 us on Linux.

    The absolutes differ by about 2x and I cannot yet account for it, so treat the title's numbers as indicative and re-measure before quoting them in the CHANGELOG. perf-fork-budget.md already warns that single runs vary and that a benchmark must be A/B'd in the same tree; this is that warning landing. The Linux figures for the same fixture are 2 to 4 ms of hook overhead per test, so the win is platform-weighted towards macOS.

    One correction to the design note above

    The issue body says a Bash 3 fork inherits both $$ and the RANDOM state. $$ yes, RANDOM no: measured, subshells get a reseeded RANDOM on 3.00.22, 3.2.57, 4.0.44, 5.2.37 and 5.3.15. See #1354.

    This does not change the recommendation. An ordinal is guaranteed unique, a reseeded RANDOM is only probably unique, and the ordinal is what #851 already established. Only the stated reason was wrong.

  2. self-assigned this
    on Sep 9, 2026
  3. Chemaclass commented on Sep 9, 2026

    @Chemaclass
    MemberAuthor

    Done in #1355 (merged).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

  • Status
    Done

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions