Repository navigation
Description
Problem
A sequential --stop-on-failure run writes none of the requested report files and does not update .bashunit/last-failed. The same run under --parallel writes all of them.
Reproduced on main 776997e9, macOS arm64, Bash 3.2.57:
printf 'function test_first_ok() { assert_same 1 1; }\nfunction test_second_ko() { assert_same 1 2; }\nfunction test_third_ok() { assert_same 1 1; }\n' > stop_test.sh
rm -rf .bashunit
./bashunit --skip-env-file -S --log-junit junit.xml --report-json report.json stop_test.sh
ls junit.xml report.json .bashunit/last-failed
./bashunit --skip-env-file -S --parallel --log-junit junit.xml --report-json report.json stop_test.sh
ls junit.xml report.json .bashunit/last-failedAfter the first run ls reports all three paths missing. After the parallel run all three exist and last-failed holds stop_test.sh:test_second_ko. Both runs print the failure, Stop on failure enabled... and exit 1.
Where
src/runner/diagnostics.sh:69-77: sequential mode leaves throughexit "$EXIT_CODE_STOP_ON_FAILURE".src/main/run.sh:83routes that exit tobashunit::main::handle_stop_on_failure_sync(src/main/run.sh:405-426), which prints the summary and cleans up, but never reaches the report writers (exec_tests,src/main/run.sh:181-256),bashunit::rerun::persist(:298) or the coverage, snapshot and GitHub annotation steps around them.- Parallel mode only marks a flag file, so
exec_testsruns to its normal end and writes everything.
Why it matters
--stop-on-failure --log-junit is a common CI pairing. The step fails, and the report the CI reads is missing or stale from the previous run. --rerun-failed after a stopped run replays the previous cache instead of the test that just stopped the run; if the previous run was green, it reports "No previously failing tests recorded" and runs the whole suite.
Suggested change
Make the sequential stop path return to exec_tests instead of exiting from inside the test loop: for example set a flag in halt_if_stop_on_failure for sequential runs too, have call_test_functions and load_test_files break out when it is set, and let exec_tests render, write reports, persist the rerun cache and exit 1 as it already does for parallel runs. If the exit path has to stay, call the same report, rerun and cleanup steps from handle_stop_on_failure_sync.
Acceptance criteria
- Sequential
-Swith--log-junit,--report-json,--report-html,--report-tap,--report-md,--log-ghawrites each requested file and it holds the tests that ran. - Sequential
-Swrites the stopping test to.bashunit/last-failed, and--rerun-failedreplays it. - The console output, exit code and
tear_down_after_scriptbehaviour of-Sare unchanged. - An acceptance test compares the files written by
-Sin both modes and fails on the current code.
Proposed fix
Make the sequential stop leave the test loop by a flag instead of an exit, so exec_tests reaches its own report, rerun and cleanup steps:
bashunit::runner::halt_if_stop_on_failure(src/runner/diagnostics.sh:69-77): for sequential runs set_BASHUNIT_STOP_REQUESTED=true(declare it next to_BASHUNIT_FILE_TEARDOWN_PENDINGinsrc/runner/hooks.sh:351) instead ofexit "$EXIT_CODE_STOP_ON_FAILURE".bashunit::runner::call_test_functions(src/runner/exec.sh:173-176): alsobreakwhen[ "${_BASHUNIT_STOP_REQUESTED:-}" = true ], so the remaining functions of the file are skipped.bashunit::runner::load_test_files(src/runner/discovery.sh:59): at the top of the loop[ "${_BASHUNIT_STOP_REQUESTED:-}" = true ] && break. The file that stopped already ran itstear_down_after_scriptat:267on the way out.bashunit::main::exec_tests(src/main/run.sh:165-168): printStop on failure enabled...when the flag is set, not only under--parallel, so the console output keeps its line and position;render_resultthen exits 1 as today.- Delete the EXIT trap at
src/main/run.sh:83andbashunit::main::handle_stop_on_failure_sync(:405-426); keeprun_pending_file_teardown, which Ctrl-C still needs (--stop-on-failureskipstear_down_after_scriptin sequential runs #1321, Ctrl-C does not runtear_down_after_scriptfor the file in flight #1323).
Bash 3.0: plain globals and [ ] tests; nothing version specific. Public behaviour that must not change: the console output of a sequential -S run (pinned by tests/acceptance/snapshots/bashunit_stop_on_failure_test_sh.*), exit code 1, tear_down_after_script running before the summary. Docs: docs/command-line.md --stop-on-failure entry, add one sentence that reports and .bashunit/last-failed are written for the tests that ran. CHANGELOG: one line under Unreleased, Fixed.
Agent brief
- Branch:
fix/1415-stop-on-failure-writes-reportsfrom freshorigin/main. Commit:fix(runner): write reports and the rerun cache when --stop-on-failure stops a sequential runwithCloses #1415in the body. - Files:
src/runner/diagnostics.sh,src/runner/exec.sh,src/runner/discovery.sh,src/runner/hooks.sh,src/main/run.sh; extendtests/acceptance/bashunit_stop_on_failure_test.sh(copytest_bashunit_when_stop_on_failure_option). - RED first:
function test_stop_on_failure_writes_the_junit_report_and_the_rerun_cache_sequentially() {
local test_file=./tests/acceptance/fixtures/test_bashunit_when_stop_on_failure.sh
local report cache_dir
report="$(bashunit::temp_file junit).xml"
cache_dir="$(bashunit::temp_dir rerun)"
BASHUNIT_RERUN_CACHE_DIR="$cache_dir" ./bashunit --no-parallel --env "$TEST_ENV_FILE" \
--stop-on-failure --log-junit "$report" "$test_file" >/dev/null 2>&1 || true
assert_file_exists "$report"
assert_file_exists "$cache_dir/last-failed"
}Both assertions fail on the current code. Run only that file: ./bashunit tests/acceptance/bashunit_stop_on_failure_test.sh.
- Non-goals: the parallel stop path (flag file) already works; do not change the summary text, exit codes, or
--rerun-failedsemantics (A green run of a subset clears .bashunit/last-failed #1418). - Gate, once, when the change is complete:
./bashunit tests/,./bashunit --parallel --simple --strict tests/,make sa,make lint.make lintis the formatting authority; nevershfmt -w; 2-space indent. - Done when: every acceptance box is checked, the gate is green, CHANGELOG and docs are updated, and a PR is open with the
prskill.
Metadata
Metadata
Assignees
Labels
Type
Projects
- StatusShow more project fieldsDone
Activity
fix(runner): finalize reports after early stop