Skip to content

Sequential --stop-on-failure writes no report files and skips the rerun cache #1415

Closed

Description

@Chemaclass

Problem

A sequential --stop-on-failure run writes none of the requested report files and does not update .bashunit/last-failed. The same run under --parallel writes all of them.

Reproduced on main 776997e9, macOS arm64, Bash 3.2.57:

printf 'function test_first_ok() { assert_same 1 1; }\nfunction test_second_ko() { assert_same 1 2; }\nfunction test_third_ok() { assert_same 1 1; }\n' > stop_test.sh
rm -rf .bashunit
./bashunit --skip-env-file -S --log-junit junit.xml --report-json report.json stop_test.sh
ls junit.xml report.json .bashunit/last-failed
./bashunit --skip-env-file -S --parallel --log-junit junit.xml --report-json report.json stop_test.sh
ls junit.xml report.json .bashunit/last-failed

After the first run ls reports all three paths missing. After the parallel run all three exist and last-failed holds stop_test.sh:test_second_ko. Both runs print the failure, Stop on failure enabled... and exit 1.

Where

  • src/runner/diagnostics.sh:69-77: sequential mode leaves through exit "$EXIT_CODE_STOP_ON_FAILURE".
  • src/main/run.sh:83 routes that exit to bashunit::main::handle_stop_on_failure_sync (src/main/run.sh:405-426), which prints the summary and cleans up, but never reaches the report writers (exec_tests, src/main/run.sh:181-256), bashunit::rerun::persist (:298) or the coverage, snapshot and GitHub annotation steps around them.
  • Parallel mode only marks a flag file, so exec_tests runs to its normal end and writes everything.

Why it matters

--stop-on-failure --log-junit is a common CI pairing. The step fails, and the report the CI reads is missing or stale from the previous run. --rerun-failed after a stopped run replays the previous cache instead of the test that just stopped the run; if the previous run was green, it reports "No previously failing tests recorded" and runs the whole suite.

Suggested change

Make the sequential stop path return to exec_tests instead of exiting from inside the test loop: for example set a flag in halt_if_stop_on_failure for sequential runs too, have call_test_functions and load_test_files break out when it is set, and let exec_tests render, write reports, persist the rerun cache and exit 1 as it already does for parallel runs. If the exit path has to stay, call the same report, rerun and cleanup steps from handle_stop_on_failure_sync.

Acceptance criteria

  • Sequential -S with --log-junit, --report-json, --report-html, --report-tap, --report-md, --log-gha writes each requested file and it holds the tests that ran.
  • Sequential -S writes the stopping test to .bashunit/last-failed, and --rerun-failed replays it.
  • The console output, exit code and tear_down_after_script behaviour of -S are unchanged.
  • An acceptance test compares the files written by -S in both modes and fails on the current code.

Proposed fix

Make the sequential stop leave the test loop by a flag instead of an exit, so exec_tests reaches its own report, rerun and cleanup steps:

  1. bashunit::runner::halt_if_stop_on_failure (src/runner/diagnostics.sh:69-77): for sequential runs set _BASHUNIT_STOP_REQUESTED=true (declare it next to _BASHUNIT_FILE_TEARDOWN_PENDING in src/runner/hooks.sh:351) instead of exit "$EXIT_CODE_STOP_ON_FAILURE".
  2. bashunit::runner::call_test_functions (src/runner/exec.sh:173-176): also break when [ "${_BASHUNIT_STOP_REQUESTED:-}" = true ], so the remaining functions of the file are skipped.
  3. bashunit::runner::load_test_files (src/runner/discovery.sh:59): at the top of the loop [ "${_BASHUNIT_STOP_REQUESTED:-}" = true ] && break. The file that stopped already ran its tear_down_after_script at :267 on the way out.
  4. bashunit::main::exec_tests (src/main/run.sh:165-168): print Stop on failure enabled... when the flag is set, not only under --parallel, so the console output keeps its line and position; render_result then exits 1 as today.
  5. Delete the EXIT trap at src/main/run.sh:83 and bashunit::main::handle_stop_on_failure_sync (:405-426); keep run_pending_file_teardown, which Ctrl-C still needs (--stop-on-failure skips tear_down_after_script in sequential runs #1321, Ctrl-C does not run tear_down_after_script for the file in flight #1323).

Bash 3.0: plain globals and [ ] tests; nothing version specific. Public behaviour that must not change: the console output of a sequential -S run (pinned by tests/acceptance/snapshots/bashunit_stop_on_failure_test_sh.*), exit code 1, tear_down_after_script running before the summary. Docs: docs/command-line.md --stop-on-failure entry, add one sentence that reports and .bashunit/last-failed are written for the tests that ran. CHANGELOG: one line under Unreleased, Fixed.

Agent brief

  • Branch: fix/1415-stop-on-failure-writes-reports from fresh origin/main. Commit: fix(runner): write reports and the rerun cache when --stop-on-failure stops a sequential run with Closes #1415 in the body.
  • Files: src/runner/diagnostics.sh, src/runner/exec.sh, src/runner/discovery.sh, src/runner/hooks.sh, src/main/run.sh; extend tests/acceptance/bashunit_stop_on_failure_test.sh (copy test_bashunit_when_stop_on_failure_option).
  • RED first:
function test_stop_on_failure_writes_the_junit_report_and_the_rerun_cache_sequentially() {
  local test_file=./tests/acceptance/fixtures/test_bashunit_when_stop_on_failure.sh
  local report cache_dir
  report="$(bashunit::temp_file junit).xml"
  cache_dir="$(bashunit::temp_dir rerun)"
  BASHUNIT_RERUN_CACHE_DIR="$cache_dir" ./bashunit --no-parallel --env "$TEST_ENV_FILE" \
    --stop-on-failure --log-junit "$report" "$test_file" >/dev/null 2>&1 || true
  assert_file_exists "$report"
  assert_file_exists "$cache_dir/last-failed"
}

Both assertions fail on the current code. Run only that file: ./bashunit tests/acceptance/bashunit_stop_on_failure_test.sh.

  • Non-goals: the parallel stop path (flag file) already works; do not change the summary text, exit codes, or --rerun-failed semantics (A green run of a subset clears .bashunit/last-failed #1418).
  • Gate, once, when the change is complete: ./bashunit tests/, ./bashunit --parallel --simple --strict tests/, make sa, make lint. make lint is the formatting authority; never shfmt -w; 2-space indent.
  • Done when: every acceptance box is checked, the gate is green, CHANGELOG and docs are updated, and a PR is open with the pr skill.

Activity

  1. added
    p2Wrong behaviour users hit in normal use
    agent-readyScoped, reproduced and testable; an agent can take it without more questions
    area:runnerCode under src or scripts for runner
    area:reportsCode under src or scripts for reports
    on Oct 28, 2026
  2. self-assigned this
    on Oct 29, 2026
  3. added a commit that references this issue on Oct 30, 2026
    fda9dfb
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

agent-readyScoped, reproduced and testable; an agent can take it without more questionsarea:reportsCode under src or scripts for reportsarea:runnerCode under src or scripts for runnerbugSomething isn't workingp2Wrong behaviour users hit in normal use

Type

No type

Projects

  • Status
    Done

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions