feat(runner): flaky is not a status — a test that only passed on retry reports as a plain pass #1012

Problem

--retry <n> hides flakiness instead of surfacing it.

src/runner/exec.sh:312-345 retries execution only, then commits a single result. When a test fails and later passes, the run records a plain pass. The only trace is a console suffix built at src/runner/exec.sh:524-527:

# A test that only passed after retrying is annotated so flakiness stays visible.
_BASHUNIT_RETRY_NOTE=""
if [ "$retries_used" -gt 0 ]; then
  _BASHUNIT_RETRY_NOTE=" (retry $retries_used/$retry_max)"
fi

That suffix exists only in the terminal. It is absent from the summary counters (src/state/counters.sh has passed/failed/skipped/incomplete/snapshot/risky — no flaky), from JUnit, TAP, JSON and HTML (src/reports/collect.sh:21-41 has no add_test_flaky), and from the GitHub Actions annotations. A CI job that retries a flaky test is indistinguishable from a clean run, so flakiness never gets triaged.

Proposal

Make flaky a first-class outcome: passed, but not on the first attempt.

• New counter bashunit::state::add_tests_flaky / get_tests_flaky, alongside the existing six.
• Summary line shows Flaky: N when non-zero (still counted inside the pass total, so exit code does not change by default).
• src/reports/collect.sh gains add_test_flaky carrying the retry count and the failure message from the first attempt (currently discarded — that message is the whole diagnostic value).
• Reporters:
  • JUnit: <flakyFailure> inside the <testcase> (Jenkins/GitLab render this natively)
  • TAP: ok N - name # TODO flaky (retried 2/3)
  • JSON: "status": "flaky", "retries": 2, "first_failure": "..."
  • HTML: distinct row styling
  • GHA: ::warning annotation (src/reports/gha.sh:31 case "$status")
• New flag --fail-on-flaky — treat flaky as failure for the exit code, mirroring the existing --fail-on-risky.

See also #1013 (--repeat), which finds flaky tests proactively.

Where to change

• src/runner/exec.sh:298-345 (capture the first attempt's failure message) and :518-530 (classification).
• src/state/counters.sh, src/state/index.sh, src/state/parallel.sh (the parallel aggregation must carry the new counter — see fix(reports): every report format is empty under --parallel #1004, where per-worker rows were dropped in the parent).
• src/reports/collect.sh, src/reports/junit.sh, src/reports/tap.sh, src/reports/json.sh, src/reports/html.sh, src/reports/gha.sh.
• src/console/summary.sh:5 render_result.

Acceptance criteria

• A test that fails once then passes under --retry 2 is reported as flaky, not as a plain pass
• Summary prints a flaky count; zero flaky tests print nothing new
• Default exit code is unchanged (flaky counts as a pass)
• --fail-on-flaky makes the run exit non-zero when any test is flaky
• The first attempt's failure message is preserved and shown in the reports
• JUnit emits <flakyFailure>; validate the XML parses
• TAP, JSON, HTML and GHA all carry the flaky status
• Counters are correct under --parallel (regression guard for fix(reports): every report format is empty under --parallel #1004)
• --retry 0 (default) can never produce a flaky result
• Unit tests in tests/unit/runner/ and tests/unit/reports/, plus an acceptance test

Repo checklist (agent)

• TDD: RED → GREEN → REFACTOR. Write the failing test first.
• Bash 3.0+ only: no printf -v, no += append, no declare -A, no [[ ]], no ${var,,}, no &>>, no ${arr[-1]}. Expanding a possibly-empty array under set -u needs ${arr[@]+"${arr[@]}"}.
• A new CLI flag must be wired in all of these or a parity test fails:
  • parse in src/main/test.sh (report-style flags need export -n, see src/main/test.sh:188-196 for why)
  • validate the value in bashunit::main::validate_config_or_exit (src/main/validate.sh:60) — unvalidated input used to run the wrong thing and exit 0 (Unknown options are silently ignored: a typo'd flag runs a different suite and exits 0 #871, --jobs with a non-integer value hangs on Bash 3.x and is silently ignored on Bash 4.3+ #873)
  • default + env var in src/config/env.sh and a documented line in .env.example
  • --help text in the same block it belongs to
  • both completions/bashunit.bash and completions/_bashunit (anti-drift test feat(cli): bash and zsh completion scripts with an anti-drift test #778 fails otherwise)
• Gates: make sa, make lint, ./bashunit tests/, ./bashunit --parallel tests/. Never run shfmt -w.
• Docs: update docs/command-line.md. Editing docs/assertions.md invalidates the bashunit doc acceptance snapshot — regenerate it.
• CHANGELOG.md: add one line under ## Unreleased.
• Fixtures under tests/acceptance/fixtures/ must not end in *test.sh.
• One issue = one PR.