Test-only failpoint in reorg stale cleanup for deterministic crash-recovery testing #248

Motivation

#217 reworks reorg handling so that a crash during stale-branch cleanup is recoverable: the common ancestor is persisted before stale history deletion, and startup stale cleanup completes the job on restart. That recovery path currently has no end-to-end test.

The recoverable-restart case is straightforward to cover externally — stop electrs for the whole reorg, restart it on its preserved DB, and verify the index converges (tip, tx status, history, balances, UTXOs over REST and Electrum). What an external test cannot do is kill electrs during stale cleanup: on regtest, cleaning up a few stale blocks takes milliseconds, so a kill from outside the process cannot land in that window reliably, and a timing-based test would be flaky by construction.

Proposal

Add a narrowly scoped, test-only failpoint (pause or panic hook) at one or more of:

1. after the rollback common ancestor is persisted, before stale history deletion begins;
2. inside process_stale(), after some stale rows have been removed;
3. after stale deletion is flushed, before replacement indexing is published.

Requirements:

• Inert in normal builds/runs — behind a cargo feature (e.g. failpoints, possibly using the fail crate) and/or an env var such as ELECTRS_FAILPOINT=<name>; zero behavior change when unset.
• Observable — emit a distinct log line (and/or touch a sentinel file in the DB dir) when the failpoint is reached, so a test kills the process only after the target point is provably hit. No sleeps, no probabilistic timing.
• Simple semantics — either block until killed (pause) or panic at the named point.

How it would be used

An integration test would: build with the feature enabled, index a chain, trigger a reorg with the failpoint armed, wait for the failpoint signal, kill the process, restart clean, and verify the index fully recovers to the daemon's best chain.

Out of scope

Marker permutations (e.g. txstore-complete/history-incomplete, out-of-order block completion) remain better suited to Rust-level Store/Indexer unit tests than to black-box failpoint testing.