Test-only failpoints for atomic-flush and tip-publication crash recovery #253

Motivation

#217 moves txstore and history into column families in one RocksDB database and relies on atomic flush before publishing the persisted tip t. The logical recovery states have Store-level coverage, but the exact process-crash boundaries around flush and tip publication are not exercised end to end.

External timing-based termination cannot reliably hit these short windows. Deterministic coverage requires narrowly scoped, test-only failpoints.

This complements #248, which covers failpoints during reorg stale cleanup. This issue is limited to forward indexing and publication durability.

Proposed failpoints

Add observable pause or panic hooks at:

1. before_block_flush — after block rows and completion markers have been written, immediately before the atomic txstore + history flush;
2. after_block_flush_before_tip — after the atomic flush completes, before persisted tip t is updated;
3. after_tip_before_publish — after persisted tip t is updated, before the in-memory HeaderList is appended/published.

Names are illustrative and can be adjusted to fit the implementation.

Requirements

• Inert in normal builds and runs, preferably behind a test-only Cargo feature and/or explicit environment variable.
• Emit an unambiguous observable signal when reached, so an external test can kill the process without sleeps or probabilistic timing.
• Support pause-until-killed or deliberate process termination semantics.
• Avoid introducing a production runtime dependency if a small test-only mechanism is sufficient.
• Reuse the same failpoint infrastructure as Test-only failpoint in reorg stale cleanup for deterministic crash-recovery testing #248 where practical.

Recovery assertions

For each failpoint, an integration test should:

1. index a chain containing confirmed spends and address history;
2. wait until the armed failpoint is reached;
3. terminate electrs abruptly with SIGKILL;
4. restart electrs on the preserved database without the failpoint;
5. verify:
   • electrs converges to the daemon tip;
   • transaction confirmation status is correct;
   • history, balances, and UTXOs are correct over both REST and Electrum;
   • no published block lacks its required history;
   • another block and spend can be indexed successfully after recovery;
   • no RocksDB background error is reported.

The three expected recovery cases are:

• before flush: unflushed WAL-disabled work may be absent or partially persisted, remains unpublished, and is safely retried;
• after flush/before t: completed durable work remains staged and is discovered/skipped or completed on restart before publication;
• after t/before in-memory publication: restart reconstructs the visible chain from durable t and completion markers.

Definition of done

• The three forward-indexing failpoints exist and are disabled by default.
• Each failpoint has a deterministic end-to-end hard-crash/restart test.
• Tests verify externally visible state and continued indexing, not only marker rows.
• The recovery behavior documented in Use REST spenttxouts for block indexing, plus consistency fixes and RocksDB multi-CF #217 is exercised at the real RocksDB durability boundaries.