Skip to content

HDDS-16462. Recon full snapshot onto an earlier OM lineage plus crash permanently serves stale derived data. - #11933

Draft
ArafatKhan2198 wants to merge 1 commit into
apache:masterfrom
ArafatKhan2198:HDDS-16462
Draft

ArafatKhan2198 wants to merge 1 commit into
apache:masterfrom
ArafatKhan2198:HDDS-16462

Conversation

@ArafatKhan2198

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

On startup, Recon rebuilds its derived tables only if a task's sequence number is behind the OM sync sequence number. But the OM sequence number can go backwards, for example after OM is restored from a backup.

Example:

  1. Recon has synced OM and built its tables up to 1000.
  2. OM is restored from an older backup and is now at 600.
  3. Recon falls back to a full snapshot, saves the OM sync number as 600, and queues a rebuild. The rebuild request is kept only in memory.
  4. Recon restarts before the rebuild finishes, so the request is lost.
  5. On startup: is 1000 behind 600? No, so Recon skips the rebuild and keeps showing data from the old OM.

Fix: rebuild when the numbers are different (!= 0) instead of only when the task is behind (< 0). The normal cases (behind → rebuild, equal → no rebuild) don't change. The check is moved into a small helper, getTasksToReprocessOnStartup, so it can be unit tested.

This is rare (OM going back in time and a Recon restart during the rebuild), but when it happens the stale data is silent and never fixes itself.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-16462

How was this patch tested?

  • New unit test testStartupReconciliationReprocessesTaskOnLineageRegression covers ahead, behind and equal.
  • TestOzoneManagerServiceProviderImpl passes (17/17), and checkstyle is clean.

… permanently serves stale derived data

Co-Authored-By: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant