[ArafatKhan2198]

What changes were proposed in this pull request?

When a Recon task returns a failure, Recon retries the batch and reinitializes the tasks if the retry fails too. But when a task throws an exception, Recon only logs it and moves on, so that batch is lost for that task.

Which errors end up as a throw, per task:

Task	Returns failure (retried)	Throws (batch skipped)
ContainerKeyMapperTask	IOException	any other exception, e.g. NullPointerException (only IOException is caught)
FileSizeCountTask	any error while reading events	any error while saving counts (writeCountsToDB rethrows it as RuntimeException)
NSSummaryTask, OmTableInsightTask	all errors (they catch Exception)	none

Example:

1. Batch 1: "key K1 created". ContainerKeyMapperTask hits a NullPointerException, which isn't an IOException, so it throws.
2. Recon logs the error. No retry, no reinitialization.
3. Batch 2 succeeds, and the task's sequence number moves past batch 1.
4. K1 is missing from the container-key mapping for good, and nothing notices.

Fix: in the exceptionally branch of processTasks, add the task to failedTasks, so a thrown exception is handled like a returned failure (retry, then reinitialize). failedTasks and retryFailedTasks also become synchronized lists, since several threads add to them at the same time.

Retrying is safe because the affected tasks save each batch in one atomic write: if the task threw, nothing from that batch was saved. This is a safety net for unexpected errors such as bugs; normal operation doesn't trigger it.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-17040

How was this patch tested?

• Updated testTaskRecordsFailureOnException: a task that always throws is retried, then requests reinitialization.
• New testTaskThrowingOnceIsRetried: a task that throws once is retried and succeeds.
• The Recon task controller and task suites pass, and checkstyle is clean.