[ivandika3]

What changes were proposed in this pull request?

This patch introduces the FCR lease mechanism similar to HDFS block report lease mechanism to bound the memory overhead of SCM due to concurrent full container reports. See https://issues.apache.org/jira/browse/HDDS-15330 for the background issues.

Overall flow

1. A datanode marks its FCR pending / deferred and requests a lease through a normal heartbeat with a request heartbeat with empty container report and fullContainerReportDeferred set.
2. SCM grants a nonzero lease ID and term if eligible and capacity is available
   1. ozone.scm.max.full.container.report.leases is the maximum number of concurrent FCR that SCM allows at one time. It acts as the rate limit
3. The datanode immediately sends a follow-up heartbeat containing the FCR and the FCR lease.
4. SCM validates the lease, then queues and processes the report
5. Completion or discard release the lease and allow new FCR to be handled.

If no lease is granted, the datanode retries on subsequent heartbeats. Other reports and commands continue normally.

Lease Lifecycle

The following is the lease lifecycleYou can refer to the SCMFullContainerReportLeaseManager (state is in LeaseState).

The main flow is as follow

NO LEASE
    |
    | Lease request granted: reserve capacity
    v
 GRANTED
    |
    | Valid leased FCR received: claimLease()
    v
 CLAIMED
    |
    | Report handler starts: startProcessing()
    v
PROCESSING
    |
    | Processing succeeds or fails: complete(processed)
    v
NO LEASE

However, there are also other transitions (e.g. lease expiry, replacement request, new registration, etc).

GRANTED --[replacement request granted; new lease ID]--> GRANTED

GRANTED --[grant expiry detected]----------------------> NO LEASE
GRANTED --[lease request prunes grants from other terms]> NO LEASE
GRANTED --[new deferred registration]------------------> NO LEASE
GRANTED --[datanode removed from SCM]------------------> NO LEASE

CLAIMED --[dispatch rejected or publication fails]-----> NO LEASE
CLAIMED --[queued report discarded or queue cleared]---> NO LEASE

Notable properties

• GRANTED, CLAIMED, and PROCESSING all reserve capacity
  • CLAIMED will be invoked before it's queued in the EventQueue to be processed
• Only GRANTED expires since once it goes to CLAIMED and PROCESSING, the FCR has been received and should be completely handled by SCM (not waiting for datanode).
• Expiry is checked during lease requests or report validation, not by a timer.
  • This is to simplify the logic. Having a thread to expire the leases would need more bookkeepings. The number of leases should not be too large so we can afford outstanding leases and we can allow lease operation to expire the existing leases.

SCM HA

The FCR lease is not replicated through Raft and the lease ID is not generated using the SCM HA sequence ID generator. This is because container report (FCR and ICR) processing is handled by both the SCM leader and followers. Therefore, we only need that the lease ID for each SCM node to be different.

Cross Cutting Concerns

• Feature flag
  • ozone.scm.fullcontainer.report.rate.limit.enabled flag is introduced with default disabled. This would allow us to revert
    back to accepting all FCR report if for example it causes SCM to be in SafeMode longer than necessary.
  • ozone.scm.fullcontainer.report.rate.limit.enabled is reconfigurable per SCM node. This allows us to evaluate the FCR lease mechanism and disable it quickly if there are issues.
• Compatibility: This patch should allow rolling upgrade
  • New SCM and old datanode: Old datanode immediately sends FCR. In that case, SCM would bypass the lease mechanism and handle the container report immediately.
  • Old SCM and new datanode: The datanode would be able to check whether SCM supports the FCR lease through HddsVersion.FULL_CONTAINER_REPORT_LEASE in the software version.

Evaluations

There are tradeoffs on enabling the FCR lease mechanism

Advantages

• The number of FCR that can exist in SCM (RPC queue, Event queue, processing) is bounded by ozone.scm.max.full.container.report.leases
  • Therefore in a cluster with a lot of dense nodes or a lot of small containers, this would reduce the memory requirement of SCM (cost saving) and allow more stable SCM (less JVM pressure from container reports)

Disadvantages

• SafeMode requires that a certain number of containers are going to be recorded. The FCR lease mechanism can increase the time it takes for SCM to exit SafeMode.
  • We can either reduce the heartbeat interval or increase the FCR lease limit for this, or disable the feature flag if it's too long
• There is an extra one RTT for normal heartbeat even if lease is disabled in SCM
  • The datanode will first send an empty FCR with the deferred flag true and only send the actual FCR once SCM grants the lease or FCR lease is disabled.
• Heartbeats processing might be delayed indefinitely (starvation). There is not fairness guarantee for the FCR lease so technically it is possible for a single datanode heartbeat to get rejected since it cannot get the lease. The fairness improvement can be handled in future patch.
  • HDFS block lease mechanism also doesn't have the fairness guarantee and seems there are no reported issues, so it might be ok for now.
  • It might be possible that we can have implementation so that SCM prioritize DN that have gone longest without successfully processed FCR. Also we might also make the DNs that has their first registration FCR to receive a special priority.

Generated-by: GPT 6.1 Sol

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-15330

How was this patch tested?

UT, IT. (Clean CI: https://github.com/ivandika3/ozone/actions/runs/41396339409)