Skip to content

streaming: wait 125% of the child reboot time before reporting it offline - #24343

Merged
stelfrag merged 1 commit into
netdata:masterfrom
stelfrag:fix-child-disc-tolerance
Oct 16, 2026
Merged

stelfrag merged 1 commit into
netdata:masterfrom
stelfrag:fix-child-disc-tolerance

Conversation

@stelfrag

@stelfrag stelfrag commented Oct 15, 2026 •

Copy link
Copy Markdown
Collaborator
Summary

When a child disconnects, the parent waits before telling Cloud the child is offline. The intended wait is 125% of the child's reboot time, capped at 30 s (#18712). However, MAX_CHILD_DISC_TOLERANCE was (125 / 100), which is integer 1, so the parent waited only 100%. A child that reconnects in the remaining 25% window caused an avoidable offline/online pair in Cloud.

The change:

  • computes the delay as reboot time × 125 / 100, capping the reboot time first so the multiplication cannot overflow;
  • floors every schedule_node_state_update() delay at 10 ms. A child with no reboot time otherwise armed a 0 ms libuv timer, which Avoid zero timeout in libuv timers #19810 avoided for the other callers.
Test Plan
  • Builds cleanly.
  • The other callers already pass 10 ms or more, so their delays are unchanged.

Summary by CodeRabbit

  • Bug Fixes
    • Scheduled node-state updates now observe a minimum delay of 10 ms, preventing shorter requested delays from being queued.
    • Child-disconnect update timing now applies a 25% tolerance to reboot time capped at 30 seconds, with the final delay also capped at 30 seconds. This avoids incorrect timing calculations and keeps disconnect delays within the intended limit.

…line

MAX_CHILD_DISC_TOLERANCE was (125 / 100), which is 1 in integer
arithmetic, so the parent waited 100% of the child's reboot time instead
of the intended 125% before telling Cloud the child is offline.

Compute the delay as reboot time * 125 / 100, capped at 30s, and floor
every schedule_node_state_update() delay at 10ms so a child with no
reboot time does not arm a 0ms libuv timer.
@coderabbitai

coderabbitai Bot commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 30ea0c50-402a-453b-ac8c-2a7459a3c86b
📥 Commits

Reviewing files that changed from the base of the PR and between c2364be and 57d9ecd.

📒 Files selected for processing: 2
  • src/database/sqlite/sqlite_aclk.c
  • src/streaming/stream-receiver.c

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 8 remain after this review.

📝 Walkthrough

Node-state update scheduling now applies a minimum delay before SQLite queues an update. Receiver reboot delays use a 125% tolerance, with a 30-second cap applied before and after the tolerance calculation.

Changes: Node-state update delay

Layer / File(s) Summary
Calculate and queue bounded update delays
src/streaming/stream-receiver.c, src/database/sqlite/sqlite_aclk.c
rrdhost_clear_receiver caps reboot time at 30 seconds, applies the 125% tolerance, then caps the delay at 30 seconds. schedule_node_state_update clamps the queued delay to at least 10 ms.

Priority: ⬇️ Low

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix

Merge Risk: ⚪ Minimal · up to 57d9e

The change bounds child-offline update delays and applies the intended reboot-time tolerance. The inspected timing paths show no concrete merge-blocking issue.

### Architecture Summary

Architecture risk: 🔵 Low · up to 57d9e

The change affects 1 system.

Changed systems: src

Architecture concerns
No architecture-level concerns identified.

Review details

Systems and components

  • observed — src (service) was modified; 2 changed files map to changed impact.

Before / after behavior

  • observed — Modified behavior in src/database/sqlite/sqlite_aclk.c: NODE_STATE_UPDATE_MIN_DELAY_MS is defined as 10 ms, and schedule_node_state_update now clamps the requested delay to at least that value before queuing it. Previously, the supplied delay was used unchanged.
  • observed — Modified behavior in src/streaming/stream-receiver.c: The tolerance constant changes from the integer ratio 125 / 100 (which evaluates to 1) to the percentage value 125.
  • observed — Modified behavior in src/streaming/stream-receiver.c: rrdhost_clear_receiver now caps total reboot time at 30 seconds before applying the 125% tolerance, then caps the scheduled delay at 30 seconds. Previously, it applied the truncated 1× ratio to uncapped reboot time before the delay cap.
Pre-merge checks: Passed 4 · Failed 1

Failed Failed checks: 1 warning

Check name Status Explanation Resolution
Docstring Coverage Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 3 functions across 2 files. Write docstrings for the functions missing them to satisfy the coverage threshold.
Passed Passed checks: 4 passed
Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed The title clearly and concisely describes the primary change: waiting 125% of the child reboot time before reporting the child offline.
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
  • Fix all pre-merge checks with AI
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 2 files

Confidence score: 5/5

  • Automated review surfaced no issues in the provided summaries.
  • No files require special attention.
Architecture diagram
sequenceDiagram
    participant CHILD as Child Node
    participant PARENT as Parent Stream Receiver
    participant SCHED as schedule_node_state_update()
    participant QUEUE as ACLK Sync Queue
    participant CLOUD as Netdata Cloud

    Note over CHILD,CLOUD: Child Disconnect Detection and State Update Flow

    CHILD-->>PARENT: CHANGED: Disconnect detected
    PARENT->>PARENT: rrdhost_clear_receiver()
    PARENT->>PARENT: Clear online flag
    PARENT->>PARENT: Read total reboot time from stream path
    
    alt CHANGED: Reboot time present
        PARENT->>PARENT: CHANGED: Cap reboot time at 30s (MAX_CHILD_DISC_DELAY)
        PARENT->>PARENT: CHANGED: Calculate delay = reboot × 125 / 100, cap at 30s
    else No reboot time (0ms)
        PARENT->>PARENT: CHANGED: Delay = 0
    end

    PARENT->>SCHED: schedule_node_state_update(host, delay)
    SCHED->>SCHED: CHANGED: Floor delay at 10 ms (NODE_STATE_UPDATE_MIN_DELAY_MS)
    SCHED->>QUEUE: queue_aclk_sync_cmd(ACLK_DATABASE_NODE_STATE, host, delay)

    alt Delay elapsed (happy path)
        QUEUE-->>CLOUD: Report child offline
    else CHANGED: Child reconnects within 25% tolerance window
        CHILD-->>PARENT: Reconnect
        PARENT->>QUEUE: Cancel pending offline update
        QUEUE-->>CLOUD: Keep child online
    end
Loading

View guided diff | Re-trigger cubic

@sonarqubecloud

Copy link
Copy Markdown

@stelfrag
stelfrag marked this pull request as ready for review October 15, 2026 21:42

@thiagoftsm thiagoftsm left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Streaming is working with notification of offline appearing as expected. LGTM!

@stelfrag
stelfrag merged commit 9fbc835 into netdata:master Oct 16, 2026
158 of 159 checks passed
@stelfrag
stelfrag deleted the fix-child-disc-tolerance branch October 16, 2026 05:03
stelfrag added a commit to stelfrag/netdata that referenced this pull request Oct 20, 2026
…line (netdata#24343)

MAX_CHILD_DISC_TOLERANCE was (125 / 100), which is 1 in integer
arithmetic, so the parent waited 100% of the child's reboot time instead
of the intended 125% before telling Cloud the child is offline.

Compute the delay as reboot time * 125 / 100, capped at 30s, and floor
every schedule_node_state_update() delay at 10ms so a child with no
reboot time does not arm a 0ms libuv timer.

(cherry picked from commit 9fbc835)
@stelfrag stelfrag mentioned this pull request Oct 16, 2026
Ferroin pushed a commit that referenced this pull request Oct 23, 2026
…line (#24343)

MAX_CHILD_DISC_TOLERANCE was (125 / 100), which is 1 in integer
arithmetic, so the parent waited 100% of the child's reboot time instead
of the intended 125% before telling Cloud the child is offline.

Compute the delay as reboot time * 125 / 100, capped at 30s, and floor
every schedule_node_state_update() delay at 10ms so a child with no
reboot time does not arm a 0ms libuv timer.

(cherry picked from commit 9fbc835)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants