Skip to content

Agent LXC filesystem collection starves later containers because every pct df shares one 10s budget #1784

Closed

Description

@pulse-triage

Problem

Reported by @avsdev-cw in #1477 (comment). On hosts with many running containers only the first handful report per-mount filesystem usage; the rest fail with signal: killed in the agent debug log. pct df takes the container config lock and costs over a second per guest, and every per-container query shared one 10-second budget with pct list in internal/hostagent/proxmox_lxc_filesystems.go, so the deadline expired after roughly five containers and the remaining queries were killed mid-run.

Fix on main

Commit 8318c74:

  • resolves per-mount usage with statfs through /proc/<pid>/root (config-declared mounts read from /etc/pve/lxc, init PID via lxc-info), which takes no config lock and needs no per-guest subprocess
  • drops config mounts whose device identity matches their parent directory instead of reporting the parent filesystem's numbers for an unmounted path
  • keeps pct df as a per-container fallback for least-privilege installs that cannot traverse /proc/<pid>/root, now with a per-container deadline inside a bounded collection window so one slow guest cannot starve the rest
  • logs a warning naming how many containers were skipped if the collection window is ever exhausted, instead of failing silently at debug level
  • regression coverage pins the statfs path being preferred over pct df, per-container fallback isolation, and config parsing stopping at snapshot sections

Retest boundary

Keep open until a release containing 8318c74 ships and the reported many-container host shows filesystem usage for all running containers (up to the 128-container inventory cap).

How Pulse handles triage

Activity

  1. pulse-triage commented on Aug 27, 2026

    @pulse-triage
    ContributorAuthor

    v6.4.0-rc.6 is now published and contains 8318c74. Please retest on the affected many-container Proxmox host and confirm that every running container reports filesystem usage through a normal collection cycle, including containers beyond the first ten.

    If any row is still missing usage, attach the agent warning/debug lines from that cycle and note whether the agent can traverse /proc//root; that distinguishes the statfs path from the bounded pct df fallback.

    I am keeping this open pending that host-level confirmation.

    How Pulse handles triage

  2. avsdev-cw commented on Aug 27, 2026

    @avsdev-cw

    Confirmed to be working on v6.4.0-rc6

  3. pulse-triage commented on Aug 27, 2026

    @pulse-triage
    ContributorAuthor

    Thanks for confirming the fix on v6.4.0-rc.6. Closing this as verified resolved; the bounded per-container fallback remains covered by regression tests.

    How Pulse handles triage

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingneeds-retest-on-latestReporter should retest on current latest stable release

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions