Repository navigation
Description
Summary
Two incidents in ~13 hours on the same Raspberry Pi 5. Both began with macb ... end0: Link is Down after hours of healthy operation. Both left the driver's hardware statistics returning garbage. Only a reboot recovered. Incident 1 affected Ethernet only (WiFi kept working). Incident 2 froze the whole machine, with a CPU stuck spinning in rp1_adc_read.
This looks like the RP1 chip (or the PCIe path to it) wedging, with Ethernet as one symptom. Possibly related to #7345 and #6948, and the forum thread "Pi 5 kernel oops at xhci_halt, intermittent FC0 timeouts on pll_sys_sec/clk_eth, RP1 defect" (forums.raspberrypi.com, p=2375780). I have not confirmed the root cause.
Environment
- Raspberry Pi 5 Model B Rev 1.0 (
Revision: d04170), 64-bit Raspberry Pi OS (Debian 12 bookworm) - Incidents on kernel
6.12.96+rpt-rpi-2712(1:6.12.96-1+rpt1); now running6.12.109+rpt-rpi-2712(not yet observed long enough to say whether it helps) - Bootloader EEPROM 2026-05-26 (1779807685), up to date per
rpi-eeprom-update; firmware086b83e3(2026/05/26);raspi-firmware1.20260907 (now 1.20260915) - Power:
vcgencmd get_throttled=0x0when checked after the fact; no undervoltage messages found in the kernel journal - Cmdline includes
pci=pcie_bus_safe;arm_boost=1; no Ethernet-related overlays or dtparams - The Ethernet link has negotiated 100 Mbps/Full on every boot observed (2026-09-26, 09-27, 09-28), never 1000, although the PHY advertises 1000baseT
- Workload: Docker (about 16 containers) incl. Jellyfin and a Netdata container with
/sysmounted read-only from the host (pid: host); this is a home server - WiFi (
wlan0) is on1001100000.mmc(SoC SDIO), not behind RP1
Incident 1: 2026-09-27 17:38, Ethernet only
Sep 26 19:28:42 kernel: macb 1f00100000.ethernet end0: Link is Up - 100Mbps/Full - flow control off
Sep 27 17:38:01 kernel: macb 1f00100000.ethernet end0: Link is Down
- Roughly 22 h of clean operation before (0 rx/tx errors, gateway ping ~0.5 ms; a watcher script logged snapshots every 5 min).
- No kernel warning before the drop; only unrelated Docker veth churn earlier at 17:02 and 17:10. SoC temperature 56-58 C.
- Immediately after the drop the counters were nonsense, e.g.
ip -s linkRX errors4294967208and, five minutes later,ethtool -S end0tx_frames: 609886253403whileq0_tx_packetsstayed at the pre-drop value (855376). ethtool end0:Link detected: no,master-slave status: resolution error.- NetworkManager later moved the device to
unavailable. WiFi stayed up and the Pi remained reachable over it. - Cable re-seated (by the owner): no effect.
ip link set end0 down/up: no effect.- Unbind/rebind of the driver made it worse:
echo 1f00100000.ethernet > /sys/bus/platform/drivers/macb/unbind (ok)
echo 1f00100000.ethernet > /sys/bus/platform/drivers/macb/bind (Invalid argument)
macb 1f00100000.ethernet: gem-ptp-timer ptp clock unregistered.
rp1-clk 1f00018000.clocks: pll_sys_sec: FC0 busy timeout
rp1-clk 1f00018000.clocks: clk_eth: FC0 busy timeout
rp1-clk 1f00018000.clocks: clk_eth_tsu: FC0 busy timeout
macb 1f00100000.ethernet end0: PHY [...] driver [Broadcom BCM54213PE] (irq=POLL)
macb 1f00100000.ethernet: gem-ptp-timer ptp clock registered.
macb 1f00100000.ethernet: gem-ptp-timer ptp clock unregistered.
macb 1f00100000.ethernet: error -ENXIO: IRQ index 1 not found
macb 1f00100000.ethernet: Unable to request IRQ -6 (error -22)
macb 1f00100000.ethernet: probe with driver macb failed with error -22
end0 no longer existed after this. A reboot restored it (link up, 100 Mbps).
Incident 2: 2026-09-28 02:44, whole-system freeze
Healthy until 02:40 (gateway ping 0.5 ms via end0, zero errors). Then:
02:44:10 kernel: macb 1f00100000.ethernet end0: Link is Down
02:44:11 (watcher) ping via end0 FAILED; rx_err=4294967288 tx_err=4294967288 <- same garbage counters
02:44:31 kernel: rcu: INFO: rcu_preempt self-detected stall on CPU
CPU: 1 UID: 0 PID: 79872 Comm: LIBSENSORS Not tainted 6.12.96+rpt-rpi-2712 #1
pc : queued_spin_lock_slowpath+0x78/0x448
lr : _raw_spin_lock+0x64/0x78
Call trace:
queued_spin_lock_slowpath
_raw_spin_lock
rp1_adc_read+0x34/0x108 [rp1_adc]
rp1_adc_show+0x44/0xc0 [rp1_adc]
dev_attr_show
sysfs_kf_seq_show
kernfs_seq_show / seq_read_iter / kernfs_fop_read_iter / vfs_read / ksys_read
- The stall was detected 21 s after the link drop (the default RCU stall timeout), which puts the start of the stall at about the time of the link drop.
LIBSENSORSis presumably the Netdata sensors collector reading/sys/class/hwmon/hwmon1(rp1_adc). I could not confirm the thread name after the reboot.- The lock holder is unknown. My hypothesis is that another context was blocked in an MMIO access to an unresponsive RP1, but I have not verified this.
- The system stayed wedged: the same stall backtrace was still being printed at 06:14, with timestamps stretching apart (minutes between consecutive lines of one backtrace), and a bash watcher loop (ethtool/ip/ping every 15 s) wrote nothing after 02:44:11. Rebooted manually at ~06:20.
- No AER/PCIe error lines were seen in the previous boot's kernel log around the event (only the normal boot-time messages; I did not search exhaustively).
What I am asking
- Is this a known RP1 hang (clock/IRQ/PCIe) with a fix in a newer kernel or firmware? Is there a debug parameter or trace that would show whether RP1 stops responding on PCIe when this happens?
- Is the
rp1_adchwmon read path expected to spin on a lock indefinitely when RP1 is unresponsive? If so, can polling hwmon during an RP1 hang take the whole system down? - Should the link-down path in
macbon RP1 reset anything else? Garbage statistics right after "Link is Down" suggest the GEM registers are already unreadable at that point. - The link negotiating only 100 Mbps on a gigabit-capable port may point to a marginal cable or a 100 Mbps-only switch port. Could a flapping physical link be what triggers the wedge?
What has been tried
- Re-seated the cable,
ip linktoggle, driver unbind/rebind (all failed, see above); reboot fixes it - Kernel/firmware update
6.12.96to6.12.109and raspi-firmware1.20260907to1.20260915(result pending) - Enabled the 15 s hardware watchdog via systemd and an automatic reboot after 5 min of unhealthy
end0, as a workaround
Attachments available
/var/log/end0-watch.log: fullethtool/ethtool -S/ip -s -d link/nmcli/dmesgdumps at each state change and every 5 min, covering both incidentsjournalctl -b -1 -kfor the incident-2 boot (contains the full RCU stall traces)
Activity
nbuchwitz commentedon Sep 28, 2026
AFAIU everything after the link drop is RP1 reading back
0xFFFFFFFF. BMCR 0xFFFF will result in the "Link is Down" etc. The RC doesn't log anything, so a dead PCIe link to RP1 would not show up in dmesg.The rp1 hwmon issue is not the cause of the freeze, but worth fixing.
When it happens again, please check the output of (before rebooting):
What is used as link partner? Did it negotiate a gbit link before? Interface name also looks a bit unusual as the udev rules should have renamed it to eth0 on stock Rpi OS. Did you disable this deliberately?
tushar-arora1 commentedon Sep 28, 2026
Thanks for looking at this.
Cable and speed
The Pi is connected with one Cat6 cable straight to my router. There is no switch in between. It was stuck at 100 Mbps on all 4 LAN ports of the router.
ethtool end0showed the other side only offering 10 and 100 (no 1000), even though the Pi offers 1000.I swapped the cable. Now it connects at 1000 Mb/s, with 0 CRC errors and 0 symbol errors. So the old cable was bad. As far as I can see, it was always 100 Mbps before.
Interface name
I did not turn off any rename rule on purpose. There is no
net.ifnamesin cmdline.txt, and nothing in /etc/udev/rules.d or /etc/systemd/network that renames it. The Pi OS network packages are all there, with no files missing. So I don't know why it is calledend0and noteth0.How the OS was installed
It is the official Raspberry Pi OS with desktop, the 2023-12-05 image (
/etc/rpi-issuesays pi-gen stage4). I have updated it since then (Debian 12, kernel 6.12.109+rpt-rpi-2712). I also put CasaOS on it at some point and removed it later, together with Docker and some desktop packages (Sept 2026). So it is not a clean image.Right now (network is working)
Next time it happens
My Pi reboots by itself 5 minutes after the network dies, so I can't run your commands by hand first. I added them to a script that runs right before the reboot: your two lspci commands, plus the full
lspci -vvfor 0002:00:00.0 and 0002:01:00.0. I will post the output the next time it drops.hwmon: rp1-adc: fix spinlock leak on errors
hwmon: rp1-adc: fix spinlock leak on errors
hwmon: rp1-adc: fix spinlock leak on errors
hwmon: rp1-adc: fix spinlock leak on errors
hwmon: rp1-adc: fix spinlock leak on errors
hwmon: rp1-adc: fix spinlock leak on errors
hwmon: rp1-adc: fix spinlock leak on errors
hwmon: rp1-adc: fix spinlock leak on errors
hwmon: rp1-adc: fix spinlock leak on errors
hwmon: rp1-adc: fix spinlock leak on errors
hwmon: rp1-adc: fix spinlock leak on errors
hwmon: rp1-adc: fix spinlock leak on errors
hwmon: rp1-adc: fix spinlock leak on errors
hwmon: rp1-adc: fix spinlock leak on errors