UEFI imaging slow (~1.2 GB/min) on Realtek RTL8168h (rev 15) - r8169 CSI fallback - possible regression from r8169 switch?
-
Hi guys,
Issue seemingly with a certain NIC type on some of our kit we’ve noticed for a while but only just been able to investigate fully. Wondering if anyone else has this issue with a similar
FOG Version: 1.5.10.2149
FOS Kernel: 6.18.38
Init Version: 20260804
Server OS: Ubuntu (master + 2 x storage nodes)
Client Hardware: Dell OptiPlex 3070ISSUE UEFI image deployments run at around 1.2 GB/min (~100 Mbps) on Dell OptiPlex 3070 machines. Legacy/BIOS deployments on the same hardware and infrastructure run at 6-10 GB/min as expected. UEFI uploads (capture) are also fast at ~15 GB/min so the issue is specific to UEFI downloads only.
THINGS ALREADY RULED OUT:
- Patch lead (swapped, no difference)
- Image compression (tested ZSTD and Partclone, identical speed)
- RAID vs AHCI (no difference)
- Storage node (tested all 3 nodes, same result)
- Dell BIOS (already on latest v1.35)
- iPXE boot file (tested ipxe.efi and snponly.efi, no difference)
- Kernel arg r8169.aspm=0 (no difference)
- Link speed (confirmed 1000Mb/s Full Duplex via ethtool)
ROOT CAUSE IDENTIFIED lspci in a FOG debug task on the affected 3070 shows: Realtek RTL8111/8168/8211/8411 PCI Express Gigabit Ethernet Controller (rev 15) dmesg | grep r816 shows: r8169 0000:01:00.0 eth0: RTL8168h/8111h r8169 0000:01:00.0 eth0: jumbo frames [frames: 9194 bytes, tx checksumming: ko] r8169 0000:01:00.0 enp1s0: No native access to PCI extended config space, falling back to CSI r8169 0000:01:00.0 enp1s0: Link is Up - 1Gbps/Full - flow control off The key issues are tx checksumming: ko and the CSI fallback, both known performance problems with the r8169 driver on RTL8168h rev 15 chips.
SUSPECTED CAUSE I noticed this recent commit in the FOS repo: Switch x64/arm64 kernels to in-kernel r8169 for RTL8168/8125/8126 I believe this switch from r8168 to r8169 may have introduced the regression. This is documented in Debian Bug #1110193 (filed July 2025): https://bugs.debian.org/1110193 The reporter has identical lspci output and the same Rx-only slowdown, resolved by installing r8168-dkms.
WORKAROUND Booting in Legacy/BIOS mode for imaging works but is not practical long term.
REQUEST Would it be possible to revert the r8169 switch for RTL8168h rev 15 chips or provide an experimental kernel build with r8168 re-enabled? Happy to test any experimental builds and report back. r8168 driver code is already in the FOS repo at: https://github.com/FOGProject/fos/tree/master/KernelPackages/drivers/net/ethernet/realtek
Thanks in advance guys.
-
@rpycroft Thank you for the writeup. The elimination you did up front — cables,
link speed, compression, RAID vs AHCI, all three storage nodes, BIOS, iPXE
files — saved me a lot of guessing, and the commit you linked is the right one.You’re right that it’s the r8169 switch, but not quite in the way it looks.
The in-kernel driver isn’t worse than the vendor one here. It exposed something
we’d had broken for nine years without knowing.Two kernel options that are on by default upstream have been off in the FOS
kernel config since 2016:# CONFIG_PCIEASPM is not set # CONFIG_PCI_MMCONFIG is not setASPM is PCIe link power management — it lets the link drop into a low power
state when idle and wake when there’s traffic. Waking up is the expensive part.Nobody noticed for nine years because the Realtek vendor drivers we used to
ship handled ASPM themselves. Our r8168 build had dynamic ASPM compiled in,
which effectively switched ASPM off any time packets were moving. It didn’t
matter that the kernel couldn’t manage it, because the driver never asked the
kernel to.r8169 does it the normal way: it asks the PCI core to disable L1 once, when the
card is probed. That’s where this bites. With CONFIG_PCIEASPM off, the function
it calls isn’t a function at all — it’s a stub that returns “success” without
doing anything. So r8169 is told the OS disabled L1, marks ASPM as being under
OS control, and then goes on to enable ASPM and L1.2 on the card — on a
kernel with no ASPM code in it that disabled nothing. Not one line in the logs
to say so.That “falling back to CSI” line you included is the second option. Without
CONFIG_PCI_MMCONFIG the kernel only sees the first 256 bytes of each device’s
config space, and the registers controlling the L1 sub-states live past that.
So even a kernel that wanted to fix this couldn’t reach them. Including that
line is what let me tie the two together, so thank you for pasting it.That also explains the two things in your report that looked strangest:
Why UEFI and not legacy — ASPM is programmed by the firmware, and Dell’s UEFI
path turns L1 on where the legacy/CSM path leaves it off. Our kernel couldn’t
change it either way, so whatever the firmware picked stuck. Legacy was never
actually faster; you were getting a machine where ASPM was already off.Why deploys and not captures — a deploy is your client receiving. The link
goes quiet between bursts from the server, drops into L1.2, and pays the wake
cost over and over. A capture is the client sending, which keeps the link
busy so it never gets the chance to sleep. That’s your 15 GB/min upload sitting
next to a 1.2 GB/min download on the same cable.Worth saying before anyone suggests it: adding pcie_aspm=off to the kernel
arguments does nothing on a FOS kernel. The code registering that argument is
inside the same #ifdef, so it’s compiled out too. It would look like you tried
the standard fix and it didn’t help.I’ve built you an experimental x64 kernel with both options turned back on:
https://github.com/FOGProject/fos/releases/tag/EXP_20260805-123232 sha256 6fadc7204889bc76fe943520319044940c24654365a523d5df7adf8327a88d25It’s the kernel only — your init is untouched and doesn’t need changing. Back
up your current /var/www/html/fog/service/ipxe/bzImage, drop this one in its
place, and deploy to one of the 3070s. Putting the old file back is the whole
rollback if it doesn’t help.If you want to confirm the diagnosis yourself first, it takes about a minute
and doesn’t need my kernel at all. Boot a 3070 to a shell in debug mode:lspci -nn | grep -i ethernet # note the Realtek address, e.g. 02:00.0 lspci -t # note the root port above it, e.g. 00:1c.5 setpci -s 02:00.0 CAP_EXP+10.w setpci -s 00:1c.5 CAP_EXP+10.wThe bottom two bits of each value are the ASPM setting — 0 is off, 2 is L1,
3 is L0s+L1. Run it once booted UEFI and once booted legacy. If I have this
right, UEFI shows a 2 or a 3 and legacy shows a 0.And to watch it fix itself, clear both (root port first) and rerun the deploy
without rebooting:setpci -s 00:1c.5 CAP_EXP+10.w=0000 setpci -s 02:00.0 CAP_EXP+10.w=0000That’s not a fix you can keep — it’s gone on the next boot — but if the speed
jumps back to 6-10 GB/min, that confirms it before you swap any files.One loose end: could you post the raw output of
ethtool -k <interface>? You
mentioned tx checksumming being off and I want to be sure I’m not waving away a
second, separate problem. It shouldn’t be able to cause a download-only
slowdown, but I’d rather look than assume.Thanks for the report and for offering to test — I’m taking you up on it.
-
This post is deleted! -
Experimental kernel EXP_20260805-123232 fixes it completely!
UEFI download speed on the OptiPlex 3070 has jumped from ~1.2 GB/min to ~6.5 GB/min. We believe the small remaining gap from our usual ~10 GB/min is down to a suspect patch lead/socket on the test machine rather than anything kernel related. We’ll confirm with a better cable shortly.
Legacy mode speeds are unchanged and still fast as expected.
One thing worth highlighting from the ethtool -k output below.
Before the experimental kernel, dmesg was showing:
tx checksumming: ko
With the experimental kernel, ethtool -k now shows: rx-checksumming: on tx-checksumming: on tx-checksum-ipv4: on
So it looks like the experimental kernel has fixed both the ASPM issue and the checksumming issue in one go. Thought that was worth calling out given you mentioned you wanted to rule out a second separate problem.
ethtool -k enp1s0 | head -50
Features for enp1s0:
rx-checksumming: on
tx-checksumming: on
tx-checksum-ipv4: on
tx-checksum-ip-generic: off [fixed]
tx-checksum-ipv6: on
tx-checksum-fcoe-crc: off [fixed]
tx-checksum-sctp: off [fixed]
scatter-gather: on
tx-scatter-gather: on
tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
tx-tcp-segmentation: on
tx-tcp-ecn-segmentation: off [fixed]
tx-tcp-mangleid-segmentation: off
tx-tcp6-segmentation: on
tx-tcp-accecn-segmentation: off [fixed]
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: on [fixed]
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-gre-csum-segmentation: off [fixed]
tx-ipxip4-segmentation: off [fixed]
tx-ipxip6-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-udp_tnl-csum-segmentation: off [fixed]
tx-gso-partial: off [fixed]
tx-tunnel-remcsum-segmentation: off [fixed]
tx-sctp-segmentation: off [fixed]
tx-esp-segmentation: off [fixed]
tx-udp-segmentation: off [fixed]
tx-gso-list: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off
rx-all: off
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
hw-tc-offload: off [fixed]Init Version: 20260804 (unchanged as expected)
Thanks again for the incredibly quick turnaround and the detailed explanation of the ASPM/CONFIG_PCIEASPM root cause.
Very much appreciated. Happy to test anything further if needed and please let me know if I missed any of your diagnostic steps happy to take another stab at it or move on to another check if it helps.
Thanks
Richard -
@rpycroft That’s the confirmation I needed, thank you. 1.2 to 6.5 GB/min is
the fix working. The change is merged now, so it’ll be in the next kernel
release rather than something you have to keep hand-installing.Two corrections from my end.
First, tx checksumming: ko was never broken. That line is about jumbo frames –
it’s the driver saying this chip can’t do TX checksum offload if you enable
jumbo, and at a normal 1500 MTU it never applies. It’s printed based on the
chip model alone, so you’ll still see it on the new kernel. Don’t let it worry
you when you spot it again. Your ethtool output was very likely the same before
the swap; there was nothing to fix there.Second, I got two details wrong in the setpci commands I posted. My apologies.
The write should be masked so it only touches the two ASPM bits – as I wrote
it, it would have cleared the whole Link Control register, including the common
clock configuration, which you don’t want to disturb on a live link. And the
order was backwards: the spec wants L1 switched off at the NIC before the root
port, not the other way round.Reading the current state (this part was fine):
setpci -s 02:00.0 CAP_EXP+10.w setpci -s 00:1c.5 CAP_EXP+10.wOnly the last hex digit matters:
0 4 8 C -> ASPM off 1 5 9 D -> L0s 2 6 A E -> L1 3 7 B F -> L0s + L1And clearing it, NIC first, masked to the ASPM bits:
setpci -s 02:00.0 CAP_EXP+10.w=0000:0003 setpci -s 00:1c.5 CAP_EXP+10.w=0000:0003If you did already run the originals, nothing you’ve reported suggests any harm
came of it, and a reboot clears the lot either way.No pressure on this next bit at all – the fix is confirmed and merged, and
this doesn’t gate anything. But if you ever have one of those 3070s spare, the
read-only pair above run once under UEFI and once under legacy would tell us
why the two differ. My theory is the Dell firmware enables L1 on the UEFI
path and leaves it off under CSM, which would explain everything you saw, but
that’s still me inferring rather than knowing. Genuinely optional – if it’s
easy, great, and if not, don’t spend your time on it.Thanks again for testing this so quickly.
