Hi guys,
Issue seemingly with a certain NIC type on some of our kit we’ve noticed for a while but only just been able to investigate fully. Wondering if anyone else has this issue with a similar
FOG Version: 1.5.10.2149
FOS Kernel: 6.18.38
Init Version: 20260804
Server OS: Ubuntu (master + 2 x storage nodes)
Client Hardware: Dell OptiPlex 3070
ISSUE UEFI image deployments run at around 1.2 GB/min (~100 Mbps) on Dell OptiPlex 3070 machines. Legacy/BIOS deployments on the same hardware and infrastructure run at 6-10 GB/min as expected. UEFI uploads (capture) are also fast at ~15 GB/min so the issue is specific to UEFI downloads only.
THINGS ALREADY RULED OUT:
- Patch lead (swapped, no difference)
- Image compression (tested ZSTD and Partclone, identical speed)
- RAID vs AHCI (no difference)
- Storage node (tested all 3 nodes, same result)
- Dell BIOS (already on latest v1.35)
- iPXE boot file (tested ipxe.efi and snponly.efi, no difference)
- Kernel arg r8169.aspm=0 (no difference)
- Link speed (confirmed 1000Mb/s Full Duplex via ethtool)
ROOT CAUSE IDENTIFIED lspci in a FOG debug task on the affected 3070 shows: Realtek RTL8111/8168/8211/8411 PCI Express Gigabit Ethernet Controller (rev 15) dmesg | grep r816 shows: r8169 0000:01:00.0 eth0: RTL8168h/8111h r8169 0000:01:00.0 eth0: jumbo frames [frames: 9194 bytes, tx checksumming: ko] r8169 0000:01:00.0 enp1s0: No native access to PCI extended config space, falling back to CSI r8169 0000:01:00.0 enp1s0: Link is Up - 1Gbps/Full - flow control off The key issues are tx checksumming: ko and the CSI fallback, both known performance problems with the r8169 driver on RTL8168h rev 15 chips.
SUSPECTED CAUSE I noticed this recent commit in the FOS repo: Switch x64/arm64 kernels to in-kernel r8169 for RTL8168/8125/8126 I believe this switch from r8168 to r8169 may have introduced the regression. This is documented in Debian Bug #1110193 (filed July 2025): https://bugs.debian.org/1110193 The reporter has identical lspci output and the same Rx-only slowdown, resolved by installing r8168-dkms.
WORKAROUND Booting in Legacy/BIOS mode for imaging works but is not practical long term.
REQUEST Would it be possible to revert the r8169 switch for RTL8168h rev 15 chips or provide an experimental kernel build with r8168 re-enabled? Happy to test any experimental builds and report back. r8168 driver code is already in the FOS repo at: https://github.com/FOGProject/fos/tree/master/KernelPackages/drivers/net/ethernet/realtek
Thanks in advance guys.