Hi guys,

Legacy (BIOS) issue I wanted to report in case anyone else experiences the same problem. I’m going on leave for a couple of weeks from today (07/08/2026) so may be slow to respond to any follow-up questions, but happy to test further on my return.

We noticed a similar thing when upgrading to 1.5.10.1667 last year but found the below workaround. Since upgrading to 1.5.10.2149 we’re unable to use the below workaround.

Environment:

FOG Version: 1.5.10.2149 (upgraded from 1.5.10.1667)
Server: Ubuntu VM on Hyper-V
DHCP boot file served to Legacy BIOS clients: ipxe.kpxe
Affected hardware: Dell OptiPlex 390, 3010, 3020 (Legacy BIOS)
Unaffected: All UEFI devices

Issue:

After upgrading from 1.5.10.1667 to 1.5.10.2149, all Legacy BIOS devices fail to exit to the local hard drive after a PXE boot when no imaging task is assigned. UEFI devices are completely unaffected.

All exit-to-hard-drive methods were tested and failed with the new ipxe.kpxe:

Exit methods tested with the new ipxe.kpxe – all failed:

GRUB_FIRST_HDD – Fatal: DHCP_ACK failure! / PXE unload fails: 106
SANBOOT – Boot from SAN 0x00 … I/O error / hangs on 0x80
refind_efi – Flashing cursor, no boot (expected as this seems to be the UEFI boot option?)
GRUB_FIRST_FOUND_WINDOWS – OptiPlex 3010 and 3020 passed, OptiPlex 390 failed to get past PXE boot and presented with a BOOTMGR error

Fix:

Restoring ipxe.kpxe from the 1.5.10.1667 stable release commit (aa524d494) resolved the issue completely on all three Legacy BIOS models tested, with GRUB_FIRST_HDD as the exit method:

Shell
1
cd /root/fogproject
2
git show aa524d494:packages/tftp/ipxe.kpxe > /tftpboot/ipxe.kpxe
3
sudo chown fogproject:root /tftpboot/ipxe.kpxe
4
sudo chmod 754 /tftpboot/ipxe.kpxe
5
sudo systemctl restart tftpd-hpa
Show more lines

Then set Exit to Hard Drive Type back to GRUB_FIRST_HDD in FOG Settings.

Confirmed working after fix:

OptiPlex 390 (Legacy BIOS)
OptiPlex 3010 (Legacy BIOS)
OptiPlex 3020 (Legacy BIOS)
UEFI devices (unaffected throughout)

This appears to be a regression in the ipxe.kpxe binary shipped with 1.5.10.2149, consistent with the UEFI-side binary issues reported in GitHub issue #978.

Thanks!

@Tom-Elliott

Thanks so much for your help with this. Happy to feed back any info I can so please do let me know if there’s anything else you need me to test. Will be on annual leave for a couple of weeks after this and have no access to the systems but I think this confirms your suspicion. Please see results below from a Dell OptiPlex 3070:

Addresses on this machine:

NIC: 01:00.0 (Realtek RTL8168h)
Root port: 00:1c.0

UEFI mode:

setpci -s 01:00.0 CAP_EXP+10.w = 0142 (last digit 2 = L1 enabled)
setpci -s 00:1c.0 CAP_EXP+10.w = 0c42 (last digit 2 = L1 enabled)

Legacy/CSM mode:

setpci -s 01:00.0 CAP_EXP+10.w = 0040 (last digit 0 = ASPM off)
setpci -s 00:1c.0 CAP_EXP+10.w = 0c40 (last digit 0 = ASPM off)

Dell’s UEFI firmware enables L1 on both the NIC and root port. The Legacy/CSM path leaves both off. The only difference between the two readings is those two ASPM bits. Everything else in the register is identical.

Thanks for the nudge to run these. Anything else needs testing please do let me know. We love using FOG and it’s invaluable in our set of schools that we work in at least so please do keep up the great work!

@Tom-Elliott

Experimental kernel EXP_20260805-123232 fixes it completely!

UEFI download speed on the OptiPlex 3070 has jumped from ~1.2 GB/min to ~6.5 GB/min. We believe the small remaining gap from our usual ~10 GB/min is down to a suspect patch lead/socket on the test machine rather than anything kernel related. We’ll confirm with a better cable shortly.

Legacy mode speeds are unchanged and still fast as expected.

One thing worth highlighting from the ethtool -k output below.

Before the experimental kernel, dmesg was showing:

tx checksumming: ko

With the experimental kernel, ethtool -k now shows: rx-checksumming: on tx-checksumming: on tx-checksum-ipv4: on

So it looks like the experimental kernel has fixed both the ASPM issue and the checksumming issue in one go. Thought that was worth calling out given you mentioned you wanted to rule out a second separate problem.

ethtool -k enp1s0 | head -50
Features for enp1s0:
rx-checksumming: on
tx-checksumming: on
tx-checksum-ipv4: on
tx-checksum-ip-generic: off [fixed]
tx-checksum-ipv6: on
tx-checksum-fcoe-crc: off [fixed]
tx-checksum-sctp: off [fixed]
scatter-gather: on
tx-scatter-gather: on
tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
tx-tcp-segmentation: on
tx-tcp-ecn-segmentation: off [fixed]
tx-tcp-mangleid-segmentation: off
tx-tcp6-segmentation: on
tx-tcp-accecn-segmentation: off [fixed]
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: on [fixed]
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-gre-csum-segmentation: off [fixed]
tx-ipxip4-segmentation: off [fixed]
tx-ipxip6-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-udp_tnl-csum-segmentation: off [fixed]
tx-gso-partial: off [fixed]
tx-tunnel-remcsum-segmentation: off [fixed]
tx-sctp-segmentation: off [fixed]
tx-esp-segmentation: off [fixed]
tx-udp-segmentation: off [fixed]
tx-gso-list: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off
rx-all: off
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
hw-tc-offload: off [fixed]

Init Version: 20260804 (unchanged as expected)

Thanks again for the incredibly quick turnaround and the detailed explanation of the ASPM/CONFIG_PCIEASPM root cause.

Very much appreciated. Happy to test anything further if needed and please let me know if I missed any of your diagnostic steps happy to take another stab at it or move on to another check if it helps.

Thanks
Richard

Hi guys,

Issue seemingly with a certain NIC type on some of our kit we’ve noticed for a while but only just been able to investigate fully. Wondering if anyone else has this issue with a similar

FOG Version: 1.5.10.2149
FOS Kernel: 6.18.38
Init Version: 20260804
Server OS: Ubuntu (master + 2 x storage nodes)
Client Hardware: Dell OptiPlex 3070

ISSUE UEFI image deployments run at around 1.2 GB/min (~100 Mbps) on Dell OptiPlex 3070 machines. Legacy/BIOS deployments on the same hardware and infrastructure run at 6-10 GB/min as expected. UEFI uploads (capture) are also fast at ~15 GB/min so the issue is specific to UEFI downloads only.

THINGS ALREADY RULED OUT:

• Patch lead (swapped, no difference)
• Image compression (tested ZSTD and Partclone, identical speed)
• RAID vs AHCI (no difference)
• Storage node (tested all 3 nodes, same result)
• Dell BIOS (already on latest v1.35)
• iPXE boot file (tested ipxe.efi and snponly.efi, no difference)
• Kernel arg r8169.aspm=0 (no difference)
• Link speed (confirmed 1000Mb/s Full Duplex via ethtool)

ROOT CAUSE IDENTIFIED lspci in a FOG debug task on the affected 3070 shows: Realtek RTL8111/8168/8211/8411 PCI Express Gigabit Ethernet Controller (rev 15) dmesg | grep r816 shows: r8169 0000:01:00.0 eth0: RTL8168h/8111h r8169 0000:01:00.0 eth0: jumbo frames [frames: 9194 bytes, tx checksumming: ko] r8169 0000:01:00.0 enp1s0: No native access to PCI extended config space, falling back to CSI r8169 0000:01:00.0 enp1s0: Link is Up - 1Gbps/Full - flow control off The key issues are tx checksumming: ko and the CSI fallback, both known performance problems with the r8169 driver on RTL8168h rev 15 chips.

SUSPECTED CAUSE I noticed this recent commit in the FOS repo: Switch x64/arm64 kernels to in-kernel r8169 for RTL8168/8125/8126 I believe this switch from r8168 to r8169 may have introduced the regression. This is documented in Debian Bug #1110193 (filed July 2025): https://bugs.debian.org/1110193 The reporter has identical lspci output and the same Rx-only slowdown, resolved by installing r8168-dkms.

WORKAROUND Booting in Legacy/BIOS mode for imaging works but is not practical long term.

REQUEST Would it be possible to revert the r8169 switch for RTL8168h rev 15 chips or provide an experimental kernel build with r8168 re-enabled? Happy to test any experimental builds and report back. r8168 driver code is already in the FOS repo at: https://github.com/FOGProject/fos/tree/master/KernelPackages/drivers/net/ethernet/realtek

Thanks in advance guys.

@kentasmith Try changing your DHCP option 67 to another one of the boot files available (link below). We had this error and this resolved it. Think changing it to ipxe.kpxe instead of undionly.kpxe helped.

https://docs.fogproject.org/en/latest/installation/network-setup/dhcp-server-settings/#option-67