FOG 1.5.10.2482 iPXE 2.0.0 - intermittent UEFI boot failures on Realtek NICs (1.21.1+ and snponly.efi unaffected)

[AUTH IT Center]

Summary

After updating FOG from 1.5.10.1903 to 1.5.10.2482, a subset of our UEFI clients with Realtek NICs began failing to boot, intermittently — about 6% of boots.
Clients with Intel NICs on the identical boot path are unaffected (0 failures in 62 boots). Reverting only /tftpboot/ipxe.efi to the binary shipped with 1.5.10.1903, while leaving FOG itself on 2482, stopped the failures.

The two iPXE builds involved are g5de5d (1.21.1+) and g12798 (2.0.0)`.

Environment

FOG (new)	1.5.10.2482, commit ff6410d242252fee5cf1370213b2bf459e858315
FOG (previous)	1.5.10.1903
OS	Ubuntu 22.04.5 LTS, kernel 5.15.0-191-generic
PHP	8.1.2
Database	MariaDB, separate host
Fleet	~2,500 clients, ~450 PXE boots/day
DHCP	ISC DHCP, per-host filename; 1,879 hosts on ipxe.efi

iPXE versions as reported by clients in the HTTP User-Agent:

• before: iPXE/1.21.1+ (g5de5d) — ipxe.efi 1,107,456 bytes
• after: iPXE/2.0.0 (g12798) — ipxe.efi 1,144,832 bytes

installfog.sh replaced every binary under /tftpboot at the time of the update.

Boot flow

DHCP (option 67) → ipxe.efi via TFTP → boot.php → exit type refind_efi → imgfetch refind.conf + chain -ar refind_x64.efi → rEFInd → local Windows Boot Manager.

rEFInd is chainloaded over HTTP (not installed on the ESP). Its config uses scanfor netboot,internal,external,optical,manual, timeout -1, use_nvram false.

Symptom

On a failing boot, every network operation succeeds:

POST /fog/service/ipxe/boot.php        200    8658 b   iPXE/2.0.0 (g12798)
GET  /fog/service/ipxe/bgdark.png      200   16523 b   iPXE/2.0.0 (g12798)
GET  /fog/service/ipxe/refind.conf     200   36624 b   iPXE/2.0.0 (g12798)
GET  /fog/service/ipxe/refind_x64.efi  200  222155 b   iPXE/2.0.0 (g12798)

…and then the client returns to PXE and repeats. Correct byte counts every time, no failed transfers, nothing in the Apache error log. Users report either a boot loop or the machine sitting on the rEFInd menu.

The failure therefore occurs after all network I/O has completed — at or after chain -ar into rEFInd, not during a transfer.

It is intermittent: the same host may succeed on one attempt and fail on the next (we observed 1 to 11 consecutive failures on one machine). Disconnecting the network cable, forcing local disk boot, always works — that is how our on-site staff were
recovering machines.

Evidence

1. Same host, same day, only the loader changed

Dell OptiPlex 3020, Realtek. From the tftpd log:

time	request	result
07:30:19	RRQ ... filename ipxe.efi (2.0.0)	failed, never reached Windows
11:30:39	RRQ ... filename snponly.efi (2.0.0)	booted, Windows in 38 s

2. Old binary via per-host DHCP override

A second OptiPlex 3020 (Realtek) served the 1.5.10.1903 ipxe.efi: 2/2 unattended boots in ~18–25 s, after previously requiring a cable pull on every attempt.

3. Failure rates

Loader determined from the tftpd log — i.e. what each client actually fetched, not what the config says:

PRE-revert — /tftpboot/ipxe.efi = iPXE 2.0.0 (g12798)
  loader           NIC       ok    failed   rate
  ipxe.efi         Realtek   131      9      6.4%
  ipxe.efi         Intel      62      0      0%
  undionly.kpxe    Realtek    96      0      0%
  undionly.kpxe    Intel       5      0      0%
  snponly.efi      any         2      0      0%

POST-revert — /tftpboot/ipxe.efi = iPXE 1.21.1+ (g5de5d), FOG still on 2482
  ipxe.efi         Realtek    14      0      0%
  ipxe.efi         Intel       9      1*     (*BitLocker recovery prompt, unrelated)
  undionly.kpxe    Realtek     8      0      0%

The cleanest comparison is Intel vs Realtek on the same binary and the same UEFI path: 0/62 vs 9/140, p ≈ 0.02.

The undionly.kpxe row (0/96 Realtek) points the same way but is confounded - those clients are BIOS-mode, so they differ in boot path as well as in which driver iPXE uses. We offer it as supporting, not primary, evidence. The snponly.efi sample is too small to stand on its own.

What we ruled out

None of these discriminate between failing and working hosts:

• Machine model — OptiPlex 3020: 26 booted fine, 7 failed. Same model, both outcomes.
• BIOS version — A02 and A19 appear among failures and among successes.
• Disk type — failures include NVMe, SATA SSD and spinning HDD.
• Optical drive presence — 70% of working hosts vs 57% of failing hosts.

We also confirmed the FOG side is unchanged: $_exitTypes, $sanboot and $refind in bootmenu.class.php are byte-identical between 1903 and 2482, and refind.conf / refind_x64.efi are served byte-identically before and after (verified by response size in the access log).

Affected hardware

All report Realtek PCIe GbE Family Controller under Windows. We do not have PCI IDs to hand but can collect them.

vendor / board	BIOS	count
Dell OptiPlex 3020	A19	4
Dell OptiPlex 3020	A02	3
Dell OptiPlex 3000	1.41.0	2
Gigabyte H110M-S2H	F22	1
Gigabyte H81M-S2V	F2	1
Fujitsu ESPRIMO D556/2	V5.0.0.12 R1.35.0	1
ASUS (System Product Name)	0803	1

Unaffected: Intel I219-LM (Dell Pro Slim QCS1250) — 0 failures in 62 boots.

Current workaround

We reverted only /tftpboot/ipxe.efi to the 1.5.10.1903 binary, leaving FOG on 2482. Every path we have tested works with that combination:

• normal boot and rEFInd chainload
• full image deploy — task script, 11 MB bzImage + 31 MB init.xz over HTTP, FOG client OS, NFS mount of /images, partclone, reboot into the deployed image, client check-in

Hypothesis and questions

snponly.efi — the same 2.0.0 build, but using the firmware’s UEFI SNP driver instead of iPXE’s own — works on hardware where ipxe.efi fails. That points at iPXE 2.0.0’s built-in Realtek driver. Since all network I/O completes before the failure, we suspect driver teardown/handover when chainloading another EFI application rather than anything in the transfers themselves. The intermittency suggests a race rather than a hard incompatibility.

Questions:

1. Are there known changes to the Realtek driver, or to EFI driver unbinding / shutdown() behaviour, between g5de5d and g12798?
2. Could the state iPXE leaves the NIC in affect a chainloaded EFI application - specifically rEFInd, which scans for netboot entries (scanfor netboot,...) and may re-enumerate SNP?
3. Is snponly.efi the recommended long-term choice for Realtek clients, or would you prefer a fix in the built-in driver?

We have ~1,200 Realtek clients and can reproduce readily. Happy to run test builds, capture verbose iPXE console output, or collect PCI IDs and chipset revisions.

AI-assisted-by: Claude Code (Opus 5)

[Tom Elliott]

@AUTH-IT-Center I do believe snponly would be the recommended, rather than iPXE’s driver.

The developers at iPXE wrote the driver on their own (of course using documentation and stuff, but for all intents/purposes it is still a handrolled driver) so anything is possible.

We shipped the native iPXE 2.0.0 mainly because of the feature it allows with actual Secureboot capabilities and instead of embedding everyfile with a custom script, a more dynamic approach for when iPXE releases new version we can upgrade more easily.

For what it’s worth, I would almost want more people to default to snponly.efi (or secureboot/snponly-shimx64.efi if using/wanting secureboot after enrolling your machines of course) because this is supposed to be using the generic driver for EFI boot protocols on the NIC rather then attempting to discover the NIC using a driver loaded.