@george1421 Thanks George. I think I mentioned in my first post - I already replicated the issue with a USB/GRUB boot disk, where it hung at the same point of handing over to the FOS kernel from GRUB. I believe I used the same method for creating the boot disk, but I could build a brand new one and re-run the test just to confirm.

I’ll also see if we can safely update iPXE without breaking anything in our environment and report back if it changes anything.

@george1421 Understood. We haven’t been able to get the automated update to work behind our corporate proxy, so it’s been manual updates - glad to hear that we are on the latest kernel.

I just set the kernel loglevel to 7 and tried a PXE boot on one of our problematic systems. It’s still falling over instantaneously as soon as it hits the FOS kernel, with no additional messaging provided. Is there some sort of logging I can turn on in iPXE which might catch something?

When I boot to one of our normally functioning systems (did this with a debug task) I can see that the loglevel is being correctly set and /var/log/messages contains the expected low level kernel log output.

Another step I might try is trying to find a distro which uses kernel 6.1.22 and seeing whether it also fails boot or if the issue is FOS exclusive. Will do some poking around and see what I can find.

@george1421 I’m not quite sure what you mean by “upgrade to the latest version of kernel 6.x”. Is there a newer version of FOS kernel than the release from Mar 30 at https://github.com/FOGProject/fos/releases? Or do you mean rebuilding FOS with a newer kernel 6.x included?

I will try changing log level to 7 and monitoring the logs while reproducing the issue, but I don’t think there’s any opportunity for the server to submit information back after it locks up. So I’m not sure if we will see anything valuable.

@george1421 Sorry for the late reply - haven’t been checking this lately due to the lack of response.

Yes, this is still an issue. We have been working around it with manual installs and disk swapping but haven’t made any headway on fixing the problem. We reached out to the system vendors as well for advice but most of their engineers are busy with launch cycles and aren’t supporting us at the moment.

I work in a testing lab where we use FOG to deploy operating systems. We have a very wide range of server platforms, from a few generations ago up to current and even a few pre-release platforms.

Generally FOG works well on all of our platforms and PXE boots with no issue. We are currently running FOG 1.5.9 to be compatible with our in-house automation, but we have the latest FOS kernel release loaded:

file /var/www/html/fog/service/ipxe/bzImage*
/var/www/html/fog/service/ipxe/bzImage:   Linux kernel x86 boot executable bzImage, version 6.1.22 (runner@fv-az565-7) #1 SMP PREEMPT_DYNAMIC Fri Mar 31 00:29:42 UTC 2023, RO-rootFS, swap_dev 0x9, Normal VGA
/var/www/html/fog/service/ipxe/bzImage32: Linux kernel x86 boot executable bzImage, version 6.1.22 (runner@fv-az576-383) #1 SMP PREEMPT_DYNAMIC Fri Mar 31 00:26:56 UTC 2023, RO-rootFS, swap_dev 0x8, Normal VGA

The issue comes with some of our newest platforms, listed here:

Platform #1: Dell R760 server, showed issues out of the box
Platform #2: Supermicro 4U server with X13DEG-OA motherboard, worked out of the box, stopped working with latest BIOS loaded.
Platform #3: Pre-release Gen4 Xeon based system, worked out of the box, stopped working with latest BIOS loaded.

They all happen to be using 4th generation Xeon Scalable processors, but I’m not sure that is the specific problem. When attempting to boot them via PXE, the system halts at the hand-off point to the FOS kernel until manually rebooted. The last visible message before hang is “EFI stub: Loaded initrd from command line option”.

This seems to be an issue with the FOS kernel rather than the iPXE chain leading up to it, since creating a USB boot disk and chainloading into the FOS kernel from GRUB also locks up the system in the same way. I’m a newbie where this sort of thing is concerned but judging by the little bit of diagnostic output I was able to get from GRUB, it looks like the failure is instant as soon as FOS takes over, and doesn’t happen anywhere while GRUB (or iPXE) is still running.

The issue also appears to be related to BIOS somehow - two of our three platforms initially worked fine with FOS and FOG deployment, but started exhibiting the issue after an update to the latest BIOS. Secure Boot is disabled on all platforms.

At this point we’ve been trying to root cause and fix this issue for a couple weeks and haven’t made any progress. Any suggestions for what we could try to resolve this or ways we could generate additional diagnostics information for you folks would be much appreciated. We have some quite capable engineers over here but nobody has much experience with low-level firmware or kernel stuff that it seems like we might be dealing with.