Hand-off to FOS kernel fails on certain Gen4 Xeon (Sapphire Rapids) based systems - Dell R760, Supermicro X13, etc

[george1421]

@asawtell Sorry for the late viewing, is this still an issue.

[asawtell]

@george1421 Sorry for the late reply - haven’t been checking this lately due to the lack of response.

Yes, this is still an issue. We have been working around it with manual installs and disk swapping but haven’t made any headway on fixing the problem. We reached out to the system vendors as well for advice but most of their engineers are busy with launch cycles and aren’t supporting us at the moment.

[george1421]

@asawtell well one way to give a bit more information if you go into FOG Configuration->FOG Settings and hit the expand all button. Then search for log set the log level to 7.

Also upgrade to the latest version of kernel 6.x. There has to be a bit more info here on why its failing. I know for the servers we may need to create a custom kernel to have drivers for the raid controllers. But if its not booting into FOS then we haven’t got to the drivers yet.

[asawtell]

@george1421 I’m not quite sure what you mean by “upgrade to the latest version of kernel 6.x”. Is there a newer version of FOS kernel than the release from Mar 30 at https://github.com/FOGProject/fos/releases? Or do you mean rebuilding FOS with a newer kernel 6.x included?

I will try changing log level to 7 and monitoring the logs while reproducing the issue, but I don’t think there’s any opportunity for the server to submit information back after it locks up. So I’m not sure if we will see anything valuable.

[george1421]

@asawtell You are on the latest kernel at the moment. But I was referring to FOG Configuration->Kernel update menu to get the latest version if you need it.

[asawtell]

@george1421 Understood. We haven’t been able to get the automated update to work behind our corporate proxy, so it’s been manual updates - glad to hear that we are on the latest kernel.

I just set the kernel loglevel to 7 and tried a PXE boot on one of our problematic systems. It’s still falling over instantaneously as soon as it hits the FOS kernel, with no additional messaging provided. Is there some sort of logging I can turn on in iPXE which might catch something?

When I boot to one of our normally functioning systems (did this with a debug task) I can see that the loglevel is being correctly set and /var/log/messages contains the expected low level kernel log output.

Another step I might try is trying to find a distro which uses kernel 6.1.22 and seeing whether it also fails boot or if the issue is FOS exclusive. Will do some poking around and see what I can find.

[george1421]

@asawtell OK lets see if we can usb boot into FOS Linux. It looks like there might be a problem with iPXE not handing off to the OS cleanly.

But first, lets see if its ipxe causing this problem. Lets update to the latest version of iPXE using this process. This is all done on the fog server. https://forums.fogproject.org/topic/15826/updating-compiling-the-latest-version-of-ipxe?_=1692123396261

If that process fails we will use this tutorial to create a usb flash drive to direct boot into FOS linux. Look over this tutorial completely to understand the caveats with this route. https://forums.fogproject.org/topic/7727/building-usb-booting-fos-image Look at the FOG Forum chat for some additional tips.

The idea of usb booting into FOS Linux will be to boot into debug mode so we can try to find out what hardware is missing from the kernel build. The default kernel build is targeted to workstation class computers not servers, we may need to make a customer (one off) kernel that has raid drivers or specialty network drivers.

[asawtell]

@george1421 Thanks George. I think I mentioned in my first post - I already replicated the issue with a USB/GRUB boot disk, where it hung at the same point of handing over to the FOS kernel from GRUB. I believe I used the same method for creating the boot disk, but I could build a brand new one and re-run the test just to confirm.

I’ll also see if we can safely update iPXE without breaking anything in our environment and report back if it changes anything.

[george1421]

@asawtell said in Hand-off to FOS kernel fails on certain Gen4 Xeon (Sapphire Rapids) based systems - Dell R760, Supermicro X13, etc:

I’ll also see if we can safely update iPXE without breaking anything in our environment and report back if it changes anything.

Upgrade iPXE will only impact pxe booting if something goes really bad. You can fix it by just recloning the installer files and reinstalling fog.

So this means that if you used the usb fos boot and its still failing to start up linux there is something fundamentally wrong with the FOS kernel. I would think that xscale processors are x64 compatible, so they should boot the fos linux kernel.

[Sebastian Roth]

@asawtell @george1421 Great to see you have been working on this while I was absent. So far to me it looks like a specific issue on this hardware with the Linux kernel.

Would be great if you can boot some kind of live Linux OS, or maybe just an installer ISO will do as well.

The other option is adding additional print output statements to the kernel source code and compile a custom kernel to boot and see exactly where it hangs. I can guide you on this if you are keen to give it a go, let me know.

Also you might see if there is an UEFI firmware update available (not sure if you talked about this already).