Client hangs at EFI stub:
-
@george1421 I have set the debug level to 7 under Fog Settings / Kernel Loglevel. I created a task to boot the system into debug. However it is still hanging at the same point. Where can I get the debug logs from the server? I was looking in /opt/fog/log but did not see anything that looks like a debug log. As for the type of system it is an Intel 4th Gen Xeon system (D50DNP) https://www.intel.com/content/www/us/en/products/details/servers/multi-node-server-systems/server-d50dnp.html
With the older firmware it was working and once updated to the newer firmware PXE is no longer working.
Link to Firmware release. https://www.intel.com/content/www/us/en/download/776474/bios-and-system-firmware-update-package-sfup-for-windows-and-linux-for-intel-server-d50dnp-family.html -
Note to myself: Linux hangs on 4th Gen Xeon Scalable processors
I know I’ve seen posts in the forum mentioning booting issues with intel scalable processor line.
@sgilbe Can you live boot a standard linux distro like ubuntu/debian/linux mint?
-
@george1421 I was able to boot into the live version of ubuntu 22.04.3 desktop as well as the ubuntu 22.04.3 server installer. I was also able to boot into the Rocky Linux 9.2 installer DVD and minimum install.
-
@sgilbe ok so if we map this out we don’t know then if its a problem with ipxe or the FOS Linux kernel. Since other distros boot it might not be the linux kernel. The only caveat is the other distros have older kernels in the 5.x series.
This next step is to build a fos linux usb boot drive. The basis of the build is this tutorial. https://forums.fogproject.org/topic/7727/building-usb-booting-fos-image
Make sure you read the entire thread so you are aware of the caveats. When you get the boot image built, boot into the Grub menu and select the debug option. The idea is to test booting into FOS linux. If it boots then the issue is with iPXE, if it doesn’t boot then we have a linux issue.
Look at the fog forum chat for a few more hints on building this boot image.
-
@george1421 I will work on making the FOS usb boot drive. Will post when done.
-
@george1421 I have been trying to create the USB drive but am having an issue with the following line.
grub2-install --removable --no-nvram --efi-directory=/mnt --boot-directory=/mnt/boot --target=x86_64-efi
I get the error: this utility cannot be used for EFI platforms because it does not support UEFI Secure Boot.
I have checked the system that I am trying to create the USB drive on and secure boot is disabled. I have also tried to add the --force parameter but that did not work. I am using Rocky Linux 9 to create the USB drive. Maybe I am doing something wrong. I could try a different OS to do this on as well if needed.Thanks for your help.
-
@george1421 Also with the system that is having issues booting into the deploy image or register host, I am able to boot into the main fog interface.
I manually Registered the host which is why it shows as registered.
I am also able to boot into the CentOS 9 Stream Installer and install that OS if I want to, as well as boot into WinPE works without any issue so I think the iPXE is working ok, but when booting the bzImage for anything but the menu it seems to hang during boot at the same screen. I have tried most of the menu items.
-
@sgilbe I can say I have not tested this process with Rocky Linux (any version). I did create the image for the tutorial using ubuntu/debian version. This issue may be related to a newer version of grub. You might want to try to add the --force switch to the command.
Did you look at the forum chat at the top of the page for additional hints?
-
@george1421 I will load ubuntu and try from that. I did try the --force switch and it still failed. I will go through the forum chat to see if there is anything there about the issue.
-
@sgilbe said in Client hangs at EFI stub::
I will go through the forum chat to see if there is anything there about the issue
Hint: little speech bubble along the fog menu bar on the right next to your avatar picture.
-
@george1421 Ok I have created the USB drive using Ubuntu 22.04.3 and it booted to the menu. I selected deploy / capture image and it is hung at the following spot. booting kernel… it has been sitting at that screen for 5 min.
-
@sgilbe ok, please pick the debug option, I think its item #6. That menu has a log level of 7.
But it does look like the linux kernel is not booting on this hardware, hopefully the additional logging level will give us some help. Also now that you have a usb drive, we can try the 5.15.x linux kernels too. It might be something in the 6.x kernel.
-
@george1421 After booting the USB drive I selected option 6 / debug and it is hanging at the same spot of booting kernel no extra information is displayed on the screen.
-
@sgilbe OK so this is telling us that the linux kernel is not booting on this hardware. Can you go to this URL https://github.com/FOGProject/fos/releases and download the 5.15.x
bzImageand copy that to the usb drive to see if 5.15 will boot on this hardware? If not working, then try the 5.10.x, if that fails then try the 4.19.x series.As for the type of system it is an Intel 4th Gen Xeon system (D50DNP)
What exact processor is installed in this computer? I might have to search this issue based on the processor. You were able to live boot ubuntu 22.04 so a linux kernel should boot on this system. There might be something missing from the FOS linux kernel. -
@george1421 Here are the CPUs that I have in the system. Intel
Xeon Platinum 8480+
Here are the results from the different kernels
5.15.98 = failed
5.15.93 = failed
5.15.68 = failed
5.10.50 = failed
4.19.145 = failed
4.19.101 = failedThey all get to the same point on the screen.
-
@sgilbe said in Client hangs at EFI stub::
Platinum 8480
OK I’ll look into the kernel configurations tonight. From what I see commercial linux can run on this hardware, there must be something missing from FOS linux. Understand that FOS Linux is a very light and fast version of linux dedicated to imaging. So it supports almost all hardware, servers them selves give us a problem since the hardware is more specific than generic.
-
@george1421 It looks like Ubuntu 22.04.3 installer uses kernel 6.2.0 which I was able to boot using the live usb media.
-
Try upgrading the PREBOOT UEFI driver which can be downloaded here:
https://www.intel.com/content/www/us/en/download/15755/intel-ethernet-connections-boot-utility-preboot-images-and-efi-drivers.html -
@george1421 were you able to look into the kernel configurations? I am trying to follow the instructions on building a custom kernel but not sure what the options are that I need to look at. Do you know how to get the .config file from the kernel build of Ubuntu 22.04.3 If I can do a compare between that and the .config file from fos then maybe I can find the difference that would help.
-
@sgilbe I haven’t had a chance to look into the settings as of now. Its kind of a complicated subject to explain in an email. My gut reaction is that it should boot even if the advanced features are not turned on. It might be what rhromkyo mentioned about the preboot environment but that also got me thinking if this is new hardware have you run through the lifecycle controllers and updated all of the firmware in that server? I’m going to keep working on the kernel bits but lets make sure there isn’t something in the firmware causing the issue.
Also as a hack and it won’t fully work, it would be interesting to take the vmlinuz from your ubuntu disk that is working and save it on the usb stick as bzImage (watch the case) and see if the kernel tries to boot when you select debug mode from the grub menu. It appears that the FOS linux kernel doesn’t even try to boot.
