Unsolved Client hangs at EFI stub:
-
@sgilbe said in Client hangs at EFI stub::
I will go through the forum chat to see if there is anything there about the issue
Hint: little speech bubble along the fog menu bar on the right next to your avatar picture.
-
@george1421 Ok I have created the USB drive using Ubuntu 22.04.3 and it booted to the menu. I selected deploy / capture image and it is hung at the following spot. booting kernel… it has been sitting at that screen for 5 min.
-
@sgilbe ok, please pick the debug option, I think its item #6. That menu has a log level of 7.
But it does look like the linux kernel is not booting on this hardware, hopefully the additional logging level will give us some help. Also now that you have a usb drive, we can try the 5.15.x linux kernels too. It might be something in the 6.x kernel.
-
@george1421 After booting the USB drive I selected option 6 / debug and it is hanging at the same spot of booting kernel no extra information is displayed on the screen.
-
@sgilbe OK so this is telling us that the linux kernel is not booting on this hardware. Can you go to this URL https://github.com/FOGProject/fos/releases and download the 5.15.x
bzImageand copy that to the usb drive to see if 5.15 will boot on this hardware? If not working, then try the 5.10.x, if that fails then try the 4.19.x series.As for the type of system it is an Intel 4th Gen Xeon system (D50DNP)
What exact processor is installed in this computer? I might have to search this issue based on the processor. You were able to live boot ubuntu 22.04 so a linux kernel should boot on this system. There might be something missing from the FOS linux kernel. -
@george1421 Here are the CPUs that I have in the system. Intel
Xeon Platinum 8480+
Here are the results from the different kernels
5.15.98 = failed
5.15.93 = failed
5.15.68 = failed
5.10.50 = failed
4.19.145 = failed
4.19.101 = failedThey all get to the same point on the screen.
-
@sgilbe said in Client hangs at EFI stub::
Platinum 8480
OK I’ll look into the kernel configurations tonight. From what I see commercial linux can run on this hardware, there must be something missing from FOS linux. Understand that FOS Linux is a very light and fast version of linux dedicated to imaging. So it supports almost all hardware, servers them selves give us a problem since the hardware is more specific than generic.
-
@george1421 It looks like Ubuntu 22.04.3 installer uses kernel 6.2.0 which I was able to boot using the live usb media.
-
Try upgrading the PREBOOT UEFI driver which can be downloaded here:
https://www.intel.com/content/www/us/en/download/15755/intel-ethernet-connections-boot-utility-preboot-images-and-efi-drivers.html -
@george1421 were you able to look into the kernel configurations? I am trying to follow the instructions on building a custom kernel but not sure what the options are that I need to look at. Do you know how to get the .config file from the kernel build of Ubuntu 22.04.3 If I can do a compare between that and the .config file from fos then maybe I can find the difference that would help.
-
@sgilbe I haven’t had a chance to look into the settings as of now. Its kind of a complicated subject to explain in an email. My gut reaction is that it should boot even if the advanced features are not turned on. It might be what rhromkyo mentioned about the preboot environment but that also got me thinking if this is new hardware have you run through the lifecycle controllers and updated all of the firmware in that server? I’m going to keep working on the kernel bits but lets make sure there isn’t something in the firmware causing the issue.
Also as a hack and it won’t fully work, it would be interesting to take the vmlinuz from your ubuntu disk that is working and save it on the usb stick as bzImage (watch the case) and see if the kernel tries to boot when you select debug mode from the grub menu. It appears that the FOS linux kernel doesn’t even try to boot.
-
@george1421 I am in the process of getting setup to do the preboot stuff I will try getting the vmlinuz from ubuntu and try as a bzImage on the usb drive.
-
@george1421 When I used the ubuntu kernel as the bzImage it started to boot but I get a kernel panic. That is a good sign though.
-
@sgilbe I realize this debugging is a lot of try this and do that, but we’ve narrowed it down to exactly the FOS Linux kernel. The ubuntu kernel error bombed out exactly where I expected it to do, at mounting the virtual hard drive (init.xz).
I think you asked earlier about the config file fog uses these are all posted on the fog project github site: https://github.com/FOGProject/fos/tree/master/configs The config file you are looking for is kernelx64.config this is the config file used to create the current kernel.
-
@sgilbe It looks like my build environment was really out of date. The last time I needed to create a one off kernel was for version 5.15.x. More to the point I updated the build environment to 6.5.3 and built this kernel. https://drive.google.com/file/d/1P-OX1LXhm-N_oBLg0PVcIjj0P3Cxm_Rp/view?usp=drive_link download this kernel and save it onto your flash drive as bzImage. I don’t expect it to work any better than the FOS standard kernel, but I want to see if the new kernel release works on your processor. If this kernel doesn’t work better than the stock FOS kernel then I compare what ubuntu is creating with this config file to see what is missing. I do have kernel options to turn on the advanced features of the Intel scalable processor, but the base x64 kernel should also run on this processor (IMO).
Also please confirm that you updated all of the firmware on the server using the lifecycle controller.
-
@george1421 This kernel is hanging at the same place as the fos kernel.
-
@sgilbe said in Client hangs at EFI stub::
This kernel is hanging at the same place as the fos kernel
In a way that’s good because it should be the same exact kernel except for 6.5.3 version instead of 6.2.x. So now the next part is I need to compare ubuntu kernel settings with FOS linux kernel settings.
-
@george1421 @sgilbe Great you guys have narrowed this down to this point. I have followed this thread but have not had the time to help you out and won’t be of much help in the next two weeks with just a little more time but no computer at hand but only the mobile phone to post and do research. Anyhow, I will try my best.
Yes it could be as “simple” as switching a kernel config setting on or off but I rather think the Ubuntu kernel might boot on the machine due to some patch Ubuntu applies to it’s kernels. That would make it a bit harder but not impossible to figure out and fix in FOS.
@sgilbe On a system with Ubuntu installed you should find the kernel config in
/boot/config-*.Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)
Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG
-
@Sebastian-Roth said in Client hangs at EFI stub::
Yes it could be as “simple” as switching a kernel config setting on or off but I rather think the Ubuntu kernel might boot on the machine due to some patch Ubuntu applies to it’s kernels. That would make it a bit harder but not impossible to figure out and fix in FOS.
The more I read the more I think I was wrong with assuming Ubuntu patches making it work. The Wikipedia article on this CPU says:
Not all accelerators are available in all processor models. Some accelerators are available under the Intel On Demand program, also known as Software Defined Silicon (SDSi), where a license is required to activate a given accelerator that is physically present in the processor. The license can be obtained as a one-time purchase or as a paid subscription. Activating the license requires support in the operating system. A driver with the necessary support was added in Linux 6.2.
So it’s not likely you still need special patches to even just boot up.
-
@Sebastian-Roth I did do a side by side comparison between ubuntu configs and FOS linux configs and there are roughly 1800 differences. Many were in drivers and options. The only one that stood out in the efi section was
CONFIG_EFI_MIXEDwhich allowed a 32 bit EFI kernel boot a 64 bit linux kernel. Seems kind of strange, but we probably should turn that on.Though a second process I started with an ia64 defconfig template and then added in the FOS required settings leaving almost all of the defconfig settings in place but adding in the fog required settings. I built this last night but haven’t had time to see if it boots. I did not add in the old ISA card network drivers or network adapters that I’m pretty sure are not in circulation like DEC Tulip network drivers. That kernel came in at 15MB as compared to the FOS kernel of 10MB. I’m not really worried about that extra 5MB kernel in size in 2023. This kernel is based on linux 6.5.3.
The other thing I need to point out is the the OPs platform is a server with an intel scalable processor. I don’t know what other hardware might be getting in the way. The FOS kernel should at least try to boot, it might not boot completely but should at least try to boot. We are not seeing that. By building the FOS usb boot drive we have eliminated all of the pxe and ipxe issues so we’ve narrowed it down to the FOS kernel, and swapping in the ubuntu kernel points directly to the FOS kernel at fault.
I hadn’t considered a ubuntu kernel patch to be the solution here either. I used linux 6.5.3 thinking that it should have all of the mainstream patches already in it.
