EFI_STUB enabled custom FOG kernel causing ipxe.efi to throw error 0x2e008081
-
@Scott-Lynch Great to hear you’ve been digging through things very intensely! I am sure you’ve learned a lot building your own kernels and iPXE binaries. Well done!
So let me try to step back a little bit to get the whole big picture. FOG comes with binaries included to make things as simple as possible for people and there are not too many cases when you actually want to build your own stuff. Don’t get me wrong, I don’t wanna sound mean. But why are you doing this at all (just trying to get an idea on where to direct you)?
As well I try to answer some of your questions let me quote one of the iPXE devs:
There is currently no way for iPXE (EFI version) to boot anything that is not a valid EFI binary.
This is from 2013 but as far as I know this is still valid. Loading the Linux kernel in bzImage format is only possible when iPXE is build for legacy BIOS -
*.(kk)pxebinaries. Modern kernels (about since around version 3.3 I think) do have something called EFI_STUB which makes it possible to chainload such a kernel image from iPXE in UEFI mode.As you are talking about kitchensink.config and sourceforge I get the feeling that you are using an older version of FOG. This is just me guessing but are you trying to make an old FOG server UEFI capable? While I am sure this can be done with a lot of effort I’d think you’d be way better of by upgrading to a newer version of FOG.
Let us know which version of FOG you use and what you are aiming for (why compile your own binaires?) and I am sure we’ll get you fixed up very soon!
Edit: Re-reading your post I noted that you are talking about the “Could not Select: Exec format error” thing as well. Possibly it’s just that making you wanna build your own kernel, hmm? Do you see this error on different client hardware? Please try different models to see if it’s always the exact same issue.
-
Lets start by collecting a bit of background info here.
- hardware are you trying to pxe boot. We should know both the manufacturer and model.
- Have you confirmed that you have the latest firmware installed on the target computer.
- Is the target computer in uefi mode or legacy (bios) mode?
- What specifically do you have defined for dhcp option 67?
- What version of FOG are you using?
To be able to help you we will probably need you to resetup your fog server to a known state so we are not chasing our tail with changes. Before trying to debug this issue. We have seen certain very buggy versions of firmware from most vendors, but Lenovo is at the top of the list for buggy firmware in my experience. We have ways of booting these one off systems but I’m more interested in understanding what is unique about this hardware you have.
-
You are right, I forgot to mention some of those points in the original post. Here is what I have:
-
Hardware I’m trying to boot:
Lenovo N23 WinBooks (NOT the ChromeBook, but the one running Windows)
Model/Type: 80UR0004US
Firmware Version: 2QCN18WW (Most recent available on the manufacturer’s website)
Firmware Date: 12/04/2016
Processor: Intel Celeron N3060
Wireless Network Adapter: Intel Dual Band Wireless-AC 7260
Ethernet Adapter: None, Wireless Networking Enabled Only
I am using a USB 3.0 to Ethernet Adapter purchased from Lenovo to to do the PXE
booting since wireless support is not enabled by default within the iPXE client that
I got off of the ipxe.org website. -
Yes, the firmware is up-to-date.
-
Target computer is UEFI mode.
IF I can get this UEFI boot issue resolved, I am going to dispense with legacy booting
altogether and just start configuring the computers for UEFI boot only. -
DHCP Option 067 = ipxe.efi
-
Version of FOG I am using is:
FOG Version: v1.4.4
SVN Revision: 6077 -
I have already set my FOG server to back to its “just out of the box” state and is ready to go.
Incidentally, I have been using FOG for my deployments for almost 5 years now. My employer, during the entirety of the 10-11 years I have been working here, has only bought Lenovo machines. That said, I have only run into 1 or 2 models of Lenovo computers that had any trouble talking to my FOG server. Both of those two machines would not PXE boot directly off the network, but would PXE boot with no issues if I loaded the ipxe client onto a bootable CD/DVD/USB Key and booted the machine off the media. I don’t remember which models of Lenovo computers had issues off the top of my head, and they aren’t relevant at this time. I have just had good luck booting “problem” machines with ipxe burned onto bootable media.
Speaking of which, I am booting my N23 WinBook off of a USB key I made containing my ipxe client. For the ipxe client, I used the following make statement:
make bin-x86_64-efi/ipxe.efiTo make the bootable media, I copied the resulting ipxe.efi of the make statement to the following directory structure on my USB key and changed its name from “ipxe.efi” to “bootx64.efi”.
Directory tree of bootable USB key:
/EFI/BOOT/bootx64.efi -
-
These N23 WinBooks are the 3rd model of Lenovo computer I’ve run across to have any issues with talking to the FOG server. The N23 WinBooks issues started with their lack of a built-in Ethernet port to which an Ethernet cable could be attached and have escalated from there. I swear these winbooks have been little more than a reminder that Murphy’s Law is in full effect.
So, in short, for the last 5 years or so, I have had very good luck with Lenovo and FOG with any hiccups being overcome by simply booting to some form of bootable media containing the ipxe client. These N23’s are the first model to give me any real issues.
Anyway, thanks for the help…
-
Hey! No problem! I understand FOG comes with binaries included. The reason I am going through all of this building the source code is the included binaries that came with FOG were the first to throw the error 0x2e00801. I began researching the issue. In the beginning, my effort was limited solely to the ipxe client, thinking it was the cause.
Now, you have confirmed what the code was “saying” when you said, “This is from 2013 but as far as I know this is still valid. Loading the Linux kernel in bzImage format is only possible when iPXE is build for legacy BIOS - *.(kk)pxe binaries. Modern kernels (about since around version 3.3 I think) do have something called EFI_STUB which makes it possible to chainload such a kernel image from iPXE in UEFI mode.”
I haven’t really tried to boot anything else other than these winbooks in UEFI mode since they all have been working up to now. I will test booting the ipxe client after I get back from luck today.
And…Yes, the “Could not select: Exec format error” is what has spurred me to go through this building the kernel bit since it doesn’t seem the the kernels that came with my installed version of FOG have EFI support enabled. -
Incidentally, what are the other possible formats in which the kernel can be built?
I have little to no experience building linux kernels of any kind. I only know that all of the documentation I have seen talks about building the kernel as “bzImage”. The only other kernel mentioned in the documentation is called something like “vmlinuz”, but I have no clue how it is built since make complains there is no recipe to build “vmlinuz”.
Thanks again for the help!
-
I just noticed your question about me trying to get an old FOG server UEFI capable. The answer is no, I only have 1 FOG server currently running on Ubuntu Server 16.04.03.
I am going through the custom kernel builds because everything I have been reading said the most likely cause for the exec format error was the kernel was lacking these two settings enabled:
EFI runtime service support = y
EFI stub support = y -
My intent is to give you a hand with this later today. I’m sorry for the delay, but my work is occupying my time today. I think the answer is in these posts. As I said before we have a last resort option of booting FOS from a usb stick. We have to go that route for some Mac systems that have had firmware updates that make it incompatible with iPXE. So by booting FOS directly from the usb stick we can bypass iPXE with certain caveats. So there ARE options here we haven’t tested.
-
My work day is about to end; however, I would be happy to go through the process with you sometime soon.
I did test loading FOG via UEFI on a completely different model of machine. The new machine doesn’t complain about the exec format error. The new machine loads to the FOG boot menu like it should; but, when a task is selected it immediately reloads ipxe and unloads it, and then reloads ipxe in an almost continues loop, when it finally stops, it loads the reFind.efi which gives restart and reboot as its two main options.
-
@scott-lynch After reviewing this thread, I think we should attempt to boot FOS directly from the usb stick (fall back plan). Right now its not clear if its the hand off between ipxe and FOS, or in the FOS kernel itself.
ref: https://forums.fogproject.org/topic/7727/building-usb-booting-fos-image
Lets stick to the tutorial with the stock FOG kernel.
-
@george1421 As far as I understand @Scott-Lynch has already tried USB booting - which worked for other models but not on this one.
I don’t have much time right now but will get back to you later on today. Most probably I can offer some help on this.
-
@Scott-Lynch Talked to George and he’s right about testing to boot FOS Linux directly from USB is good to test. We have seen crappy UEFI firmware where even that failed.
Other than that I am wondering if you’ve disabled secure boot? I know this questions sounds stupid but just wanna make sure as the error sounds a bit like it could be on. I do remember one case where a Lenovo device had some kind of extra security chip which needed to be disabled - read through this and this.
-
Ok, I’ll go through the tutorial now. I have the FOG server reset to the stock kernel, so everything should be good to go. Will post back with results.
-
No problem. I do have secure boot disabled on the WinBook and the 2nd test computer. Neither computer would boot to the USB stick with the secure boot option turned on. I will go through the FOS tutorial and get back with the results.
-
@Scott-Lynch The security chip mentioned is something else. Please read through the links I posted.
-
My bad. The Intel TPM chip was still enabled. I just disabled it and, just for kicks, ran the PXE boot sequence. I made it to the FOG boot menu. It correctly showed the host as being already registered. (I had manually entered the information into the FOG console since I am having to use a usb-to-ethernet adapter to get things rolling client side.) I then attempted to have it run the Client System Information (Compatibility) at which point it successfully loaded the stock bzImage. It threw a few ACPI errors before going into kernel panic. Exact message for the kernel panic is:
Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(1,0)
In truth, this is a bit of headway given that, before, it wouldn’t even load the kernel, much less read it and attempt to execute it.
-
@scott-lynch said in EFI_STUB enabled custom FOG kernel causing ipxe.efi to throw error 0x2e008081:
Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(1,0)
It may not fit exactly this case, but historically every time we’ve seen this the FOG admin had somehow tried to pxe boot with pxelinux.0, which was how fog 1.1 use to pxe boot before FOG switching over the iPXE. Please ensure you are using fog delivered pxe boot files.
The other implications of this is that the init.xz file is damaged in some way.
The APCI messages are only warnings and not something to be concerned about.
-
I am still working through the FOS build, but have run into a few snags I am attempting to get through. Namely, should I be looking at the multiboot instructions? They seem to be the only set that mentions EFI…
-
@scott-lynch Look at the FOG chat bubble above on the fog tool tray. If you follow the process the usb drive will be universal both uefi and bios booting.
-
Well, to get everything set up for all of this, I deleted the bzImage and bzImage32 files located in /var/www/fog/service/ipxe, but I didn’t touch either the init.xz nor the init_32.xz. I also deleted anything with ‘ipxe’ in its filename from /tftpboot/
I then ran the FOG installfog.sh in fog_1.4.4/bin to have it put everything back that I’d removed.
