EFI_STUB enabled custom FOG kernel causing ipxe.efi to throw error 0x2e008081

[Scott Lynch]

Hello,

I’m having an issue doing an iPXE UEFI boot to FOG. Specifically, no matter what I do, I get the “Could not Select: Exec format error (http://ipxe.org/2e008081)” error.

I have read everything at ipxe.org and the FOG Wiki concerning the issue and I have also scoured the Google and Bing search results for anything and everything related to the issue. All of the reference sources from ipxe.org, the FOG Wiki, the FOG forum posts I was able to locate, and the online documentation from my web-searches agree that the issue is caused by ipxe.efi attempting to load a non-EFI image (bzImage) and to fix it, I needed to build a new bzImage with EFI Stub support enabled. However, every single custom kernel I have built fails with the same 0x2e008081 error code.

I have even booted to the ipxe command line interface and tried various configuration settings to try and determine exactly where the issue comes up. In all cases, I only receive the error when the call to load the bzImage file occurs, regardless of whether or not the EFI_STUB is enabled within the kernel.

Incidentally, with my Windows DHCP options 066 and 067 set as follows, my client will successfully load the FOG boot menu:
option 066: <ip address of FOG deployment server>
option 067: ipxe.efi

However, any task involving either bzImage or bzImage32 to be loaded causes the above mentioned error to be thrown.

Kernel config files used for building the kernels are both TomElliott.config.32 and TomElliott.config.64 located at:
https://svn.code.sf.net/p/freeghost/code/trunk/kernel/

I have even attempted to build using the kitchensink.config and TomElliott.config.32/64 files the FOG 1.4.4 installation archive lists in its directory
./fog_1.4.4/kernel/

I am at a complete loss as to what else to do. Architectures of the ipxe.efi builds I have attempted are x86_64 (for 64-bit kernel) and x86/i386 (for the 32-bit kernel).

This one issue has had me stumped for a long while, and I think I’m too close to the issue now to properly see the solution.

Also, I have built the ipxe.efi from source, also. I have been using the following modified header files to build ipxe.efi:

#ifndef CONFIG_CONSOLE_H
#define CONFIG_CONSOLE_H

/** @file
 *
 * Console configuration
 *
 * These options specify the console types that iPXE will use for
 * interaction with the user.
 *
 */

FILE_LICENCE ( GPL2_OR_LATER_OR_UBDL );

#include <config/defaults.h>

/*
 * Default console types
 *
 * These are all enabled by default for the appropriate platforms.
 * You may disable them if needed.
 *
 */

//#undef	CONSOLE_PCBIOS		/* Default BIOS console */
//#undef	CONSOLE_EFI		/* Default EFI console */
//#undef	CONSOLE_LINUX		/* Default Linux console */

/*
 * Additional console types
 *
 * These are not enabled by default, but may be useful in your
 * environment.
 *
 */

//#define	CONSOLE_SERIAL		/* Serial port console */
#define	CONSOLE_FRAMEBUFFER	/* Graphical framebuffer console */
//#define	CONSOLE_SYSLOG		/* Syslog console */
//#define	CONSOLE_SYSLOGS		/* Encrypted syslog console */
//#define	CONSOLE_VMWARE		/* VMware logfile console */
//#define	CONSOLE_DEBUGCON	/* Bochs/QEMU/KVM debug port console */
//#define	CONSOLE_INT13		/* INT13 disk log console */

/*
 * Very obscure console types
 *
 * You almost certainly do not need to enable these.
 *
 */

//#define	CONSOLE_DIRECT_VGA	/* Direct access to VGA card */
//#define	CONSOLE_PC_KBD		/* Direct access to PC keyboard */

/* Keyboard map (available maps in hci/keymap/) */
#define	KEYBOARD_MAP	us

/* Control which syslog() messages are generated.
 *
 * Note that this is not related in any way to CONSOLE_SYSLOG.
 */
#define	LOG_LEVEL	LOG_NONE

#include <config/named.h>
#include NAMED_CONFIG(console.h)
#include <config/local/console.h>
#include LOCAL_NAMED_CONFIG(console.h)

#endif /* CONFIG_CONSOLE_H */

#ifndef CONFIG_GENERAL_H
#define CONFIG_GENERAL_H

/** @file
 *
 * General configuration
 *
 */

FILE_LICENCE ( GPL2_OR_LATER_OR_UBDL );

#include <config/defaults.h>

/*
 * Banner timeout configuration
 *
 * This controls the timeout for the "Press Ctrl-B for the iPXE
 * command line" banner displayed when iPXE starts up.  The value is
 * specified in tenths of a second for which the banner should appear.
 * A value of 0 disables the banner.
 *
 * ROM_BANNER_TIMEOUT controls the "Press Ctrl-B to configure iPXE"
 * banner displayed only by ROM builds of iPXE during POST.  This
 * defaults to being twice the length of BANNER_TIMEOUT, to allow for
 * BIOSes that switch video modes immediately before calling the
 * initialisation vector, thus rendering the banner almost invisible
 * to the user.
 */
#define BANNER_TIMEOUT		20
#define ROM_BANNER_TIMEOUT	( 2 * BANNER_TIMEOUT )

/*
 * Network protocols
 *
 */

#define	NET_PROTO_IPV4		/* IPv4 protocol */
#undef	NET_PROTO_IPV6		/* IPv6 protocol */
#undef	NET_PROTO_FCOE		/* Fibre Channel over Ethernet protocol */
#define	NET_PROTO_STP		/* Spanning Tree protocol */
#define	NET_PROTO_LACP		/* Link Aggregation control protocol */

/*
 * PXE support
 *
 */
//#undef	PXE_STACK		/* PXE stack in iPXE - you want this! */
//#undef	PXE_MENU		/* PXE menu booting */

/*
 * Download protocols
 *
 */

#define	DOWNLOAD_PROTO_TFTP	/* Trivial File Transfer Protocol */
#define	DOWNLOAD_PROTO_HTTP	/* Hypertext Transfer Protocol */
#define	DOWNLOAD_PROTO_HTTPS	/* Secure Hypertext Transfer Protocol */
#define	DOWNLOAD_PROTO_FTP	/* File Transfer Protocol */
#undef	DOWNLOAD_PROTO_SLAM	/* Scalable Local Area Multicast */
#define	DOWNLOAD_PROTO_NFS	/* Network File System Protocol */
//#undef DOWNLOAD_PROTO_FILE	/* Local filesystem access */

/*
 * SAN boot protocols
 *
 */

//#undef	SANBOOT_PROTO_ISCSI	/* iSCSI protocol */
//#undef	SANBOOT_PROTO_AOE	/* AoE protocol */
//#undef	SANBOOT_PROTO_IB_SRP	/* Infiniband SCSI RDMA protocol */
//#undef	SANBOOT_PROTO_FCP	/* Fibre Channel protocol */
//#undef	SANBOOT_PROTO_HTTP	/* HTTP SAN protocol */

/*
 * HTTP extensions
 *
 */
#define HTTP_AUTH_BASIC		/* Basic authentication */
#define HTTP_AUTH_DIGEST	/* Digest authentication */
//#define HTTP_ENC_PEERDIST	/* PeerDist content encoding */
//#define HTTP_HACK_GCE		/* Google Computer Engine hacks */

/*
 * 802.11 cryptosystems and handshaking protocols
 *
 */
#define	CRYPTO_80211_WEP	/* WEP encryption (deprecated and insecure!) */
#define	CRYPTO_80211_WPA	/* WPA Personal, authenticating with passphrase */
#define	CRYPTO_80211_WPA2	/* Add support for stronger WPA cryptography */

/*
 * Name resolution modules
 *
 */

#define	DNS_RESOLVER		/* DNS resolver */

/*
 * Image types
 *
 * Etherboot supports various image formats.  Select whichever ones
 * you want to use.
 *
 */
//#define	IMAGE_NBI		/* NBI image support */
//#define	IMAGE_ELF		/* ELF image support */
//#define	IMAGE_MULTIBOOT		/* MultiBoot image support */
//#define	IMAGE_PXE		/* PXE image support */
#define	IMAGE_SCRIPT		/* iPXE script image support */
//#define	IMAGE_BZIMAGE		/* Linux bzImage image support */
//#define	IMAGE_COMBOOT		/* SYSLINUX COMBOOT image support */
#define	IMAGE_EFI		/* EFI image support */
//#define	IMAGE_SDI		/* SDI image support */
//#define	IMAGE_PNM		/* PNM image support */
#define	IMAGE_PNG		/* PNG image support */
#define	IMAGE_DER		/* DER image support */
#define	IMAGE_PEM		/* PEM image support */

/*
 * Command-line commands to include
 *
 */
#define	AUTOBOOT_CMD		/* Automatic booting */
#define	NVO_CMD			/* Non-volatile option storage commands */
#define	CONFIG_CMD		/* Option configuration console */
#define	IFMGMT_CMD		/* Interface management commands */
//#define	IWMGMT_CMD		/* Wireless interface management commands */
#define IBMGMT_CMD		/* Infiniband management commands */
#define FCMGMT_CMD		/* Fibre Channel management commands */
#define	ROUTE_CMD		/* Routing table management commands */
#define IMAGE_CMD		/* Image management commands */
#define DHCP_CMD		/* DHCP management commands */
#define SANBOOT_CMD		/* SAN boot commands */
#define MENU_CMD		/* Menu commands */
#define LOGIN_CMD		/* Login command */
#define SYNC_CMD		/* Sync command */
#define NSLOOKUP_CMD		/* DNS resolving command */
#define TIME_CMD		/* Time commands */
#define DIGEST_CMD		/* Image crypto digest commands */
#define LOTEST_CMD		/* Loopback testing commands */
#define VLAN_CMD		/* VLAN commands */
//#define PXE_CMD		/* PXE commands */
#define REBOOT_CMD		/* Reboot command */
#define POWEROFF_CMD		/* Power off command */
#define IMAGE_TRUST_CMD	/* Image trust management commands */
#define PCI_CMD		/* PCI commands */
#define PARAM_CMD		/* Form parameter commands */
#define NEIGHBOUR_CMD		/* Neighbour management commands */
#define PING_CMD		/* Ping command */
#define CONSOLE_CMD		/* Console command */
#define IPSTAT_CMD		/* IP statistics commands */
//#define PROFSTAT_CMD		/* Profiling commands */
//#define NTP_CMD		/* NTP commands */
//#define CERT_CMD		/* Certificate management commands */

/*
 * ROM-specific options
 *
 */
#undef	NONPNP_HOOK_INT19	/* Hook INT19 on non-PnP BIOSes */
#define	AUTOBOOT_ROM_FILTER	/* Autoboot only devices matching our ROM */

/*
 * Virtual network devices
 *
 */
#define VNIC_IPOIB		/* Infiniband IPoIB virtual NICs */
//#define VNIC_XSIGO		/* Infiniband Xsigo virtual NICs */

/*
 * Error message tables to include
 *
 */
#undef	ERRMSG_80211		/* All 802.11 error descriptions (~3.3kb) */

/*
 * Obscure configuration options
 *
 * You probably don't need to touch these.
 *
 */

#undef	BUILD_SERIAL		/* Include an automatic build serial
				 * number.  Add "bs" to the list of
				 * make targets.  For example:
				 * "make bin/rtl8139.dsk bs" */
#undef	BUILD_ID		/* Include a custom build ID string,
				 * e.g "test-foo" */
#undef	NULL_TRAP		/* Attempt to catch NULL function calls */
#undef	GDBSERIAL		/* Remote GDB debugging over serial */
#undef	GDBUDP			/* Remote GDB debugging over UDP
				 * (both may be set) */
//#define EFI_DOWNGRADE_UX	/* Downgrade UEFI user experience */
#define	TIVOLI_VMM_WORKAROUND	/* Work around the Tivoli VMM's garbling of SSE
				 * registers when iPXE traps to it due to
				 * privileged instructions */

#include <config/named.h>
#include NAMED_CONFIG(general.h)
#include <config/local/general.h>
#include LOCAL_NAMED_CONFIG(general.h)

#endif /* CONFIG_GENERAL_H */

#ifndef CONFIG_SETTINGS_H
#define CONFIG_SETTINGS_H

/** @file
 *
 * Configuration settings sources
 *
 */

FILE_LICENCE ( GPL2_OR_LATER_OR_UBDL );

#define	PCI_SETTINGS	/* PCI device settings */
//#define	CPUID_SETTINGS	/* CPUID settings */
//#define	MEMMAP_SETTINGS	/* Memory map settings */
//#define	VMWARE_SETTINGS	/* VMware GuestInfo settings */
//#define	VRAM_SETTINGS	/* Video RAM dump settings */
//#define	ACPI_SETTINGS	/* ACPI settings */

#include <config/named.h>
#include NAMED_CONFIG(settings.h)
#include <config/local/settings.h>
#include LOCAL_NAMED_CONFIG(settings.h)

#endif /* CONFIG_SETTINGS_H */

One other thought is that, since ipxe will not build with both IMAGE_EFI and IMAGE_BZIMAGE enabled, the entire reason for my issue is my FOG kernels are all bzImage files. It is a stretch, I know. The only way I have found for ipxe.efi to build is if the IMAGE_BZIMAGE option is disabled as an image type ipxe.efi will be able to process. Again, I’m not prone to thinking that particular “possible cause” actually holds water, but I can’t rule it out with how little I actually know about the ipxe code.

Anyone have an idea why the kernel will still throw the exec format error in spite of the EFI_STUB setting have been turned on?

[Sebastian Roth]

@Scott-Lynch Nice! Keeping my fingers crossed for it to finish without another hick up. I’ll mark this solved now. Just posting what I said earlier so people may find it again.

Solution here:

Other than that I am wondering if you’ve disabled secure boot? I know this questions sounds stupid but just wanna make sure as the error sounds a bit like it could be on. I do remember one case where a Lenovo device had some kind of extra security chip which needed to be disabled - read through this and this.

[Sebastian Roth]

@Scott-Lynch Great to hear you’ve been digging through things very intensely! I am sure you’ve learned a lot building your own kernels and iPXE binaries. Well done!

So let me try to step back a little bit to get the whole big picture. FOG comes with binaries included to make things as simple as possible for people and there are not too many cases when you actually want to build your own stuff. Don’t get me wrong, I don’t wanna sound mean. But why are you doing this at all (just trying to get an idea on where to direct you)?

As well I try to answer some of your questions let me quote one of the iPXE devs:

There is currently no way for iPXE (EFI version) to boot anything that is not a valid EFI binary.

This is from 2013 but as far as I know this is still valid. Loading the Linux kernel in bzImage format is only possible when iPXE is build for legacy BIOS - *.(kk)pxe binaries. Modern kernels (about since around version 3.3 I think) do have something called EFI_STUB which makes it possible to chainload such a kernel image from iPXE in UEFI mode.

As you are talking about kitchensink.config and sourceforge I get the feeling that you are using an older version of FOG. This is just me guessing but are you trying to make an old FOG server UEFI capable? While I am sure this can be done with a lot of effort I’d think you’d be way better of by upgrading to a newer version of FOG.

Let us know which version of FOG you use and what you are aiming for (why compile your own binaires?) and I am sure we’ll get you fixed up very soon!

Edit: Re-reading your post I noted that you are talking about the “Could not Select: Exec format error” thing as well. Possibly it’s just that making you wanna build your own kernel, hmm? Do you see this error on different client hardware? Please try different models to see if it’s always the exact same issue.

[george1421]

Lets start by collecting a bit of background info here.

1. hardware are you trying to pxe boot. We should know both the manufacturer and model.
2. Have you confirmed that you have the latest firmware installed on the target computer.
3. Is the target computer in uefi mode or legacy (bios) mode?
4. What specifically do you have defined for dhcp option 67?
5. What version of FOG are you using?

To be able to help you we will probably need you to resetup your fog server to a known state so we are not chasing our tail with changes. Before trying to debug this issue. We have seen certain very buggy versions of firmware from most vendors, but Lenovo is at the top of the list for buggy firmware in my experience. We have ways of booting these one off systems but I’m more interested in understanding what is unique about this hardware you have.

[Scott Lynch]

@george1421 :

You are right, I forgot to mention some of those points in the original post. Here is what I have:

1. Hardware I’m trying to boot:
   Lenovo N23 WinBooks (NOT the ChromeBook, but the one running Windows)
   Model/Type: 80UR0004US
   Firmware Version: 2QCN18WW (Most recent available on the manufacturer’s website)
   Firmware Date: 12/04/2016
   Processor: Intel Celeron N3060
   Wireless Network Adapter: Intel Dual Band Wireless-AC 7260
   Ethernet Adapter: None, Wireless Networking Enabled Only
   I am using a USB 3.0 to Ethernet Adapter purchased from Lenovo to to do the PXE
   booting since wireless support is not enabled by default within the iPXE client that
   I got off of the ipxe.org website.

2. Yes, the firmware is up-to-date.

3. Target computer is UEFI mode.
   IF I can get this UEFI boot issue resolved, I am going to dispense with legacy booting
   altogether and just start configuring the computers for UEFI boot only.

4. DHCP Option 067 = ipxe.efi

5. Version of FOG I am using is:
   FOG Version: v1.4.4
   SVN Revision: 6077

6. I have already set my FOG server to back to its “just out of the box” state and is ready to go.

Incidentally, I have been using FOG for my deployments for almost 5 years now. My employer, during the entirety of the 10-11 years I have been working here, has only bought Lenovo machines. That said, I have only run into 1 or 2 models of Lenovo computers that had any trouble talking to my FOG server. Both of those two machines would not PXE boot directly off the network, but would PXE boot with no issues if I loaded the ipxe client onto a bootable CD/DVD/USB Key and booted the machine off the media. I don’t remember which models of Lenovo computers had issues off the top of my head, and they aren’t relevant at this time. I have just had good luck booting “problem” machines with ipxe burned onto bootable media.

Speaking of which, I am booting my N23 WinBook off of a USB key I made containing my ipxe client. For the ipxe client, I used the following make statement:
make bin-x86_64-efi/ipxe.efi

To make the bootable media, I copied the resulting ipxe.efi of the make statement to the following directory structure on my USB key and changed its name from “ipxe.efi” to “bootx64.efi”.

Directory tree of bootable USB key:
/EFI/BOOT/bootx64.efi

[Scott Lynch]

@scott-lynch :

These N23 WinBooks are the 3rd model of Lenovo computer I’ve run across to have any issues with talking to the FOG server. The N23 WinBooks issues started with their lack of a built-in Ethernet port to which an Ethernet cable could be attached and have escalated from there. I swear these winbooks have been little more than a reminder that Murphy’s Law is in full effect.

So, in short, for the last 5 years or so, I have had very good luck with Lenovo and FOG with any hiccups being overcome by simply booting to some form of bootable media containing the ipxe client. These N23’s are the first model to give me any real issues.

Anyway, thanks for the help…

[Scott Lynch]

@sebastian-roth :

Hey! No problem! I understand FOG comes with binaries included. The reason I am going through all of this building the source code is the included binaries that came with FOG were the first to throw the error 0x2e00801. I began researching the issue. In the beginning, my effort was limited solely to the ipxe client, thinking it was the cause.

Now, you have confirmed what the code was “saying” when you said, “This is from 2013 but as far as I know this is still valid. Loading the Linux kernel in bzImage format is only possible when iPXE is build for legacy BIOS - *.(kk)pxe binaries. Modern kernels (about since around version 3.3 I think) do have something called EFI_STUB which makes it possible to chainload such a kernel image from iPXE in UEFI mode.”

I haven’t really tried to boot anything else other than these winbooks in UEFI mode since they all have been working up to now. I will test booting the ipxe client after I get back from luck today.

And…Yes, the “Could not select: Exec format error” is what has spurred me to go through this building the kernel bit since it doesn’t seem the the kernels that came with my installed version of FOG have EFI support enabled.

[Scott Lynch]

@sebastian-roth :

Incidentally, what are the other possible formats in which the kernel can be built?

I have little to no experience building linux kernels of any kind. I only know that all of the documentation I have seen talks about building the kernel as “bzImage”. The only other kernel mentioned in the documentation is called something like “vmlinuz”, but I have no clue how it is built since make complains there is no recipe to build “vmlinuz”.

Thanks again for the help!

[Scott Lynch]

@sebastian-roth :

I just noticed your question about me trying to get an old FOG server UEFI capable. The answer is no, I only have 1 FOG server currently running on Ubuntu Server 16.04.03.

I am going through the custom kernel builds because everything I have been reading said the most likely cause for the exec format error was the kernel was lacking these two settings enabled:
EFI runtime service support = y
EFI stub support = y

[george1421]

My intent is to give you a hand with this later today. I’m sorry for the delay, but my work is occupying my time today. I think the answer is in these posts. As I said before we have a last resort option of booting FOS from a usb stick. We have to go that route for some Mac systems that have had firmware updates that make it incompatible with iPXE. So by booting FOS directly from the usb stick we can bypass iPXE with certain caveats. So there ARE options here we haven’t tested.

[Scott Lynch]

@george1421 :

My work day is about to end; however, I would be happy to go through the process with you sometime soon.

I did test loading FOG via UEFI on a completely different model of machine. The new machine doesn’t complain about the exec format error. The new machine loads to the FOG boot menu like it should; but, when a task is selected it immediately reloads ipxe and unloads it, and then reloads ipxe in an almost continues loop, when it finally stops, it loads the reFind.efi which gives restart and reboot as its two main options.

[george1421]

@scott-lynch After reviewing this thread, I think we should attempt to boot FOS directly from the usb stick (fall back plan). Right now its not clear if its the hand off between ipxe and FOS, or in the FOS kernel itself.

ref: https://forums.fogproject.org/topic/7727/building-usb-booting-fos-image

Lets stick to the tutorial with the stock FOG kernel.

[Sebastian Roth]

@george1421 As far as I understand @Scott-Lynch has already tried USB booting - which worked for other models but not on this one.

I don’t have much time right now but will get back to you later on today. Most probably I can offer some help on this.

[Sebastian Roth]

@Scott-Lynch Talked to George and he’s right about testing to boot FOS Linux directly from USB is good to test. We have seen crappy UEFI firmware where even that failed.

Other than that I am wondering if you’ve disabled secure boot? I know this questions sounds stupid but just wanna make sure as the error sounds a bit like it could be on. I do remember one case where a Lenovo device had some kind of extra security chip which needed to be disabled - read through this and this.

[Scott Lynch]

@george1421 :

Ok, I’ll go through the tutorial now. I have the FOG server reset to the stock kernel, so everything should be good to go. Will post back with results.

[Scott Lynch]

@sebastian-roth :

No problem. I do have secure boot disabled on the WinBook and the 2nd test computer. Neither computer would boot to the USB stick with the secure boot option turned on. I will go through the FOS tutorial and get back with the results.

[Sebastian Roth]

@Scott-Lynch The security chip mentioned is something else. Please read through the links I posted.

[Scott Lynch]

@sebastian-roth :

My bad. The Intel TPM chip was still enabled. I just disabled it and, just for kicks, ran the PXE boot sequence. I made it to the FOG boot menu. It correctly showed the host as being already registered. (I had manually entered the information into the FOG console since I am having to use a usb-to-ethernet adapter to get things rolling client side.) I then attempted to have it run the Client System Information (Compatibility) at which point it successfully loaded the stock bzImage. It threw a few ACPI errors before going into kernel panic. Exact message for the kernel panic is:

Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(1,0)

In truth, this is a bit of headway given that, before, it wouldn’t even load the kernel, much less read it and attempt to execute it.

[george1421]

@scott-lynch said in EFI_STUB enabled custom FOG kernel causing ipxe.efi to throw error 0x2e008081:

Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(1,0)

It may not fit exactly this case, but historically every time we’ve seen this the FOG admin had somehow tried to pxe boot with pxelinux.0, which was how fog 1.1 use to pxe boot before FOG switching over the iPXE. Please ensure you are using fog delivered pxe boot files.

The other implications of this is that the init.xz file is damaged in some way.

The APCI messages are only warnings and not something to be concerned about.

[Scott Lynch]

@george1421 :

I am still working through the FOS build, but have run into a few snags I am attempting to get through. Namely, should I be looking at the multiboot instructions? They seem to be the only set that mentions EFI…

[george1421]

@scott-lynch Look at the FOG chat bubble above on the fog tool tray. If you follow the process the usb drive will be universal both uefi and bios booting.