EFI_STUB enabled custom FOG kernel causing ipxe.efi to throw error 0x2e008081
I’m having an issue doing an iPXE UEFI boot to FOG. Specifically, no matter what I do, I get the “Could not Select: Exec format error (http://ipxe.org/2e008081)” error.
I have read everything at ipxe.org and the FOG Wiki concerning the issue and I have also scoured the Google and Bing search results for anything and everything related to the issue. All of the reference sources from ipxe.org, the FOG Wiki, the FOG forum posts I was able to locate, and the online documentation from my web-searches agree that the issue is caused by ipxe.efi attempting to load a non-EFI image (bzImage) and to fix it, I needed to build a new bzImage with EFI Stub support enabled. However, every single custom kernel I have built fails with the same 0x2e008081 error code.
I have even booted to the ipxe command line interface and tried various configuration settings to try and determine exactly where the issue comes up. In all cases, I only receive the error when the call to load the bzImage file occurs, regardless of whether or not the EFI_STUB is enabled within the kernel.
Incidentally, with my Windows DHCP options 066 and 067 set as follows, my client will successfully load the FOG boot menu:
option 066: <ip address of FOG deployment server>
option 067: ipxe.efi
However, any task involving either bzImage or bzImage32 to be loaded causes the above mentioned error to be thrown.
Kernel config files used for building the kernels are both TomElliott.config.32 and TomElliott.config.64 located at:
I have even attempted to build using the kitchensink.config and TomElliott.config.32/64 files the FOG 1.4.4 installation archive lists in its directory
I am at a complete loss as to what else to do. Architectures of the ipxe.efi builds I have attempted are x86_64 (for 64-bit kernel) and x86/i386 (for the 32-bit kernel).
This one issue has had me stumped for a long while, and I think I’m too close to the issue now to properly see the solution.
Also, I have built the ipxe.efi from source, also. I have been using the following modified header files to build ipxe.efi:
#ifndef CONFIG_CONSOLE_H #define CONFIG_CONSOLE_H /** @file * * Console configuration * * These options specify the console types that iPXE will use for * interaction with the user. * */ FILE_LICENCE ( GPL2_OR_LATER_OR_UBDL ); #include <config/defaults.h> /* * Default console types * * These are all enabled by default for the appropriate platforms. * You may disable them if needed. * */ //#undef CONSOLE_PCBIOS /* Default BIOS console */ //#undef CONSOLE_EFI /* Default EFI console */ //#undef CONSOLE_LINUX /* Default Linux console */ /* * Additional console types * * These are not enabled by default, but may be useful in your * environment. * */ //#define CONSOLE_SERIAL /* Serial port console */ #define CONSOLE_FRAMEBUFFER /* Graphical framebuffer console */ //#define CONSOLE_SYSLOG /* Syslog console */ //#define CONSOLE_SYSLOGS /* Encrypted syslog console */ //#define CONSOLE_VMWARE /* VMware logfile console */ //#define CONSOLE_DEBUGCON /* Bochs/QEMU/KVM debug port console */ //#define CONSOLE_INT13 /* INT13 disk log console */ /* * Very obscure console types * * You almost certainly do not need to enable these. * */ //#define CONSOLE_DIRECT_VGA /* Direct access to VGA card */ //#define CONSOLE_PC_KBD /* Direct access to PC keyboard */ /* Keyboard map (available maps in hci/keymap/) */ #define KEYBOARD_MAP us /* Control which syslog() messages are generated. * * Note that this is not related in any way to CONSOLE_SYSLOG. */ #define LOG_LEVEL LOG_NONE #include <config/named.h> #include NAMED_CONFIG(console.h) #include <config/local/console.h> #include LOCAL_NAMED_CONFIG(console.h) #endif /* CONFIG_CONSOLE_H */
#ifndef CONFIG_GENERAL_H #define CONFIG_GENERAL_H /** @file * * General configuration * */ FILE_LICENCE ( GPL2_OR_LATER_OR_UBDL ); #include <config/defaults.h> /* * Banner timeout configuration * * This controls the timeout for the "Press Ctrl-B for the iPXE * command line" banner displayed when iPXE starts up. The value is * specified in tenths of a second for which the banner should appear. * A value of 0 disables the banner. * * ROM_BANNER_TIMEOUT controls the "Press Ctrl-B to configure iPXE" * banner displayed only by ROM builds of iPXE during POST. This * defaults to being twice the length of BANNER_TIMEOUT, to allow for * BIOSes that switch video modes immediately before calling the * initialisation vector, thus rendering the banner almost invisible * to the user. */ #define BANNER_TIMEOUT 20 #define ROM_BANNER_TIMEOUT ( 2 * BANNER_TIMEOUT ) /* * Network protocols * */ #define NET_PROTO_IPV4 /* IPv4 protocol */ #undef NET_PROTO_IPV6 /* IPv6 protocol */ #undef NET_PROTO_FCOE /* Fibre Channel over Ethernet protocol */ #define NET_PROTO_STP /* Spanning Tree protocol */ #define NET_PROTO_LACP /* Link Aggregation control protocol */ /* * PXE support * */ //#undef PXE_STACK /* PXE stack in iPXE - you want this! */ //#undef PXE_MENU /* PXE menu booting */ /* * Download protocols * */ #define DOWNLOAD_PROTO_TFTP /* Trivial File Transfer Protocol */ #define DOWNLOAD_PROTO_HTTP /* Hypertext Transfer Protocol */ #define DOWNLOAD_PROTO_HTTPS /* Secure Hypertext Transfer Protocol */ #define DOWNLOAD_PROTO_FTP /* File Transfer Protocol */ #undef DOWNLOAD_PROTO_SLAM /* Scalable Local Area Multicast */ #define DOWNLOAD_PROTO_NFS /* Network File System Protocol */ //#undef DOWNLOAD_PROTO_FILE /* Local filesystem access */ /* * SAN boot protocols * */ //#undef SANBOOT_PROTO_ISCSI /* iSCSI protocol */ //#undef SANBOOT_PROTO_AOE /* AoE protocol */ //#undef SANBOOT_PROTO_IB_SRP /* Infiniband SCSI RDMA protocol */ //#undef SANBOOT_PROTO_FCP /* Fibre Channel protocol */ //#undef SANBOOT_PROTO_HTTP /* HTTP SAN protocol */ /* * HTTP extensions * */ #define HTTP_AUTH_BASIC /* Basic authentication */ #define HTTP_AUTH_DIGEST /* Digest authentication */ //#define HTTP_ENC_PEERDIST /* PeerDist content encoding */ //#define HTTP_HACK_GCE /* Google Computer Engine hacks */ /* * 802.11 cryptosystems and handshaking protocols * */ #define CRYPTO_80211_WEP /* WEP encryption (deprecated and insecure!) */ #define CRYPTO_80211_WPA /* WPA Personal, authenticating with passphrase */ #define CRYPTO_80211_WPA2 /* Add support for stronger WPA cryptography */ /* * Name resolution modules * */ #define DNS_RESOLVER /* DNS resolver */ /* * Image types * * Etherboot supports various image formats. Select whichever ones * you want to use. * */ //#define IMAGE_NBI /* NBI image support */ //#define IMAGE_ELF /* ELF image support */ //#define IMAGE_MULTIBOOT /* MultiBoot image support */ //#define IMAGE_PXE /* PXE image support */ #define IMAGE_SCRIPT /* iPXE script image support */ //#define IMAGE_BZIMAGE /* Linux bzImage image support */ //#define IMAGE_COMBOOT /* SYSLINUX COMBOOT image support */ #define IMAGE_EFI /* EFI image support */ //#define IMAGE_SDI /* SDI image support */ //#define IMAGE_PNM /* PNM image support */ #define IMAGE_PNG /* PNG image support */ #define IMAGE_DER /* DER image support */ #define IMAGE_PEM /* PEM image support */ /* * Command-line commands to include * */ #define AUTOBOOT_CMD /* Automatic booting */ #define NVO_CMD /* Non-volatile option storage commands */ #define CONFIG_CMD /* Option configuration console */ #define IFMGMT_CMD /* Interface management commands */ //#define IWMGMT_CMD /* Wireless interface management commands */ #define IBMGMT_CMD /* Infiniband management commands */ #define FCMGMT_CMD /* Fibre Channel management commands */ #define ROUTE_CMD /* Routing table management commands */ #define IMAGE_CMD /* Image management commands */ #define DHCP_CMD /* DHCP management commands */ #define SANBOOT_CMD /* SAN boot commands */ #define MENU_CMD /* Menu commands */ #define LOGIN_CMD /* Login command */ #define SYNC_CMD /* Sync command */ #define NSLOOKUP_CMD /* DNS resolving command */ #define TIME_CMD /* Time commands */ #define DIGEST_CMD /* Image crypto digest commands */ #define LOTEST_CMD /* Loopback testing commands */ #define VLAN_CMD /* VLAN commands */ //#define PXE_CMD /* PXE commands */ #define REBOOT_CMD /* Reboot command */ #define POWEROFF_CMD /* Power off command */ #define IMAGE_TRUST_CMD /* Image trust management commands */ #define PCI_CMD /* PCI commands */ #define PARAM_CMD /* Form parameter commands */ #define NEIGHBOUR_CMD /* Neighbour management commands */ #define PING_CMD /* Ping command */ #define CONSOLE_CMD /* Console command */ #define IPSTAT_CMD /* IP statistics commands */ //#define PROFSTAT_CMD /* Profiling commands */ //#define NTP_CMD /* NTP commands */ //#define CERT_CMD /* Certificate management commands */ /* * ROM-specific options * */ #undef NONPNP_HOOK_INT19 /* Hook INT19 on non-PnP BIOSes */ #define AUTOBOOT_ROM_FILTER /* Autoboot only devices matching our ROM */ /* * Virtual network devices * */ #define VNIC_IPOIB /* Infiniband IPoIB virtual NICs */ //#define VNIC_XSIGO /* Infiniband Xsigo virtual NICs */ /* * Error message tables to include * */ #undef ERRMSG_80211 /* All 802.11 error descriptions (~3.3kb) */ /* * Obscure configuration options * * You probably don't need to touch these. * */ #undef BUILD_SERIAL /* Include an automatic build serial * number. Add "bs" to the list of * make targets. For example: * "make bin/rtl8139.dsk bs" */ #undef BUILD_ID /* Include a custom build ID string, * e.g "test-foo" */ #undef NULL_TRAP /* Attempt to catch NULL function calls */ #undef GDBSERIAL /* Remote GDB debugging over serial */ #undef GDBUDP /* Remote GDB debugging over UDP * (both may be set) */ //#define EFI_DOWNGRADE_UX /* Downgrade UEFI user experience */ #define TIVOLI_VMM_WORKAROUND /* Work around the Tivoli VMM's garbling of SSE * registers when iPXE traps to it due to * privileged instructions */ #include <config/named.h> #include NAMED_CONFIG(general.h) #include <config/local/general.h> #include LOCAL_NAMED_CONFIG(general.h) #endif /* CONFIG_GENERAL_H */
#ifndef CONFIG_SETTINGS_H #define CONFIG_SETTINGS_H /** @file * * Configuration settings sources * */ FILE_LICENCE ( GPL2_OR_LATER_OR_UBDL ); #define PCI_SETTINGS /* PCI device settings */ //#define CPUID_SETTINGS /* CPUID settings */ //#define MEMMAP_SETTINGS /* Memory map settings */ //#define VMWARE_SETTINGS /* VMware GuestInfo settings */ //#define VRAM_SETTINGS /* Video RAM dump settings */ //#define ACPI_SETTINGS /* ACPI settings */ #include <config/named.h> #include NAMED_CONFIG(settings.h) #include <config/local/settings.h> #include LOCAL_NAMED_CONFIG(settings.h) #endif /* CONFIG_SETTINGS_H */
One other thought is that, since ipxe will not build with both IMAGE_EFI and IMAGE_BZIMAGE enabled, the entire reason for my issue is my FOG kernels are all bzImage files. It is a stretch, I know. The only way I have found for ipxe.efi to build is if the IMAGE_BZIMAGE option is disabled as an image type ipxe.efi will be able to process. Again, I’m not prone to thinking that particular “possible cause” actually holds water, but I can’t rule it out with how little I actually know about the ipxe code.
Anyone have an idea why the kernel will still throw the exec format error in spite of the EFI_STUB setting have been turned on?
So zstd is the preferred compression method for the images?
Yes it is much faster at decompression. There is a small performance hit on compression speeds, but for that initial capture penalty you get a much faster decompression rate.
If I remember correctly the gzip engine is (now) only use for image capture. Zstd is used as the default decompresser because it can accept both gzip and zstd images. You only get the performance increase with a zstd native image.
BTW: Wonderful that you are now capturing an image with FOS. So that tells us the issue isn’t with the FOS kernel (bzImage) but with the hand off between Firmware, iPXE and the FOS linux kernel.
So zstd is the preferred compression method for the images?
It’s meant to be much faster…
@Scott-Lynch Nice! Keeping my fingers crossed for it to finish without another hick up. I’ll mark this solved now. Just posting what I said earlier so people may find it again.
Other than that I am wondering if you’ve disabled secure boot? I know this questions sounds stupid but just wanna make sure as the error sounds a bit like it could be on. I do remember one case where a Lenovo device had some kind of extra security chip which needed to be disabled - read through this and this.
Thanks for mentioning the gzip/compression settings of the image in the FOG console. I would never have thought they were a cause of one of my issues. So zstd is the preferred compression method for the images?
Ok, I adjusted my image settings and restarted the capture process. This time it is behaving as it “should”. At least everything seems to be optimal on the capture process.
Sorry for the late reply. Other work kept me busy. I will verify the image settings and make the adjustments you mentioned and will try again. It will be Monday before I will likely get the results for you.
Ok, thanks heaps @george1421 to have pushed this quite far! We know FOS is running properly on this hardware. @Scott-Lynch Just let me know when you have time and we can tackle the kernel panic thing. I am fairly sure this is not a big thing. Were you still able to boot up other clients in this scenario? Would be great if you could take a picture (or even better a steady 60 fps video) of the kernel panic on screen and post here. Quite often there is more information hidden.
@scott-lynch Ok this is a image definition configuration error.
FOG supports two different image compression tool. The legacy gzip and the newer zstd. From what I understand from the post you have gzip selected but you are using a compression level higher than 9. I’m suspecting that you are using a compression index greater than 9 as defined in the image definition
That error probably needs to be trapped in the web ui to keep people from making that same setting @Developers
BUT you getting this far also tells us that this computer is not having a problem with the FOS kernel itself, it maybe related to the hand off between iPXE and FOS (bzImage).
Here is the error.
At this point I need to get to my doctor appointment. Thanks for your help.
Capture task started and client booted to FOS, selected option 1, and it is currently resizing the filesystem. So far, everything looks optimal. It didn’t even hiccup on the call to DHCP for an IP Address. I think the issue with it “failing” had to do with the client sending 4 requests for an IP address.
However, the client just threw a garbled error message. I’ll try and upload a picture. One sec.
No problem. I will start a task and do as you say on booting the client.
@scott-lynch Sorry my screen wasn’t refreshing. So I missed a few questions
I don’t think there is a need, I ran the ifconfig command and it shows that eth0 has an IP address and the link is up.
OK then its important to understand why on this. My intuition is telling me its either spanning tree or you did not key in your fog server IP address correctly. If you could snap a picture of it saying it can’t get the IP address we can find out which.
In your case now how to use this usb boot
On your fog server you must first schedule a capture/deploy then usb boot from this stick. At the grub menu pick option 1 and it will do the rest.
I am about to leave for the day. Doc appointment. I will continue in the morning. Is there a dictionary of commands for the fos boot system somewhere I can look at to familiarize myself with the next bit of testing?
ip link show gives:
eth0: <BROADCAST, MULTICAST, UP, LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000 link/ether 3c:18:a0:0c:d6:05 brd ff:ff:ff:ff:ff:ff
The network connection looks sound. (Besides, with the trouble I went through to find a usb-to-ethernet adapter that the computer would recognize during boot, it better work… LOL)
I don’t think there is a need, I ran the ifconfig command and it shows that eth0 has an IP address and the link is up. No idea why it would say it failed to get an IP address. Regardless, it seems to be okay.
So, to begin the debug process, since I am NOT having an issue with eth0, I use the command ‘fog’ to start an image capture?
@george1421 Just off the top of my head I can think of two reasons why its not getting dhcp.
- FOS doesn’t support your network adapter. This can be proven by
ip link show
- spanning tree hasn’t started forwarding data by the time the network is ready. We see this issue if someone has standard spanning tree enabled and not one of the fast protocols.
- Its really getting an IP address, its just failing to reach the fog server. It will try this 3 times during booting then give up.
- FOS doesn’t support your network adapter. This can be proven by
@scott-lynch so you are at the FOS linux command prompt?
If so there is a command to restart the dhcp process I think it
udhcpc -i eth0
I updated the grub.cfg with my server’s IP address and booted the client. It loaded to the DHCP testing, still threw a could not get IP address from DHCP on the 4th query of eth0, but loaded to the FOG splash screen when I hit the key to continue.